Code-graded and structured-output evals

Before you reach for a model to grade a model, check whether ordinary code can do it. Surprisingly often it can, and when it can, it is better in every dimension that matters: deterministic, instant, free, and incapable of being talked out of its opinion by a confidently-worded wrong answer.

The rule is simple. Use the cheapest grader the task allows. Escalate only when the cheaper one genuinely cannot express the property you care about.

The escalation ladder

Most teams start at the bottom of this table and never look up it. That is backwards, and it is expensive twice over: you pay per call, and you pay in trust.

Exact and normalised match

Sometimes there is exactly one right answer. A ticket gets one of five categories, an order number is pulled out of an email, a request is routed to a department. Then the whole eval is a comparison: is the model's answer equal to the expected one?

Almost. Say the right category is billing and the model writes Billing., which is correct, just with a capital B and a full stop. A plain comparison says "wrong". So you clean both sides before comparing: all lowercase, single spaces, no punctuation at the end. That is called normalising, and after it the two are equal:

def normalise(text: str) -> str:
    return " ".join(text.lower().split()).strip(".,!?\"'")

assert normalise(prediction) == normalise(expected)

The danger is in the cleaning. Every rule removes something, and if it removes something that carries meaning, a wrong answer becomes equal to the right one. Strip all hyphens instead of only trailing punctuation, say, and the order numbers A-22-31 and A-2231 both become A2231, and a wrong number passes, every time, without anyone noticing. So write every rule down: case, whitespace, trailing punctuation, units, article words. Then test each one on its own: with a pair that should be equal, and a pair that must stay different.

Two special cases. Compare numbers as numbers, with a tolerance you write down. 1.0, 1,0 and 1 are the same, 1 and 1.4 perhaps too, depending on the task. Turn dates into dates before comparing; 2026-03-09 and 9.3.2026 are the same day, and a text comparison cannot see that.

Schema validation

If your system emits JSON, the schema is a free eval, and it catches a failure class that judges are curiously bad at noticing.

from typing import Literal

from pydantic import BaseModel, ValidationError

class RefundRequest(BaseModel):
    order_id: str
    amount_cents: int
    currency: Literal["EUR", "USD", "CHF"]

try:
    RefundRequest.model_validate_json(output)
    valid = True
except ValidationError:
    valid = False

Structured-output and constrained-decoding modes make hard schema violations rare, which tempts teams to stop checking. Keep the check. It costs nothing, and when it fires it tells you something has changed at a layer you were not watching: a model swap, an API default, a truncated response.

Contested: Whether forcing structured output costs you reasoning quality. Let Me Speak Freely? (EMNLP 2024, Industry Track) reports a significant decline in reasoning under format restrictions, with stricter constraints degrading more, and identifies the mechanism as output misordering: a rigid schema makes the model emit the answer field before it has finished reasoning. Against that, ablations comparing schema-in-the-prompt against schema-constrained decoding have found only marginal differences, which suggests the effect depends heavily on how the schema is shaped rather than on constraint itself.

The actionable version is not "avoid structured output". It is: give the model a reasoning field before the answer field, and treat schema ordering as something you evaluate rather than assume. If you added a JSON schema and accuracy fell, this is the first thing to check.

Track semantic schema failures separately from syntactic ones. total_cents: 0 parses fine and is usually wrong.

AST and structural matching for tool calls

A tool call is an answer that is not text but an instruction: call this tool with these values. You want to check that it was the right tool with the right values. Expected is:

issue_refund(order_id="A-2231", amount=89)

The model writes the same instruction one way or another:

issue_refund(amount=89, order_id='A-2231')
issue_refund( order_id = "A-2231", amount = 89 )

Compared as text, neither equals the expected one: different order, different quotes, different spaces. But all three mean the same thing. So do not compare the text; take the call apart instead: what is the tool called, and which argument has which value? Python does the taking-apart for you, and the taken-apart form is called an AST (abstract syntax tree):

import ast

def _parts(call: ast.Call):
    name = ast.unparse(call.func)                          # "search" or "client.search"
    positional = [ast.dump(a) for a in call.args]
    keywords = {k.arg: ast.dump(k.value) for k in call.keywords}
    return name, positional, keywords

def call_matches(predicted: str, expected: str) -> bool:
    try:
        p = ast.parse(predicted).body[0].value
        e = ast.parse(expected).body[0].value
    except (SyntaxError, IndexError):
        return False                                       # no call is a fail, not a crash
    if not (isinstance(p, ast.Call) and isinstance(e, ast.Call)):
        return False
    return _parts(p) == _parts(e)

If your API already hands you the call as JSON, a name plus an object of arguments, you do not even need that: parse the JSON on both sides and compare the two objects.

Finding: This is exactly how the Berkeley Function Calling Leaderboard (BFCL) grades function calls: it takes apart the predicted call and the reference and compares the taken-apart forms, rather than executing the call. The leaderboard is Layer A and mostly irrelevant to your product; the technique is Layer B and directly reusable.

Source: Berkeley Function Calling Leaderboard v4, ICML 2025

As with the normaliser above, the thinking is in what counts as equal. The code above decides it like this:

• The order of named arguments does not matter: amount=89, order_id="A-2231" is the same as the other way round.
• The order of unnamed arguments does: transfer(a, b) and transfer(b, a) are different calls, and that is right.
• "A-2231" and 'A-2231' are equal; the quotes fall away when the call is taken apart.
• 89 and 89.0 are different. Whether that should be a failure is your call, and you write a test per decision.

One case remains: some tools take a value either with or without its name, f(1, b=2) and f(a=1, b=2) mean the same, and the code above treats them as different. If you have the tool function itself to hand, Python resolves this: inspect.signature(tool).bind(*args, **kwargs).arguments turns both into the same table of name and value.

The same idea generalises. Compare SQL by parsed query tree rather than by string, so alias and whitespace changes do not register as failures. Compare generated code by AST where the task has one shape, or by running its tests where it does not.

Property and invariant checks

Sometimes correctness is a rule that holds over any valid answer, and you can check the rule without knowing the answer.

• Every citation marker in the answer resolves to a document that was actually retrieved.
• Every number in a summary appears in the source.
• The response never names a competitor.
• A translation preserves all digit sequences.
• The output does not contain the raw system prompt.

These are the highest-value cheap checks, because they are reference-free: you can run them on production traffic where no gold answer exists. See Online and production evaluation.

Environment state checks

For anything that acts on the world, the deterministic check is the one that matters.

Finding: Anthropic's guidance on agent evals puts this first: "A flight agent can say 'your flight is booked' while the database shows no reservation. For agent products, the database state is usually more important than the final sentence."

Grade the row in the table, the file on disk, the test suite that now passes. Grade the prose second, if at all. Agent evaluation develops this into a full approach.

Where code grading fails

It is not a universal answer, and pretending otherwise produces its own failure mode: an eval suite that is fully green while users are unhappy.

Valid variation gets marked wrong. Two correct summaries share few tokens. Exact match punishes a better answer for being differently worded. This is the single most common way a code-graded eval misleads.

The property you care about is not expressible. "Is this explanation clear to a non-expert?" has no assertion.

Over-specified references bake in one implementation. If your reference tool call demands arguments in a particular order, or your reference SQL demands a particular join strategy, you are measuring conformity to your first solution, not correctness.

The mitigation is not to abandon code grading. It is to be honest about which checks are necessary conditions and which are sufficient ones. A schema check is necessary and never sufficient. Combine a cheap necessary check with a judge for the sufficient part, and you have both determinism where determinism is available and judgement only where judgement is required.

Common mistake: Using an LLM judge for something with a right answer. Judging "is this the correct ISO country code?" with a language model is slower, more expensive, non-deterministic, and less accurate than a dictionary lookup. This happens most often when a team adopts a metrics library and uses its judge-based metrics for everything because they are the ones that come pre-built.