Agent and tool-use evaluation

An agent does things. That single property changes evaluation more than anything else in this guide, because a system that acts on the world can be graded against the world rather than against an opinion about text.

Most teams do not take that opportunity. They grade the final message.

Grade the state, not the sentence

Finding: Anthropic's guidance on agent evals leads with this: "A flight agent can say 'your flight is booked' while the database shows no reservation. For agent products, the database state is usually more important than the final sentence."

The distinction separates a demo eval from a production eval. A demo eval asks whether the answer sounds right. A production eval verifies the system did the job.

Concretely, grade:

• The row in the database. Does the refund exist, with the right amount, against the right order?
• The file on disk. Does it exist, parse, and contain what it should?
• The test suite. For coding agents, this is the whole grader and it is free.
• The external side effects. Email sent, ticket transitioned, calendar entry created, and nothing else touched.

That last clause matters as much as the rest. An agent that books the flight and cancels an unrelated reservation has failed, and an outcome-only check that looks at one row will call it a pass. Assert on what should not have changed.

Grade the final text second, if at all. It is the part users read, so it is not worthless, but it is also the part most easily made to look correct.

Agent evals are systems tests

Grading a chat model is a pure function: text in, text out, score it. Grading an agent is not, because the agent changes things, and the next test run inherits those changes. That makes an agent eval a systems test: you are testing a whole running environment, not a single function.

Here is the failure that teaches this. Your eval has one task: "book the last available seat on flight LH441."

• Trial 1 runs. The agent books the seat. Pass.
• Trial 2 runs against the same environment. There is no seat left. Trial 1 took it. The agent fails.
• Trial 3 fails for the same reason.

Your agent scored 33%, and its actual ability had nothing to do with it. Worse, reorder the trials and you get a different score from identical code. That is what "results depend on execution order" means, and it is genuinely miserable to debug because nothing is broken. Each individual piece behaves correctly.

The practical consequences follow from that framing:

Environments must be resettable and isolated. Every trial starts from a known state and cannot see another trial's side effects. (A transactional fixture is the cheap version: open a database transaction before the trial, roll it back afterwards, and the changes vanish. A container per trial is the thorough version.) Without this, results depend on execution order and the suite becomes unreproducible in a way that is very hard to debug. In practice this means a container or a transactional fixture per trial, not a shared staging database.

Trials are slow and expensive. An agent trial is minutes and many model calls, not one call. This shapes everything about how you tier the suite. See Evals in CI.

Store every transcript. A transcript is the complete record of one attempt: every message, every tool call with its arguments and result, in the order they happened. It is for an agent what the trace is for a chat system. When an aggregate score moves, the transcripts are the only way to find out why. Anthropic's guidance recommends sampling transcripts weekly, and especially after any score change.

Reliability is a separate measurement from capability

Run the same task once and the agent might get lucky. Run it once more and it might not. A single attempt tells you almost nothing, so you run the same task k times (k is just "how many attempts", usually 3, 5 or 8) and score the batch. Each attempt is called a trial.

There are two ways to score that batch, and the notation is unhelpfully similar:

• pass@k: did at least one of the k attempts succeed?
• pass^k: did all k attempts succeed?

The caret is a useful reminder: for a single task, pass^k is literally the pass rate raised to the power of k, because all k have to come off. That exponent is what makes it collapse so fast.

They answer different questions.

pass@k: at least one of k attempts succeeded. Measures whether the agent can do it. Appropriate when a human retries, or when the product itself retries and verifies.

pass^k: all k attempts succeeded. Measures whether the agent reliably does it. This metric was introduced by τ-bench precisely because pass@k flatters agents that are occasionally brilliant.

Take one agent that succeeds 75% of the time, run it three times, and score it both ways:

pass@3 = 1 − (chance all three fail)
       = 1 − 0.25³  = 1 − 0.016  ≈ 98%     "it can do this"

pass^3 = chance all three succeed
       = 0.75³      = 0.42       ≈ 42%     "it reliably does this"

Same agent, same trials, same day. 98% and 42%. Neither number is wrong. They answer different questions, and which one you quote decides whether your agent sounds production-ready or unusable.

One caveat the arithmetic does not show: the 42% holds for one task where every trial is an independent coin flip at 75%. Across a task set, pass^k is the mean of p^k per task, and that is always above the calculation from the average, in the extreme by a lot: an agent that solves 75% of tasks every time and 25% never has pass^3 = 75%, not 42%. The coin-flip arithmetic is the worst case. Where your agent sits in between, no formula tells you; that is the reason to actually run k trials rather than extrapolate them from the single-run rate.

Finding: The arithmetic is brutal and worth internalising. τ-bench, which introduced pass^k, reported state-of-the-art agents scoring pass^8 below 25% in its retail domain: agents that look competent on a single attempt and are unusable eight times running.

Source: τ-bench (Yao et al., 2024)

An agent that succeeds three times in four looks acceptable in a demo and fails more often than not when a user needs it to work three times in a row. For anything running unattended, pass^k is the honest number, and the gap between the two metrics, measured rather than extrapolated, is the size of the reliability problem.

Report both. They are not competing; they bound different product claims.

Trajectory versus final answer

Final-answer grading scores the outcome. Trajectory grading scores the path: which tools were called, with what arguments, in what order, how many steps, whether it recovered from an error.

A trajectory is just the agent's log of what it did, in order. For a refund request it might read:

1. search_orders(customer_id="4417")          → 3 orders
2. get_order(order_id="A-2231")               → €89.00, delivered
3. read_policy("refunds")                     → returns within 14 days
4. issue_refund(order_id="A-2231", amount=89) → ok

Final-answer grading reads only the last line and the message to the customer. Trajectory grading reads all four, and would notice if step 4 had refunded €890, or if the agent had skipped step 3 entirely, or if steps 1–3 had repeated eleven times.

Both are necessary and each misses what the other catches. A correct answer reached in 20 steps with two policy-violating intermediate calls is a failing trajectory with a passing outcome. For an agent with real permissions, the trajectory is the part that matters.

What is worth grading in a trajectory:

Contested: How strictly to grade trajectories. Requiring an exact tool sequence catches real process failures and also penalises novel-but-valid solutions, which makes the eval a conformity test against your first implementation. The workable compromise is to assert on invariants rather than on an exact sequence: tools that must be called, tools that must not be, budgets that must hold. Leave "was this path sensible?" to sampled human review rather than an automated judge.

Cost and latency are eval metrics

For agents these are not operational footnotes. An agent that solves the task in 90 steps and four minutes has failed a product requirement even if the outcome row is correct.

Track per trial: steps, tool calls, total tokens, wall-clock, and cost. Report them alongside success rate, never instead of it.

Finding: This is the argument the Holistic Agent Leaderboard makes about public agent benchmarks: headline success scores conceal large differences in cost and reliability between agents that appear comparable.

Source: Holistic Agent Leaderboard, 2025

The same holds internally. A prompt change that raises success by 2 points and doubles step count is usually a regression.

Two suites, not one

Anthropic's guidance separates them, and the separation is load-bearing:

Regression suite. Stable, fast, tied to real product failures. Deterministic graders wherever possible. Blocks releases. Should be near 100% and any drop is an incident.

Capability suite. Hard tasks, deliberately including ones the agent fails. Reports rather than blocks. Should not be near 100%. If it is, it has stopped measuring the frontier of what your agent can do and needs harder tasks.

Teams that maintain one suite end up with something that is bad at both jobs: too flaky to gate on, too easy to learn from.

Public agent benchmarks, and what to take from them

These are Layer A. They measure a foundation model's agentic skill, not your agent. Your agent's scaffold, tools, prompts, and domain are the majority of its behaviour, and none of that is in the benchmark.

They are still worth knowing, because the techniques transfer even when the tasks do not:

Take τ-bench's pass^k, BFCL's AST matcher, and OSWorld's state-based grading. Do not take their tasks and do not report their scores as evidence about your product.

Common mistake: Building an agent eval that only checks the final message, because that is what a chat eval looks like and the agent has a chat interface. The interface is text; the product is the state change. If your eval never queries the database, it is not evaluating the agent.