Glossary

Terms as practitioners actually use them, with the confusion each one invites. Where usage is genuinely unsettled, that is noted rather than smoothed over.

Layers and scope

Layer A / B / C: This guide's shorthand for foundation-model evals, system evals, and process evals. See The three layers. Not standard vocabulary elsewhere; the distinction is widely made, the labels are ours.

Offline eval: Evaluation against a fixed dataset before deployment. Do not confuse with batch inference; offline refers to the timing relative to deployment, not to how the requests are issued.

Online eval: Scoring live production traffic after deployment. Do not confuse with a guardrail, which blocks; an online eval only records.

Component eval: Scores one stage of a pipeline in isolation. Do not confuse with unit test; the component is still a model and still stochastic.

End-to-end eval: Scores the whole system's output. Tells you whether a component failure mattered; cannot tell you which component failed.

Graders

Grader / scorer: Anything that turns an output into a score. Used interchangeably; "scorer" is the term in Inspect, "grader" is more common in prose.

Code-graded: Scored by deterministic code. Do not confuse with "easy"; the normalisation function is where the difficulty lives.

LLM-as-judge: A model prompted to grade another model's output. Do not confuse with a reward model, which is trained to output a scalar preference score and is a different artefact used for a different purpose.

Pointwise: Scores one output on its own against a criterion. Required for absolute thresholds and CI gates.

Pairwise: Picks the better of two outputs. More reliable for subjective qualities, and where position bias is strongest.

Jury / panel: Multiple judges aggregated. Reduces variance; does not remove biases shared by all members.

Reference-based: Compares against a gold answer. Do not confuse with exact match; a reference-based judge can accept a paraphrase.

Reference-free: Judges an output on its own terms, with no gold answer. Convenient, and the reason faithfulness can mislead.

Rubric: Written criteria a judge applies. Do not confuse with a scale; a rubric can and usually should produce a binary verdict.

Process

Eval (the artefact): A directory of case files, a module of check functions, a runner, and one result file per run. An eval in five files shows the smallest complete one. Do not confuse with a benchmark, which is somebody else's eval of a model.

Trace: The full record of one interaction: input, retrieved context, tool calls, intermediate steps, output, metadata. Do not confuse with a log line, and do not confuse with a span, which is one step within a trace.

Open coding: Writing free-text notes about failures, without a predefined category list. Borrowed from qualitative research.

Axial coding: Grouping those notes into a named failure taxonomy. The step that turns reading into knowledge.

Failure taxonomy: The named set of distinct ways your system fails, with counts. The output of error analysis.

Theoretical saturation: The point where new traces stop revealing new failure categories. A real stopping rule, roughly 20 consecutive clean traces.

Criteria drift: Shankar et al.'s term for the catch-22 that you need criteria to grade outputs, but grading outputs is how you discover the criteria. The formal reason to read data before writing a rubric.

Benevolent dictator: One domain expert as sole arbiter of quality, rather than a committee. Husain's term.

Development set / test set: The two parts of an eval set. The development set is looked at constantly and may wear out; the test set is touched only for a ship decision and replaced by fresh traces once it has been seen. See Dataset design.

Build pile: The 30 to 40 labelled answers a judge's examples are taken from.

Sealed pile: The labelled answers a judge is measured on and never shown. Do not confuse with the test set, which measures the system, not the judge.

Metrics

TPR / TNR: True positive rate and true negative rate. For a judge, TPR is usually recall on failures, the number that matters, because the judge exists to catch failures. Do not confuse with accuracy, which is dominated by the majority class.

Cohen's κ: Chance-corrected agreement between two annotators. Do not confuse with raw agreement, which is systematically flattering under class imbalance.

Fleiss' κ: The same idea for more than two annotators.

Krippendorff's α: Agreement that tolerates missing ratings and non-binary scales. The right choice for messy real annotation workflows.

pass@k: At least one of k attempts succeeded. Measures capability.

pass^k: All k attempts succeeded. Measures reliability. Introduced by τ-bench. Do not confuse with pass@k. The superscript is doing a lot of work, and at 75% per-trial success pass^3 is about 42% for a single task; higher across a task set, see Agents.

Faithfulness: Fraction of claims in an answer that are entailed by the retrieved context. Do not confuse with correctness: an answer can be perfectly faithful to context that was the wrong context.

Context recall: Whether retrieval found everything needed to answer. Requires ground truth, which is why teams skip it, which is why the faithfulness trap keeps happening.

Context precision: Whether the retrieved chunks were relevant.

Answer relevancy: Whether the answer addresses the question asked.

Recall@k: Of the documents needed, how many appear in the top k retrieved. The most diagnostic single RAG number.

Attack success rate (ASR): Fraction of adversarial attempts that produced the disallowed behaviour.

Minimum detectable effect (MDE): The smallest difference your eval can reliably detect at a given n and power. If you have never computed yours, it is probably larger than the differences you have been acting on.

Standard error (SE): How much a measured rate moves on the strength of which examples happened to land in the sample. It shrinks with the square root of n: four times as many examples halve it.

Confidence interval (CI): The range around your measured value, built so that in 95 of 100 repeated measurements on fresh samples it covers the true value. If two systems' intervals overlap substantially, the difference between them has not been demonstrated.

Clustered standard error: Standard error corrected for examples arriving in correlated groups. Ignoring clustering makes intervals too narrow, which manufactures significance.

Precision / recall: Recall: of all real failures, how many the judge flagged (the TPR above). Precision: of everything the judge flagged, how much was a real failure. This guide reports TPR and TNR instead, because precision moves with the failure rate.

Wilson interval: The confidence interval to use for pass rates near 0% or 100%, where the simple formula gives a zero or impossible spread.

Bootstrap: Estimating a spread by resampling your own cases with replacement many times. The default whenever a formula becomes unwieldy.

Non-inferiority margin: The largest quality loss you would accept for a cheaper model, decided before the numbers are in. See Model selection.

Slice: A subset of results split by kind of request, language or persona. Wide error bars; hypotheses, not findings.

Systems

Trajectory: The sequence of steps an agent took. Do not confuse with the final answer; a correct answer can come from a failing trajectory.

Environment state: What the world looks like after the agent acted. The primary thing to grade for agents.

Guardrail: A runtime control that blocks, rewrites, or routes. Optimised for precision. Do not confuse with an online eval, which is optimised for recall and blocks nothing.

Contamination: Benchmark data present in a model's training set. Do not confuse with leakage, which is your own eval examples reaching your prompts or your tuning loop.

Saturation: A benchmark where top models cluster near the ceiling, so remaining differences are noise and test defects. Do not confuse with theoretical saturation above. The concepts are unrelated, the collision unfortunate.

Translationese: Text that is grammatically in one language but structurally in another. Why translating an eval set does not produce a valid eval set.

Floor: A check with no noise in it: deterministic checks and named incident cases. A failure on a floor is an incident, never a statistical question.

Interleaving: Showing every user one list that mixes two systems' results and counting whose results get chosen. Far more sensitive than an A/B test, but only where the user picks from a list.

Shadow run: Running a new system on real traffic without anyone seeing its answers, to compare offline. Catches the real input mix; cannot measure user reaction.