The three layers of evaluation

Two engineers can argue about evaluation for an hour without disagreeing about anything. One means "does GPT-5 beat Claude on reasoning", the other means "does our support bot stop inventing refund policies". Both call it evals. Almost every confused discussion in this field is this confusion.

Naming the layers fixes it.

The three layers

Layer A: foundation-model evals. Measure a model's general capability, independent of any product. MMLU, GPQA, SWE-bench, ARC-AGI, LMArena. Run by labs and benchmark authors, published as leaderboards, consumed by everyone.

Layer B: system evals. Measure whether your application does its job. Your traces, your gold set, your judges, your definition of a good answer. Nobody else can run these, because nobody else has your data or your definition of correct.

Layer C: process evals. Measure whether your team's evaluation loop is working at all. Are humans still reviewing traces? Does the judge still agree with them? Is the eval set still representative of live traffic? Layer C is the layer that decays silently.

Finding: The split is not our invention, even though the A/B/C labels are. Towards More Standardized AI Evaluation: From Models to Agents (2026) frames the same discontinuity as two eras: a model-centric one built on "static benchmarks, aggregate scores, and one-off success criteria", and an agent-centric one where evaluation becomes "a core control function" for non-deterministic systems. Its charge against current practice is blunt: "high benchmark scores routinely mislead teams", and evaluation is treated as "performance theater" rather than a measurement discipline.

The labels Layer A / B / C are this guide's shorthand and are not standard vocabulary. The distinction they name is widely made; the naming is ours.

Source: Towards More Standardized AI Evaluation, 2026

Why the distinction is expensive to miss

A Layer A score is a rough first guess at Layer B performance, and a weak one. A model can top a coding leaderboard and be the wrong choice for your product because your bottleneck is retrieval, latency, or a tool schema it handles badly. The correlation is real but much looser than procurement decisions usually assume.

The reverse error is worse and more common: teams treat a leaderboard as a substitute for looking at their own data. Choosing a model is a decision you make in an afternoon. Knowing whether your system works is a discipline you maintain for the life of the product. They are not the same activity and the first cannot replace the second.

Common mistake: Answering "how good is our AI feature?" with a model benchmark. The honest answer requires Layer B data you may not have collected yet. Saying so is better than substituting a number that means something else.

Vocabulary that has actually stuck

The terms below are used consistently enough across industry writing to be safe. Terms that have not stabilised are flagged where they appear, and the glossary carries all of them with the confusion each one invites.

Offline vs online. Offline runs against a fixed dataset before deploy. Online scores live traffic after deploy. Both are Layer B; they answer different questions and they routinely disagree. Offline: the 200 saved support tickets you re-run on every release. Online: the actual conversations that happened yesterday. See Online and production evaluation.

Component vs end-to-end. Component evals score one stage: the retriever, the classifier, the tool-call formatter. End-to-end scores the whole system's output. Component: did the search step find the right help article? End-to-end: was the reply the user got any good? Both can pass while the other fails: a perfect retrieval the model then ignores, or a good answer reached despite retrieving the wrong document. Component evals localise a failure; end-to-end evals tell you whether it mattered. You need both, and RAG evaluation is where this bites hardest.

Code-graded vs LLM-graded vs human-graded. Ordered by cost and by trustworthiness, in the same direction. Code-graded wherever the task allows, LLM judges for what is left, humans to validate the judges.

Pointwise vs pairwise. Pointwise scores one output on its own. Pairwise picks the better of two. Pointwise: "is this summary acceptable, yes or no?" Pairwise: "here are two summaries, which is better?" People, and judges, are far more consistent at the second question than the first, which is why pairwise is more reliable for subjective qualities. But pointwise is required whenever you need an absolute threshold, which includes every CI gate: "better than the other one" cannot tell you whether either is good enough to ship.

Reference-based vs reference-free. Reference-based compares against a gold answer. Reference-free judges the output on its own terms. Reference-based: you wrote out the correct refund policy in advance and check the answer against it. Reference-free: you ask a judge "does this answer cite a real policy?" without having written one yourself. Most application evals start reference-free because writing gold answers is expensive, then acquire references for the subset that matters most.

Trajectory vs final answer. Specific to agents. The final answer can be right while the path was wrong, expensive, or unsafe. The agent told the customer the correct delivery date, after calling the refund tool twice and reading another customer's order. The answer passes; the trajectory should not. See Agent evaluation.

Product metrics vs proxy metrics. A product metric is something the business already cares about: resolution rate, escalation rate, time to first correct answer. A proxy metric is something you invented that you hope tracks it. Evals are proxies. Their value depends entirely on how well they track the product metric, and that relationship should be checked rather than assumed.

Where this guide sits

Chapters 2 through 14, and 16, are Layer B and C. Chapter 15 is the single Layer A chapter, and it is about how to read those numbers, not how to produce them.

That weighting is deliberate. Layer A is well covered, well funded, and updated hourly by people whose full-time job it is. Layer B and C are where teams actually lose months, and they are covered mostly by vendor documentation that has an interest in the answer being "buy our platform".