The Evals Guide
How to find out whether your AI feature actually works, and whether a change made it better.
Most teams shipping an LLM feature discover the same thing in the same order. The demo works. The launch goes fine. Then support tickets arrive describing failures nobody can reproduce, someone proposes "adding evals", and the team spends three weeks integrating a platform that produces a dashboard of numbers no one trusts or acts on.
That sequence is avoidable, and avoiding it has almost nothing to do with which tool you pick.
This guide is about evaluating your application: the retrieval pipeline you built, the agent you deployed, the support bot answering your customers. It is not about ranking foundation models. That distinction runs through every chapter, because conflating the two is the single most expensive mistake in this field.
Who this is for
Engineers and applied scientists shipping production LLM features: RAG systems, tool-using agents, copilots, classifiers, support automation. You have a system in front of users, or you are about to. You need to know whether a change made it better.
It assumes you can read Python and think in terms of precision and recall. It assumes nothing about your familiarity with the evaluation literature.
The three layers, and which one this is about
Almost every argument about evals dissolves once you notice the participants are talking about different layers.
| Layer | What it measures | Who runs it | Example |
|---|---|---|---|
| A. Foundation-model evals | A model's general capability | Labs, benchmark authors | MMLU, GPQA, SWE-bench, LMArena |
| B. System evals | Whether your application does its job | You | Your traces, your gold set, your judges |
| C. Process evals | Whether your team's evaluation loop works | You | Error analysis, annotation, CI, monitoring |
The spine of this guide is B and C. Layer A appears exactly once, in Reading foundation-model benchmarks, framed as what it actually is: a skill for interpreting other people's numbers.
The reason for that weighting is empirical. A model that scores well on MMLU tells you close to nothing about whether your retrieval pipeline surfaces the right statute, and a great deal of wasted effort comes from teams treating a leaderboard as a substitute for looking at their own data.
What this guide will not do
- Rank models. Those numbers are stale within weeks and several sites already do it hourly.
- Recommend a vendor. The tooling landscape is descriptive and unranked, which is unusual enough to be worth stating plainly: nearly every high-ranking "best eval tools" comparison is published by one of the tools it compares.
- Teach prompt engineering. promptingguide.ai covers it well and this guide does not duplicate it.
- Pretend things are settled when they are not. Where practitioners genuinely disagree, both positions appear, marked Contested.
The shape of the argument
If you read nothing else, read these four claims. Everything downstream is detail.
1. Look at your data before you measure anything. Error analysis, reading actual traces and categorising actual failures, determines which evals are worth building. Teams that skip it build elaborate measurement for problems they do not have. Hamel Husain and Shreya Shankar teach the most-subscribed practitioner course on LLM evals, and have trained more than 2,000 engineers and product managers across 500+ companies. They put 60–80% of development time in this bucket.
2. Use the cheapest grader the task allows. A schema check is deterministic, instant, free, and cannot be talked out of its opinion. An LLM judge is none of those things. Code-graded evals come first; judges are for what is genuinely left over.
3. A judge you have not validated is a random number generator with good manners. The step people skip is measuring the judge against human labels it has never seen. Without it you have automated the production of scores, not the production of knowledge. Building an aligned judge is the how; where judges break is the why it matters.
4. Your eval score is an estimate, and estimates have error bars. An eval set is a sample. A three-point improvement on 100 examples is usually noise. Statistics for small eval sets covers the arithmetic, which is less intimidating than its absence from most practitioner writing suggests.
How to read it
The chapters are ordered the way work actually happens, not alphabetically and not by taxonomy. Read in order the first time. When a term is unfamiliar, the glossary defines each one in a line and names the thing it is most often confused with.
If you are in a hurry, Your first eval in a week is a five-day route through the guide that ends with a regression suite running in your CI. It skips most of the subject on purpose. It is also the highest-return week most LLM teams have available.
If you have never seen what an eval is as files, start with An eval in five files: three cases, three checks, a runner, one command and its output, on one screen. Every later chapter adds to those files and replaces none of them.
On sourcing
Numbers in this guide come in three kinds, and it is worth knowing which you are reading. Sourced findings sit in callout boxes and carry a link to the source. Worked arithmetic, the confidence intervals in Statistics and the pass^k calculation in Agents, is derived in the text so you can check it. Illustrative figures in examples are invented to make a shape concrete and are labelled as such. Where the underlying evidence is thin the text says so rather than rounding it into confidence, and the bibliography notes how much weight each source can bear, including dated corrections where earlier drafts of this guide got a number wrong.
That is a deliberate constraint, and it removed several claims that would have made better copy. Evaluation content that cites nothing is asking you to take on faith exactly the thing the field exists to stop you doing.
Common mistake: Treating "we need evals" as a tooling decision. It is a process decision with a small tooling component. Teams routinely install a platform before they can name a single failure mode their system has, then wonder why the dashboard is not actionable.