Your first eval in a week
Five days, at the end of which your project automatically checks whether a change broke something.
Most of this guide is reference. This page is not: it is one route through it, sized for a week, that ends with something running in your CI that would have caught a regression you actually shipped.
This page tells you what to do. The next one, One week, worked through, shows a team doing it, the same five days, with their tickets, files, code and numbers left in. Plan from this page; check your work against that one. If you have never seen an eval as files, read An eval in five files first. It is one screen, and day 3 builds on it.
It is deliberately incomplete. You will skip agents, RAG specifics, multilingual, safety, and most of the statistics. Those are in the rest of the guide for when you need them. What you cannot skip is the order. Every day depends on the one before it.
If you have no production traffic yet, do Day 1 differently and the rest as written.
When you do not need this week
This guide argues for more measurement throughout, so the cases where it is not worth it belong at the front. Four:
- Your system answers so few requests that one person can read them all. Below about 50 answers a week you need no automation: read every one. Day 1 and Day 2 are still worth doing, because that is exactly what they are, with a notepad. A judge is not worth it. A model grading 50 answers a week costs more effort than the reading it would replace.
- A human corrects every output before it goes out. Then you have a product metric that beats any eval: how much of the draft the person had to change. Measure that first. A judge becomes necessary only when you want to know what was changed, not whether. The team in the worked example is exactly this case, and would have started with the edit rate on day zero had it read this paragraph.
- A prototype before the first user. Day 1 with synthetic cases, Day 3 with the smoke checks, nothing else. Everything beyond that measures failure modes you guessed.
- A one-off batch run. Sample, read, done. An eval is a question you ask repeatedly; without repetition there is nothing to automate.
In all four cases one thing remains: read what the system produces. That is the part of this week that never goes away.
Day 1: Read 100 traces
Do: Pull 100 real interactions. Each one is called a trace: the user's input, whatever documents your system looked up, which tools it called, what it finally answered. Read them. For each one, write one sentence about the first thing that is wrong. Do not categorise yet. Do not fix anything yet.
This assumes you log all three parts. If today you only keep the final answer, extend your logging to the input, the retrieved documents and the tool calls first, and come back when a week of them exists. Online evaluation has the standard field names. Final answers alone tell you that something went wrong, not where.
Time: Three to four hours. It feels slow and it is the highest-value day of the week.
No traffic yet? Generate a structured set instead: name the dimensions your inputs vary along, build 20 tuples by hand, expand them to natural language in a separate prompt. Dataset design has the recipe and the four cases where synthetic data will lie to you.
End of day: 100 free-text notes and an uncomfortable sense of how your product actually behaves.
Chapter: Error analysis first
Day 2: Turn notes into a taxonomy
Do: Group the notes into named failure modes. A failure mode is one kind of mistake your system makes, named so that everyone recognises it: "invents prices", "answers in the wrong language", "ignores the second question". The list of them is your failure taxonomy, with a count next to each. Paste them into an LLM, ask for a clustering, then correct it. The grouping is tedious, the naming is yours. The notes contain customer content: use a model covered by your data-processing agreement, or pseudonymise first. Dataset design explains why that is not a formality. Count how many traces fall into each category. Sort by count.
Then split the list in two: failures that are ordinary bugs (a truncated context, a tool returning an error string the model treats as data) and failures that are behavioural and will recur after you fix the obvious things.
Time: Half a day, plus however long the bug fixes take. Fix the bugs now.
End of day: A ranked failure taxonomy with counts, and a shorter list of what actually needs measuring. Expect the top item to surprise you.
Chapters: Error analysis first · Dataset design and annotation
Day 3: Build the set, and grade what code can grade
Do: Assemble at least 100 cases. A case is one frozen situation plus what to check about it, one JSON file each, exactly as in An eval in five files. They come from three sources: every real incident from day 1 that you can reconstruct, the ordinary requests of your core path, and the edge cases your taxonomy from day 2 surfaced. Commit the files to the repository and tag the set, so that two runs can be compared on the same cases. Then move about a fifth of them, around 20, into a directory CI never runs. That part is the test set. What it is for and when you touch it is in Dataset design.
Then write the cheap graders, the bits of code that decide whether one answer passed or failed. Anything with a right answer, a shape, or a rule gets ordinary code rather than a model: schema validation, tool-call matching, "every citation resolves to a retrieved document", "the response never contains the system prompt". These cost nothing to run and return the same verdict every time.
Time: A day.
End of day: A versioned eval set and a scorer module, in the shape of
An eval in five files: a cases/ directory, checks.py and run.py.
Run it with uv run --with openai python evals/run.py. Some of your rule checks
("every citation resolves") will already be failing on production traffic; that is normal and it is free signal.
Chapters: Dataset design and annotation · Code-graded and structured-output evals
Day 4: One judge, validated
Do: Build one judge for one failure mode, and measure how good it is before you trust it. A judge is a second model with a single job: it reads one answer and says pass or fail. Six steps, with one example running through them.
-
Choose one failure mode from your Day 2 list. It has to be one that code cannot catch, because recognising it means reading the answer. "Invents prices" qualifies. "Citation does not resolve" does not, since day 3 already catches that. Take the most frequent one you will keep working on. One, not four: every judge needs its own labels and its own measurement. Turn it into a single question. For "invents prices" the question is: does the cited passage support the number in this draft?
-
Collect 100 to 150 answers your question applies to. Take them from real traffic, not from the Day 3 eval set. If you labelled the eval cases and used the explanations as examples in the judge, you would have written the test cases into the prompt. That is the leak Dataset design warns about. Keep only answers the question fits: for "is the number supported?", answers with a number in them.
-
Have your domain expert label them. Each answer gets pass or fail and one written sentence saying why. At a minute and a half each that is three to four hours. It is the largest item of the day and the one that cannot be trimmed, because these sentences are what the judge is built from and measured against.
-
Split the labels into two piles that never touch. The build pile holds 30 to 40 answers, deliberately half pass and half fail. From it you write the judge's instruction: the question from step 1, the expert's sentences as examples, and the rule that it must give its reason before its verdict. The sealed pile is everything else. It must contain at least 20 to 25 real failures. Fewer, and the measurement in step 5 says nothing: a catch rate computed from five failures can sit anywhere between 0 and 100 %, and even from 25 it still carries a spread of ±15 to 20 points. If you have fewer failures, label more before you measure.
-
Measure the judge on the sealed pile. The judge has never seen it, not even "for a quick look at whether that one is tricky". Report two numbers, each with its count: how many of the real failures it caught ("17 of 25"), and how many of the good answers it left alone ("50 of 55"). Do not report "how often it agreed with us". That number flatters a judge that says pass to everything.
-
If the catch rate disappoints, read the missed cases. Not the total. Revise the instruction, then measure again on the same sealed pile. Two or three rounds are normal. The worked example shows every step with its numbers, including a first version that caught only 14 of 25.
Time: A day, most of it labelling. If it does not fit, labelling is the part that may spill into Day 5, not the measuring.
End of day: One judge with a known error rate, and a known spread on that rate. If it catches only half the real failures, you have learned something worth knowing before you trusted it, which is the point.
Chapters: Building an aligned LLM judge · Where LLM judges break
Day 5: Make it run without you
Do: Two tiers, both wired into CI, the system that already runs your tests whenever someone pushes code.
A smoke suite of the deterministic checks on every push: fast, free, and allowed to block a merge. A regression suite of your Day 3 set on every pull request, which blocks only on the deterministic checks and, for the judge's scores, compares against the previous result case by case and with its margin of error rather than as a bare number. A two-point drop on 100 cases is usually noise, and a gate that fails on noise gets switched off.
Pin the model version, the judge version, and the dataset version in configuration that lives in the repo. Cache generations so a scorer change does not pay to regenerate.
Time: Half a day if your CI already exists.
End of day: A pipeline that will tell you when a prompt change breaks something.
Chapters: Evals in CI · The offline harness · Statistics for small eval sets
What you have, and what you do not
After a week you have a failure taxonomy grounded in real traces, a versioned eval set, deterministic graders running on every push, one validated judge, and a regression suite attached to pull requests. That is more evaluation infrastructure than most teams shipping LLM features have, and all of it is aimed at failures you have actually observed.
What you do not have, and should not pretend to:
| Missing | Read when |
|---|---|
| Anything measured in production | Users are hitting it and you are guessing why · Online and production evaluation |
| Retriever-level diagnosis | Your answers are grounded but incomplete · RAG and retrieval evaluation |
| State and trajectory grading | Your system does things rather than saying them · Agent and tool-use evaluation |
| Non-English quality | You serve a second language and are reporting one number · Multilingual evaluation |
| A safety floor | You expose tools, retrieval, or user-supplied content · Safety and red-teaming |
| Confidence you are not overfitting | Your scores climb while user metrics stay flat · Evals in CI |
| A model decision | Someone asks whether the cheaper model is good enough · Model selection |
The week after
Put a recurring 30 minutes in the calendar to skim 10–20 traces, weighted toward outliers. Every two to four weeks, re-run Day 1 on 100 fresh traces and see whether the taxonomy still holds.
This is the part that decays silently. Nothing breaks when a team stops looking at traces. The dashboards keep producing numbers. They just stop corresponding to anything.
Common mistake: Doing Day 4 first because building a judge feels like the real work. A judge built before error analysis measures a failure mode you guessed at, and you will not find out it was the wrong one, because it will keep returning plausible scores.