Error analysis first

There is one activity that reliably separates teams whose AI products get better from teams whose AI products plateau, and it is not a technique. It is sitting down and reading what your system actually produced for real users.

Hamel Husain and Shreya Shankar, who have taught this process to more than 2,000 practitioners across 500+ companies, state it plainly: error analysis is the most important activity in evals, and it is what tells you which evals to write in the first place. Their working estimate is that 60–80% of development time on an LLM product goes to error analysis and evaluation, not as a separate phase but as the substance of the work.

That number sounds wrong the first time you read it. It stops sounding wrong once you notice how much of the alternative, guessing at prompt changes and shipping them, is uncompensated rework.

Why it has to come first

An eval is a question you ask repeatedly. Writing one before you know your failure modes means guessing the question. The guesses are predictable and predictably wrong: teams reach for fluency, helpfulness, tone, and a 1–5 "quality" score, then discover months later that their actual problem was that the retriever never surfaced documents from the last 30 days.

There is also a deeper reason, and it has a name.

Finding: Criteria drift. Shankar et al. (UIST 2024) documented a catch-22 in LLM evaluation: to grade outputs you must externalise your criteria, but the act of grading outputs is how you discover what your criteria are. Some criteria turn out to depend on the specific outputs you saw. They are not definable in advance at all. This is the formal reason a rubric written before you read your data is worth less than one written after.

So the order is not a preference. Writing the rubric first is attempting something the research says cannot be done well.

The loop

1. Get traces in front of a human

A trace is the full record of one interaction: input, retrieved context, tool calls, intermediate steps, final output, and enough metadata to reconstruct what happened. Not just the final answer.

Sample from real production traffic if you have it. If you do not, generate a structured synthetic set. Dataset design covers how to do that without fooling yourself.

Start with 50–100 traces. That is the number Husain and Shankar give, and it is usually enough to find the failure modes that dominate.

2. Open coding

Read each trace and write a short free-text note about what is wrong. Not a category but a sentence, in your own words. For traces with nothing wrong, write that down, and now and then a sentence on why the answer was good. Those sentences later become the passing examples in your labelling guideline; without them a judge only learns to find fault.

Two rules that matter more than they look:

• Note the first failure you see, and further ones only if they are not a consequence of the first. Downstream problems are frequently caused by the first one; recording all of them inflates the count of consequences and hides the cause. But a second failure unrelated to the first is not a consequence, as when the price is wrong and the second question goes unanswered. Note it, marked as secondary. The ranking counts the primary ones; the secondary ones show you which classes you would otherwise underestimate.
• Do not use a predefined category list. The whole point is to discover categories you would not have written down.

Contested: Husain and Shankar state the first rule more strictly: note the first failure, close the trace. That is simpler and protects against double-counting consequences. The price: failure classes that typically occur alongside another land systematically too low in the ranking. In the worked example that is why "ignores the second question" was nearly dropped. The two-level note is our trade-off, not a rule from the source.

Four notes from a real-shaped review session, written exactly as you would write them, rough, specific, no categories yet:

#0142  Quoted a €50 fee. The linked policy page says €0 for Plus members.
#0147  User asked in German, answered in English. Content was fine.
#0151  Said "I've cancelled it" but never called the cancel tool. Nothing happened.
#0158  Cited the returns policy for a question about delivery. Wrong doc entirely.

Note how unglamorous these are. That is correct. The value is in the specificity: "quoted €50, policy says €0" is a note you can act on; "hallucination" is not.

3. Axial coding

Group the free-text notes into a failure taxonomy, a set of distinct, named failure modes. This is the step that turns reading into knowledge, and it is the step people skip.

An LLM is genuinely useful here: paste the notes, ask for a clustering, then correct it. Deciding the categories is yours; the tedious grouping is not.

A useful taxonomy has categories a fix could target. "Bad answer" is not a category. "Cites a document that does not contain the claim" is.

4. Count

Count traces per category. This is the whole reason for the exercise: it converts "the model sometimes hallucinates" into a ranked list.

Continuing the four notes above, after reviewing 100 traces and grouping them:

Two things to read off this table, and both are the point of the whole exercise.

First, the denominator. 42 of 100 traces failed, but the percentages are shares of the failures, not of all traffic. "31% citation errors" means 13 traces out of 100, not 31. Mixing these up is the single most common way error-analysis numbers get misquoted in a standup.

Second, the ranking is usually not what the team expected. Wrong-language replies are the kind of thing that generates a loud complaint and feels like the main problem; here they are fourth. Citation errors generate no complaints at all (the user cannot tell), and they are first.

That inversion is normal, and it is why counting beats remembering.

5. Stop at saturation

Finding: Stop when new traces stop revealing new categories. The heuristic Husain and Shankar give is roughly 20 consecutive traces with no new failure mode, with an absolute minimum of about 100 traces reviewed before you trust the picture.

Source: Husain & Shankar, LLM Evals FAQ

This is theoretical saturation, borrowed from qualitative research, and empirically studied there, not merely asserted. Guest, Bunce and Johnson systematically analysed 60 in-depth interviews and found that no new themes emerged after twelve; the basic elements were present after six. The order of magnitude behind the practitioner rule of thumb above is therefore not invented: it lines up with a measured result from an entirely different discipline.

Source: Guest, Bunce & Johnson, How Many Interviews Are Enough?, Field Methods 18(1), 2006, pp. 59–82 · DOI 10.1177/1525822X05279903

It is a real stopping rule, not a vibe, and it is why error analysis is finite work rather than an open-ended commitment.

What to do with the result

Here is the part that surprises people: most of what you find is not an eval.

A large share of discovered failures are ordinary bugs: a truncated context window, a tool returning an error string the model treats as data, a prompt that contradicts itself. You fix those. No metric required.

Build an automated evaluator only for failures that persist after the obvious fix and that you expect to iterate on repeatedly. The cost hierarchy is steep:

An LLM judge is a real engineering asset with a real maintenance burden. Do not build one for a failure mode you saw twice.

Who does the reading

For most small and medium teams, appoint one domain expert as the decision-maker, Husain's "benevolent dictator". One person sets the standard, others advise, and annotation conflicts stop consuming meetings. Multiple annotators are justified when you genuinely serve multiple domains or cultural contexts, not as a default fairness gesture.

Whoever it is, reduce friction: a single screen showing the input, the trace, and two buttons will get you ten times the review volume of a spreadsheet plus a log viewer. A purpose-built viewer is one of the highest-return internal tools an LLM team can build, and it is usually an afternoon's work.

Keeping it alive

Error analysis is not a phase. Traffic shifts, users learn to use the product differently, and you ship changes.

• Every 2–4 weeks: review 100+ fresh traces and re-run the taxonomy. Categories will appear and disappear.
• Weekly, between cycles: skim 10–20 traces, weighted toward outliers: unusual latency, retries, user-flagged responses.

This is Layer C work, and it is the layer that fails silently. Nothing breaks when a team stops looking at traces. The dashboards keep producing numbers. They just stop corresponding to anything.

Common mistake: Jumping from "we found failures" straight to "we need a judge for each one". Most failures are bugs, and most of the rest are better caught by a code-graded check. The judge is for what genuinely requires reading and interpreting text.