Dataset design and annotation

An eval set is a claim about what your users do. Every property it has is a claim you are making: size, composition, how it was gathered, how it is labelled. Most eval sets make claims their authors never examined.

This chapter is about making those claims deliberately.

How big does it need to be?

There are two answers and you need both.

The practitioner heuristic. For a CI set, the one that runs on every change, Husain and Shankar suggest 100+ examples, purpose-built to cover core features, regression cases from real incidents, and known edge cases. Not a random sample; a curated one.

The statistical answer. The size you need depends on the difference you want to detect and how noisy your scores are. A 5-point improvement on a 100-example set is often inside the noise band. Statistics for small eval sets has the arithmetic; the short version is that you should calculate this once rather than adopting a number from a blog post.

The two answers do not conflict. 100+ is where useful CI signal starts. Whether it is enough for your decision is a calculation.

Common mistake: Growing the eval set to feel rigorous. A larger set of unrepresentative examples is a more expensive way to be wrong. Composition beats size until you are trying to resolve small differences.

Where examples come from

Mining production traces

The best source, by a wide margin, is traffic that already happened. It is free, and it already has the right mix of requests, which is the property synthetic data spends the most effort trying to fake.

Do not pick traces at random. Say your support bot handled 1,000 conversations last week. About 600 of them were "hi" and "reset my password". A few request types always make up most of the traffic. Pick 100 at random and roughly 60 are those easy ones. The system already handles them, so they tell you nothing. The failures you want are in the other 40, the rare and awkward requests. So decide what goes in, instead of drawing a random sample. Ways to do that:

• Stratify by the failure taxonomy. Stratifying means: first divide into groups, then draw deliberately from each group. The groups are the failure classes from error analysis, and from each you take a fixed number of traces that show exactly that failure, so that every class is measurable, not just the ones chance happened to deliver.
• Oversample the tail. Long inputs, rare tools, retries, timeouts, and non-English traffic, which needs its own treatment rather than a few extra rows here.
• Include the boring cases anyway. If your eval contains only hard cases, you cannot detect a regression that breaks the easy path, and that is the regression users notice.
• Take user-flagged responses. Thumbs-down is a free label with a strong signal and a well-known bias: it captures dissatisfaction, not incorrectness.
• Attach metadata to every case: intent, language, user type, source. Cheap when the case is created and expensive to add later, and without it there are no slices to segment by afterwards.

Handle this data properly. Production traces contain user content, and using them as an eval set is a processing purpose that needs a legal basis, a retention rule, and usually pseudonymisation. This is not an afterthought bolted on at the end; the retention window determines whether the eval set can exist at all.

Longer conversations as cases

In a chat with several messages, a case is not "one message, one answer". A case is: the chat so far, stored as it was, plus the user's next message. The system only has to produce the one answer to that, and that answer is what you grade. As a file it looks like this:

{
  "id": "case-217",
  "history": [
    { "user": "Where is order A-2231?" },
    { "system": "It is with the carrier and due on Thursday." },
    { "user": "Can you send it to my work address instead?" },
    { "system": "Yes. What is the address?" }
  ],
  "input": "Hauptstraße 12, 80331 München.",
  "expect": { "mentions": ["A-2231"] },
  "note": "Original answer: 'Which order was that again?' The number from message 1 was forgotten."
}

history is frozen text. The system reads it but does not answer it again; otherwise you would be testing a different conversation on every run. Only the answer to input is new, and expect says what it has to get right.

A chat where something went wrong becomes a case like this: history up to the message before the failure, input is that message. So when you take notes in error analysis, write down at which message it happened. And if you cut several cases from one chat, they are related; Statistics says why those count for less than independent cases.

Synthetic generation, done structurally

When you have no traffic, as with a pre-launch product or a new feature, generate examples. The naive approach ("write me 100 customer questions") produces 100 paraphrases of the same three questions, because that is what sampling from a mode gives you.

Finding: The collapse of aligned models onto a few typical outputs has been studied independently. One cause sits in the preference data itself: annotators systematically favour familiar-sounding text ("typicality bias"). Unfolding the diversity deliberately, rather than simply sampling more, raises it by 1.6–2.1× over direct prompting in creative writing.

Source: Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity

The structured recipe from Husain and Shankar avoids this:

1. Name the dimensions along which real inputs vary. For a recipe assistant: dietary restriction, cuisine, complexity, user expertise, ingredient availability.
2. Build tuples by choosing one value per dimension. Write about 20 by hand first. You will discover dimensions you missed.
3. Generate in two steps. First produce the tuples structurally, then convert each tuple to natural language in a separate prompt. Doing both at once collapses the diversity you just engineered.
4. Run them through the system and error-analyse the results as normal.

The two-step split is the part that matters. A single prompt asked for both structure and phrasing will optimise for fluent phrasing and quietly abandon the structure.

A worked example. Say you are building that recipe assistant. Step 1 gives you five dimensions with three or four values each:

Step 2 draws a tuple from these, for instance (nut allergy, Thai, 15 minutes, beginner, leftovers only). That is not yet a user question, only its blueprint. Five dimensions with three to four values each yield more than 500 combinations; you do not need all of them, but you can see immediately how narrow the yield of a single "write me 100 questions" is by comparison.

Step 3 turns this one tuple into natural language in its own call. The prompt contains nothing but the tuple and the instruction to write one realistic user question: no list, no further examples, so the model cannot fall back into its default mode. What comes out looks like:

"I've got coconut milk and some leftover veg, but I have to avoid nuts entirely. Anything Thai I can do in fifteen minutes? I don't cook much."

That single sentence shows the difference: restriction, cuisine, time budget, ingredient situation and the user's hesitancy are all in there because they were fixed as structure beforehand, not because the model happened to think of them. Repeat step 3 per tuple and you have an eval set whose diversity you can name.

When synthetic data will mislead you

Finding: Four situations where synthetic generation produces unrealistic data and should not be trusted: complex domain-specific content (legal filings, medical records), low-resource languages, high-stakes domains requiring manual validation, and representing underrepresented user groups.

Source: Husain & Shankar, LLM Evals FAQ

The common thread is that the generator's competence is the ceiling on the data's realism. Where the model is weak, the synthetic data is confidently wrong in exactly the way the model is, and your eval set inherits the blind spot you were trying to measure.

Contamination and leakage

Both come down to the same thing: the answers are known in advance, so your score measures something other than what you think. Only the route there differs.

Public benchmark contamination: the benchmark was in the model's training data. It saw the test questions before you tested it. The score then measures recall rather than capability, and says nothing about how the model handles your inputs. For public Layer A benchmarks, assume this has happened rather than treating it as a possibility.

Your own leakage: knowledge of the test cases seeps into the system you are trying to assess. This is the likelier of the two, because you cause it yourself. An example: you take three particularly tricky cases from your eval set and drop them into the prompt as few-shot examples so the model handles them better. It then passes exactly those three. What you measured is that a model can reproduce what is in its own prompt.

The same thing happens when you tune the prompt until the eval score looks right: the prompt is now fitted to your eval set, and the number says nothing about new inputs. Or when a synthetic set is generated by the very model under test. Then you are testing the model against its own assumptions.

Prevention is unglamorous and effective:

• Keep a test set: a portion of the examples you never look at while developing. You touch it only when a decision is due ("do we ship?"), not while experimenting. How large, for how long, and what comes after is in the next section.
• Never draw few-shot examples from the eval set.
• Rotate in fresh production traces on the same cadence as your error analysis.
• Version the set, and record which version produced which number.

Separating development and test data

If you change the prompt until your cases pass, the cases no longer tell you anything about new inputs. It is like a student who has seen the exam questions in advance: they pass, but you do not know whether they know the subject.

So you split your cases into two parts. For the 147 cases the team in the worked example ends up with, roughly like this:

"You" in this section means the team that builds the system and writes the cases. The model never sees either set as a whole; on every run it only receives individual inputs.

Do not put incidents into the test set, that is, cases from real tickets on which the system already got it wrong. Incidents have to run in CI every time, because each one on its own signals a regression. Evals in CI explains why an incident needs no error bar. Incidents therefore always stay in the development set.

The team may read the results on the development set as often as it likes: which cases fail, why, what the draft says. The results on the test set the team reads only before a decision whether to ship. If the test set comes out badly, the team goes back to the development set and looks for the cause there. It does not fix things against the test set's cases. Otherwise it tunes the system, piece by piece, to exactly those cases, like a student who knows the exam questions in advance. The test set then no longer measures how good the system is, only how well it knows those 20 cases.

After a few decisions the team has read the test cases often enough anyway. Then they become ordinary development cases, and the fresh traces from the next error analysis become the new test set. Nothing is thrown away, and there is always a test set nobody has yet used for fixing.

In the five-day week you set the test set aside on Day 3; it is not needed until the first ship decision.

Labelling

Labelling means going through your eval examples and recording, for each one, whether the answer was acceptable or not. Those judgements are the reference everything else is measured against, including an LLM judge you align to them. Without them you have an eval set but nothing to measure it against.

How much of this section applies to you comes down to one question: is one person labelling, or several? Working alone, you only need the first two subsections: binary rather than a scale, and writing the guidelines down. The third, measuring agreement, only becomes relevant once more than one person is passing judgement.

Binary, not Likert

Finding: The practitioner consensus is binary pass/fail over 1–5 scales. The reasons: a scale hides uncertainty in the middle values, annotators disagree about where 3 becomes 4, detecting a difference on a graded scale needs a larger sample, and binary decisions are simply faster.

Source: Husain & Shankar, LLM Evals FAQ

If you need nuance, get it by decomposing rather than grading. Instead of "quality: 3", use three binary checks: cites a real source, answers the question asked, stays within policy. Each is separately actionable; the average of the three is not.

A third form besides binary and scale is the category: the judge assigns each answer one failure mode from a fixed list, such as "wrong plan", "outdated section", "no source", "no failure". That is not a scale, because the categories have no order. Measure them like several binary criteria: one catch rate per category against the labels. A scale stays defensible for one job only: putting two answers in order, as in a pairwise comparison. As a gate or as an absolute number it is no good.

Contested: The academic literature continues to report Likert correlations, and reference-based scoring often assumes graded output. This is a genuine split between research practice and product practice, and both sides have a point: graded scores carry more information per item, binary labels carry more reliable information per item.

A second piece of evidence against the 1–5 scale comes from an entirely different corner, annotation research. Kiritchenko and Mohammad had sentences rated for their sentiment, once on a scale and once like this: the raters get four sentences and name only the best and the worst. For the same effort, the second method was more reliable, because people answer "which one is better?" more consistently than "how good is this one, from 1 to 5?". The method is called best–worst scaling. So both findings say the same thing: the scale is the problem, not the raters. They differ only in the replacement: pass/fail here, best/worst there.

Source: Kiritchenko & Mohammad, Best-Worst Scaling More Reliable than Rating Scales, ACL 2017

Write the guidelines down

Annotators are the people who assign labels: they read an answer and write pass or fail next to it, with a reason. In the worked example that is the support lead. It can also be a model, if you let a model label. The guideline is the text they decide by: the criterion in one sentence, plus examples. Whoever labels cannot ask anyone mid-task how a case is meant. That holds for people and for models alike. A written guideline is therefore the only way to make every label follow the same rule.

The guideline is not finished at the first label. While labelling you hit cases nobody thought of, and then you change the guideline and relabel the earlier cases under the new version. That is criteria drift: you only find the criteria while grading. It is not a process failure but the normal case.

A workable guideline has, per criterion: a one-sentence definition, two passing examples, two failing examples, and at least one explicit edge case with its resolution. The edge cases are the part that actually transfers.

Here is what one looks like. Say your criterion is "cites a real source":

Definition. The answer refers to at least one document that was actually retrieved.

Pass. "According to the 2026 price list (document PL-2026), the plan costs €49." Document was in context, identifier checks out.

Pass. "It's in the price list you sent me." No title given, but unambiguously about a retrieved document.

Fail. "Plans like this typically run about €50." Plausible, but sourceless.

Fail. "Per section 5 of the terms …" when the terms were not in context. Sounds like a source, is invented. The expensive case.

Edge case, resolved. The answer cites two sources, one of them invented. → counts as fail. Reasoning: an invented source sitting next to a real one is harder for a reader to catch than one on its own, not easier.

That last block is the actual point of the exercise. Two people read the same definition and rule differently on this one answer, until somebody writes the case down and gives the reason.

Measure agreement, if you have more than one annotator

What this is for: you are not testing the annotators, you are testing your guidelines. If two people rate the same answers differently, the rule is usually unclear rather than the person unreliable. And if you later align an LLM judge to these labels, it inherits exactly that fuzziness. A judge never becomes more consistent than the labels it was built from.

Raw agreement percentages are misleading when classes are imbalanced. With 90% passes, two annotators who both always say "pass" agree 90% of the time and have demonstrated nothing.

κ (kappa) answers exactly one question: how much agreement is left once you subtract the agreement two people would have reached by guessing alone? 0 means "no better than chance", 1 means "in complete agreement". Three variants are in common use:

• Cohen's κ: two annotators, chance-corrected. The default.
• Fleiss' κ: more than two annotators.
• Krippendorff's α: handles missing ratings and non-binary scales; the right choice for messy real annotation workflows where not everyone rates everything.

For calibration: the common bands go back to Landis and Koch (1977): 0.41–0.60 "moderate", 0.61–0.80 "substantial", above that "almost perfect". The authors themselves call these cut-offs arbitrary; they are a convention, not a measurement.

Contested: How high the bar should sit is disputed across disciplines. In computational linguistics the rule of thumb since Artstein and Poesio has been κ > 0.8 for reliable, 0.67–0.8 for tentative conclusions. That is considerably stricter than the κ ≥ 0.5 we propose here as a working bar for eval labels. We think the lower threshold is defensible because eval labels support a decision rather than a publication, but it is a trade-off, not a rule. If you are publishing or being audited, take the stricter bar.

Source: Artstein & Poesio, Inter-Coder Agreement for Computational Linguistics, Computational Linguistics 34(4), 2008; Landis & Koch, Biometrics 33(1), 1977

Human–human κ on a clean task can reach 0.97 (see the TriviaQA figure in judge failure modes), so low agreement usually means the guidelines are underspecified, not that the task is inherently subjective.

The number is only the start. What comes after it is what makes the labels better: take the cases the annotators disagreed on, discuss them, and change the guideline, one more resolved edge case, not one person's opinion. Then both relabel the disputed cases. If a case is still disputed after that, mark it ambiguous and take it out of the pile you measure the judge on. A judge cannot be better than humans who do not agree. Your annotators' agreement on the same cases is the ceiling against which to read its catch rate.

If you have a single decision-maker, you do not need these numbers. You need the guidelines anyway: for the day that person is on holiday, and for the judge you are about to build from their labels.

Versioning

Treat the eval set as source code, because that is what it is.

• It lives in the repository, or in a store with immutable versions.
• Every reported number carries the dataset version that produced it.
• Adding examples creates a new version; comparing scores across versions is invalid and should be made hard to do by accident.
• Removing an example requires a reason in the commit message. Silently deleting examples your system fails is how an eval set becomes decorative.