Statistics for small eval sets

Your eval score is not a measurement. It is an estimate produced from a sample, and like every estimate it has a spread. Most eval reporting omits the spread, which is how teams end up shipping a prompt change on a three-point improvement that was noise.

The arithmetic here is undergraduate statistics. It is in this guide because it is almost entirely absent from practitioner writing about evals, and its absence is expensive.

Your eval set is a sample

You care about how your system performs on the population of inputs your users will send. You measured it on 100 of them. The gap between those two things is sampling error, and it is quantifiable.

For a binary eval, meaning pass/fail as dataset design recommends, the standard error of the pass rate is:

SE = sqrt( p * (1 - p) / n )

where p is the observed pass rate and n is the number of examples. The 95% confidence interval is p ± 1.96 * SE.

A 95% confidence interval is built so that it covers the true value if you repeated the same measurement with fresh samples, in 95 repetitions out of 100. It does not tell you how good your system is; it tells you how much your number moves on the strength of which examples happened to land in the set.

Work an example. You run 100 examples and 70 pass.

Step by step, one line at a time:

p = 0.70          70 of 100 passed
n = 100           that is how many examples you ran

Step 1, standard error:
  p * (1 - p)     = 0.70 * 0.30 = 0.21
  divided by n    = 0.21 / 100  = 0.0021
  square root     = sqrt(0.0021) = 0.0458

Step 2, half the interval width:
  1.96 * 0.0458 = 0.090

Step 3, form the interval:
  lower bound = 0.70 - 0.090 = 0.610
  upper bound = 0.70 + 0.090 = 0.790

The 1.96 is not arbitrary: in a normal distribution, 95% of all values lie within 1.96 standard errors of the mean. For a 99% interval the number would be 2.58.

Converted to percentages, 0.610 and 0.790 are exactly 61% and 79%.

Your 70% is therefore somewhere between 61% and 79%. That is the honest report. A change that moves the number to 74% has not been shown to have done anything at all.

This single calculation, applied once, changes how a team talks about eval results permanently.

For a continuous score, the equivalent is SE = σ / sqrt(n), where σ is the standard deviation of the per-item scores.

A limit of this formula: near 0% and near 100% it stops working. At 100 of 100 it gives a standard error of zero, at 98 of 100 an upper bound above 100%. That is exactly where regression suites live. From about 90% pass rate, or below 10%, use the Wilson interval; for the same numbers it is asymmetric and stays between 0 and 100%. At 98 of 100 it reads 93% to 99.4%. Every statistics library has it; in statsmodels it is proportion_confint(k, n, method="wilson").

Three things that make the interval narrower

Three words are about to do a lot of work, so here they are in plain terms:

• The interval is the "somewhere between 61% and 79%" you just computed. It is your uncertainty, written down.
• Error bars are the same thing drawn on a chart: the little whiskers above and below a bar. Same idea, different clothes. Miller's paper title uses this name.
• Variance is how much your number bounces around for reasons that are not your system getting better or worse. More variance, wider interval.

Narrower is better. A narrow interval means you can tell a real improvement from noise. A wide one means you cannot, and you will either ship changes that did nothing or discard changes that worked.

The obvious way to narrow it is more examples, the sqrt(n) in the formula. That is expensive. The three techniques below narrow it without collecting more examples.

Compare in pairs

A version is one state of your system. Version A is the prompt running today. Version B is the changed prompt you want to ship. It can also be today's model and a new one. In the worked example it is the old and the stricter system prompt of day 5. The question is always: is B better than A?

Pairing means: both versions run over the same cases. Then you look at every case on its own and note whether A and B passed it. You do not compare the two percentages at the end.

An example. Version A passes 70 of 100 cases. Version B passes 74. Case by case it looks like this:

The first two rows are 88 cases. On those, A and B come out the same. These cases say nothing about which version is better. Only the last two rows count: 12 cases where A and B differ. B wins 8 of them, A wins 4.

Now the spread. The spread is the range the true difference can lie in. Measured is +4 points for B. There are two ways to compute the spread for it:

Without pairing:  +4 points ± 12.4
With pairing:     +4 points ±  6.8

Without pairing you compute as if the two percentages had come from two different sets. Then the noise of both numbers adds up: 1.96 × sqrt(0.70×0.30/100 + 0.74×0.26/100) = 0.124. With pairing only the 12 cases where A and B differ count: 1.96 × sqrt(8 + 4) / 100 = 0.068. That second formula is the one to remember: 1.96 times the square root of the number of cases where the versions differ, divided by the number of all cases. The gate in Evals in CI computes exactly this.

The same 100 cases, and the spread is almost half the size. Only because you did not throw away how each case went under each version.

Two things about that. First: the spread ±6.8 runs from −2.8 to +10.8. So zero is still inside. 4 points on 100 cases do not prove B is better, even with pairing. They are only closer. Second: how much pairing buys you depends on how often A and B have the same result. The more often, the more it buys. The numbers above are one example.

Finding: Miller's Adding Error Bars to Evals (Anthropic, 2024) reports that paired-difference analysis reduces variance by roughly one third in relative terms compared with treating the two runs as independent samples.

For practice this means: comparing two versions is much easier than saying how good one version is on its own. Most eval decisions are comparisons. So most of the time you are in the easy case. As long as you pair.

Resample per question

Ask a model the same question twice and you can get two different answers. That is what stochastic means, and it means some of the bouncing in your score is the model disagreeing with itself, not a difference between your examples.

Ask each question K times, say 5, and average the 5 scores for that question. Use that average as the question's score instead of a single roll of the dice. This reduces the sampling component of variance without needing more questions, which matters when questions are expensive to label and API calls are not.

Miller also notes that where the eval is a next-token choice, such as multiple choice or classification, you can read the token probabilities directly instead of sampling, which eliminates that variance component entirely. Temperature 0 is the cruder version of the same idea. When it is appropriate and when not is in the harness chapter.

Do not confuse more cases with more information

Suppose your 200 cases are 40 base situations, each worded five different ways. Then you do not have 200 independent cases. You have something closer to 40. Because when the system cannot handle a base situation, it fails all five wordings at once.

Cases that hang together like this are called clusters. If you compute the spread as though all 200 cases were independent, it comes out too narrow. That is the dangerous direction of the mistake: a difference looks certain that is not.

Two words that appear in the findings below. The standard error is the measure of the noise in a measured number; the spread is roughly twice it. Naive means: computed as though all cases were independent.

Finding: Miller reports that correctly clustering standard errors can increase them by up to a factor of three relative to the naive calculation.

Source: Miller, Adding Error Bars to Evals

The effect is nothing special about language models. Survey research has known it since the 1960s. Whoever interviews 200 people but visits only 20 households does not have 200 independent opinions. The spread grows by a factor that Kish (1965) called the design effect. Even a weak link within the groups pushes that factor well above one. Moulton (1986) worked it through for econometrics.

Finding: Independent 2026 work reaches the same conclusion for LLM evaluation and goes further: naive standard errors come out 40–60% smaller than ones that also account for variance from judge model, temperature and prompt wording. On Chatbot Arena data, the coverage of naive 95% intervals falls as n grows, while corrected intervals stay at 95%.

Source: Messing, Hidden Measurement Error in LLM Pipelines, 2026

A reading aid: coverage means how often an interval that promises 95 % actually contains the true value. If it falls as n grows, naive intervals do not get better with more data; they get reliably too narrow. And that is the more uncomfortable of the two findings: even if you handle the clusters correctly, you have not yet captured the noise that comes from the choice of judge model and the wording of the prompt.

You have clusters more often than you think. Several messages from the same conversation. Questions generated from the same document. Synthetic cases built from the same combination of properties. If your cases come in such groups, compute the score per group and treat the groups as the thing you count.

At some point your score is no longer a simple pass rate: judge scores on a scale, clusters, paired differences across groups. Then there is no simple formula for the spread any more. The answer is then the bootstrap, and the idea is simple: you pretend your n cases are the whole world and keep drawing new samples from them. Concretely: draw n cases from your n, where the same case may be drawn more than once. With clusters, draw whole groups. Compute the score. Repeat a thousand times. Sort the thousand values. The 25th and the 975th are the limits of your spread. Twenty lines of code, no assumption about the distribution, the same code for every metric. It is the standard answer whenever a formula becomes unwieldy.

How many examples do you actually need?

Enough to detect the smallest difference you would act on. That is the definition, and it is a calculation, not a convention.

Decide three things first:

• δ (delta): the smallest improvement worth shipping. Be honest: if you would not act on 2 points, do not design to detect 2 points.
• α (alpha): how often you are willing to call a difference real when it is not. Conventionally 0.05: a 1-in-20 false alarm rate.
• 1 − β (power): if the improvement really is there, how often do you want your eval to actually notice it? Conventionally 0.80. Yes, that means the standard is missing a real improvement one time in five.

Say your current version passes 70% and the new one passes 75%. You want to know how many examples it takes to tell those apart reliably. Every number below is one of your three decisions, converted:

n per arm ≈ (1.96 + 0.84)² × [p₁(1−p₁) + p₂(1−p₂)] / δ²

And the arithmetic, one step at a time:

(1.96 + 0.84)²   = 2.80²          = 7.84
0.70 × 0.30      = 0.21                    ← spread of the old version
0.75 × 0.25      = 0.1875                  ← spread of the new one
0.21 + 0.1875    = 0.3975
0.05²            = 0.0025
7.84 × 0.3975 / 0.0025            ≈ 1247

About 1,250 examples per arm to reliably detect a five-point difference, unpaired. That number is shocking the first time you compute it, and it explains a great deal about why teams see eval results that do not replicate.

Read the shape of that formula and you can predict what makes evals cheaper without ever running it again. δ is squared in the denominator: halving the difference you want to detect quadruples the examples you need. Chasing a 2.5-point difference instead of a 5-point one costs you not 2,500 examples but 5,000.

Two ways out, and you should take both:

1. Pair. That is, run both versions over the same cases and compare case by case, as in the section Compare in pairs. The calculation above assumes two separate sets. With pairing the spread gets smaller, and so clearly fewer cases suffice for the same difference.
2. Stop chasing small deltas. A change worth shipping is often worth 15 points on a targeted slice, not 3 points on a general set. Design the eval around the failure mode from error analysis and the effect gets larger while n stays small.

Finding: Miller's worked example shows that increasing the number of sampled answers per question moved the minimum detectable effect from 13.2% down to 7.5%, variance reduction buying real sensitivity without new labelled questions.

Source: Miller, Adding Error Bars to Evals

Slices need their own spreads, and they are wide

A slice is a subset of your eval set: only the billing questions, only the tickets from new customers, only the cases with two questions. Splitting results this way is essential, because only then do you see where the system fails. But every slice has fewer cases than the whole set. And the fewer cases, the wider the spread.

An example. A slice has 20 cases, 12 of them passed, so 60 %. The spread for that is about ±21 points. The true value can therefore lie anywhere between 39 % and 81 %. From 60 % on 20 cases almost nothing can be concluded.

So the rule is: a bad value in a slice is a suspicion, not proof. If a slice looks bad, gather more cases for exactly that slice and measure again. You do not declare a regression on the spot.

A second reason for caution: the more slices you look at, the more certain it is that one of them looks bad by chance alone. If a slice looks bad one time in twenty even though nothing is broken, and you look at twelve slices, one such false alarm is in there on average. You just do not know which one. That too is why slices are suspicions, not findings.

What to report

A defensible eval result contains:

• The point estimate
• A confidence interval, or the n and enough to compute one
• The dataset name and version
• The model and judge versions, pinned
• Whether the comparison was paired
• For judge-scored results, the judge's TPR/TNR against human labels

That is six lines. It is the difference between a number a team can act on and a number a team can argue about.

Common mistake: Reporting eval scores to two decimal places with no n. "Faithfulness improved from 0.82 to 0.86" is not a finding until you know it was measured on the same 400 examples and that the interval does not straddle zero. Precision in the display is not precision in the estimate, and the extra digits actively mislead readers into treating noise as signal.