Evals in CI

An eval that runs when someone remembers to run it is a document. An eval that runs automatically is infrastructure. The difference is whether regressions get caught before users find them.

Finding: The case for gating prompt changes is empirical, not hygienic. When Generic Prompt Improvements Hurt (2026) shows that generic prompt additions do not produce monotonic improvements: stronger output-contract instructions improved strict extraction for both models tested, while appending the same generic rules dropped one model's RAG citation-compliance from 26/30 to 9/30 passing cases. The authors' conclusion is the one this chapter is built on: "prompt changes should be treated as potential regression risks and tested against task-specific suites before deployment".

A prompt edit is a deploy. That is the whole argument for putting evals in CI.

The design problem is that LLM evals violate the assumptions CI is built on. They are slow, they cost money per run, and they are non-deterministic, so a naive port of unit test habits produces a pipeline that is expensive, flaky, and eventually disabled.

Tier the suite

Three tiers, with different jobs, is the shape that works.

Smoke exists to catch the broken pipeline, not the degraded answer. Schema validation, tool calls parsing, no empty responses, no leaked system prompt, latency within an order of magnitude. Deterministic, seconds, effectively free. This tier catches most of what actually breaks in a change.

Regression is your curated set of real past incidents plus core-path coverage. Every production bug you fix should add an example here. This is the tier that earns its keep. It is a test suite of things that genuinely went wrong. Two kinds of case live here, and they are read differently. The named incidents should sit at 100%: a new failure among them is an incident, not noise. The core-path and judge-scored cases are a rate with a margin, 71% in the worked example, and that rate is reported, never gated. Day 5 uses "regression suite" in exactly this sense.

Full is where slice metrics, expensive judges, and statistical comparison live. It runs on a schedule because it costs too much per invocation to attach to a push, and it reports rather than blocks because its signal needs a human to interpret.

Gate on what is deterministic

The hard part is deciding what fails a build. Getting this wrong in either direction is expensive: too strict and the team disables the check, too loose and it is decorative.

Hard-fail on deterministic checks. Schema violations, unparseable tool calls, policy regex hits, crashes. These are binary and reproducible. Treat them exactly like unit tests.

Never hard-fail just because the judge score dropped a little. An example: 200 cases, 75 % passed. That number has a spread of about ±6 points, simply because 200 cases are few. A gate with the rule "the score must not drop" then fires several times a week without anything having changed. The team learns to re-run until green. After that it ignores the gate. A gate everyone ignores is worse than none.

What the gate checks instead: the paired comparison. Two runs exist as two result files, the baseline and the new run, each with one line per case, the way run.py from An eval in five files writes them. Paired means: you put the two files side by side and compare the same case line by line. Per case there are four possibilities:

Only the last two rows decide. From them the gate computes two numbers. The drop: worse minus better, divided by all cases. And the noise: how much drop can arise just from individual cases coming out one way or the other. The build fails only if the drop is larger than the noise plus a tolerance you set:

# Both runs on the same cases; the comparison is case by case.
worse  = sum(1 for c in cases if baseline[c].passed and not new[c].passed)
better = sum(1 for c in cases if new[c].passed and not baseline[c].passed)
n = len(cases)

drop  = (worse - better) / n                    # observed drop
noise = 1.96 * math.sqrt(worse + better) / n    # what individual cases flipping can explain on their own

assert drop <= noise + TOLERANCE

With the numbers from the worked example: 147 cases, 11 better, 7 worse. Drop = (7 − 11) / 147 = −2.7 points, so an improvement. Noise = 1.96 × sqrt(18) / 147 = 5.7 points. Only a drop above 5.7 points plus tolerance fails the build. Why the arithmetic works this way is explained in Statistics.

For this the baseline has to exist as a file with one line per case, not as a percentage. A percentage cannot be paired with anything. The baseline is the last run on the main branch with the same versions of model, judge and dataset. After every merge the new run replaces it. If model, judge or dataset changes, it is re-run.

What the gate does not catch: a real drop smaller than noise plus tolerance. In the worked example that is up to 5.7 points. That is the price of a gate that does not fire on noise. If you want to see smaller drops, you need more cases, not a stricter gate.

If you report the pass rate itself with a spread and it sits near 100 %, use the Wilson interval instead of the simple formula from Statistics. The simple formula gives a spread of zero at 100 of 100 and an upper bound above 100 % at 98 of 100. Both are wrong. In Python: proportion_confint(k, n, method="wilson") from statsmodels.

Always hard-fail on an incident. An incident is a case from a real ticket on which the system once got it wrong. Its expected behaviour is a contract. Check it deterministically wherever you can. There is no spread for incidents: an incident that passed yesterday and fails today is a relapse, not noise.

Managing flakiness

Non-determinism is a property of the system, not a defect in the pipeline. Handle it explicitly:

• Distinguish infrastructure failures from evaluation failures. A 503 is not a wrong answer. Retry infrastructure errors; never retry a scored failure.
• Fail the run, not the score, when error rates are high. A run with 10% API errors should be reported as invalid rather than as a 10% regression.
• Quarantine, do not delete. An example that flips between runs is telling you something is genuinely borderline. Move it to a tracked quarantine list with an owner; deleting it removes the signal and the memory.
• Cache generations, and set temperature=0 where only the shape is checked. See The offline harness on when production parameters are the better choice. Much apparent CI flakiness is uncached sampling.

Cost control

• Cache generations across runs, keyed on model, params, and prompt. A PR that changes only the scorer should not pay to regenerate.
• Run the expensive tier on merge to main, not on every push to a branch.
• Cap spend per run and fail loudly when the cap is hit, rather than truncating silently. A truncated run reports a score for a dataset that was not fully evaluated.
• Track cost per run over time. Suites grow monotonically and nobody notices until the bill does.

Pinning in CI

CI is where an unpinned model version does the most damage, because the pipeline reports a change that has no corresponding commit.

Pin the model version, the judge version, and the dataset version in configuration that lives in the repository. Then a model upgrade becomes a pull request: visible, reviewable, and diffable against the eval results it changes. That is exactly what you want: the upgrade shows up as a change in scores attached to a change in code.

Preventing eval-set overfitting

The failure mode at the end of this road is a suite that is fully green and no longer predicts anything. Every iteration against a fixed set moves you slightly toward fitting that set.

Contested: How badly this actually bites is genuinely unsettled, and the honest answer is less alarming than the folklore. Adaptive reuse of a holdout is a well-established statistical problem. Dwork et al. built Thresholdout, a noise-adding mechanism that provably supports quadratically more adaptive queries than naive reuse. But when Recht et al. rebuilt the ImageNet test set from scratch and re-evaluated every published model, they found an absolute accuracy drop with the adaptive-overfitting component "limited to non-existent". The ranking held. Later work attributes that robustness partly to ImageNet being a many-class problem, which makes test-set reuse much harder to exploit.

The transfer to LLM application evals is not automatic and cuts against you: your eval set is small, often binary rather than many-class, and iterated against far more aggressively per example than an ImageNet submission ever was. Take the precautions below because your setting is the unfavourable one, not because ImageNet proved they were necessary.

• Keep a test set that CI never touches. Run it before a ship decision, not on every PR, and do not reuse it at every release; retire and replace it by the rule in Dataset design.
• Rotate in fresh production examples on the same cadence as error analysis, every 2–4 weeks.
• Watch for a rising score with flat user metrics. That divergence is the signature of overfitting, and it is the most useful alarm you can build. Online evaluation is where you see it.
• Be suspicious of 100%. Husain and Shankar make the point directly: if you pass all your evals, they are not challenging enough. A pass rate around 70% carries far more information than one at 99%. That holds for the rate-scored part of a suite and for a capability suite, not for the named incident cases, which should sit at 100%.

Contested: Whether evals belong in the merge-blocking path at all. The case against: they are slow and probabilistic, and blocking merges on a probabilistic signal trains developers to bypass it. The case for: a regression suite of real past incidents is not meaningfully more probabilistic than an integration test against a flaky external service, and nobody argues those should be advisory. The workable resolution is the tiering above: block on the deterministic, report on the probabilistic.

A minimal pipeline

The evals command below stands in for whatever runner you use; the shape is what matters.

name: evals
on: [pull_request]

jobs:
  smoke:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -e .
      - run: python -m evals run --suite smoke --fail-on-error
        env:
          MODEL: ${{ vars.EVAL_MODEL_VERSION }}

  regression:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/cache@v4
        with:
          path: .eval-cache
          key: evals-${{ vars.EVAL_MODEL_VERSION }}
      - run: python -m evals run --suite regression --baseline main --gate interval
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: eval-results
          path: results/

Two details do most of the work. --gate interval compares against the baseline using a confidence interval rather than a point estimate. Uploading results if: always() means a failed run still produces the per-example rows someone needs to diagnose it. Without that, a red build tells you something broke and nothing about what.

Common mistake: Putting the full eval suite on every push because that is what you do with unit tests. It is slow and expensive, so someone adds if: label == 'run-evals', and within a month nobody applies the label. Tier from the start.