The offline harness

A harness is the plumbing that turns "we have an eval set and some graders" into "we can re-run this and compare". It is unglamorous, it is mostly bookkeeping, and getting the bookkeeping wrong is the most common way eval numbers become uninterpretable.

The runner in An eval in five files is the smallest harness there is. This chapter is what it grows into.

What a harness is made of

Four parts, whatever you build it with:

1. A dataset: versioned examples, with expected outputs or grading criteria.
2. A runner: executes your system against each example, with concurrency, retries, and caching.
3. Scorers: one or more graders per example, from code checks to judges.
4. A result store: every run, with enough metadata to reproduce it and to compare it with another.

Inspect names the same four parts slightly differently, and its naming is worth borrowing even if you never use the tool:

The important property is that the four parts are separable, and that pays off in two everyday situations. You want to try a new model: swap the solver, and the dataset and scorers stay untouched. You think of a new check: add a scorer and re-mark yesterday's saved outputs, with no need to pay for generating them again.

A harness where these are tangled together makes both of those a rewrite instead of a one-line change.

The fourth part is the one teams skip and the one that determines whether the harness is worth having.

Pin everything

An eval score is only meaningful relative to the conditions that produced it. Every unpinned variable is a way for two runs to differ for reasons nobody recorded.

The rule of thumb: if you cannot reconstruct a six-month-old number from what you stored, you did not store enough.

Pin the prompt format too

This is the pin people do not think of, and there is a striking result behind it.

First, what is being pinned. Many evals show the model a few solved examples before the real question, the "in-context examples". Something has to separate them. That separator is the delimiter, and it is usually chosen without a thought:

Question: What is the capital of France?
Answer: Paris
###                          <- this is the delimiter
Question: What is the capital of Spain?
Answer: Madrid
###
Question: What is the capital of Italy?
Answer:

Swap that ### for a blank line, or a ---, or \n\n, and nothing about the task has changed. The finding is that the score does.

Finding: A Single Character can Make or Break Your LLM Evals (2025) found that the delimiter separating in-context examples changes MMLU performance by up to ±23%, with drops of 18.3–29.4% across Llama, Qwen, and Gemma families. Model rankings could be reordered to put any model in the lead by changing that one character. The effect pervades topics and model families and does not improve with scale. Naming the chosen delimiter explicitly in the prompt improved robustness.

Put that number next to what you are usually trying to detect. A prompt improvement worth shipping might move your score by 3 to 5 points. That the separator moves MMLU by 23 does not mean it moves yours by 23. The finding comes from few-shot prompts to open-weight models, a Layer A setup, and your application eval through a chat API looks different. What it shows is the order of magnitude a formatting detail can reach. How large it is for you, you find out only by varying it once. Until then, treat it as a variable whose effect you do not know: pin it.

The implication is not that benchmarks are worthless. It is that the prompt template is an experimental condition whose effect size can exceed most of the changes you are trying to measure. Treat it as pinned configuration, version it with the dataset, and never let a "harmless formatting tidy-up" land in the same change as a model comparison.

If you are comparing models, hold the format fixed across them, and be aware that a format favouring one model is a real confound rather than a hypothetical one.

Determinism, and its limits

Temperature is the dial that controls how much randomness a model uses when it picks its next word. At every step the model has a ranked list of candidates. At temperature=0 it always takes the top one. Turn the temperature up and it will sometimes take the second or third. That is what makes output feel varied and creative, and what makes the same question produce a different answer each time.

temperature = 0     same question → (almost) the same answer, every run
temperature = 1     same question → a different answer each run

For an eval you want 0. You are trying to measure whether your system changed, and random variation between runs is noise sitting on top of that measurement. Creativity is a product feature; in a measurement it is interference.

Set temperature=0 in offline runs if that is what your product runs at. Where the eval is a constrained choice, read token probabilities instead of sampling. See Statistics.

Contested: Whether an eval should run at temperature 0 when the product runs at 0.7. For: less noise, cheaper runs, reproducible diffs. Against: you are then measuring a different system from the one users see, which is exactly the fifth case in Online evaluation, and failure modes that only arise through sampling never show up in the eval. This guide's position: production parameters, K samples per case, the mean as the case's score. Temperature 0 is the shortcut for tiers that check only the shape and not the answer (the smoke tier) and for products that run at 0 anyway.

Do not expect bitwise reproducibility. Hosted inference varies with batching, hardware, and silent backend changes, and temperature=0 reduces variance without eliminating it. Plan for it rather than fighting it:

• Report a mean over K samples for anything you will make a decision on.
• Treat a small score change with no code change as measurement noise until proven otherwise.
• Log the raw outputs, not just the scores. When a number moves, the outputs are the only way to find out why.

Cache aggressively

Two caches, doing different jobs:

Generation cache, keyed on (model, params, prompt). Lets you add or fix a scorer and re-score without re-generating, which is most of the cost, and most of the wait.

Judge cache, keyed on (judge model, judge prompt, output). Judge calls dominate the cost of a mature eval suite.

Both must key on everything that affects the result. A cache keyed only on the prompt text will serve you yesterday's model's answers after a model upgrade, and it will do so silently. This is a genuinely common bug and it produces the worst possible symptom: an eval that says nothing changed.

Budget

Evaluation costs real money and real time, and a suite nobody runs because it takes forty minutes is not a suite.

• Tier the suite. A fast subset on every change, the full set nightly. See Evals in CI.
• Put the cheap graders first. If schema validation fails, you rarely need to pay a judge to assess prose quality.
• Use a smaller judge where you have validated one. Judge quality is measured, not assumed. If a smaller model hits your TPR/TNR bar on your sealed pile, it is a valid judge and it costs a fraction.
• Track cost per run as a first-class metric. Suites grow, and the moment of "why is CI suddenly expensive" comes later and more painfully than the moment of noticing.

Concurrency and failure

Practical details that determine whether the harness is usable:

• Bound concurrency and handle 429s with backoff and jitter. An eval run is a burst load.
• Distinguish system failures from evaluation failures. A timeout is not a model getting the answer wrong. Count them separately; a run with 12% infrastructure errors is not a valid measurement, and averaging them in as failures quietly understates your system.
• Make runs resumable. With a generation cache this is nearly free, and it turns a crashed hour into a retried minute.
• Fail loudly on partial runs. A suite that silently scores 340 of 400 examples reports a number for a dataset that does not exist.

Store results so they can be compared

The minimum record per run:

run_id, timestamp, git_sha
dataset_name, dataset_version
model, model_version, decoding_params
judge_model, judge_version, judge_prompt_hash
per_example: input_id, output, scores{}, latency, tokens, cost, error
aggregate: score, n, standard_error

Per-example rows are the part that pays off. Aggregates tell you a number moved; per-example rows let you diff two runs and see which examples changed, which is the first question anyone asks and the one an aggregate cannot answer.

Common mistake: Building the harness before doing error analysis. The harness's shape should follow from what you are measuring. Teams that build infrastructure first end up with an elegant runner wired to metrics they chose because the framework provided them.