Online and production evaluation

Offline evaluation tells you how your system handles inputs you chose. Production tells you how it handles inputs your users write. Those are two different questions, and they regularly give different answers.

The last row changes everything. Offline you catch a failure before it reaches anyone. Online you count failures that have already happened, in order to prevent the next ones. Only offline can you check a change before you ship it. Only online do you learn what your users really ask. And that is never quite what you assumed.

This chapter says what you build for that: four things, in this order. Then what to do when online and offline disagree, and how to compare changes in production.

Guardrails and evals are not the same thing

A guardrail is a check between your model and your user. It may stop an answer. If the answer breaks a rule, the user never sees it, but gets a refusal, a rewritten version, or a human. An online eval sees the same violation and writes it down. The user gets the answer anyway.

Concretely: your model drafts a reply that contains a customer's full credit-card number. A guardrail spots it and blocks the message. An online eval spots it and counts it. Same detection, opposite job.

A guardrail that blocks 2 % of legitimate answers is a serious incident. A judge that mislabels 2 % of sampled traces is a rounding error. Build them as separate components with separate thresholds. Never let a check built for measuring block traffic without re-tuning it for precision first.

What you build: four things

1. Traces you can query. Every call writes a record with fixed field names.
2. Cheap checks on every trace. The same functions as offline, every night over the whole day.
3. The judge on a sample. The same judge as offline, afterwards, over a small share.
4. A weekly report a person reads. Numbers with spread, plus 20 to 30 traces to read.

The four sections below follow the same pattern: what you do, the file, what you see.

1. Store traces you can query

What you do. On every call of your system you write one line to a trace file or your trace database. The line contains everything a judge or a person later needs to assess the answer. That is at least these fields:

The "standard name" column points at an agreed list of field names, so that every tool can read your traces, not only the one you wrote them for:

Finding: There is now a vendor-neutral standard for this: the OpenTelemetry GenAI Semantic Conventions, developed by the GenAI Special Interest Group, formed in April 2024 under the Semantic Conventions SIG and backed by the CNCF. The standard defines gen_ai.* span attributes: gen_ai.request.model, gen_ai.usage.input_tokens and gen_ai.usage.output_tokens, gen_ai.response.finish_reasons, and structured content capture via gen_ai.input.messages / gen_ai.output.messages. Its scope has grown from bare LLM calls to agent orchestration, MCP tool calls, and quality evaluation. It is implemented by Google Cloud, AWS, Azure and Datadog, among others.

This is the single highest-leverage decision in production evaluation, and it is not an evaluation decision at all. Instrument to the standard, and your traces survive whichever eval platform you pick this year. See The tooling landscape on exit costs.

Source: OpenTelemetry GenAI Semantic Conventions

The file. One line from traces/2026-09-04.jsonl, for the support assistant from the worked example:

{"trace_id":"tr-88a1","timestamp":"2026-09-04T09:12:31+02:00","model":"<vendor>/<model>@2026-07-11","customer":{"plan":"pro","locale":"de"},"input":"Was kostet ein zusätzlicher Platz im Monat?","retrieved":[{"ref":"pricing#plans","text":"Starter: 39 € per seat and month. Pro: 45 € per seat and month."}],"output":"Ein zusätzlicher Platz kostet 45 € pro Monat [doc:pricing#plans].","tokens_in":2310,"tokens_out":41,"latency_ms":1840,"flags":[],"signals":{"escalated":false,"edit_distance":0.04}}

The fields customer, retrieved and output are the same three things the judge sees in the worked example. That is why the same judge can run over production traces without any change.

What you see. Nothing yet. But from now on every question in this chapter can be answered with a query over this file.

2. Cheap checks on every trace

What you do. The deterministic checks from An eval in five files run every night over all traces of the day. In production there is no expected answer, so checks that need expect drop out. The rule checks stay:

• Every citation points at a document retrieval actually returned
• Every money amount has a citation
• The output validates against the schema
• No fragment of the system prompt, no competitor name, no personal data in the answer

These checks cost nothing and run on 100 % of traffic.

The file. A script that runs the checks from checks.py over the day's traces and counts per check:

# evals/online_checks.py
import json
import pathlib
import sys

from checks import CHECKS

ONLINE = ["citations_resolve", "no_unsourced_number"]         # checks without expect

day = sys.argv[1]                                             # e.g. 2026-09-04
traces = [json.loads(line) for line in pathlib.Path(f"traces/{day}.jsonl").read_text().splitlines()]
failures = {name: 0 for name in ONLINE}

for trace in traces:
    for name in ONLINE:
        if not CHECKS[name](trace["output"], trace):
            failures[name] += 1
            trace["flags"].append(name)

pathlib.Path(f"traces/{day}.jsonl").write_text("".join(json.dumps(t, ensure_ascii=False) + "\n" for t in traces))
print(f"{day}  {len(traces)} traces")
for name, count in failures.items():
    print(f"  {name:24} {count:5}  ({count / len(traces):.1%})")

The checks read trace["retrieved"], exactly as they read case["retrieved"] in the offline eval. That is why the fields in step 1 are named the way they are.

What you see. A summary every night:

2026-09-04  3,412 traces
  citations_resolve           31  (0.9%)
  no_unsourced_number         88  (2.6%)

A trace with a failure gets the check's name in flags, and the file is written back with those marks. That is the mark step 3 uses.

What the user did afterwards is counted the same night. These signals are free, come in volume, and are biased in known ways:

Escalation rate is usually the most honest product metric a support-shaped product has. It needs no labels.

3. The judge on a sample

What you do. The judge you measured offline against human labels runs every night over a share of the traces. Not over all of them, because that would be too expensive. Suppose you handle 100,000 conversations a day and run the judge over 5 % of them. That is 5,000 judged and 95,000 not. You know less. In return you get a bill you can pay.

Which traces get drawn is decided by three rules:

• A base rate of 1 to 10 %, drawn at random. It gives the unbiased number for "how good are we overall".
• Every trace with a mark in flags, at 100 %. Those are the interesting ones.
• More from the corners you want to watch. New features, rare languages, long conversations, valuable customers. If 2 % of your traffic is Italian, a random sample of 500 gives you only 10 Italian conversations. That is enough for no conclusion. Draw 100 on purpose. Keep those deliberately drawn traces separate from the base rate in the report, or the overall number is neither one thing nor the other.

The 5 % above is an example, not a recommendation. There is no study that fixes a correct rate. The tools that store production traces, such as Langfuse or Braintrust from the tooling landscape, all ship this pattern as their default, with a base rate of 1 to 10 %. Work out your own base rate the way Statistics shows: large enough to see the change you would act on, and that per slice you watch. On top comes a check on the judge itself: 5 to 10 % of its verdicts are shown again to a human. That is how you notice when the judge gets worse in production than it was on the sealed pile.

The file. A script that draws the sample and calls the judge from run.py:

# evals/online_judge.py
import json
import pathlib
import random
import sys

from run import judge

day, judge_model, judge_name = sys.argv[1], sys.argv[2], sys.argv[3]
BASE_RATE = 0.05

traces = [json.loads(line) for line in pathlib.Path(f"traces/{day}.jsonl").read_text().splitlines()]
random.seed(day)
sample = [t for t in traces if t["flags"] or random.random() < BASE_RATE]

out = pathlib.Path(f"runs/online-{day}.jsonl")
with out.open("w") as f:
    for trace in sample:
        case = {"judge": judge_name, "customer": trace["customer"], "retrieved": trace["retrieved"]}
        passed = judge(trace["output"], case, judge_model)
        f.write(json.dumps({"trace_id": trace["trace_id"], "flagged": bool(trace["flags"]), "passed": passed}) + "\n")

base = [json.loads(line) for line in out.read_text().splitlines()]
base = [r for r in base if not r["flagged"]]
failed = sum(1 for r in base if not r["passed"])
print(f"{day}  base rate: {len(base)} traces, {failed} failed ({failed / max(len(base), 1):.1%})")

The judge runs afterwards over stored traces, never between request and answer. There it would delay every answer by its full latency, and its outages would be your outages. Work out what it costs beforehand: 5 % of 100,000 conversations is 5,000 judge calls a day. At the worked example's numbers, about 3,000 tokens per call, that is roughly €20 a day. If that is too much, lower the base rate, or let a smaller judge pre-sort and run the large one only over what the small one marks.

What you see. One line per night, and one result file per trace:

2026-09-04  base rate: 168 traces, 13 failed (7.7%)

The worked example shows on day 31 what this number is worth: there 8 % failed in production, 2 % in the eval set. The difference was the news.

4. The weekly report, and the reading

What you do. Once a week you sum up the nights and read 20 to 30 traces yourself: everything the judge and a check marked, plus some from the base rate. That is the same activity as the error analysis of day 1, only without an end. It keeps everything else honest, because only that way do you notice failure modes for which there is no check and no judge yet.

The file. The report has a fixed shape. Every number sits next to last week's and with its spread, because a number on its own does not say whether anything moved:

Week 36                        this week        last week    spread
Traces                         23,910           22,480
citations_resolve              0.8 %            0.9 %        ±0.1
no_unsourced_number            2.4 %            2.7 %        ±0.2
Judge, base rate               7.1 %            7.9 %        ±1.5      (n = 1,180)
Judge, Italian                 14.0 %           n/a          ±6.8      (n = 100, drawn on purpose)
Escalation rate                3.1 %            3.0 %        ±0.2
Draft edited (median)          6 %              6 %

Read: 28 traces. Newly noticed: 3 × "customer asks about cancelling, assistant
answers with the price list". No check for it yet. Added to the taxonomy.

What you see, and what you do with it. Three rules for what raises an alarm and what belongs in the report:

• Alarm on deterministic checks. If schema failures or unresolvable citations jump, that is a real incident with a real cause. That may wake someone up.
• Alarm on the escalation rate. It is a product metric, not a proxy. If it rises, something changed for users.
• No alarm on the judge score. If it drops below a threshold for one hour, that is noise. If the seven-day trend sits outside the spread, that belongs in the report. The judge's value in production is diagnosis, not alarm.

When online and offline disagree

This is the most revealing moment. The offline number is good and the online number is not, or the other way round. Check the five causes in this order, because the first ones are the most common and the easiest to check.

1. Your eval set does not look like your traffic. Check: draw 50 production traces and 50 eval cases, mix them, and try to tell them apart. If that is easy, you have your answer. Fix: read 100 fresh traces and refresh the set, as the worked example does on day 31.

2. The system is not what you evaluated offline. Check: compare model version, prompt hash, retrieval index and context assembly between harness and production. Usual culprits: a middleware truncating prompts, a rate limiter degrading to a smaller model, an index that is older offline than online.

3. The judge is worse in production. It was measured on clean examples and now sees messier ones. Check: the 5 to 10 % of judge verdicts a human re-checks. Fix: measure the judge again, this time on labelled production traces.

4. Overfitting to the eval set. Months of iteration against the same set. The signature: the offline score climbs, user signals stay flat. The remedy is in Evals in CI.

5. The eval measures something users no longer care about. Faithfulness rose, users wanted completeness. This is the hardest failure, because every number involved behaves correctly. Check: read the week's escalated conversations and ask whether your criterion sees the reason for the escalation at all.

Common mistake: Trusting offline over online when they disagree, because offline is the controlled measurement. Production is the ground truth by definition. When they diverge, the offline eval is the thing that needs fixing.

Comparing changes in production

A/B testing is the default. It answers the question you actually have: did this change the outcome users experience? Three rules: split users per user, not per request, or a conversation switches systems mid-thread. Measure the product metric, such as the escalation rate, not the judge score. And expect to need more traffic than you think, because product metrics are noisier than eval scores.

Interleaving is the alternative wherever the user picks from a list. An A/B test shows half your users the results of system A and the other half the results of system B, then compares how often each group clicks. That needs a lot of traffic, because the two groups are different people with different needs, and that difference is noise. Interleaving instead shows every user one list that mixes both systems' results: A's first result, B's first, A's second, and so on. It counts whose results get clicked. Every user compares both systems on the same query. So a real difference shows up with far fewer users.

Finding: Interleaving was validated at scale by Chapelle, Joachims, Radlinski and Yue (TOIS, 2012) across two commercial search engines and a scientific-literature retrieval system. The sensitivity figure usually quoted alongside it is from Radlinski and Craswell, Optimized interleaving for online retrieval evaluation (WSDM 2013): across 38 large-scale online experiments and over 3 billion clicks, optimised interleaving maintained sensitivity one to two orders of magnitude above A/B tests while improving agreement with A/B metrics by up to 22%.

"One to two orders of magnitude" is the defensible form. Treat the tidier "10–100×" you will see repeated as a rounding of this result, not as independent corroboration of it.

Source: Chapelle et al., TOIS 2012 · Radlinski & Craswell, WSDM 2013

One to two orders of magnitude means the same conclusion from ten to a hundred times less traffic. The catch: it only works where the user picks from a list. Search results, recommendations, the documents a RAG system retrieves. A chat answer is not a list. The retrieval half of a RAG system produces exactly such a list, though. There interleaving is under-used.

Shadow running. The new system runs alongside on real traffic, without users seeing its answers. You store both answers and compare them offline, with the same checks and the same judge as above. No user risk, and the real distribution of requests. That is the best bridge between the two worlds. Only a user reaction it cannot measure.

Handling the data

Production traces contain user content. Sampling them for evaluation is a processing purpose. It needs a legal basis, a retention period, and an answer to who may read them. Under the GDPR, "we keep everything forever so we can improve the model" is not such an answer.

The practical shape: pseudonymise when writing the trace line. Keep raw traces for a short window only. Retain derived labels and metrics for longer. Separate the surface in which people read traces from general log access. Decide this before you build step 4. The retention window determines what the report may show. Retrofitting it is much harder than designing for it from the start.