One week, worked through

The previous chapter says what to do. This chapter shows one team doing it. Every day follows the same pattern: what the team did, the file that came out of it, the result in numbers, and what it means. The files are the ones from An eval in five files. They only grow.

Everything here is invented. The team does not exist and no number was measured. The numbers are chosen to look the way real numbers look: a judge that starts out bad, an improvement that turns out to be noise, an eval set that quietly goes stale.

One ticket runs through the whole chapter: T-48213. On day 1 it is a note, on day 3 a case, on day 4 a label, and it is the reason the first judge had to go.

The words this chapter uses

Twelve words, one line each. Every other term is explained where it appears.

The system

A SaaS for shift planning. About 4,000 business customers, six people in support, about 2,800 tickets a month in German and English. The assistant is a single model call: it takes the ticket, searches the product documentation and writes a draft reply with citations. A person reads the draft, edits it and sends it. Nobody has ever measured it.

This is what a trace looks like. Three parts. The ticket, as the helpdesk exports it:

{
  "id": "T-48213",
  "created": "2026-03-09T08:41:00+01:00",
  "customer": { "account": "acc_7731", "plan": "pro", "seats": 34, "locale": "de" },
  "subject": "Zusatzplätze und Nachtschicht-Zuschlag",
  "body": "Guten Tag,\n\nwir brauchen ab April 6 weitere Plätze. Was kostet ein zusätzlicher Platz im Monat?\n\nAußerdem: rechnet Ihr System den Nachtschicht-Zuschlag automatisch, wenn eine Schicht um 21:45 beginnt?\n\nViele Grüße\nM. Brandt"
}

The three documentation passages retrieval found:

pricing#plans           | Starter 39 € per seat/month · Pro 45 € per seat/month
pricing#seats           | "Additional seats are billed pro rata from the day
                        |  they are activated."
rules#night-surcharge   | "The night surcharge applies to shifts beginning at
                        |  22:00 or later."

The assistant's draft:

Guten Tag Herr Brandt,

zusätzliche Plätze kosten 39 € pro Platz und Monat [doc:pricing#plans] und
werden ab dem Tag der Aktivierung anteilig berechnet [doc:pricing#seats].

Viele Grüße
Ihr Support-Team

Two mistakes. The customer is on the Pro plan, so the price is 45 €, not 39 €. The number is on the cited page, just in the wrong row. And the second question, about the night surcharge at 21:45, is not answered. The answer would have been no.

Tools. No framework. Everything is Python and files in Git.

A framework would have been a reasonable choice too. The tooling landscape describes what frameworks do.

Day 1: Read 100 traces

What the team did. It pulled 100 consecutive tickets from one week, each with its draft and the retrieved passages. Two people read 50 each and wrote one sentence per trace about the first thing that was wrong. No categories, no fixes.

The file. A text file with 100 lines. Four of them:

#7    (T-48213) Says the Pro plan is €39 per seat. It's €45. The number is
      on the cited page but it's the Starter row. Also never answers the
      night-shift question.
#23   Customer wrote in German, draft came back in English. Agent rewrote
      the whole thing by hand.
#41   Points at "Settings → Shift templates → Bulk edit". There is no
      Bulk edit.
#58   "I've escalated this to our billing team." It cannot escalate
      anything. Nobody was escalated.

Result. Three and a half hours per person. Both had independently written "invents prices" about a dozen times.

What it means. The sent replies were fine, because support had been repairing the drafts by hand for months. That is why there was no complaint. The measure that would have shown the failure earlier is the distance between draft and sent reply. Nobody had ever computed it.

Day 2: Notes become failure modes

What the team did. It pasted the 100 notes into a model and asked for groups. It threw half the groups away and renamed the rest. Each trace counts once, under the first thing that was wrong. So T-48213 counts as a price failure.

The result. The failure modes, sorted by frequency:

What it means. Two of the rows were ordinary bugs and were fixed the same afternoon. The helpdesk strips Markdown tables, so the prompt now forbids them: 4 traces gone. The search index was six weeks old and served anchors to renamed sections. After a rebuild, 11 dropped to 3. That left 51 traces with behavioural failures that no bug fix removes. The first target is clear: the largest row is also the one that costs money once a customer acts on it.

Common mistake: Building the list around what is easy to fix. The team nearly dropped "ignores the second question" because nobody knew what to do about it. The row stayed, with its count, as an unmeasured failure. That is the correct treatment for a real failure you cannot yet address.

Day 3: The eval set and the checks

Where the 147 cases come from

An eval set has to answer two questions. Does the thing we know is broken get better? And does anything that already worked get worse? No single source gives you both. The team used three.

Source 1: 41 incidents, for the first question. Day 2 ended with 51 traces with behavioural failures. Ten of them were repeats, four customers asking the seat price in four wordings and the like. One case per situation is enough. 51 minus 10 leaves 41 incidents, each a real ticket, frozen with the documentation of that day. T-48213 is one of them.

Source 2: 78 ordinary tickets, for the second question. On day 1, 37 of the 100 traces had nothing wrong with them. A set built only from failures cannot tell you whether a fix breaks those 37. So the team sampled 78 tickets from three months, in the same proportions as the ticket types occur in real traffic:

Source 3: 28 edge cases written by hand. Situations the day 2 list marks as hard, but which had not shown up in one week of traffic:

41 + 78 + 28 = 147 cases, committed to evals/cases/ and tagged support-drafts@v14. The tag matters on day 5: two runs can only be compared if both used the same 147.

What is missing: a test set CI never sees. The team only created one on day 31. Dataset design explains why it should exist earlier.

A case as a file

This is T-48213 as a case, in evals/cases/case-041.json:

{
  "id": "case-041",
  "source": "incident T-48213, 2026-03-09",
  "ticket": "tickets/T-48213.json",
  "fixture": "docs@a91f4c7",
  "checks": ["citations_resolve", "no_unsourced_number", "mentions_expected", "fits_the_helpdesk_field"],
  "expect": { "mentions": ["\\b45\\s?€"] },
  "unmeasured": ["ignores_second_question"],
  "note": "Pro customer. pricing#plans lists 39 and 45; only 45 applies."
}

Compared with the case in An eval in five files, three fields are new:

Two rules sit in this file. First: checks is per case, because a check that does not apply to a case would otherwise pass silently. Four of the edge cases are bug reports where the correct reply cites nothing. There citations_resolve would return true without checking anything. So those four list only fits_the_helpdesk_field. Second: expect exists only on the 41 incidents. The 78 traffic cases and the edge cases have no expected answer, they carry only checks. A price in expect does not rot, because the case runs against the frozen documentation. A check asserting today's price would rot. Hence: rules in the checks, fixed values only in incidents.

The checks

Three of the four checks are unchanged from An eval in five files: citations_resolve, no_unsourced_number, mentions_expected. A fourth is new:

# evals/checks.py, what was added on day 3
MAX_CHARS = 1200


def fits_the_helpdesk_field(draft: str, case: dict) -> bool:
    """The helpdesk editor renders no tables and truncates past 1200 chars."""
    return len(draft) <= MAX_CHARS and "|" not in draft


CHECKS = {f.__name__: f for f in (citations_resolve, no_unsourced_number, mentions_expected, fits_the_helpdesk_field)}

The runner gets two additions. A loader turns the ticket path and the fixture into the same retrieved list the checks already read. And the model call is cached on disk, so a repeated run with nothing changed costs nothing:

# evals/run.py, what was added on day 3
CACHE_DIR = pathlib.Path("runs/cache")


def load_case(path: pathlib.Path) -> dict:
    case = json.loads(path.read_text())
    ticket = json.loads(pathlib.Path(case["ticket"]).read_text())
    case["customer"] = ticket["customer"]
    case["ticket"] = ticket["body"]
    case["retrieved"] = retrieve(ticket["body"], fixture=case["fixture"])   # the product's own search, against the pinned docs
    return case


def call_model(case: dict) -> str:
    ...                                          # the body of generate() from the five-files chapter, unchanged


def generate(case: dict) -> str:
    cached = CACHE_DIR / f"{case['id']}_{MODEL}_{PROMPT_SHA}.txt"
    if not cached.exists():
        cached.write_text(call_model(case))
    return cached.read_text()

retrieve is the product's own search function, imported from the application. The eval does not reimplement search.

The first run

The team ran the runner over all 147 cases the same afternoon:

What it means. 50 failures, found by code that runs in under a second and costs nothing. And one gap: on T-48213 there is a citation right after the 39 €, so no_unsourced_number passes. Only mentions_expected catches the case, because it carries a fixed expectation. The 78 traffic cases carry none. There, nothing catches the price of the wrong plan. That is what day 4 is for.

Day 4: One judge, measured

The same six steps as in the previous chapter, with this team's numbers.

1. One failure mode, one question

The failure mode is the largest row of day 2: a number the cited page does not support. Code catches only the easy part of it, the number without a source. The hard part, a number with a source that says something else, has to be read by someone. The question to the judge is:

Does the cited passage support the number in this draft, for this customer?

2. 120 drafts from traffic

The team took one month of tickets and ran the assistant over all of them. It kept every draft with at least one money amount, because the question has nothing to say about the others. That left 120 drafts. They do not come from the day 3 eval set. Otherwise the test cases would later sit in the judge's instruction as examples.

The file. Every draft is stored with what the judge will later need to see: plan, retrieved passages, draft. One line per draft in evals/drafts.jsonl:

{"draft_id":"d-0311","source":"T-48213","customer":{"plan":"pro"},"retrieved":[{"ref":"pricing#plans","text":"Starter: 39 € per seat and month. Pro: 45 € per seat and month."},{"ref":"pricing#seats","text":"Additional seats are billed pro rata from the day they are activated."}],"draft":"Guten Tag Herr Brandt, zusätzliche Plätze kosten 39 € pro Platz und Monat [doc:pricing#plans] und werden ab dem Tag der Aktivierung anteilig berechnet [doc:pricing#seats]."}

3. Labelling

The support lead read all 120 and answered only the one question for each. The result is a file evals/labels.jsonl, one line per draft, with a reason. The field pile is added in step 4 and says which pile the draft belongs to:

{"draft_id":"d-0311","source":"T-48213","pile":"build","verdict":"fail","reason":"Says 39 € for a Pro customer. pricing#plans lists 39 € for Starter and 45 € for Pro. The number is real, the plan is wrong."}
{"draft_id":"d-0312","source":"T-48377","pile":"sealed","verdict":"pass","reason":"Says the first 5 seats are included, cites pricing#included-seats: 'every plan includes 5 seats'. Matches."}
{"draft_id":"d-0313","source":"T-48401","pile":"build","verdict":"fail","reason":"Says 42 € and cites pricing#archive-2025. That was the 2025 price. The current section says 45 €."}

Result. About 90 seconds per draft, three hours in total. 45 of the 120 drafts failed, 75 passed. Twice along the way the support lead changed the wording of the question and relabelled the drafts already done. The reasons are the material the instruction is written from in step 4.

4. Two piles

The build pile is balanced because you are looking for examples of both sides of the distinction. The sealed pile is deliberately not balanced: 25 to 55 is the mix the judge will see in production. The judge never gets to see the 80, not even as an example.

5. Measuring: version one

The measuring script. It reads labels and drafts, runs the judge over the sealed 80 and counts the four cells of the matrix from chapter 5:

# evals/measure.py
import json
import math
import pathlib
import sys

from run import judge

HERE = pathlib.Path(__file__).parent
judge_model, judge_name = sys.argv[1], sys.argv[2]


def rows(name: str) -> list[dict]:
    return [json.loads(line) for line in (HERE / name).read_text().splitlines()]


drafts = {row["draft_id"]: row for row in rows("drafts.jsonl")}
sealed = [label for label in rows("labels.jsonl") if label["pile"] == "sealed"]

true_positive = false_negative = false_positive = true_negative = 0
for label in sealed:
    draft = drafts[label["draft_id"]]
    case = {"judge": judge_name, "customer": draft["customer"], "retrieved": draft["retrieved"]}
    passed = judge(draft["draft"], case, judge_model)
    if label["verdict"] == "fail":
        false_negative += passed
        true_positive += not passed
    else:
        true_negative += passed
        false_positive += not passed


def rate(hits: int, total: int) -> str:
    p = hits / total
    spread = 1.96 * math.sqrt(p * (1 - p) / total)
    return f"{hits} of {total} = {p:.2f}  ±{spread:.2f}"


print("Catch rate          ", rate(true_positive, true_positive + false_negative))
print("True-negative rate  ", rate(true_negative, true_negative + false_positive))

Call and output for the first version of the instruction, written from the 40 drafts of the build pile with two reasons as examples:

uv run --with openai python evals/measure.py "<vendor>/<model>@2026-06-02" number_supported_by_citation

Catch rate           14 of 25 = 0.56  ±0.19
True-negative rate   53 of 55 = 0.96  ±0.05

The same numbers as a matrix:

Catch rate 0.56, that is 14 of 25. True-negative rate 0.96, that is 53 of 55. The judge found only half of the real failures. On T-48213 it returned PASS, with the reason: "The section pricing#plans contains 39 €, so the number is supported." The judge had learned "is the number somewhere on the page?" The question was "does the passage support the number for this customer?"

6. Revising: version two

The team read the 11 missed cases. All were numbers that were on the page but belonged to the wrong thing. Version two got three reasons from the build pile as examples, one for each kind of this mistake.

Critique 1: right number, wrong plan

The draft says 39 € per seat and cites pricing#plans. That section lists
39 € for Starter and 45 € for Pro. The customer record says plan "pro".
The number is on the page; it belongs to the other plan. FAIL.

Critique 2: right number, wrong time

The draft says 42 € and cites pricing#archive-2025. 42 € was the 2025
price and that section is archived. Current pricing#plans says 45 €.
A number from an archived section is not support. FAIL.

Critique 3: right number, wrong unit

The draft says 540 € and cites pricing#plans, which lists 45 € per seat
per month. 540 € is 12 × 45, arithmetically correct, but the customer
asked for the monthly price and the passage states no annual figure.
A number that had to be computed is not quoted. FAIL.

Plus one rule that, in the team's estimate, did as much as the three examples: the judge must quote the supporting sentence before it gives a verdict. If it finds none, the verdict is FAIL. The finished instruction, in evals/judges/number_supported_by_citation.txt:

You check one thing: does the cited passage support the number in the
draft, FOR THIS CUSTOMER? The customer's plan is given to you.

Quote the sentence from the passage that supports it BEFORE you give a
verdict. If you cannot quote one, the verdict is FAIL.

A number is NOT supported if it is on the page but belongs to a different
plan, a different period, or a different unit. Three examples:

<the three critiques above, verbatim>

Output: {"quote": string | null, "verdict": "PASS" | "FAIL", "reason": string}

The same script, the same sealed 80, nothing else changed:

Catch rate           21 of 25 = 0.84  ±0.14
True-negative rate   50 of 55 = 0.91  ±0.08

Catch rate 0.84, that is 21 of 25. True-negative rate 0.91, that is 50 of 55. The judge now misses 4 real failures and accuses 5 good drafts. Both numbers are known, and that is the point of the exercise. With 25 failures the catch rate has a spread of about ±14 points. That does not change the order of the two versions, but it makes the second decimal meaningless. On T-48213 the judge now returns FAIL and quotes the Starter row.

One number the team deliberately does not report: agreement. It went from 67 to 71 of 80. A judge that says PASS to everything would score 55 of 80 without reading. Agreement hides exactly what the judge was built for.

The file. The case from day 3 grows by one line, the name of the instruction file:

  "judge": "number_supported_by_citation",

And the runner gets the function that calls the judge. It shows it the same three things the expert saw, and reads the verdict:

# evals/run.py, what was added on day 4
def judge(draft: str, case: dict, judge_model: str) -> bool:
    instruction = (HERE / "judges" / f"{case['judge']}.txt").read_text()
    passages = "\n".join(f"[doc:{doc['ref']}] {doc['text']}" for doc in case["retrieved"])
    user = f"Customer plan: {case['customer']['plan']}\n\nDraft:\n{draft}\n\nPassages:\n{passages}"
    response = OpenAI().chat.completions.create(
        model=judge_model,
        temperature=0,
        response_format={"type": "json_object"},
        messages=[{"role": "system", "content": instruction}, {"role": "user", "content": user}],
    )
    return json.loads(response.choices[0].message.content)["verdict"] == "PASS"

The judge runs on the 41 incidents and on any other case whose draft contains a number. The remaining cases never call it. That keeps the cost down.

The same judge in German

The team repeated steps 3 to 5 with 80 German drafts and the translated instruction:

Catch rate 0.63, that is 17 of 27. That sits well below the 0.84 in English. Whether the difference is real cannot be said yet with 27 and 25 failures; the spreads of the two numbers overlap. Statistics works it out. What the team knows: the judge is not demonstrably as good in German, and that is enough not to trust it there until more labels exist. That judges do worse outside English is known. Multilingual evaluation has the evidence.

Day 5: Into CI

Two tiers

A case counts as passed when all of its checks pass, in the regression tier including the judge. The runner gets two flags for this:

# evals/run.py, what was added on day 5
parser.add_argument("--tier", choices=["smoke", "regression"], default="smoke")
parser.add_argument("--judge")                       # model name; see day 4

            results = {name: CHECKS[name](draft, case) for name in case["checks"]}
            if args.tier == "regression" and "judge" in case:
                results[case["judge"]] = judge(draft, case, args.judge)

The cost sheet sat next to the configuration from the start:

Generation    147 cases × ~2,500 input tokens + ~300 output   ≈ €0.40   only after a prompt or model change, otherwise cached
Judge         ~60 cases with a number × ~3,000 tokens         ≈ €0.25   on every regression run
One regression run                                            ≈ €0.65
About 8 PRs a day plus the nightly run                        ≈ €6 a day, roughly €130 a month

The amounts age with the price list, the arithmetic does not. Every additional judge doubles the second line.

The file. The CI configuration:

# .github/workflows/evals.yml
name: evals

on:
  push:
  pull_request:
  schedule: [{ cron: "0 3 * * *" }]

env:
  MODEL: "<vendor>/<model>@2026-07-11"
  JUDGE_MODEL: "<vendor>/<model>@2026-06-02"
  DATASET: "support-drafts@v14"
  PROMPT_SHA: "b3f1c02"
  DOCS_FIXTURE: "docs@a91f4c7"

jobs:
  smoke:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: uv run --with openai python evals/run.py --tier smoke
        # the deterministic checks only; non-zero exit blocks the merge
  regression:
    if: github.event_name != 'push'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: uv run --with openai python evals/run.py --tier regression --judge "$JUDGE_MODEL"
        continue-on-error: true       # judge scores comment, they do not gate

Five things are pinned in the env block: model version, judge version, dataset tag, hash of the prompt template, documentation fixture. Anything not listed there can change between two runs without anyone noticing.

The first real test

Somebody rewrote the system prompt to be stricter about citations. The regression tier reported: 71.4 % → 74.1 % of cases passed. Ship it?

No. Both runs use the same 147 cases, so compare them case by case:

n = 147 cases          105 passed before  →  109 after     (+4 net)

Naive error bar on one run
  SE  = sqrt(0.714 × 0.286 / 147) = 0.0373   →  ±7.3 points

But both runs use the same 147 cases, so compare them pairwise:
  129 cases unchanged     11 fail → pass     7 pass → fail
  SE  = sqrt(11 + 7) / 147 = 0.0289          →  ±5.7 points

Observed change: +2.7 points.  Inside the margin either way.

What it means. Plus 2.7 points against a spread of ±5.7 is noise. The overall rate says nothing. What was not noise in the same run: no_unsourced_number failures dropped from 22 to 3, and not a single case got worse. The stricter prompt did exactly what it was meant to. The overall rate was just too blunt to show it. How the arithmetic works in detail is in Evals in CI.

Common mistake: Gating the merge on the overall pass rate. It is the least sensitive number in the whole suite. A real improvement on one failure mode disappears among the modes that did not move. Gate on the deterministic checks. Report the rest with its spread, and read it.

Day 31: The set goes stale

What the team did. Every week it pulled 200 drafts from live traffic and ran the same judge over them.

Result. In production 8 % of drafts failed the judge. In the eval set it was 2 %.

What it means. Marketing had shipped a new pricing page in week three. Twelve of the 41 incidents now pointed at sections that no longer existed, and customers were asking about the new plans, for which the set had no case. The set was not wrong. It described March. Two day 3 decisions saved it: the checks check rules, so no_unsourced_number was still right. And the incidents are frozen against a fixture, so they keep running, just against old documentation.

The refresh took a morning: 100 fresh traces, the same pass as on days 1 and 2, DATASET in CI bumped. The 100 fresh traces also became the first test set CI does not see. The pass is now a monthly calendar entry.

Common mistake: Treating the eval set as finished. When it goes stale, nothing breaks. CI stays green, the numbers keep coming. They just gradually stop being about your product.

What the week produced

A list of failure modes from real traces. 147 versioned cases with frozen documentation. Four checks that run for free on every push. A judge with a known catch rate in two languages. And a prompt change you can argue about with numbers instead of impressions.

Three things this example does not show: how to grade actions instead of sentences, how to tell a broken search from a broken answer, and how to handle hostile input. The closing table of the previous chapter maps those gaps to the chapters that cover them.

The most valuable result was day 1: the discovery that support had been quietly repairing the same failure for months. Until a customer saw the reply, it looked fine.