An eval in five files

An eval is not a table and not a line of code. It is four things in one directory: case files, check functions, a runner, and one result file per run. This page shows the smallest complete example, in the same product as the worked example: an assistant that drafts replies to support tickets. Everything here is plain Python and JSON. You can type it in, run it, and read the output before you read any other chapter. No model grades the answers here. That comes in chapter 5, and for good reason only then.

evals/
  cases/
    case-001.json
    case-002.json
    case-003.json
  prompt.txt       The instruction to the model. Its hash goes into every result row.
  checks.py        Three check functions. No model call.
  run.py           The runner: loads cases, calls the model, checks, writes results.
runs/
  2026-09-03T14-22_650de45.jsonl   Created by running. One line per case.

One case

A case is one frozen situation plus the record of what gets checked about it:

{
  "id": "case-001",
  "source": "ticket T-48213, 2026-03-09",
  "customer": { "plan": "pro", "locale": "de" },
  "ticket": "Guten Tag, wir brauchen ab April 6 weitere Plätze. Was kostet ein zusätzlicher Platz im Monat? Außerdem: rechnet Ihr System den Nachtschicht-Zuschlag automatisch, wenn eine Schicht um 21:45 beginnt?",
  "retrieved": [
    { "ref": "pricing#plans", "text": "Starter: 39 € per seat and month. Pro: 45 € per seat and month." },
    { "ref": "pricing#seats", "text": "Additional seats are billed pro rata from the day they are activated." },
    { "ref": "rules#night-surcharge", "text": "The night surcharge applies to shifts beginning at 22:00 or later." }
  ],
  "checks": ["citations_resolve", "no_unsourced_number", "mentions_expected"],
  "expect": { "mentions": ["\\b45\\s?€"] },
  "note": "Pro customer. pricing#plans lists 39 and 45; only 45 applies. Word boundary in the pattern, or the 45 in 21:45 matches."
}

The two other cases have the same fields. case-002 asks about pro-rata billing and expects the pattern pro rata. case-003 asks for the current price, and retrieval also returns the 2025 archive price. It expects \b45\s?€ again.

The checks

Three functions with one signature. None of them names a price; they check the shape of the answer, not its content. Only mentions_expected reads the case.

# evals/checks.py
import re

CITATION = re.compile(r"\[doc:([a-z0-9-]+#[a-z0-9-]+)\]")
MONEY = re.compile(r"\d[\d.,]*\s?(?:€|EUR)|(?:€|EUR)\s?\d[\d.,]*")


def citations_resolve(draft: str, case: dict) -> bool:
    """Every citation points at a section retrieval actually returned."""
    available = {doc["ref"] for doc in case["retrieved"]}
    cited = CITATION.findall(draft)
    return bool(cited) and all(ref in available for ref in cited)


def no_unsourced_number(draft: str, case: dict) -> bool:
    """Every money amount has a citation within the next 120 characters."""
    return all(CITATION.search(draft[m.end() : m.end() + 120]) for m in MONEY.finditer(draft))


def mentions_expected(draft: str, case: dict) -> bool:
    """Every pattern under expect.mentions occurs in the draft."""
    return all(re.search(pattern, draft) for pattern in case["expect"]["mentions"])


CHECKS = {f.__name__: f for f in (citations_resolve, no_unsourced_number, mentions_expected)}

The instruction to the model

A text file, so that its hash sits in every result and a changed prompt shows up as a changed prompt.

You are the support assistant of a shift-planning tool. You receive a customer
ticket, the customer's plan, and excerpts from the documentation.

Rules:
- Answer every question in the ticket. If the information is missing, say so.
- Use only the documentation. Invent nothing.
- Put the source right after every number, in the form [doc:section].
- Answer in the customer's language, at most 120 words, no tables.

The runner

This is the piece the worked example calls "a script". It is sixty lines.

# evals/run.py
import argparse
import datetime
import hashlib
import json
import os
import pathlib
import sys

from checks import CHECKS

HERE = pathlib.Path(__file__).parent
SYSTEM_PROMPT = (HERE / "prompt.txt").read_text()
PROMPT_SHA = hashlib.sha1(SYSTEM_PROMPT.encode()).hexdigest()[:7]
MODEL = os.environ["EVAL_MODEL"]


def generate(case: dict) -> str:
    from openai import OpenAI  # reads OPENAI_API_KEY and OPENAI_BASE_URL from the environment

    passages = "\n".join(f"[doc:{doc['ref']}] {doc['text']}" for doc in case["retrieved"])
    user = (
        f"Customer plan: {case['customer']['plan']}\n\n"
        f"Ticket:\n{case['ticket']}\n\n"
        f"Documentation:\n{passages}"
    )
    response = OpenAI().chat.completions.create(
        model=MODEL,
        temperature=0,
        messages=[{"role": "system", "content": SYSTEM_PROMPT}, {"role": "user", "content": user}],
    )
    return response.choices[0].message.content


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--cases", default=str(HERE / "cases"))
    parser.add_argument("--out", default="runs")
    args = parser.parse_args()

    cases = [json.loads(p.read_text()) for p in sorted(pathlib.Path(args.cases).glob("*.json"))]
    run_id = datetime.datetime.now().strftime("%Y-%m-%dT%H-%M") + "_" + PROMPT_SHA
    out = pathlib.Path(args.out) / f"{run_id}.jsonl"
    out.parent.mkdir(parents=True, exist_ok=True)

    failures = 0
    with out.open("w") as f:
        for case in cases:
            draft = generate(case)
            results = {name: CHECKS[name](draft, case) for name in case["checks"]}
            passed = all(results.values())
            failures += not passed
            row = {
                "run_id": run_id,
                "model": MODEL,
                "prompt_sha": PROMPT_SHA,
                "case": case["id"],
                "passed": passed,
                "checks": results,
                "draft": draft,
            }
            f.write(json.dumps(row, ensure_ascii=False) + "\n")
            failed = ", ".join(name for name, ok in results.items() if not ok)
            print(f"{'PASS' if passed else 'FAIL'}  {case['id']}  {failed}")

    print(f"\n{len(cases) - failures}/{len(cases)} passed  ->  {out}")
    sys.exit(1 if failures else 0)


if __name__ == "__main__":
    main()

Running it

Three environment variables, one command. The endpoint is any OpenAI-compatible endpoint. The model is pinned with a date, as the harness chapter requires.

export OPENAI_API_KEY="…"
export OPENAI_BASE_URL="https://<endpoint>/v1"
export EVAL_MODEL="<vendor>/<model>@2026-07-11"

uv run --with openai python evals/run.py

What you see afterwards

In the terminal:

FAIL  case-001  mentions_expected
PASS  case-002  
PASS  case-003  

2/3 passed  ->  runs/2026-09-03T14-22_650de45.jsonl

And the first line of the result file:

{"run_id": "2026-09-03T14-22_650de45", "model": "<vendor>/<model>@2026-07-11", "prompt_sha": "650de45", "case": "case-001", "passed": false, "checks": {"citations_resolve": true, "no_unsourced_number": true, "mentions_expected": false}, "draft": "Guten Tag, zusätzliche Plätze kosten 39 € pro Platz und Monat [doc:pricing#plans] und werden anteilig berechnet [doc:pricing#seats]. Der Nachtzuschlag gilt erst ab 22:00 Uhr [doc:rules#night-surcharge], für 21:45 also nicht."}

How to read it. Each terminal line is one case. A function name after it is the failure class. case-001 cited every source correctly and backed every amount, and still quoted the Starter price. The two shape checks cannot see that. Only the pinned expectation did, and a case taken from live traffic carries none. For those cases a judge comes later. The draft itself is read in the result row, not in the terminal. Exit code 1 is what blocks a merge in CI. Two runs are compared through their two files, case by case over case; the arithmetic is in the CI chapter.

What this eval does not have

No cache for model answers (the harness). No judge (chapter 5). No paired comparison of two runs (CI). No sealed pile (dataset design). Everything in those chapters builds on these five files and replaces none of them. The worked example shows what a team adds to them in a week.

Common mistake: An expectation written as a substring. The first version of case-001 expected "45". A draft quoting the wrong price of 39 € passed anyway, because the ticket's 21:45 is in the answer. A check that goes green when it should go red is worse than no check. Run every case once with an answer you know is wrong before you trust it.