Building an aligned LLM judge

An LLM judge is a model prompted to grade another model's output. It is the only practical way to automate evaluation of properties that require reading prose, and it is also the component teams most often deploy without checking whether it works.

The distinction between a useful judge and a decorative one is a single step: measuring the judge against human labels on data it has never seen. Everything else in this chapter exists to make that step produce a good number.

The seven-step loop

This is Hamel Husain's method, which is the most concrete published process for building a judge that a business will actually act on.

1. Pick one principal domain expert

One person whose judgement defines correct for your product. A clinician for a health tool, a lawyer for contract analysis, a support director for a service bot. Not a committee.

This person's role is not to approve a rubric someone else wrote. They read outputs and decide, and in doing so they surface acceptance criteria nobody had written down. Domain experts routinely discover their own inconsistency during this process. One reported that seeing the model's reasoning laid out "made me realise I wasn't being consistent". That discovery is the value, not a problem with the method.

2. Collect the answers to be labelled

The expert does not label inputs. They label answers. An answer has three parts: the user's request, whatever the system retrieved or did, and what it replied. Those are the same three parts as a trace. So this step has two halves. First you choose the requests. Then you run the current system over them. The current system is your product as it runs today: today's prompt, today's model.

This is not day 1 again. On day 1 you read 100 arbitrary traces with an open question: what is the first thing that is wrong? Out of that comes the list of failure modes. Here you read traces with the single fixed question from step 1, such as "is the number supported?", and only traces that question applies to. Out of that comes the judge. You do not reuse the day 1 traces: they were read for a different question, some of them have since become eval cases, and the system has answered differently since the day 2 bug fixes.

The requests. Take 100 to 150 requests from real traffic, written by real users. Do not take them from your eval set; Your first eval in a week says why. Keep only requests where the question from step 1 has something to check. For "is the number supported?" that means answers containing a number.

Coverage. So that the 100 to 150 are not all of the same kind, sort every request by three properties:

Every combination is one cell of a grid. One of them, written out:

Feature:  order tracking
Scenario: order not found
Persona:  new customer
-> "Where's my order? I ordered yesterday, first time using this."

The team writes the grid. The users write the requests in it. Sort the real requests into the cells, and most cells are filled afterwards, because real users produce these cases anyway. Only for the cells that stay empty do you write two or three requests yourself. Dataset design shows how. The point of the grid: you can point at an empty cell and say "we have never tested this". And later at a full one: "this is where we fail 75 % of the time".

The answers. Then run the current system over all requests once and store each answer next to its request. Those stored answers are what the expert reads in the next step.

3. Binary judgments with written critiques

The expert labels each output pass or fail, never 1–5, and writes a short critique explaining why.

The critique is not documentation. It is the raw material for the judge prompt, and it has two jobs:

• Specific enough that a new colleague would understand the decision
• Detailed enough to serve as a few-shot example

A good critique names the primary objective, says whether it was achieved, and identifies the specific thing that decided the outcome. Passing critiques matter as much as failing ones. A judge trained only on failures learns to find fault.

Two real-shaped examples, for a support assistant:

Fail. User asked where their order was. The assistant explained the returns policy instead. The primary objective, locating the order, was never attempted. The tracking tool was available and not called.

Pass. User asked where their order was; tracking returned no match. The assistant said so plainly, gave the likely reason (order placed under 30 minutes ago), and offered to email an update. Objective achieved: the user knows the state and what happens next.

Notice what the passing critique does. It does not say "good answer". It names why this counted as a pass, which is the part a judge can learn from. "Good answer" teaches nothing.

Budget for 100 to 150 labels in total and split them into two piles that never touch. The build pile is 30 to 40 of them, hand-picked so that half pass and half fail. It is where the judge's examples come from; keep adding to it until the expert stops encountering new failure modes. The sealed pile is everything else, at its natural pass/fail mix, with at least 20 to 25 real failures. It is for measuring in step 6. The judge never sees it, not even to check whether one case is tricky.

4. Fix the obvious bugs first

Reviewing outputs will surface plain defects. Fix them and regenerate before continuing. Building a judge to detect a bug you could fix this afternoon is a category error, and it also pollutes your label set with failures that will never occur again.

5. Build the judge and iterate

The judge gets, per case, everything the expert had in front of them: the input, the retrieved context or tool results, the answer, and any fact the criterion depends on, such as the customer's plan. It produces one verdict. The prompt has four parts, in this order:

1  Task        You check one thing about a support answer: <one sentence>.
2  Criterion   PASS if <definition>. FAIL if <definition>.
               Edge case: <one case the expert resolved, with the reason>.
3  Examples    Two or three of the expert's critiques from step 3, verbatim,
               each with its verdict, at least one PASS and one FAIL.
4  Output      {"reason": "<two to four sentences naming what you checked>",
                "verdict": "PASS" | "FAIL"}

That is Eugene Yan's checklist in one block: a single task and a single criterion, precise definitions including edge cases, few-shot examples for calibration, a fixed output format. Parts 2 and 4 filled in, for a citation criterion:

Criterion:
PASS if the response cites at least one retrieved document and every cited
document contains a statement supporting the claim it is attached to.
FAIL otherwise.

Output:
{"reason": "<Two to four sentences. Name the specific claim and citation you checked.>",
 "verdict": "PASS" | "FAIL"}

The output format is always the same JSON. The runner reads only verdict from it. The worked example adds a field quote, because its criterion demands a verbatim quotation, and shows the function that calls the judge and reads the JSON.

The order inside part 4 is not cosmetic. Require the judge to write its reasoning before its verdict. A verdict emitted first and justified afterwards is a rationalisation, and it measurably reduces quality.

Then iterate. Run the judge over the sealed pile from step 3, list every case where it disagrees with the expert, and read the judge's reasoning on each. Every disagreement falls into one of three boxes:

Change one thing per round, re-run on the same sealed pile, and stop when the catch rate in the next step is where you need it. Two or three rounds are normal. Husain reports reaching >90% agreement in three iterations on a production case. That is raw agreement, which the next step refuses as a measure for good reason; the criterion for "done" is the catch rate, not this number. If you are on iteration ten, the criterion is probably underspecified rather than the judge being incapable. The worked example shows one full round: a first version that passed T-48213 with the reasoning "the page contains 39 €", three critiques added, and a second version that catches it.

6. Measure properly

This is the step that determines whether you have anything.

Finding: Report true positive rate and true negative rate, or precision and recall, rather than raw agreement. Husain is explicit that raw agreement misleads under class imbalance, and it is almost always imbalanced: if 90% of outputs pass, a judge that says "pass" unconditionally scores 90% agreement while detecting nothing.

Source: Hamel Husain, Creating a LLM-as-a-Judge

Concretely, on the sealed pile from step 3, labelled by the expert, never seen by the judge:

The four cells read like this: true positive means the judge flagged a failure and it really was one. False negative means it waved a real failure through. Two rates come out of these cells:

• TPR (true positive rate, "recall on failures"): of all the actual failures, how many did the judge find?
• TNR (true negative rate): of all the genuinely good answers, how many did it correctly let through?

One convention you need when reading other sources: in this guide, positive means "the judge flags a failure". Many libraries and papers take the other class as positive, and then TPR and TNR swap places. Say which one is positive for you.

Work it with numbers. You have hand-labelled 200 examples; 20 are real failures, 180 are fine. The judge produces:

TPR = 8 / (8 + 12)    = 8 / 20   = 40%
TNR = 171 / (171 + 9) = 171 / 180 = 95%
Raw agreement = (8 + 171) / 200 = 89.5%

That 89.5% looks healthy, and that is exactly the trap: the judge waved through 12 of 20 real failures. It catches 40% of what it was built for. Because good answers dominate, the high TNR pulls the headline number up and hides this.

TPR is therefore usually the number that matters: the judge exists to catch failures, not to confirm that the good stuff is good. And it is itself an estimate from few cases: 8 of 20 is 40%, with a spread of about ±21 points. So report TPR and TNR with the number of cases behind them, "8 of 20", rather than as a bare percentage; Statistics has the arithmetic.

Set the bar by consequence. A judge gating a release needs high TPR. A judge sampling production for a weekly report can tolerate more error, because you will look at the flagged cases anyway. As a working bar, in the spirit of κ ≥ 0.5 in dataset design: for a release gate a catch rate of at least 0.8, measured on at least 25 real failures. For reports and samples less is enough, as long as somebody reads the flagged cases. That is a trade-off, not a rule. How to compute both rates from the labels and the judge's verdicts is shown by the measuring script in the worked example.

7. Segment the errors, then specialise

Segmenting means you stop looking at the failure rate as one number and split it by the features your requests differ along, the same dimensions you fixed for the dataset in step 2. For a support assistant those might be intent, search result and user type.

An example. The judge runs over 2,000 conversations and the overall failure rate is 22%, a number nobody can act on. Broken down:

Now it is no longer a diffuse quality problem but a nameable case: when order tracking finds nothing and the user is new, the system fails three times out of four. That is a bug you can fix. The 22% was not. This is precisely what aggregate scores conceal.

Build additional narrow judges only for specific failure modes that survive this analysis. One judge per criterion, not one judge that scores everything.

What the judge's error does to your eval score

A judge with TPR 0.84 and TNR 0.91 is wrong in both directions. It lets 16 % of the bad drafts through. And it rejects 9 % of the good drafts. Both shift the pass rate it reports. The shift is not random. It is the same size on every run, as long as the judge stays the same.

An example with 100 drafts, of which 70 are in truth good and 30 bad:

Of the 70 good ones the judge passes 91 %:       0.91 × 70 = 63.7
Of the 30 bad ones it passes 16 %:               0.16 × 30 =  4.8
Reported pass rate:                               63.7 + 4.8 = 68.5 %

The judge reports 68.5 %. The truth is 70 %. As a formula, the way other texts write it:

observed rate = TNR × true rate + (1 − TPR) × (1 − true rate)

When it does not matter. When you compare two versions of your system under the same judge, both numbers are shifted by the same amount. The difference between them stays almost unchanged. That is the most common case, and there you can ignore the shift.

When it counts. When you want to claim an absolute number, such as "70 % of our drafts are correct" in a report. Then you take the shift back out:

true rate = (observed − (1 − TPR)) / (TNR − (1 − TPR))
          = (0.685 − 0.16) / (0.91 − 0.16) = 0.70

The formula is called the Rogan–Gladen correction. It comes from epidemiology, where it has been used since 1978 to estimate the true frequency of a disease from the results of an imperfect test. A judge is exactly such a test.

One condition: TPR and TNR must have been measured on data that looks like the data the judge then runs over. Otherwise the 0.84 and 0.91 do not hold for this data, and the correction computes with wrong numbers. That is one more reason to draw the sealed pile from real traffic and not from the eval set.

Source: Rogan & Gladen, Estimating prevalence from the results of a screening test, American Journal of Epidemiology 107(1), 1978 · DOI 10.1093/oxfordjournals.aje.a112510

Choosing the judge model

Contested: Husain's position is that the same model powering your product is usually fine as a judge, provided you validate its alignment against human labels. If alignment is poor, try a different model. The self-preference literature points the other way: gpt-4 has been measured awarding itself a 10% higher win rate, and claude-v1 25%, when judging their own outputs.

These reconcile better than they first appear. Self-preference is measured in pairwise comparison of models. A pointwise judge applying a written criterion to a single output is a different task with less room for it. The safe rule: if the judge is comparing outputs from different models, do not use one of those models as the judge. If it is applying a fixed criterion pointwise, validate and proceed.

Source: Husain & Shankar, LLM Evals FAQ; Self-Preference Bias, 2024

In practice, proceed like this. Start with the model that powers your product and measure the catch rate. Take a stronger model only when the catch rate is not enough after two or three prompt rounds; that is the third row of the table in step 5. Take a smaller, cheaper model only if its catch rate holds on the same sealed pile. Work out what a judge costs beforehand: cases that call the judge, times tokens per call, times price per token. In the worked example that is 60 cases times about 3,000 tokens, about €0.25 per run. At eight runs a day about €60 a month for this one judge. Every further judge comes on top.

Do not fine-tune a judge as a first move. Fine-tuned evaluator models behave as task-specific classifiers: strong in-domain, poor at generalising, and prone to collapsing when the evaluation scheme changes. Effort spent fine-tuning a judge is usually better spent improving the system under test.

Maintaining it

A judge is a model-dependent artefact. When the judge model version changes, the alignment you measured no longer applies.

• Pin the judge model version explicitly, including any dated snapshot suffix.
• Keep the sealed pile. Re-run alignment whenever the judge model, the criterion, or the task distribution changes.
• Treat a re-alignment run as a required step of a model upgrade, not a nice-to-have.
• Recheck periodically anyway. Task distribution drifts even when nothing in your stack does.
• Re-measure per language if you serve more than one. An alignment number is only valid for the language it was measured in. See Multilingual evaluation.

Common mistake: Shipping a judge without a sealed pile of labels, then quoting its scores in a planning meeting. Without alignment numbers those scores have unknown sign. You cannot say whether a change that improved them improved anything else.