Where LLM judges break

The previous chapter was about building a judge that works. This one is about knowing what "works" can and cannot mean, because the failure modes of LLM judges are systematic, measurable, and largely invisible in the output.

A judge does not return an error when it is biased. It returns a score.

Calibrating your expectations

Before the biases, one question: how good can a judge realistically get? You need the answer before you polish your judge. Otherwise you keep working on a catch rate of 0.84 that is already as good as the task allows. Or you settle for 0.6 where 0.9 is within reach. The limit is known from studies that measured judges against humans. This section sums them up.

Picture two people rating the same 100 summaries as acceptable or not. They agree on 85 of them: 85% agreement. Now replace one of them with a model. Researchers have done exactly that, many times over, and two results are worth remembering:

• On open-ended questions, a judge can agree with humans as often as humans agree with each other. On MT-Bench, conversational questions with no single right answer, gpt-4 agreed with human experts 85% of the time, and the experts agreed with each other only 81% of the time. That is the number everyone quotes. It is real, and it says as much about the task as about the judge: where humans are inconsistent, "as good as a human" is a low bar.
• On questions with a right answer, a judge gets close to humans but not level. On TriviaQA, plain knowledge questions, two humans agree almost perfectly, and gpt-4 agrees with them clearly less. That gap is the realistic ceiling for a judge on a well-defined task.

The TriviaQA numbers are missing from that second point on purpose, because reading them takes one more idea.

Plain agreement flatters. If 90 of 100 answers are good, two people who say "good" to everything agree on 90 of them, without having judged anything. So researchers subtract the agreement you would get by guessing. The result is called Cohen's κ (kappa): 0 means no better than guessing, 1 means perfect. On TriviaQA the humans reach κ 0.97, gpt-4 0.84. As plain percentages both would read "over 90%", and the gap would be invisible.

Correlation asks a different question. Some studies do not ask "did they give the same verdict?" but "did they rank the answers the same way?" When the human rates one answer above another, does the judge too? Those numbers (Pearson, Spearman ρ, Kendall's τ) also run from 0 to 1, but they measure ordering, not verdicts, and a row using one cannot be compared with a row using the other.

For the record, what roughly two dozen studies found. Read each row only against its own human baseline in the last column, never against another row:

Rule of thumb for any of them: above 0.8 is strong, 0.5 to 0.8 is usable with care, below 0.5 means judge and human often disagree.

What to take from this. First: a good judge on a well-defined task sits at about κ 0.8 against humans, not at 1.0. The catch rate of 0.84 in the worked example is therefore not an interim result but what you may expect. Second: before you compare a number from a study with your own, check which metric it is. 85 % agreement, κ 0.84 and ρ 0.67 answer three different questions. Third: report your own measurement the way chapter 5 requires, as catch rate and true-negative rate with their counts. Then you can compare it with the row for your kind of task, and with nothing else.

Finding: The same effect appears in search relevance, where judges reach Cohen's κ of only 0.3–0.5 while Kendall's τ and Spearman's ρ on the same data sit at 0.8–0.9. The correlation metric and the agreement metric disagree about whether the judge is good. Report both, or state which you chose and why.

Source: Eugene Yan, Evaluating LLM-Evaluators

The three classical biases

Position bias

In pairwise comparison, judges favour a position rather than a response.

What that looks like in practice. You show the judge two answers and ask which is better:

Round 1:  A = the concise answer   B = the verbose one   -> judge picks A
Round 2:  A = the verbose answer   B = the concise one   -> judge picks A again

Same two answers, same question, opposite verdict. The judge chose the slot, not the answer, and you would never notice, because a single run only ever shows you one of the two rounds.

Finding: gpt-3.5-turbo preferred the first-presented response in 50.0% of pairs, claude-v1 in 75.0%. Swap the order and the verdict holds for gpt-3.5-turbo in 46.2% of cases, for claude-v1 in 23.8%.

Source: Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023, Table 2

The standard mitigation is to run each comparison twice with the order swapped and count only consistent verdicts, treating flips as ties. It costs double and it works for this bias.

Contested: It has been widely assumed that moving from pairwise to pointwise rubric scoring removes position bias. Work from 2026 examining rubric-based LLM-as-judge finds position effects persist in the pointwise setting. If you switched to rubrics specifically to escape position bias, that assumption needs testing on your own data rather than inheriting.

Source: Position Bias in Rubric-Based LLM-as-a-Judge, 2026

Verbosity bias

What "padding" means here is concrete and slightly absurd. The attack takes a correct answer and restates its own content as a list, adding nothing:

Original: "Paris is the capital of France."

Padded:   "Paris is the capital of France. Key points:
           1) Paris is the capital of France.
           2) France's capital city is Paris.
           3) The seat of government is located in Paris."

No new information. One is not better than the other by any standard a user would apply.

Finding: Against exactly this attack ("repetitive list"), claude-v1 and gpt-3.5-turbo preferred the padded version 91.3% of the time, gpt-4 only 8.7%.

Source: Zheng et al., MT-Bench, Table 3

The spread is the real finding: verbosity bias is not a law of the technique but a property of the particular judge model. Someone else's numbers will not tell you how badly yours is affected. You have to measure that on your own.

This one is dangerous because it survives into optimisation. If you tune prompts against a judge with verbosity bias, you will reliably produce a more verbose product and a rising score, and users will experience it as worse. Control for it: include length in your reporting, or normalise, or explicitly instruct the judge that length is not a quality signal, and verify the instruction actually worked.

Self-preference

Finding: gpt-4 awarded itself a 10% higher win rate on its own outputs, claude-v1 25%. The authors themselves caution that the data is not sufficient to establish genuine self-enhancement beyond doubt.

Source: Zheng et al., MT-Bench, Figure 2(b)

Finding: Later independent work supplies a mechanism: the strength of self-preference correlates linearly with a model's ability to recognise its own outputs. Strengthen that ability by fine-tuning and the bias grows with it.

Source: Panickssery, Bowman & Feng, LLM Evaluators Recognize and Favor Their Own Generations

The practical rule from the previous chapter holds: never let a model be the judge in a comparison it is competing in.

The failure modes that matter more

The three classical biases are well known and mostly mitigable. These are worse because they are quieter.

Recall collapse on the failures

Finding: Evaluating factual consistency, gpt-3.5-turbo identified more than 95% of consistent summaries but only 30–60% of inconsistent ones.

Source: Eugene Yan, Evaluating LLM-Evaluators

In the vocabulary of the previous chapter, that is a TNR above 95% paired with a TPR between 30 and 60%. Put concretely: of 100 summaries that genuinely contradict their source, the judge flags somewhere between 30 and 60 and lets the rest through.

Read that again in terms of your use case. The judge is excellent at confirming that good output is good, and close to a coin flip at catching the bad output you deployed it to catch. Since catching failures is the entire point, a headline accuracy figure on an imbalanced set is actively misleading, which is exactly why the previous chapter insists on TPR rather than agreement.

Blindness to deliberate corruption

Finding: gpt-4-turbo failed to assign lower scores to deliberately perturbed answers more than 50% of the time on coherence, factuality, and instruction-following. On distinguishing factual from hallucinated summaries, the best model tested reached 58.5% accuracy, barely above chance.

Source: Eugene Yan, Evaluating LLM-Evaluators

This is the strongest available argument against using an LLM judge as your primary hallucination detector. If a judge cannot reliably notice an answer that was corrupted on purpose, it will not notice one corrupted by accident.

Alignment with non-experts

Finding: LLM evaluators have been observed to align better with non-expert human annotators than with expert ones.

Source: Eugene Yan, Evaluating LLM-Evaluators

This has an uncomfortable implication for the whole literature: published correlation numbers derived from crowdworker labels may be systematically optimistic relative to what you would get against a domain expert. It is also a direct argument for the principal-domain-expert method. Expert labels are the harder target, and the one that matters.

G-Eval, and what most implementations actually are

What G-Eval is. A recipe for building a judge, published in 2023. You will meet the name on the metric list of almost every eval platform. The recipe became known through one number: its verdicts correlated with human ratings at 0.514. On that scale 0 is "no relationship" and 1 is "perfect agreement". For summaries and dialogues, the tasks in the paper, that is decent.

The recipe. It has two parts. The first is ordinary: the judge gets a scoring form and has to give its reasons before it scores. That is the same rule as in building a judge. The second part is the actual idea. When a model picks a score from 1 to 5, it internally holds a confidence in each of the five scores and reports the strongest. Those confidences are called logprobs. G-Eval asks for them and computes a weighted average over all five instead of taking only the winner:

The model's internal confidence in each score:
  1 → 1%     2 → 4%     3 → 15%     4 → 60%     5 → 20%

Plain judge  → reports the winner                       → 4
G-Eval       → (1×0.01)+(2×0.04)+(3×0.15)+(4×0.60)+(5×0.20)  → 3.94

A judge torn between 3 and 4 thus produces 3.4. A judge certain of 4 produces 3.95. These fine-grained values correlate better with human ratings than whole numbers do. That is where the 0.514 comes from.

The catch. You only get the logprobs from an API that hands them out. Most chat APIs do not. So what eval platforms ship as "G-Eval" is usually only the first part of the recipe:

Finding: G-Eval's published contribution is a form-filling prompt with chain-of-thought plus probability-weighted summation over the token logits of the 1–5 options. Production deployments overwhelmingly go through chat APIs that do not expose logit distributions, so the shipped pattern collapsed to a discrete integer score with a CoT preamble, sometimes rescaled to look continuous. Vendors then market that as a "G-Eval implementation" and cite the paper's 0.514 correlation.

Source: Liu et al., G-Eval, 2023

What this means for you. Three things.

1. If a metric in your tool is called "G-Eval", check whether it asks for logprobs. In code that is a search for logprobs. With a vendor it is a question to support.
2. If it does not ask for logprobs, you have an ordinary judge with reasons. That is not bad. But the 0.514 does not apply to it, because the number came from the part that is missing. How good this judge is you only know once you measure it yourself: catch rate and true-negative rate on a sealed pile, as in chapter 5.
3. If it does ask for logprobs, the 0.514 still holds only for summaries and dialogues, the tasks in the paper. For your task you have to measure either way.

In short: the name G-Eval replaces no measurement. An ordinary judge you measured yourself is worth more than a "G-Eval" whose only number comes from somebody else's paper.

Where not to use a judge at all

Some properties should not be delegated to an LLM judge regardless of how well you prompt it:

• Factual correctness on long-tail knowledge. The judge shares the base model's knowledge gaps and will confidently confirm the same wrong facts. Use retrieval against a source of truth, or humans.
• Tool-call correctness. Code-graded AST matching is exact, free, and instant. A judge here is strictly worse.
• Anything with a deterministic answer. Schema validity, arithmetic, IDs, dates, enum membership.
• Safety decisions with real consequences. A thin judge-based screen is a reasonable signal; it is not a control. See Safety.
• Sole arbiter of a release gate. Combine with deterministic checks and a human sample.

Juries

Using several judges and aggregating is a genuine improvement. A panel of smaller, diverse models has been reported to correlate better with humans than a single strong judge on reference-based tasks, and it costs less than the obvious alternative.

It is not a cure. Ensembling reduces variance within the judge population; it does not remove biases that all the judges share. If every member shares the same verbosity bias, then so does the jury. Among models of one generation that is the rule rather than the exception, even though its strength varies by model, as seen above. Use a jury for stability, not for absolution, and validate the jury against human labels exactly as you would a single judge.

Common mistake: Reporting a judge score to two decimal places. The judge has a measured error rate against humans and the eval set is a sample, so most of those digits are decoration. Report the score with a confidence interval and the judge's TPR/TNR beside it, or report a range. See Statistics.