Multilingual evaluation

This chapter exists because almost no practitioner guide includes one, and the evidence says it should. If your product serves users in more than one language and your evaluation is in English, you do not know how it performs for most of your users, and the standard techniques degrade in ways that are measured, specific, and rarely mentioned.

What the evidence says

Three findings, all from 2025–2026 work, all pointing the same direction.

Finding: How Reliable is Multilingual LLM-as-a-Judge? (EMNLP Findings 2025) found that LLM judges struggle to produce consistent judgments across languages, reporting an average Fleiss' κ of approximately 0.3, with worse performance on low-resource languages.

Judge the same content in five languages and you get meaningfully different verdicts. That is not a subtle effect; κ ≈ 0.3 is weak agreement.

Finding: Does the Judge Prefer English? (2026) tested four API-accessible judges on all 419 LLMBar response pairs in English, Chinese, and Chinese-English language-switched variants: 13,408 pairwise judgments. Chinese and language-switched presentations induced 10.7–14.4% preference flips relative to English, and all judges achieved their highest accuracy in English.

The same paper reports a real counterweight, and it belongs here: translation-equivalent tie probes did not reveal a systematic English preference. Most were judged as ties, and non-tie decisions more often favoured Chinese. The robust finding is instability under language switching, not a simple across-the-board bias toward English.

More than one judgment in ten changes when you change nothing but the language, and that is a two-language comparison using strong models on a curated benchmark. Treat it as a floor, not a worst case.

Finding: Work on multilingual and low-resource settings (2026) identifies three compounding problems: inconsistent outcomes depending on the prompt language, performance that is overestimated for low-resource languages, and a widespread reliance on a single judge without ensemble or multi-judge assessment.

Source: Challenges and Recommendations for LLMs-as-a-Judge in Multilingual Settings, 2026

The middle one is the dangerous one. The failure is not that scores are noisy. It is that they are biased optimistic exactly where you have the least ability to check them.

Why translating your eval set does not fix it

The obvious move is to machine-translate the English eval set. It is cheap and it is worse than it looks.

Translationese is not your traffic. Translated English carries English syntax, English discourse structure, and English-shaped questions. Real German users do not write translated English. You have built a set that is in German and distributed like English.

The failure modes do not translate. Your English set has no examples of the failures specific to another language, because those failures were never in the source. German compound nouns break entity extraction and chunking in ways English never surfaces. The formal/informal distinction (Sie / du) is a correctness property in German business communication with no English counterpart, so no English test case covers it.

Reference answers become doubly unreliable. A translated gold answer is a translation of one valid English answer, not the best answer in the target language.

Machine translation is a reasonable way to bootstrap coverage, and a poor basis for measurement. Treat a translated set as a smoke test and build the real set from real traffic in each language.

Failure modes that only appear outside English

Worth writing test cases for explicitly:

Most of these are code-graded and cost nothing to run. Output language identification alone catches a surprising amount, and it needs no judge.

A workable approach

1. Measure judge alignment per language. Not once, globally. The TPR/TNR measurement has to be repeated for each language you serve, against labels from a native speaker. A judge validated in English is validated in English. Expect it to be worse elsewhere and budget for the result. How much worse, the evidence above does not say: it measures disagreement across languages and preference flips in pairwise comparison, not the decay of a pointwise judge's catch rate. That number exists for your task only once you collect it, and until then any assumption about it is an assumption.

2. Fix the judge's prompt language and record it. Prompt language changes outcomes, so it is an experimental condition. Pin it like everything else in the harness. Whether to judge in English or in the content's language is an empirical question for your task. Test both, once, and then stop varying it.

3. Prefer language-independent graders wherever possible. Schema validation, citation resolvability, tool-call matching, and numeric checks work identically in every language. This is the strongest argument for pushing work down the grader ladder: the cheap graders are also the ones that do not degrade across languages.

4. Use a jury where you must use a judge. Single-judge reliance is specifically called out as a problem here. A panel reduces variance, though it does not remove biases common to all members. See Where LLM judges break.

5. Get native speakers for the labels. There is no substitute. One native speaker labelling 100 real traces per language will tell you more than any amount of translated tooling. This is the expensive part, and it is the part that works.

6. Report per language, never pooled. A pooled score dominated by your English traffic hides a failing language entirely. Slice by language in every report, and see Statistics for why small per-language slices need wide error bars.

The cost, stated honestly

Doing this properly multiplies your annotation cost by roughly the number of languages you take seriously. That is a real budget, and most teams cannot spend it on every language they nominally support.

The reasonable compromise is a tier list rather than a pretence:

• Tier 1: full treatment. Native-speaker labels, per-language judge alignment, per-language reporting. Your top one or two non-English markets.
• Tier 2: language-independent graders on real traffic, plus a small native-speaker review sample. Deterministic checks and human spot-checks, no validated judge.
• Tier 3: output language identification and crash monitoring only. You are not measuring quality and you should say so internally rather than reporting a number you do not believe.

Naming the tiers is the useful part. It converts "we support 12 languages" into an honest statement about which ones you can make claims about.

Contested: Whether to judge in English or in the content's language. Judging in English exploits the models' strongest capability but requires the judge to reason about text in another language, and the language-switching results suggest that is exactly where invariance breaks. Judging in the target language keeps the task monolingual but uses a weaker capability. There is no settled answer, the correct choice appears task-dependent, and the useful move is to test both on 50 labelled examples rather than to inherit someone else's default.

Common mistake: Reporting one global quality score for a multilingual product. It is an average dominated by your largest language, and the worse a language performs, the less it moves the number, because it usually also has less traffic. The metric is structurally blind to exactly the failure you need it to surface.