Bibliography

Everything this guide draws on, with what it is good for and how much weight it can bear. Confidence reflects how well-sourced a claim is, not how much we agree with it.

Where a figure in a chapter came from secondary coverage rather than a verified primary source, the chapter says so and the entry below repeats it.

The practitioner method

Hamel Husain & Shreya Shankar, LLM Evals: Everything You Need to Know (evals FAQ), May 2025 · hamel.dev · Layer C · Confidence: high The densest practitioner source that exists, distilled from teaching thousands of engineers. Source of nearly every concrete number in Error analysis and Dataset design: 50–100 traces to start, saturation at ~20 clean traces, 100+ per review cycle, 60–80% of development time, 100+ examples for CI.

Hamel Husain, Creating a LLM-as-a-Judge That Drives Business Results · hamel.dev · Layer B/C · Confidence: high The seven-step judge method in Building an aligned LLM judge. Binary judgments with critiques, critiques as few-shot examples, precision and recall over raw agreement, error analysis by segment.

Eugene Yan, Evaluating the Effectiveness of LLM-Evaluators, Aug 2024 · eugeneyan.com · Layer B · Confidence: high A synthesis of roughly two dozen papers, and the source of every correlation figure and bias effect size in Where LLM judges break. The single most useful document for calibrating expectations about judges.

Eugene Yan, An LLM-as-Judge Won't Save The Product — Fixing Your Process Will · eugeneyan.com · Layer C · Confidence: high The counterweight to tool-shopping.

Shankar, Zamfirescu-Pereira, Hartmann, Parameswaran, Arawjo, Who Validates the Validators?, UIST 2024 · arXiv:2404.12272 · Layer B/C · Confidence: high Names and demonstrates criteria drift. The formal argument for reading data before writing a rubric.

Anthropic, Demystifying evals for AI agents, Jan 2026 · Anthropic Engineering Blog · Layer B/C · Confidence: high Agent evals as systems tests. State over sentence, resettable environments, multiple trials, separate regression and capability suites, weekly transcript sampling.

Anthropic, Define your success criteria and the evals cookbook · docs.anthropic.com · Layer B · Confidence: high

LLM-as-judge

Liu et al., G-Eval, Mar 2023 · arXiv:2303.16634 · Layer B · Confidence: high Form-filling with chain-of-thought and probability-weighted scoring; average Spearman ρ 0.514. Historical context. Read it specifically to understand why most implementations calling themselves G-Eval are not.

A Survey on LLM-as-a-Judge · arXiv:2411.15594 · Layer B · Confidence: high · Use for taxonomy, not for numbers.

Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge, Oct 2024 · arXiv:2410.02736 · Layer B · Confidence: high

Self-Preference Bias in LLM-as-a-Judge, Oct 2024 · arXiv:2410.21819 · Layer B · Confidence: high

Judging the Judges: A Systematic Study of Position Bias, IJCNLP 2025 · ACL Anthology · Layer B · Confidence: high

Am I More Pointwise or Pairwise? Position Bias in Rubric-Based LLM-as-a-Judge, Feb 2026 · arXiv:2602.02219 · Layer B · Confidence: medium Position bias survives the move to pointwise rubrics. Relevant if you switched to rubrics to escape it.

Multilingual

How Reliable is Multilingual LLM-as-a-Judge?, EMNLP Findings 2025 · ACL Anthology · Layer B · Confidence: high Average Fleiss' κ ≈ 0.3 across languages.

Does the Judge Prefer English?, Jun 2026 · arXiv:2606.14278 · Layer B · Confidence: high Four API judges, 419 LLMBar pairs, 13,408 judgments across English, Chinese, and language-switched variants. 10.7–14.4% preference flips relative to English; all judges most accurate in English. Note the paper's own counterweight, which this guide reproduces: translation-equivalent tie probes showed no systematic English preference, and non-tie decisions more often favoured Chinese. The finding is instability under switching, not blanket English bias.

Challenges and Recommendations for LLMs-as-a-Judge in Multilingual Settings and Low-Resource Languages, Jul 2026 · arXiv:2607.02235 · Layer B · Confidence: high Performance is overestimated for low-resource languages.

Towards Reliable Multilingual LLMs-as-a-Judge, May 2026 · arXiv:2605.28710 · Layer B · Confidence: medium

Agents

τ-bench (Yao et al., 2024) · arXiv:2406.12045 · Layer A · Confidence: high Introduced pass^k, "a new metric (pass^k) to evaluate the reliability of agent behavior over multiple trials", and reported that state-of-the-art agents were "quite inconsistent (pass^8 <25% in retail)". Take the metric, not the benchmark.

τ²-bench (Barres et al., Jun 2025) · arXiv:2506.07982 · GitHub · Layer A · Confidence: high for the dual-control design; the per-domain task counts circulating in secondary write-ups (airline 50, retail 114, telecom 114) are not stated in the abstract or README and are not repeated in this guide. Extends τ-bench to a dual-control environment where "both agent and user make use of tools to act in a shared, dynamic environment".

Beyond the Final Answer: Evaluating the Reasoning Trajectories of Tool-Augmented Agents, Oct 2025 · arXiv:2510.02837 · Layer B · Confidence: high

Holistic Agent Leaderboard, Oct 2025 · arXiv:2510.11977 · Layer A · Confidence: medium Headline agent scores conceal cost and reliability differences.

Berkeley Function Calling Leaderboard v4, ICML 2025 · gorilla.cs.berkeley.edu · Layer A/B · Confidence: high Deterministic AST matching against a reference invocation. The technique is directly reusable for your own tool-call evals.

OSWorld, WebArena, VisualWebArena, GAIA, BrowseComp, AssistantBench · Layer A · Confidence: medium · The browser and computer-use landscape; OSWorld covers 369 tasks across a full desktop. OSWorld 2.0 (arXiv:2606.29537) extends to long-horizon tasks.

Coding and benchmark decay

OpenAI, Why we no longer evaluate SWE-bench Verified, Feb 2026 · openai.com · Layer A · Confidence: high, read from the primary source Verified from the primary source, 2026-08-28. Saturation: the top score moved only 74.9% → 80.9% over six months. Defects: 138 problems, a 27.6% subset of the 500, that o3 "did not consistently solve over 64 independent runs", each reviewed independently by at least six experienced software engineers, with anything flagged re-verified by a further team. 59.4% contained material issues in test design and/or problem description: 35.5% narrow test cases enforcing unspecified implementation details, 18.8% wide test cases checking functionality the description never mentioned, 5.1% miscellaneous. Contamination evidence: models reproduced gold patches and verbatim problem statements, and "models that have seen the problems during training are more likely to succeed, because they have additional information needed to pass the underspecified tests".

SWE-bench Pro · Layer A · Confidence: medium · 1,865 tasks, 41 repositories, large held-out split. Recommendation withdrawn by OpenAI in July 2026, see below.

OpenAI, Separating signal from noise in coding evaluations, July 2026 · openai.com · Layer A · Confidence: high, read from the primary source OpenAI retracted its own recommendation of SWE-bench Pro, stating it "no longer reliably measures frontier coding capability" and that "30% of SWE-Bench Pro tasks" are broken. Audit method: model-based investigator agents plus five independent experienced software engineers. Automated pipeline flagged 200 tasks (27.4%); human reviewers 249 (34.1%). Defect classes in order of size: overly strict tests, underspecified prompts, low-coverage tests, misleading prompts. Independently corroborated at the same ~30% by Faros AI, a vendor with a disclosed product interest, working from 152 manually reviewed disagreement cases. The generalisable point is the most reusable finding in this bibliography for teams mining their own repository history: a test written to validate one pull request is not a specification of correctness.

UTBoost: Rigorous Evaluation of Coding Agents on SWE-Bench, Jun 2025 · arXiv:2506.09289 · Layer A · Confidence: medium

Contamination

A Survey on Data Contamination for Large Language Models, Feb 2025 · arXiv:2502.14425 · Layer A · Confidence: high Mitigations: label protection, canary strings, encryption, post-hoc detection, dynamic benchmarking.

AntiLeakBench, ACL 2025 · ACL Anthology · Layer A · Confidence: high

LLM Benchmark Datasets Should Be Contamination-Resistant, May 2026 · arXiv:2605.19999 · Layer A · Confidence: medium

Rethinking Benchmark and Contamination for Language Models with Rephrased Samples, 2023 · arXiv:2311.04850 · Layer A · Confidence: high 8–18% of HumanEval overlaps with RedPajama-Data-1T and StarCoder-Data. Rephrasing or translating a test item bypasses n-gram decontamination, after which "a 13B model can easily overfit a test benchmark and achieve drastically high performance, on par with GPT-4".

A Careful Examination of Large Language Model Performance on Grade School Arithmetic (GSM1k), Scale AI, 2024 · arXiv:2405.00332 · Layer A · Confidence: high Hand-written replica of GSM8k matched on style, difficulty, human solve rate, solution steps and answer magnitude. Accuracy drops of up to 8% with systematic overfitting in several model families; frontier models mostly clean. Spearman's r² = 0.36 between a model's probability of generating a GSM8k example and its GSM8k-to-GSM1k gap.

Correction, 2026-08-28: earlier drafts of this guide cited "~40% of HumanEval" and a "~13 point" GSM8K drop from secondary summaries. Both are wrong. The verified figures are 8–18% and up to 8% respectively, and the GSM1k result is a fresh-benchmark comparison rather than a decontamination experiment. The BIG-Bench canary-GUID reproduction claim could not be traced to a primary source and has been removed.

RAG

RAGAS documentation · docs.ragas.io · Layer B · Confidence: high · Source of the standard metric vocabulary.

RAGChecker, NeurIPS 2024 · arXiv:2408.08067 · Layer B · Confidence: high Claim-level entailment separating retriever-side from generator-side failure. Verified in the paper's meta-evaluation table: Pearson 61.93 with human preference on overall assessment, against the strongest baseline RAGAS at 48.31, and ROUGE-L 43.10, TruLens 35.15, BLEU-avg 35.14, BERTScore 33.51, ARES 17.81.

ARES · Layer B · Confidence: medium · Synthetic data plus lightweight judges, calibrated with prediction-powered inference.

Statistics and harness

Evan Miller, Adding Error Bars to Evals, Nov 2024 (Anthropic) · arXiv:2411.00640 · Layer C · Confidence: high Five recommendations: standard errors via the CLT, clustered standard errors for grouped questions, variance reduction via resampling and next-token probabilities, paired inference when comparing two models, and power analysis. Verified against the paper text: "Clustered standard errors can be over 3X larger than naive standard errors" (DROP shows a ratio of 3.05); "using paired differences will reduce the variance of the estimator by 1/3 in relative terms"; and the worked example where "increasing KA=KB from 1 to 10 reduces the Minimum Detectable Effect from 13.2% to 7.5%".

A Single Character can Make or Break Your LLM Evals, Oct 2025 · arXiv:2510.05152 · Layer A/C · Confidence: high Delimiter choice moves MMLU by up to ±23%. Verified per-model in the paper: Llama-3.1-8B 18.3%, Qwen2.5-7B 23.5%, Gemma-2-9B 29.4%. Also tested on ARC-Challenge, CommonsenseQA, Banking77, and TACRED. "One can manipulate model rankings to put any model in the lead by only modifying the single character separating examples", and the brittleness "doesn't improve with scale". Naming the delimiter in the prompt improved robustness by 1.5–27.9%. The strongest argument for pinning prompt format.

The Leaderboard Illusion, Apr 2025 · arXiv:2504.20879 · Layer A · Confidence: high And LMArena's reply: news.lmarena.ai. Read both.

Tooling

Inspect (UK AI Security Institute) · inspect.aisi.org.uk · GitHub · Confidence: high

EleutherAI lm-evaluation-harness, HuggingFace lighteval · Confidence: high

OpenAI Evals (deprecated) · deprecation notice · migration to promptfoo · Confidence: high Announced 3 June 2026; read-only 31 October 2026; shutdown 30 November 2026.

promptfoo · promptfoo.dev · Confidence: high

DeepEval, Braintrust, LangSmith, Arize Phoenix, Langfuse, W&B Weave, Evidently, MLflow · Confidence: high that they exist and do what they say; low for any published comparison Every high-ranking comparison of these tools is published by one of them. Treat comparison pages as marketing and verify claims against primary documentation.

Safety

HarmBench · overview · Layer A · Confidence: high

AILuminate (MLCommons, Dec 2024) · Layer A · Confidence: high · 12 hazard categories, 24,000 prompts.

AgentHarm · overview · Layer A · Confidence: high · 11 harm categories; measures propensity to complete malicious multi-step tasks.

CyberSecEval (Wan et al., 2024) · Layer A · Confidence: medium

International AI Safety Report 2026, Feb 2026 · arXiv:2602.21012 · Layer A · Confidence: high

Added in the second research pass (2026-08-28)

Sources brought in to close the thin chapters: CI, production, safety, and the structured-output section of the code-graded chapter.

Towards More Standardized AI Evaluation: From Models to Agents, Feb 2026 · arXiv:2602.18029 · Layer A/B · Confidence: high Frames the model-centric → agent-centric shift and argues evaluation has become "a core control function". Source of the framing in The three layers.

Let Me Speak Freely? A Study on the Impact of Format Restrictions, EMNLP 2024 Industry Track · ACL Anthology · arXiv:2408.02442 · Layer A/B · Confidence: high Reasoning declines under format restrictions; stricter constraints degrade more; mechanism is output misordering. Counter-evidence from schema ablations means this is presented as contested in Code-graded evals.

When Generic Prompt Improvements Hurt: Evaluation-Driven Iteration for LLM Applications, 2026 · arXiv:2601.22025 · Layer B/C · Confidence: high Proposes the Minimum Viable Evaluation Suite. Generic prompt additions are not monotonic: one model's RAG citation-compliance fell from 26/30 to 9/30 when generic rules were appended. The empirical basis for gating prompt changes in Evals in CI.

Dwork, Feldman, Hardt, Pitassi, Reingold, Roth, Generalization in Adaptive Data Analysis and Holdout Reuse, 2015 · arXiv:1506.02629 · Layer C · Confidence: high Thresholdout: noise-based holdout reuse supporting quadratically more adaptive queries.

Recht et al., ImageNet test-set reproduction, and Mania et al., Model Similarity Mitigates Test Set Overuse, 2019 · arXiv:1905.12580 · Layer C · Confidence: medium Absolute accuracy dropped on a rebuilt test set but the adaptive-overfitting component was "limited to non-existent"; many-class problems resist test-set reuse. The reason Evals in CI presents eval-set overfitting as contested rather than settled.

OpenTelemetry GenAI semantic conventions · opentelemetry.io · Layer C · Confidence: high CNCF-backed gen_ai.* span attributes; GenAI SIG formed April 2024; scope now covers agent orchestration, MCP tool calling, and quality evaluation. Implemented by Google Cloud, AWS, Azure, Datadog. The instrumentation recommendation in Online evaluation.

Chapelle, Joachims, Radlinski & Yue, Large-scale validation and analysis of interleaved search evaluation, TOIS 30(1), Article 6, Feb 2012 · 10.1145/2094072.2094078 · Layer C · Confidence: high · the large-scale validation of interleaving.

Radlinski & Craswell, Optimized interleaving for online retrieval evaluation, WSDM 2013 · Semantic Scholar · Layer C · Confidence: high The actual source of the sensitivity claim: 38 large-scale online experiments, 3+ billion clicks, sensitivity one to two orders of magnitude above A/B tests with up to 22% better agreement with A/B metrics. Linked via Semantic Scholar: ACM's DOI page sits behind a bot challenge and could not be verified from here, and after the mis-citation noted below we are not printing a DOI we have not resolved.

Correction, 2026-08-28: an earlier draft cited DOI 10.1145/2071389.2071390 for the Chapelle et al. paper. That DOI is a different work entirely (Carpineto & Romano, "A Survey of Automatic Query Expansion in Information Retrieval"). It also attributed the sensitivity figure to Chapelle et al. rather than to Radlinski & Craswell. Both fixed.

OWASP Top 10 for LLM Applications, 2025 · owasp.org · Layer B · Confidence: high Prompt injection at LLM01 for the second consecutive edition; excessive agency at LLM06; system prompt leakage added.

XSTest, 2023 · arXiv:2308.01263 · Layer A · Confidence: high · 250 safe prompts across ten types, 200 unsafe contrasts.

OR-Bench, ICML 2025 · arXiv:2405.20947 · Layer A · Confidence: high · 80,000 over-refusal prompts, 10 categories, ~1,000 hard subset, 600 toxic prompts, 32 LLMs across 8 families. Verified in the abstract.

AgentDojo, NeurIPS 2024 · arXiv:2406.13352 · Layer A/B · Confidence: high 97 realistic user tasks across email, e-banking and travel; 629 security test cases. "State-of-the-art LLMs fail at many tasks (even in the absence of attacks)" and "existing prompt injection attacks break some security properties but not all".

InjecAgent, 2024 · arXiv:2403.02691 · Layer A/B · Confidence: medium · the single-step indirect-injection counterpart to AgentDojo.

Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents, 2025 · arXiv:2503.00061 · Layer B · Confidence: high Published defences fail against attackers aware of them. The reason Safety ranks permission limits above detection.

ARES, 2023 · arXiv:2311.09476 · Layer B · Confidence: medium · synthetic data plus lightweight judges, calibrated with prediction-powered inference.

Added in the third research pass (2026-08-30)

Purpose of this pass: put chapters that rested on a single source onto independent footing. Affected were dataset design (Husain/Shankar only), statistics (Miller only) and the judge failure modes (predominantly Eugene Yan).

Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, NeurIPS 2023 · arXiv:2306.05685 · Tier A · Confidence: high The primary source for position, verbosity and self-enhancement bias. The figures in judge failure modes come from here and were checked against the full text: position 50.0% / 75.0% (Table 2), "repetitive list" 91.3% against 8.7% for gpt-4 (Table 3), self-enhancement +10% / +25% (Figure 2b, described by the authors themselves as not established beyond doubt).

Panickssery, Bowman & Feng, LLM Evaluators Recognize and Favor Their Own Generations · arXiv:2404.13076 · Tier A · Confidence: high Independent mechanism behind self-preference: linear correlation with self-recognition capability.

Artstein & Poesio, Inter-Coder Agreement for Computational Linguistics, 2008 · ACL Anthology J08-4004 · Tier A · Confidence: high The standard survey of agreement coefficients. Supplies the stricter bar (κ > 0.8) and thus the counterpoint to the practitioner threshold in dataset design. The bands themselves go back to Landis & Koch, Biometrics 33(1), 1977, whose authors call them arbitrary.

Kiritchenko & Mohammad, Best-Worst Scaling More Reliable than Rating Scales, ACL 2017 · arXiv:1712.01765 · Tier A · Confidence: high Empirical evidence against graded rating scales from annotation research: a different remedy from the binary label, the same diagnosis.

Guest, Bunce & Johnson, How Many Interviews Are Enough?, Field Methods 18(1), 2006 · Semantic Scholar · Tier A · Confidence: high The empirical basis for theoretical saturation: saturation after twelve interviews, basic elements after six. Linked via Semantic Scholar because SAGE blocks automated retrieval; DOI 10.1177/1525822X05279903.

Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity · arXiv:2510.01171 · Tier A/B · Confidence: medium Independent evidence that naive prompting collapses onto a few typical outputs, with the cause traced to preference data (typicality bias).

Messing, Hidden Measurement Error in LLM Pipelines, 2026 · arXiv:2604.11581 · Tier A/B · Confidence: medium Independent second source for statistics: naive standard errors 40–60% too small when variance from judge choice, temperature and prompt wording is left out.

Kish, Survey Sampling, 1965 · Moulton, 1986 · Tier A · Confidence: high Classical origin of the design effect. Establishes that the clustering problem in statistics is not an LLM peculiarity.

Added in the review pass (2026-09-02)

Rogan & Gladen, Estimating prevalence from the results of a screening test, American Journal of Epidemiology 107(1), 1978 · DOI 10.1093/oxfordjournals.aje.a112510 · Tier A · Confidence: high The correction used in building a judge to recover a true pass rate from a judge with known TPR and TNR. Fifty years of use in epidemiology for exactly this shape of problem: a screening test with known sensitivity and specificity.

Walker & Nowacki, Understanding Equivalence and Noninferiority Testing, J Gen Intern Med 26(2), 2011 · PMC3019319 · Tier A · Confidence: high The clinical-trials framing behind model selection: the non-inferiority margin as a decision made in advance, and why narrowing it costs sample size out of proportion. DOI 10.1007/s11606-010-1513-8.

Chen, Zaharia & Zou, FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance, 2023 · arXiv:2305.05176 · Tier A/B · Confidence: medium LLM cascades: the "up to 98% cost reduction" while matching the best single model is on the paper's tasks. Cited in model selection for the technique, not the figure.

Ong et al., RouteLLM: Learning to Route LLMs with Preference Data, 2024 · arXiv:2406.18665 · Tier A/B · Confidence: medium Routers trained on human preference data; "over 2 times" cost reduction "in certain cases" without quality loss, on the paper's benchmarks. Same caveat as FrugalGPT.