Reading foundation-model benchmarks

This is the one chapter about other people's numbers. It is here because you cannot avoid reading them, and because reading them well is a distinct skill from building your own evals. Model cards, procurement decisions, and your CTO forwarding a leaderboard link are all facts of life.

The short version: public benchmarks are useful for coarse capability screening and almost useless for predicting your application's performance. Treat them the way you would treat a candidate's degree classification.

What benchmarks are actually for

A Layer A benchmark answers "is this model broadly capable at this kind of task". That is a real question with a real use: narrowing a field of twenty models to three worth testing on your own data.

They cannot answer whether a model will work in your product, because your product is mostly prompt, retrieval, tools, and domain, none of which the benchmark contains.

Used well, the workflow is: benchmarks narrow the field, your Layer B eval picks the winner. Used badly, the benchmark is the decision and you discover the mismatch in production.

Contamination is now the default assumption

A benchmark is an exam. Contamination is when the student has already seen the exam paper.

Not through deliberate cheating. The paper was published on the internet, and the model was trained on the internet. Benchmarks are public by design. That is what makes them comparable, and that same publication is what puts them in the next model's training data.

Why it matters: a high score is supposed to mean "this model can solve problems of this kind." If the model saw the exact problems during training, the score might instead mean "this model remembers the answer key." From the outside, the two look identical. You cannot tell them apart by looking at the number.

The usual defence is called n-gram decontamination. An n-gram is just a run of n consecutive words, and the defence is a search-and-delete: scan the training data for word runs that also appear in the test set, and remove those documents. It catches verbatim copies.

It does not catch a paraphrase. "What is 2 + 2?" and "Compute the sum of two and two" share almost no word runs and are the same question. That gap is exactly what the first finding below is about.

Finding: Contamination is measured, not just suspected, and the measurements are smaller and more interesting than the round numbers that circulate.

Rethinking Benchmark and Contamination with Rephrased Samples found that 8–18% of HumanEval overlaps with pre-training sets such as RedPajama-Data-1T and StarCoder-Data. Its more alarming result is about the defence rather than the disease: paraphrasing or translating a test item bypasses n-gram decontamination entirely, and once rephrased variants are in the training data, "a 13B model can easily overfit a test benchmark and achieve drastically high performance, on par with GPT-4".

Source: Rethinking Benchmark and Contamination, 2023

Finding: The cleanest natural experiment is Scale AI's GSM1k: 1,000-odd grade-school maths problems written by hand to mirror GSM8k's style, difficulty, human solve rate, solution length, and answer magnitude, then used to re-score everyone. The result was accuracy drops of up to 8%, "with several families of models showing evidence of systematic overfitting across almost all model sizes", while "many models, especially those on the frontier, show minimal signs of overfitting". A model's probability of generating a GSM8k example correlated with its GSM8k-to-GSM1k gap at Spearman's r² = 0.36, a measure of how closely two rankings move together, where 0 is no relationship and 1 is perfect. 0.36 is a moderate link: real and unlikely to be chance, but far from one-to-one.

That last correlation is the part worth keeping: it links the score gap to memorisation directly, rather than leaving contamination as an accusation. And the finding that frontier models were mostly clean is a real check on the assumption that every headline number is inflated.

Source: A Careful Examination of LLM Performance on Grade School Arithmetic, 2024

What the field does about it, in plain terms:

None of this fully solves it. Contamination is a structural consequence of publishing an evaluation, and the only complete answer is a private eval set, which is exactly what your Layer B work produces.

Benchmarks die, and that is normal

Benchmarks have a life cycle, and both ways they end are worth recognising.

Saturation is the benchmark becoming too easy. When every serious model scores 80% and above, the remaining 20% is mostly broken questions, and the benchmark can no longer tell good models from better ones. The exam has stopped discriminating, not because the students got worse, but because they all now pass it.

Defects are the questions themselves being wrong: ambiguous, or graded against a test that demands something the question never asked for.

Both happened to SWE-bench Verified at once.

Finding: On 23 February 2026 OpenAI announced it would stop reporting SWE-bench Verified. Two reasons, both of which generalise.

Saturation. State-of-the-art progress had slowed to a move "from 74.9% to 80.9% in the last 6 months", raising the question OpenAI puts directly: "do the remaining failures reflect model limitations or properties of the dataset itself?"

Defects. Of 138 problems o3 "did not consistently solve over 64 independent runs", 59.4% contained material issues in test design and/or problem description, "rendering them extremely difficult or impossible even for the most capable model or human to solve". Two thirds of those were tests that were too narrow, enforcing implementation details the task never specified. (Full breakdown in the bibliography.)

Source: OpenAI, Why SWE-bench Verified no longer measures frontier coding capabilities, 23 Feb 2026

The contamination half of that post contains the subtler finding, and it is the one worth carrying into your own work.

Finding: Models under test could "reproduce the original, human-written bug fix used as the ground-truth reference, known as the gold patch, or verbatim problem statement specifics for certain tasks". In one case GPT-5.2 solved 31 tasks OpenAI had identified as near-impossible; on django__django-14725, where the tests demand an edit_only parameter the problem statement never requires, its chain of thought showed knowledge of the Django release notes and correctly placed the parameter's introduction in Django 4.1.

The mechanism is the part to remember: "models that have seen the problems during training are more likely to succeed, because they have additional information needed to pass the underspecified tests." Contamination and underspecification are not two independent flaws. Contamination supplies the missing specification. A benchmark with underspecified tasks does not merely tolerate contamination. It rewards it.

Source: OpenAI, 23 Feb 2026

OpenAI pointed the community at SWE-bench Pro instead.

The replacement broke too, in five months

This is where the lifecycle stops being a theory. The benchmark that was the answer to the broken benchmark was itself audited and withdrawn before the year was out.

Pro saturated on the same curve: on its 731-task public split, frontier models went "from a pass rate of 23.3% to 80.3% in eight months".

Finding: On 8 July 2026 OpenAI retracted its own recommendation: "Given the issues uncovered in this analysis, we retract our earlier recommendation to adopt SWE-Bench Pro." Its estimate is that ~30% of SWE-bench Pro tasks are broken.

The audit ran in three stages. An automated filter flagged 286 potentially broken tasks; those went to both a Codex-based investigator-agent review with a researcher making the final call, and a human annotation campaign with five independent reviewers per task. The two disagreed in the expected direction: the agent pipeline flagged 200 tasks (27.4%), human reviewers 249 (34.1%), both out of the 731-task public split. Their category judgments overlapped in 74% of cases, and "in no flagged task was 'not broken' the most common human label".

By share of the dataset (agent / human): overly strict tests 14.4% / 17.8%, the largest class by a wide margin, low-coverage tests 4.1% / 9.4%, misleading prompts 6.3% / 7.5%, miscellaneous 1.9% / 1.2%, and underspecified prompts the smallest at well under 1%.

Source: OpenAI, Separating signal from noise in coding evaluations, 8 July 2026 · independently corroborated at ~30% by Faros AI (a vendor, disclosed)

Note what OpenAI did not do: name a successor. The February post had one ready; the July post asks the community to build "new benchmarks built by experienced software developers specifically to test model capabilities". If you are waiting for the next recommended coding benchmark, there currently is not one.

The root cause is the transferable part, and OpenAI states it plainly:

"Issues and pull requests from open-source repositories were originally created for human collaboration… problem descriptions, merged code, and unit tests do not always line up to form clean, isolated tasks for evaluating models reliably. In particular, tests included in pull requests can be overly strict because they are written to validate a specific change, rather than to define an implementation-agnostic standard for solving the task."

A test written to validate one pull request is not a definition of correctness. It was written to prove that this change worked, by the person who made that change, against the implementation they chose. Harvest it into a benchmark and you have silently promoted one author's implementation decisions into the grading criterion. Every functionally correct solution that took a different route now fails.

If you build eval cases by mining your own repository history, which is a genuinely good idea and one Dataset design recommends, you are exposed to exactly this. The mitigation is the one from Code-graded evals: be explicit about which assertions are necessary conditions and which encode one implementation. A test suite makes a fine grader and a poor specification.

The generalisable lesson is bigger than SWE-bench. A benchmark has a lifecycle: it is informative when new, saturates as models improve and training sets absorb it, and eventually its residual failures are dominated by defects in the benchmark rather than the model.

The practical consequences when reading a score:

• Check the benchmark's age. A 2023 benchmark quoted in 2026 is a historical note.
• Check how tightly the top is bunched. Once the leading models sit within a few points of each other, the remaining gap is noise and test defects rather than capability, which is precisely the argument OpenAI made about both SWE-bench benchmarks above.
• Check whether a serious lab has publicly abandoned it. That is the strongest available signal that a benchmark is finished. Note that OpenAI abandoned two in five months, including the one it had just recommended.
• Check the date of the replacement, not just the benchmark. "We moved to the newer one" is a claim with a half-life.

Measurement fragility

Even uncontaminated, unsaturated benchmarks are more fragile than their published precision implies.

Finding: A Single Character can Make or Break Your LLM Evals (2025) found MMLU performance varying by up to ±23% with the choice of delimiter between in-context examples, and showed that model rankings could be reordered to place any model first by changing that single character. The effect does not diminish with scale.

Two consequences. When comparing published scores from different sources, you are usually comparing different harnesses, and a 1–2 point difference means nothing. And when someone reports their own run of a public benchmark, the harness and prompt format are part of the result. See The offline harness.

Arenas and the incentive problem

An arena works differently from a benchmark. There is no question bank and no correct answer. A visitor types a prompt, gets two anonymous answers from two different models, and votes for the better one. Millions of those votes are turned into a ranking, the way chess ratings are built from match results.

That measures something a benchmark cannot: what people actually prefer. It also inherits every quirk of what people prefer, including several that are not quality. And because a public ranking is worth money, it attracts the kind of optimisation described below.

Human-preference arenas measure something benchmarks cannot: what people prefer. They also have structural dynamics worth knowing.

Finding: The Leaderboard Illusion (April 2025) argued that Chatbot Arena rankings are distorted by undisclosed private testing with selective disclosure, and by asymmetric data access. On the first count, the paper identifies 27 private LLM variants tested by Meta in the lead-up to the Llama-4 release. On the second, Google and OpenAI are estimated to have received 19.2% and 20.4% of all arena data respectively, against 29.7% for 83 open-weight models combined. It estimates that even limited additional data can yield relative performance gains of up to 112% on the arena distribution. LMArena published a rebuttal stating its testing policy is applied uniformly and that any lab may submit as many variants as it can run.

Read both. The disagreement is partly empirical and partly about what a leaderboard owes its readers, and neither side is obviously wrong.

Separately from the dispute, arenas measure preference, and preference has known correlates that are not quality: responses with bulleted lists and particular lengths perform better. This is the verbosity bias you already know about, appearing in human raters rather than model judges.

Aggregate indices

Composite scores, Artificial Analysis-style intelligence indices and similar, are convenient and add a layer of opacity. The composite's value depends entirely on the weighting, the weighting is a judgement call, and it is rarely the judgement call you would make for your use case.

If you use one, look at the component scores. If the index moved because of a benchmark irrelevant to you, the index did not move for you.

How to actually use a benchmark

Before the process, a short reading guide. When a benchmark number is put in front of you, four questions settle how much of it to believe:

None of these require expertise. They require asking, and most published comparisons do not survive all four.

A defensible model-selection process:

1. Shortlist on benchmarks, weighted toward tasks resembling yours. Take three or four candidates, not one. Ignore differences under a few points. After the SWE-bench Pro audit, treat "a few points" on a coding benchmark as generous rather than cautious.
2. Check the non-capability constraints: context window, latency, structured-output support, tool-calling reliability, price, and where the model is hosted. These eliminate candidates faster than capability does, and they are facts rather than estimates.
3. Run your own eval set. This is the decision. A hundred examples from your own dataset will separate the shortlist more decisively than any public number, enough to eliminate candidates. Certifying a close call takes more, and Model selection shows how much, and what else has to happen before a cheaper model goes live.
4. Pilot on live traffic. Shadow-run or A/B. See Online evaluation.

Step 3 is where the actual information is, and it is cheap. A hundred examples against four models is an afternoon.

Common mistake: Choosing a model on a benchmark that does not resemble your task: picking a competition-mathematics leader for a customer-support product. Capability is not one-dimensional, and the correlation between benchmark families is weaker than the single-number framing suggests.

Contested: Whether public benchmarks retain enough signal to be worth reading. The pessimistic view is that contamination and saturation have made them marketing artefacts. The optimistic view is that they remain the only comparable cross-model measurements in existence, and that a coarse, noisy signal beats none. Both are defensible. The position this guide takes is narrower and, we think, uncontroversial: whatever their value for screening, they are not evidence about your application, and the only thing that is, is your own eval set.