Model selection: the cheapest model that clears the bar

The question arrives as a one-line message: "Model X is a third of the price. Can we switch?" There are three ways to answer it badly. Point at a leaderboard, which measures somebody else's task. Run your hundred cases once, read off two percentages, and pick the higher one. Or switch in production and wait for the tickets.

This chapter is the fourth way. Nothing in it is new: every piece is already in this guide. But the model question is where the pieces have to work together, and it is the decision most eval sets are actually used for. In order:

1. Work out what the switch could save at most. If that is small, stop here.
2. Decide in advance how much worse the new model may be.
3. Run both models on your cases, and read the new model's failures before its score.
4. Check the floors: the checks that allow no noise.
5. Split the result by kind of request.
6. Re-check the judge on the new model's answers.
7. Collect enough cases to decide, and decide.
8. Count cost per request, not per token.

The whole chapter follows one case. Everything in it is invented, as in the worked example. Same company as there, different feature: the assistant inside the product that answers schedulers' questions about rota rules, such as "can Meier take the night shift on Friday if he worked Thursday?", from the customer's own rule set plus the documentation.

First, what the switch is worth

Before running anything, compute the most the switch could save. If the candidate could simply replace the current model with no loss: 6,800 € down to about 2,300 €, so 4,500 € a month. That is worth a week of work.

Do this step because it often ends the exercise. The support-draft assistant from the worked example handles 2,800 drafts a month; the same switch there saves under 10 €, and the week is not worth it. Your first eval in a week lists the cases where evals are not worth building; a model swap that saves less than the evaluation costs is one of them.

The question is "not worse", not "better"

You are not trying to show that the candidate is better. You are trying to show that it is not worse by more than an amount you decided in advance. Decide that amount before you see any numbers: how much quality would you trade for the saving? Two points? Five? Statisticians call this the non-inferiority margin; the plain question is "how much worse may it be?", and it is a product decision, not a statistical one.

Finding: This is a standard problem in clinical trials, where a cheaper or safer treatment has to be shown not worse than the established one. Walker and Nowacki define the margin as "the maximum clinically acceptable difference that one is willing to accept in return for the secondary benefits of the new therapy", and show that the sample size is far more sensitive to the margin than in a superiority trial. In their example, narrowing the margin from 12 to 8 percentage points nearly doubled the required n.

Source: Walker & Nowacki, Understanding Equivalence and Noninferiority Testing, J Gen Intern Med 26(2), 2011 · DOI 10.1007/s11606-010-1513-8

The test itself is the paired comparison from Statistics: both models on the same cases, compared case by case. Count the cases only the old model passed and the cases only the new one passed; the difference is the loss. Then add the noise, the part of the loss that could be chance, and check whether loss plus noise stays below the margin. For a margin of 5 points:

loss   = (old-only passes − new-only passes) / n
noise  = 1.96 × sqrt(old-only + new-only) / n
passes if  loss + noise ≤ 0.05

Two things follow. The margin has to be written down first, because once the numbers are in, it will drift to wherever they landed. And this test needs more cases than a "which is better" comparison, because you are trying to pin a small difference down rather than spot a large one. Reading benchmarks said a hundred cases against four models is an afternoon. It is, for eliminating models. For certifying a close call within a few points it is nowhere near enough, and the case below shows by how much.

Read the candidate's failures before its score

The team ran both models over the 200 cases with the same prompt. The current model passed 152 (76%), the candidate 131 (65.5%). That is the number most teams stop at.

Instead they read all 69 of the candidate's failures the way Error analysis reads traces: one sentence each, first failure, no categories. Two kinds of failure had never appeared before:

Here is why the score alone misleads: your eval set describes the old model's failures. It was built from that model's traces, its judge was tuned on that model's answers, and its cases were chosen around that model's weak spots. A different model fails differently, often in kinds of failure the set has no cases for. Until you have read its failures, its score only says how well it does at the old model's problems.

Then sort the new failures the way Day 2 does: which are bugs, which are the model? The prose-before-JSON failure went away with the provider's structured-output mode and one line in the format instruction, changed on the development set, never on the test set (Dataset design explains the two). Re-run: 143 of 200. The invented rule sections did not move. That is the model.

Floors first, then slices

A floor is a check with no noise in it: the deterministic checks, and the named incident cases. Evals in CI explains why they get no error bar. A case that passed yesterday and fails today is an incident, not chance. A candidate has to clear the same floors as a release, and one that fails them is out before any statistics.

The candidate cleared the tenant check (0 of 200, same as the current model) and, after the format fix, the schema check. It did not clear the citation check: 6 of 200 cited sections that do not exist, against 0 for the current model, and nothing in production would catch that today. Of the 52 named incident cases it passed 46 where the current model passed 50.

As a straight replacement for all traffic, the candidate is out at this point. Most teams would stop here, and it is a defensible place to stop. The team went one step further and sliced the result by kind of request, the way the judge chapter does with its 22%:

The collapse is entirely in one row, and so are all six invented sections. On plain rule lookups, more than half the traffic, the candidate is two cases behind the current model. The integrations row looks like a win for the candidate; on 40 cases, four cases is noise, and the team treated it as such.

This is the shape to expect. A cheaper model rarely fails evenly. It holds on the routine requests and collapses on the hard ones, and the overall score averages the two into a number that is wrong about both.

The judge was tuned on the old model

The support lead labelled 60 of the candidate's answers on rule lookups, pass or fail, the way Day 4 describes. 19 were real failures. The judge found 15 of the 19 and cleared 38 of the 41 good ones. On the current model's answers it had found 23 of 28, about the same, and both numbers carry the wide spread that 19 and 28 failures bring. Reading the four misses showed no pattern tied to length or format.

Why this afternoon was necessary: the judge's catch rate, how many real failures it finds, was measured on the current model's answers. The candidate writes differently, shorter here and formatted differently, and verbosity bias alone can shift a judge's verdicts between two models of equal correctness. Had the four misses all been short answers, the judge would have been unusable for this comparison. And if the candidate were also the judge model, or the current model were, the self-preference findings apply on top.

Skip this step and every judge number that follows has unknown sign for the candidate.

Enough cases to decide

Now the test from the second section, on the rule-lookup row, which is where the candidate might go. The margin was set before the run: 5 points. The reason was written next to it: the answer shows the cited rule and the scheduler reads it before acting, so a wrong lookup costs a click, not a shift.

Ninety cases cannot decide this. On 90 cases the two models were two cases apart, and the noise alone was about 8 points, more than the whole margin. So the team pulled 300 fresh rule-lookup traces the eval set had never contained, ran both models with the adapted prompt, and scored them with the re-checked judge. The current model passed 231 of the 300, the candidate 227. Case by case: 17 cases only the current model passed, 13 only the candidate, a difference of four cases on 300.

loss   = (17 − 13) / 300            = 0.013   →  1.3 points
noise  = 1.96 × sqrt(17 + 13) / 300 = 0.036   →  3.6 points

loss + noise = 4.9 points  ≤  5    →  not worse than the margin allows

Just inside. Had the margin been 3 points, the same result would have needed roughly 1,400 cases. The margin decides the bill, which is why it is written down first and by the product owner, not by whoever runs the numbers.

Cost per request, not per token

The price list said a third. The harness records tokens per case, so the team could check:

The candidate writes longer answers even after the format fix, so it costs 40% of the current model per request, not a third. For agents the gap is usually larger: a cheaper model that needs two more steps or a retry per task can cost more per task than the expensive one. Count tokens, retries, steps and wall-clock per case. The harness has them already.

For this case: 0.23 cents saved on 55% of 60,000 requests a day is about 76 € a day, roughly 2,300 € a month. Half the ceiling from the first section, because half the traffic is not going anywhere.

Routing: the slice table is the routing table

The result is not "switch" or "do not switch". It is: send rule lookups to the candidate, keep conflict resolution on the current model. The table with the three rows is that routing table, and it exists only because every case carried its kind of request as metadata. Dataset design says to attach that on day one.

Finding: Cascades and routers are a studied way of getting most of a strong model's quality at a fraction of its cost. FrugalGPT (2023) learns "which combinations of LLMs to use for different queries" and reports matching the best individual model "with up to 98% cost reduction" on its tasks, or beating it by 4% at the same cost. RouteLLM (2024) trains routers on human preference data and reports reducing costs "by over 2 times in certain cases — without compromising the quality of responses". Both numbers are on the papers' benchmarks, not on your traffic; the technique transfers, the figures do not.

Source: FrugalGPT, Chen, Zaharia & Zou, 2023 · RouteLLM, Ong et al., 2024

Two things about routing that teams discover late. The router is a component with its own error rate. Something has to decide which requests are rule lookups, and if that is a classifier or a small model call, it sends some conflict questions to the cheap model, exactly the row it is bad at. Measure the router like any other component, and run the end-to-end eval with routing switched on, because that is the system users get. Here the router was a small model call at about 0.02 cents on every request, roughly 360 € a month off the saving.

A cascade needs a checker you already trust. FrugalGPT's cheapest pattern tries the cheap model first and escalates when a checker is unsure. That checker is a judge in the request path, which makes it a guardrail, tuned for precision, as Online evaluation explains, and not the judge you measure with. Do not put an unvalidated judge in the hot path to save money on the model behind it.

Same prompt or adapted prompt?

Contested: Whether to compare models on the same prompt or let each have a prompt tuned for it. Same prompt measures the straight replacement, which is the question most teams are actually asking, but it handicaps the candidate: the prompt was iterated against the current model for months. Adapted prompts measure the potential, but the adaptation is an optimisation against your eval set and has to stay on the development set. The delimiter finding makes the choice matter more than it looks. The workable rule: compare as a straight replacement first; if the candidate fails narrowly, adapt on the development set and decide on the test set; and say which of the two you measured when you report the number.

The answer has a half-life

The team's result is to send 55% of traffic to the candidate and save about 1,900 € a month net. It is true for two pinned model versions, one prompt hash and one price list. All three change. Providers ship new snapshots, cheap models get better on hard rows, and prices move.

So the model decision is not a document. It is an eval run with both models in the harness, pinned, that a pull request can re-execute when any of the three moves. The routing table lives in configuration next to the pinned versions. And the decision is closed the way Online evaluation describes: first a shadow run, where the new model answers real requests without anyone seeing them; then an A/B test on the routed row with a product metric such as follow-up question rate, thumbs or escalation, because the eval score is a stand-in, and this is where you find out whether it tracked.

Common mistake: Deciding a model swap on the overall score of the existing eval set. It measures the old model's failure modes, its judge was tuned on the old model's answers, and its hundred-odd cases can eliminate a candidate but cannot certify one. Read the candidate's failures, slice, re-check the judge, set the margin first, and count the cost per request, in that order.