How to test an LLM change without fooling yourself

There's a new model or reasoning setting almost every week, and it's easy to try one, glance at the result, and decide it's better. That glance is the problem. A single run tells you very little, and it will cheerfully confirm whatever you were hoping to see. So we built a small rig to check properly. It is not a study — the samples are far too small — but it has caught us being wrong more than once.

Most bad comparisons move two things at once. We move one. The evidence stays frozen — the same eight vetted sources on every run — and the prompt is the real one, imported straight from the app instead of retyped, so we're testing what actually ships. Then we change a single knob: the model, the reasoning effort, or the length instruction. Whatever shifts in the output came from that knob. It's cheap, too. There's no live search to pay for, so a full grid is minutes and a few dollars.

A model grading its own output tends to like what it sees. So we stripped the labels off every candidate and had both families grade — Opus marking Sonnet's work, Sonnet marking Opus's. The two defaults came out level, 4.68 to 4.68. What mattered was the next part: change the grader and the order stayed put. Opus scored at least as high as Sonnet whether Opus or Sonnet held the pen. Had only one judge called it a tie, we wouldn't have trusted it.

Here is the moment. At maximum reasoning effort, the first output caught two methodological points the default had skipped — a 75% jump on our depth measure, and enough to justify paying for the expensive setting. Then we ran it three more times. Those two points never came back once. Over four runs the gap collapsed into ordinary run-to-run scatter, and the methodology that actually carries the brief showed up at both effort levels every time. One run is a story you tell yourself. You only see the pattern when you repeat it.

The judge's blind spot. To save tokens, we handed the graders a shortened version of the sources. They flagged "fabrications" — except the flagged details were sitting right there in the full sources, just missing from our short copy. The eval was manufacturing the very problem it was meant to catch. Fix: give the graders the whole source, not your abridged one.

The cost mirage. We first read cost straight off the dev CLI. It drags along its own cached system prompt, which pushed the reported number up 3–4× and even flipped which model looked cheaper. You have to recompute against list prices for whatever you'll actually deploy on. The figure your dev tool prints is not that figure.

1. 01Freeze everything except the one variable you are testing.
2. 02Import the real production prompt — never retype a clean copy.
3. 03Anonymize candidates; grade across model families, not just your own.
4. 04Replicate. One good run on its own proves nothing.
5. 05Give judges the complete evidence, not a convenient summary.
6. 06Measure cost on the real deployment shape, not the dev CLI.