The checker that can't read a verdict
Our report has a last line of defense: a step that re-reads the finished draft against the raw data and flags where the words and the numbers disagree. We swapped the cheapest model into it and planted six known errors. It caught two — and never once caught a result described by the wrong verdict.
Post A ended on a promise: reasoning effort earns its cost only where the thinking is the product — a fact-check, a verdict, a proof. Our report pipeline has exactly such a step. Before a research report is sealed, a checker re-reads the prose against the structured results and flags contradictions: a number that doesn't match the run, a hypothesis the text calls "supported" that the record says wasn't. It had been running on the cheapest model. So we did to it what the flagship post says to do — held everything fixed and varied one thing, the model — and measured whether it still catches what it's there to catch.
1 · A checker is a recall problem — so plant the errors
gold set · 6 planted
You can't grade a fact-checker on a clean report; a clean report has nothing to find. So we took a real, internally-consistent report and planted six errors we chose ourselves — three numeric (a stated figure changed to disagree with the run data) and three verdict (a hypothesis's outcome reworded to the wrong conclusion). Now the answer key is known, and every model gets scored the same way: of six planted errors, how many does it catch? No taste, no rubric — just recall against a gold set.
2 · It reads the number, not the verdict
0 / 5 on two of three
The pattern under the bar is sharper than the bar. Haiku catches a wrong number some of the time — a mortality rate off by ten points, an odds ratio moved. But on two of the three planted verdicts — a hypothesis the text calls "supported" that the record marks not-supported — it scored zero out of five. It cannot tell that the prose is claiming the opposite of the result. And this is the one miss you can't see downstream: when the checker finds nothing, it returns an empty list — exactly what it also returns when it crashes or simply misses. A silent miss and a healthy pass are the same shape. That's the failure we care about most, and the cheap model produces it quietly.
3 · Nobody cried wolf — and the pedants were right
the false positives were real
The flip side of recall is noise: does a model invent problems on a clean report? Here the surprise ran the other way. On the un-planted report the two smaller models each flagged one thing — a risk difference written as −10.3% where 31.2 − 21.0 = 10.2. That isn’t a hallucination; it's a real rounding slip that had been sitting in the report the whole time, that opus and our own shipping checker both waved through. So the honest scoreboard is: no model cries wolf, and the "false positives" were the small models being more careful, not less. (It also means our clean baseline wasn't perfectly clean. We left the note in.)
Match the model to the step
and gold-set your checkers
Post A said turn the effort dial up where the thinking is the product. This is the same rule one level up: on a step whose entire job is to catch a contradiction, the model is the recall, and the cheapest one couldn't read a verdict. We moved this checker off it. The broader lesson is the cheap one to act on: any step you rely on to catch something — a checker, a guard, a filter — deserves a gold set of planted failures before you trust it, because the one thing it can't tell you on its own is what it quietly let through.
Bench note · honest edges
One report, one step, six planted errors; haiku/sonnet n = 5, opus n = 4. Matching is by signature token, not blind human review. Costs are CLI-measured and run high — the relative shape is what holds. Directional field notes, not a study.