The number everyone quotes is measuring the wrong thing

Someone asked me a fair question this week: by how much does a well-built harness actually improve a coding agent? Give me a number.

So I went and got the numbers.

On the public leaderboards you can occasionally find the same model running under two different harnesses. The gap is three to eight percentage points. Meanwhile, keep the harness fixed and swap the model underneath, and the score moves twenty-five.

Read only that, and the conclusion writes itself: the harness barely matters, buy a better model, go home.

Except that is not what any of us are actually worried about

Those benchmarks measure one task, in one container, that either passes or fails. Nobody in production is losing sleep over one task in one container.

What people lose sleep over is a system that falls over every few days. A result that cannot be reproduced next Tuesday. A verdict nobody can audit after the fact. A run that reports success while quietly having invented the file it claims to have edited.

For that — for stability, reproducibility, auditability — there is no public benchmark at all. The closest attempt I could find is three weeks old, and its own preregistered audit came back inconclusive.

Where the numbers do exist, they are not small. One study cut regressions by 70% by handing the agent a single context file. Another measured a 9.5% drop in failed tasks from adding a critic that checks the plan before execution. Both of those are harness changes. Neither shows up on a leaderboard.

Which means for now the burden is on you. Measure the thing you actually care about — crash frequency, rework rate, how often a verdict survives review — because nobody is going to publish it for you.