Then I asked it a question it could not answer: what did this cost?

It knew the artifact was good enough. It had no idea whether reaching that verdict took forty minutes or four hours, twelve dollars or two hundred. It could not tell a clean first-pass approval from a run that limped over the line on its tenth attempt at the same unresolved issue.

Those two runs are not the same result. Only one of them is a working system.

The cap that bounds the wrong thing

I had iteration caps — ten loops per phase, maximum. That felt like control. It is not. An iteration cap bounds how many times a loop runs, not what each turn costs, and in my pipeline one review round fans out to eight verifiers running in parallel. That single step can cost more than every phase before it combined, and no cap anywhere notices.

So I shipped a ledger. A budget declared before work starts. Checked when crossing between phases and on every loop back, never mid-work. Cross an edge already over the ceiling and the run stops with an explicit verdict instead of quietly finishing and presenting you with a bill.

The part I underestimated is the route analysis. It measures the spend that did not move the run forward — everything consumed by attempts that ended in a loop back — and flags reasons that repeat. That last number is the uncomfortable one. A run that reached a pass on the tenth identical retry is a failing run wearing a passing badge, and nothing in my old reports would have shown it.

Why this is worth more than it looks

There is a study from March that made me rethink how I add anything to an agent at all.

Researchers gave a coding agent one extra thing: a map of which tests cover which code, handed over as a plain text file the agent could read. Regressions — tests that used to pass and stopped — fell from 6.08% to 1.82%. A 70% drop from one file.

The same study ran a second branch. Instead of the map, they gave the agent procedural instructions: work test-first, follow this discipline. Regressions rose to 9.94%. Worse than doing nothing at all.

Which is also why the budget lives in the state file rather than in a prompt asking the agent to be careful with money. An agent cannot be careful with a number it cannot see.

So I went and got the numbers.

On the public leaderboards you can occasionally find the same model running under two different harnesses. The gap is three to eight percentage points. Meanwhile, keep the harness fixed and swap the model underneath, and the score moves twenty-five.

Read only that, and the conclusion writes itself: the harness barely matters, buy a better model, go home.

Except that is not what any of us are actually worried about

Those benchmarks measure one task, in one container, that either passes or fails. Nobody in production is losing sleep over one task in one container.

What people lose sleep over is a system that falls over every few days. A result that cannot be reproduced next Tuesday. A verdict nobody can audit after the fact. A run that reports success while quietly having invented the file it claims to have edited.

For that — for stability, reproducibility, auditability — there is no public benchmark at all. The closest attempt I could find is three weeks old, and its own preregistered audit came back inconclusive.

Where the numbers do exist, they are not small. One study cut regressions by 70% by handing the agent a single context file. Another measured a 9.5% drop in failed tasks from adding a critic that checks the plan before execution. Both of those are harness changes. Neither shows up on a leaderboard.

Which means for now the burden is on you. Measure the thing you actually care about — crash frequency, rework rate, how often a verdict survives review — because nobody is going to publish it for you.

My own site spent three weeks serving a version two commits behind, and I had no idea.

Here is what happened. On 6 August I pushed two changes to a subpage. Three deploys ran. All three failed. Each one recorded a build duration of zero milliseconds and the message "Page build failed." — no detail, no notification, nothing in my inbox.

Zero milliseconds means the build never even started, so it was not my code. Something upstream fell over.

And the site? The site was fine. HTTP 200, valid certificate, everything loading. Because the last successful build was still being served, exactly as it had been the day before. The failure had no visible surface at all. From the outside, from my own browser, from anyone else's — a healthy website.

I only found it because I was poking at something unrelated and checked the deploy status out of curiosity.

The uncomfortable parallel

This is the same failure mode I build gates against all day, arriving from a direction I did not defend.

An agent reports success. The tests pass. The diff looks clean. Everything on the surface says the work is done, and the only way to know otherwise is to check something the reporting party does not control — run the actual application, query the actual state, compare against something the optimiser cannot touch.

I had that instinct about agents. I did not have it about my own deployment pipeline, which had been quietly serving stale content for twenty-one days.

The fix, incidentally, was deleting work rather than adding it. The site is plain HTML, but every deploy was running it through a template engine it never needed — 165 seconds to publish half a megabyte. One empty file later, deploys are a straight file copy and take 19 seconds. Nine tenths of the time was spent on a step whose only real contribution was being able to fail.