Market review · as of 27 August 2026
Who Checks the Agent
Seventeen harnesses reviewed against the same seven axes. What separates them turns out to be neither the model nor the subagent count, but one question: does anyone other than the author judge the result — and does that judge touch a running application. The last row of the table is an in-house pipeline, so it can be read against the market.
I assembled this on 27 August 2026 from vendor documentation, changelogs, source code and public leaderboards, rather than from commentary about them. The final row of the table is a pipeline I built myself, kept in so it sits under the same seven questions as everything else. Every claim points back to a primary source, so the numbers can be checked.
The verdict
First
Claude Code / Agent SDK
The only one where an independent judge is a product feature: after every turn, /goal sends the condition and the transcript to a separate model that rules met / not yet met / impossible. Plus a published, costed methodology and first place on the external Terminal-Bench 2.1 leaderboard at 83.8%.
Second
OpenAI Codex
Better plumbing: server-side compaction, enterprise-grade permissions and audit, cheaper models. Weaker on one thing that matters here — auto-review guards safety, not correctness. Claims 88.8% on the same benchmark, but that number does not appear on the public leaderboard.
Third
Cursor
Bets on parallelism and selection rather than the loop: /best-of-n runs several models at one problem, each in its own worktree. Its reviewer, Bugbot, sits on pull requests rather than in the loop — and does not run tests. No published benchmarks at all.
In fairness: the winner is the vendor whose model compiled this review. That is why the case rests on externally checkable things — a public leaderboard and product documentation — rather than on impressions. If OpenAI's 88.8% were confirmed on an independent leaderboard at a comparable token budget, the top two places would swap.
Seventeen harnesses, seven axes
Cell shorthand: N not confirmed in a primary source, ND no data. Everything else comes from documentation, changelogs, or source code.
| Harness | Who judges the result | Context | Touches a running app | Orchestration and limits | Definition of done | Since |
|---|---|---|---|---|---|---|
| AnthropicClaude Code | Subagents with their own context; /goal — a separate small model rules after every turn |
Auto-compaction plus a fresh context inside each subagent | Yes — Playwright MCP, in-app browser on desktop, computer use in the CLI | 20 subagents in parallel, nesting depth 3; workflows: 16 at once, up to 1000 per run; worktree isolation | A condition up to 4000 characters, verdict met / not yet / impossible | 07.2025 → 05.2026 |
| AnthropicManaged Agents | Writer plus an automatically provisioned grader in a separate window — it sees the rubric and the artifact, not the reasoning | The grader starts clean | Inspects the artifact, not the application | max_iterations defaults to 3, capped at 20 |
A markdown rubric, scored criterion by criterion | 04.2026 beta |
| OpenAICodex (app) | Orchestrator plus explorer / worker plus reviewer | Threads per project, forking, memories | Yes — built-in browser and computer use, clicks through a local dev server | Parallel threads, built-in git worktrees | Goal mode: objective plus constraints plus how it will be verified | 02.2026 → 05.2026 |
| OpenAICodex cloud / CLI | Auto-review is a safety gate, not a quality one | /compact, session resume, AGENTS.md |
Tests inside a container; no browser in the CLI | Sandbox per task; circuit breaker at 3 refusals in a row or 10 out of 50 | Diff plus whatever tests the user defined | GA 10.2025 |
| GoogleAntigravity | Manager plus Editor; artifacts for a human to review, no built-in critic | Knowledge Items, rules capped at 12,000 characters | Yes — a subagent drives Chrome, takes screenshots and video recordings | Nesting up to 10 levels, native worktrees, sandbox off by default | Procedural only — "request review" versus "always proceed" | 11.2025 → 05.2026 |
| GoogleJules | Planning Critic before execution and a Critic Agent after it | A fresh virtual machine per task — a hard reset | Yes — renders the frontend and returns a screenshot; a CI fixer reads logs and repairs its own PR | Free tier: 15 tasks per day, 3 in parallel | Nothing measurable; the only published number is a 9.5% drop in failed tasks | GA 08.2025 |
| GoogleGemini CLI | Plan mode separated from execution, four built-in subagents; no evaluator — a hook plays the part of the gate | /clear and /compress, auto-compaction at half the window, a hierarchy of context files |
Yes — a browser agent with Chrome DevTools MCP bundled in, but off by default; capped at 100 actions per task | Parallel subagents capped at 30 turns and 10 minutes each; worktrees, choice of sandbox | The AfterAgent hook — a "deny" verdict or exit code 2 forces another attempt with feedback |
06.2025 → 02.2026 |
| GoogleADK | A "generate and review" pattern built into the framework | Session state plus event compaction | Conformance tests against a recorded baseline; Playwright in the toolset | max_iterations defaults to no limit at all; exit by escalation |
12 metrics, thresholds of 1.0 for trajectory and 0.8 for response | 2.0 GA 05.2026 |
| CognitionDevin | A planning subagent, MultiDevin, and Devin Review with auto-fix | A 200k cap on a million-token window; machine snapshots | Yes — a full desktop, screenshots and video recordings of end-to-end tests | A micro-VM per session; one manager and up to 10 workers; iteration limits ND | Confidence scores: green, yellow, red | GA 12.2024 |
| CursorCursor 3 | Bugbot on pull requests; several agents judged against each other | Automatic summarisation, memories, rules | A native browser tool, but Bugbot does not run tests | /worktree, /best-of-n, subagents on separate machines; iteration limits ND |
Bugbot as a blocking status check on the PR | 10.2025 → 04.2026 |
| GitHubCopilot coding agent | A separate code review plus security scanning | AGENTS.md, compression only in the CLI |
Yes — Playwright MCP built in, screenshots in the PR description | Ephemeral environments on Actions, many sessions at once | Human approval is mandatory — the agent cannot merge its own PR | GA 09.2025 |
| Open sourceOpenHands | An explicit critic scoring 0 to 1 | A context condenser, with a hard reset as the fallback | Yes — browser control on by default | max_iteration_per_run = 500 |
Critic threshold of 0.6 | SDK 12.2025 |
| Open sourceSWE-agent | Repeated attempts plus a separate model choosing the best one | History processors, observation truncation | A linter gate on edits; Playwright in the multimodal variant | No step limit — the budget is purely monetary, 3 dollars per instance | A "submit" action plus a four-point self-check | 1.0 02.2025 |
| Open sourceAider | Architect plus editor — two models, but the judge is a human | A repository map, summarisation by a weaker model | No — never launches the application or a browser | Strictly sequential, no subagents, max_reflections = 3 |
Lint and tests exit zero, then commit | 09.2024 |
| Open sourceRalph loop | Nobody — a human judges at the end | A fresh context every iteration, state kept in plan files | Build and tests as backpressure; browser ND | A loop with no iteration limit | No stop condition — a green build and a tag | 07.2025 |
| Open sourceLangGraph | The framework verifies nothing; interrupts to a human | Checkpointers plus a summarisation layer | No — verification lives outside the framework | Fan-out, subgraphs, recursion limit of 10007 | No built-in success criterion | 1.0 10.2025 |
| In-houseInternal pipeline | A QA agent plus 8 parallel verifiers, a PROCEED / LOOP_BACK verdict and GO / NO-GO gates | A fresh agent every iteration, state kept in phase files | Yes — Playwright as the gate into the next stage, later an agent driving the browser over MCP | 8 verifiers at once, max_per_phase ≤ 10; no cost budget |
MUST / SHOULD / MAY criteria plus scored gates with numeric thresholds | 01–02.2026 |
Two of these products are being wound down: Gemini CLI is becoming Antigravity CLI (announced 19.05.2026), and Gemini Code Assist dropped individual accounts on 18.06.2026. Their rows stay, because the mechanisms they describe carried over to the successor.
What arrived when
Harness events only — roles, verification, orchestration, stop criteria. Model releases are left out.
2024-09-26AiderReasoning split from editing — architect plus editor, the first public division of roles
2024-11-04MicrosoftMagentic-One: an orchestrator with a progress ledger asking whether the request has been satisfied
2024-12-24CognitionDevin generally available
2025-04-17OpenHandsA critic model — the first explicit, numeric judge in open source
2025-06-30AnthropicHooks in Claude Code — external gates wired into the agent's cycle
2025-07-02GitHubThe agent gets its own browser and puts screenshots in the PR description
2025-07-14HuntleyThe Ralph loop — a fresh context every iteration, state kept in files
2025-07-24AnthropicSubagents with their own context window
2025-08-08GoogleJules: an interactive plan and a Critic Agent
2025-09-10ReplitAgent 3 tests its own application in a browser, 200 minutes of autonomy
2025-10-16AnthropicAgent Skills; declared an open standard that December
2025-11-18GoogleAntigravity — artifacts and a browser subagent that records video
2025-11-24In-houseAn
ai-sdlc repository is created at 13:12 UTC. Its manifest already carries delegate–review–own with ownership that never transfers, deny-by-default permissions with tools declared alongside their access limits, layered prompts, evaluations per agent, and decision logs kept in version control2025-11-26In-house"AI SDLC sync" — the roles, the handoff templates between them and the review model written down
2025-11-26Anthropic"Effective harnesses for long-running agents" — context resets and a feature list as state
2025-12-16OpenHandsAn SDK with the critic as a first-class citizen
2026-01-21In-houseInternal pipeline — roles, gates, evaluator separated from generator
2026-01-26GoogleJules: the Planning Critic, a 9.5% drop in failed tasks
2026-02-11OpenAI"Harness engineering" — repository knowledge as the system of record
2026-02-17In-houseThe pipeline runs a real production project; Playwright written in as the E2E gate
2026-02-24CognitionDevin 2.2: a full desktop and video recordings of end-to-end tests
2026-03-24Anthropic"Harness design for long-running application development" — planner, generator, evaluator
2026-04-30OpenAIAuto-review: a separate agent judges actions, with a circuit breaker after repeated refusals
2026-05-06AnthropicOutcomes — a grader scoring against a rubric in its own context window
2026-05-11Anthropic
/goal — a loop toward a condition, verified by a separate model2026-05-19GoogleAntigravity 2.0 with a CLI and SDK; ADK 2.0; Gemini CLI folded into Antigravity CLI
2026-05-21OpenAIGoal mode — an objective pursued "for hours or even days"
2026-07-04SimmonsGraph engineering named, with seven design practices
Corrections
Numbers that circulate in industry threads and do not match the primary source. Useful if someone throws one of them across a meeting.
OpenHands caps runs at 100 iterations100 → 250 (04.2025) → 500 (06.2025)
LangGraph: recursion limit of 2510007 in the code on main. The 25 comes from a different library, and LangGraph's own docs say 1000 — contradicting the code
SWE-agent has a step limitIt does not. The budget is monetary — 3 dollars per instance. The 250-step limit belongs to mini-swe-agent
The Ralph loop is
claude -p --dangerously-skip-permissionsThe original is a plain shell loop feeding a prompt file. The flag variant comes from a later field reportAider supports MCPIt does not — and it never launches an application or a browser
AutoGen is a live alternative to LangGraphIn maintenance mode since 10.2025; the successor is Microsoft Agent Framework
There is a SWE-agent 2.xThe 2.x numbering belongs to mini-swe-agent; SWE-agent has been on 1.1.0 since May 2025
Jules went generally available at I/O 2026It went GA a year earlier, in August 2025
Ralph has a published failure rateNo such figure exists in the primary sources — only qualitative claims
Antigravity 3.0Absent from the official changelog; the latest release is 2.11, dated 26.08.2026
What could not be established
Iteration limits for Devin, Cursor and the GitHub agent — undocumented in all three.
Compaction thresholds in Antigravity — the figures in circulation come only from community blogs.
A hard timeout for Codex in the cloud and in the CLI — not in the documentation.
A formal definition of done at GitHub — the only gate is human approval.
Role architecture in Replit Agent 3 and orchestration in Amazon Kiro — no public data.
SWE-bench Verified — the leaderboard renders client-side and could not be retrieved.
Assembled 27.08.2026 from vendor documentation, changelogs, source code and leaderboards — roughly 70 direct fetches, with no search engine involved.
A companion note compares the same in-house pipeline against the publications by OpenAI, Anthropic, Simmons and Perez.