Your AI Coding Agent Benchmark Is Wrong If You Ignore the Harness

A coding agent score names a model but measures a model plus a harness. In a 100-task SWE-bench Verified subset, one May 2026 position paper found harness-induced variance of 18.48 pp² against 2.37 pp² for model-induced variance, a ratio of 7.80x. A September 2026 study found no average gap between vendor-native and neutral harnesses. Both results are reasons to stop reading single-number leaderboards as model rankings.

What is a harness, and why does it change coding benchmark scores?

The harness is everything between the model and the task: prompts, tools, the agent loop, context handling and verification. Zhang et al. define it as the layer governing "context construction, tool interaction, orchestration, and verification."

Change any of those and the same weights solve a different number of tasks. A harness that retries failed edits, trims a long context or runs the tests before declaring success is doing work the leaderboard credits to the model.

How big is the harness effect, measured directly?

In the one controlled grid in the position paper, the effect was large: 8.5 to 13.0 points per model. The authors ran three models on a 100-task, difficulty-stratified subset of SWE-bench Verified, with two runs per cell and a 50-step budget. They varied only the harness, from a minimal H1 to a full H3 with self-checks, drift checks and rollback.

All figures are independent measurements by the paper's authors, published 7 May 2026, on SWE-bench Verified (a 100-task subset, not the full set). Within a fixed harness, swapping the model moved scores by only 3.0, 2.5 and 5.0 points across H1, H2 and H3. Six of nine model-pair and harness-pair comparisons reversed the model ranking.

The authors say they do not claim 7.80x is universal. With 100 tasks, two runs and three models, it is a single small sample. Also note that H1 is a deliberately bare baseline built by the authors, so the gain partly reflects how weak H1 was.

What does public leaderboard data show?

The same paper collects public comparisons, and they point the same way, but they are observational. The paper itself notes that harnesses differ on many dimensions at once, so these gaps cannot say which component caused them.

According to Zhang et al., these are second-hand numbers and use different benchmark versions, so do not put them in one column of a ranking. The pattern is the point: double-digit swings from scaffold alone appear on every benchmark checked.

Does a vendor's own harness beat a neutral one?

On average, not in the one paired study we found. Arjmandi (8 September 2026) ran 80 tasks from a private, contamination-controlled 256-task suite under each vendor's harness and under the neutral deepagents harness.

These are independent measurements on a private suite, so nobody else can rerun them. Neither contrast resolves an advantage, though the intervals are wide enough (about ±10 points per cell) that the study does not prove the harnesses are equal.

The Opus result hides a split. The native harness trailed by 9.0 points on 61 repository tasks (37.7% vs 46.7%) and led by 23.7 points on 19 contest tasks (84.2% vs 60.5%). The author states the partition was chosen after seeing the data and needs a designed replication, so treat it as a hypothesis.

Does the harness change cost, not just accuracy?

Yes, and the cost view can reverse the accuracy view. Arjmandi re-priced runs from raw per-turn usage at frozen list prices and found the neutral harness cost 1.3 to 1.6 times as much per solved task on Opus 4.8, and 1.2 times as much on GPT-5.5.

The author flags a caveat: on the Anthropic account, 58 runs left no usage record, and allocating that spend either way would move the Opus ratio between 0.7 and 2.3. The billed ordering is unresolved, and an earlier manuscript had wrong cost figures because of a telemetry bug in the author's own tooling.

A second finding bears on any timed benchmark: 22 of 81 runs cancelled at the 1,200 second wall-clock ceiling had already produced a passing patch. A harness with a tighter timeout can score lower without the model being worse.

Why do some benchmarks fix the harness?

Fixing the harness removes one variable. The Terminal-Bench paper (submitted 17 January 2026) built Terminus 2 as a neutral testbed, a scaffold with a single headless-terminal tool. Terminal-Bench 2.0 has 89 tasks, and the paper evaluated six harnesses: Claude Code, Codex CLI, Gemini CLI, OpenHands, Mini-SWE-Agent and Terminus 2.

Independent evaluators follow the same logic. Artificial Analysis says it runs Terminal-Bench 2.1 "with the Terminus 2 agent harness in an e2b sandbox and report[s] pass@1 averaged over 3 repeats per task." As captured on 13 October 2026, its page lists Claude Fable 5.1 (Max, Default Fallback) at 91.4%, Xhigh at 91.0% and High at 89.9%, all independent measurements on Terminal-Bench 2.1.

Do not compare those numbers with 2.0 results from the paper. They are different benchmark versions. A lab-reported number from a tuned production harness and an independent Terminus 2 number answer different questions.

Is the model or the harness more important?

The sources disagree. The Terminal-Bench authors write that "model selection is usually more important than agent scaffold when optimizing for performance." The same paper reports Gemini 2.5 Pro gaining 17% when paired with Terminus 2 instead of OpenHands, and its Figure 1 uses the scaffold chosen to maximize each model's score, so scaffold choice is already baked into the headline.

The position paper claims the opposite for long-horizon tasks across models of comparable frontier capability. Arjmandi's result sits in between: swapping to a neutral harness changed little on average, while one stratum moved by more than 20 points.

A fair reading: when models differ a lot in capability, model choice dominates. When frontier models cluster within a few points, the harness can decide the ordering. These are different benchmarks, model sets and months, so the disagreement is not settled.

How should you report or read a coding agent score?

Treat the harness as part of the score and report it. Zhang et al. propose a seven-layer disclosure called ETCSOVG: execution, tool, context, scheduling, observability, verification and governance. In practice, a score is only comparable if these are stated:

1. Harness name and version, and whether it is vendor-native or neutral.
2. Benchmark name and exact version (Terminal-Bench 2.0 and 2.1 are not interchangeable).
3. Tool list, step budget and wall-clock timeout.
4. Sandbox and number of repeats, with pass@1 defined.
5. Whether the number is lab-reported or independently measured, and its capture date.

For your own comparisons, the paper suggests a 2×2 grid of two models and two harnesses with task order, environment and grading held fixed. If the ranking flips across harnesses, the leaderboard order was not a model property.

What's still unclear?

Three things limit every claim above. First, the cleanest evidence is small: 100 tasks and two runs in the position paper, 80 tasks with wide intervals in the private-suite study, whose tasks are not public.

Second, the public gaps are confounded. A scaffold swap changes prompts, tools and parsing together, so no one number says how much each component matters.

Third, the two direct experiments point in different directions on whether harness choice matters on average, and the Terminal-Bench authors weigh the model more heavily. We cannot say which finding generalizes to current models. A larger designed replication with public tasks would resolve more than another leaderboard row.