Claude Opus 5.5 on Terminal-Bench 4.0: Anthropic Says 66.4%, Independent Test Says 59.6%

Anthropic reports 66.4% for Claude Opus 5.5 on Terminal-Bench 4.0, run at xhigh effort. Artificial Analysis, running the same benchmark with its own harness, lists 59.6% as of 12 October 2026. That is a 6.8 point gap between a lab-reported and an independent number for the same model and benchmark version.

The gap is real, but it is not evidence of misreporting. The two figures come from different run protocols, and neither is wrong on its own terms. This post lays out what each side did, where the numbers can and cannot be compared, and what a careful reader should take from them.

What does Terminal-Bench 4.0 measure?

It measures whether an agent can finish complex terminal work. Artificial Analysis describes it as a 66-task benchmark covering software, machine learning, science, operations, security, hardware and media. Each task has its own tests, and a task passes only if every test passes.

Version 4.0 removed 8 tasks (2 saturated, 2 refusal-prone, 2 with public solutions, 2 with unresolved quality or platform issues) and fixed 19. Every task has a flat 8-hour agent timeout. Version 4.0 is a different task set from 2.x and 3.0, so scores from earlier versions are not interchangeable with it.

How is it scored?

The metric is resolution rate: the share of tasks an agent resolves. Aggregation depends on who runs it. Anthropic states that its public-leaderboard runs use 5 trials per task with the Claude Code harness. Artificial Analysis reports pass@1 averaged over three repeats per task.

There is no judge model. Grading is deterministic test suites, which removes one common source of noise. It leaves harness, repeat count and agent scaffolding as the main free variables.

Who runs it, and are there conflicts of interest?

Terminal-Bench is hosted by Stanford, Harbor and the Laude Institute. The 4.0 announcement says leaderboard experiments are supported by grants from OpenAI, Anthropic, Z.ai, SpaceX AI and the Laude Institute. So the labs being ranked also fund part of the work.

Artificial Analysis says all of its evaluations are conducted independently. Terminal-Bench 4.0 is one of ten components of its Intelligence Index, so a result on this test also feeds a composite that vendors then cite.

What's the current leaderboard?

Two sources, two protocols, one table. Treat rows from different sources as separate measurements, not a single ranking.

The Anthropic figures for GPT-6 Astra are what OpenAI reported, at high effort, reproduced in Anthropic's table. Anthropic's page describes these as each model's highest score.

Where do lab-reported and independent scores disagree, and why?

They disagree by 6.8 points for Opus 5.5. The usual suspect, effort setting, does not explain it here: Artificial Analysis lists its Max and Xhigh variants at the same 59.6%, so moving to xhigh did not lift the score on its harness.

What differs is the run protocol. Artificial Analysis uses mini-swe-agent, a minimal harness, with three repeats. Anthropic's footnote ties its public-leaderboard numbers to the Claude Code harness and five trials. NYU Shanghai's technology office, summarising the launch, says the two figures come from different test harnesses and should not be compared directly.

Anthropic's own stated uncertainty is ±2.6 points on the Opus 5.5 score. The 6.8 point gap is roughly 2.6 times that figure. The reported errors cover sampling noise inside one protocol, not the difference between protocols, so this arithmetic does not settle which number is closer to the model's capability.

There is also a direction-of-bias question in the other way. Anthropic says Opus 5.5 was evaluated with production safeguards enabled. When safeguards intervened, older Claude models completed the affected cybersecurity and biology-adjacent tasks, which the company says likely reduces Opus 5.5's performance on these benchmarks. Artificial Analysis lists its variants as "Default Fallback", but the page does not define the term, so whether its runs involve the same routing is unknown.

What are the weaknesses?

• Harness sensitivity. The same weights can score differently under different scaffolds. This gap is a live example, though the cause is not isolated.
• Sample size. At 66 tasks, one task is about 1.5 percentage points. Anthropic's ±2.6 standard error shows how much run-to-run variance matters.
• Version drift. Task changes between 3.0 and 4.0 mean cross-version comparisons are not like-for-like. Never put 3.0 or 2.x scores in the same column as 4.0.
• Funding overlap. Labs ranked on the leaderboard also fund its experiments.
• Effort confounds. Lab figures are often each model's best setting. Anthropic's page says GPT-6 Astra's number is at high effort and Opus 5.5's at xhigh, so the comparison is best-versus-best, not equal-compute.
• Contamination. Version 4.0 removed two tasks for public solutions, which shows the issue is live. Neither source gives a contamination estimate for the remaining 66.

Should you trust it, and for what?

Trust it as a relative signal within one protocol, not as an absolute capability number. Scores below are my judgement from the published methodology, on a 1 to 5 scale.

Use the independent 59.6% to compare Opus 5.5 with other models Artificial Analysis has run under the same harness. Use Anthropic's 66.4% as an upper-end figure for a tuned setup. Anthropic itself notes that benchmark margins have become a less reliable guide to real-world differences, which is a fair reason to run your own tasks.

What would a better version look like?

A better report would publish both numbers side by side from the same run: one under a fixed neutral harness and one under the vendor's preferred scaffold, each with a confidence interval over the repeats. It would state the harness for every row, not only in footnotes.

It would also report cost and tokens per resolved task. The 4.0 post cites token counts for some leaderboard runs, so the data exists. Pairing resolution rate with cost would show whether a higher score is bought with more compute.

What's still unclear?

• Which harness produced 66.4%. Anthropic's footnote describes the public leaderboard's Claude Code harness and says its setup reproduced Opus 5's published score within noise. The page does not say outright which scaffold produced the Opus 5.5 figure.
• What "Default Fallback" means in Artificial Analysis's model labels, and whether it matches Anthropic's safeguard routing.
• Ranking at the top. Artificial Analysis's page summary names Claude Sonnet 5.5 (Max, Default Fallback) as highest at 63.6%, ahead of Opus 5.5 at 59.6%. NYU Shanghai's summary, dated 23 September 2026, says Opus 5.5 ties GPT-6 Astra at 59.6%. Leaderboard snapshots may differ by date.
• Whether the gap signals misreporting. Nothing here shows Anthropic misreported. The numbers answer different questions, and the gap is within what a harness change could plausibly produce, though no controlled test in the sources above proves that.

Opus 5.5 is priced at $4 per million input tokens and $20 per million output tokens as of 12 October 2026, against $5 and $25 for Opus 5. On Artificial Analysis's Intelligence Index v4.3.2, it scores 58, first of 227 models on the page.