Terminal-Bench 4.0 Explained: What Changed From 2.1 and 3.0, and Why Scores Do Not Carry Over

Terminal-Bench 4.0 is a maintenance-style release of the terminal agent benchmark: 8 tasks removed, 19 fixed, resources recalibrated and a flat 8-hour agent timeout. Top scores moved from the mid-30s on 3.0 to the 60s, with Anthropic reporting 66.4% for Claude Opus 5.5 at xhigh effort and Artificial Analysis measuring 59.6% independently. Those numbers are not comparable with 2.1 or 3.0 results, and the release pages themselves do not declare any earlier version retired.

What is Terminal-Bench 4.0?

It is the current numbered release of Terminal-Bench, run through the Harbor framework with the command harbor run -d terminal-bench/terminal-bench@4.0.0, per the 4.0 release page. Snorkel AI, Artificial Analysis and BenchLM each describe 66 tasks. The release page does not print the total, but it matches the 74 tasks in 3.0 minus the 8 removed.

Snorkel lists the release as "Aug 2026"; the release page carries no date. The page credits grants from OpenAI, Anthropic, Z.ai, SpaceX AI and the Laude Institute for leaderboard experiments. Keep that in mind for the conflicts-of-interest question below.

What changed from 2.1 to 3.0 to 4.0?

Each version changed the task set or the run conditions, so each needs its own column. The maintainers' own pages give the following.

The 8 removals in 4.0 break down as 2 for saturation, 2 for refusals, 2 for public solutions and 2 for unresolved quality or platform-compatibility issues. The page defines a task as saturated when all classes within all families of the latest generation of models solve it 5/5 times. Snorkel counts 20 revised tasks rather than the 19 on the release page, a small conflict that is unexplained.

Why is it 4.0 and not 3.1?

The maintainers say Terminal-Bench is now a continuous benchmark with semantic versioning, and that the resource changes and task removals are "breaking changes that require re-running trials". The scheme on the 4.0 page works like this:

• Incrementing the first number means rerun: new tasks, prompt fixes, resource adjustments.
• Incrementing the second means regrade: verifier changes.
• Incrementing the third means reuse: metadata changes.

The page says 4.1 will bring verifier improvements such as tamper-resistant verifiers, and 5.0 will add new tasks. So a score is only meaningful with its full version string, for example 4.0.0.

Is Terminal-Bench 2.1 actually dead?

Not according to the maintainers' pages. None of the 2.1, 3.0 or 4.0 release pages says an earlier version is deprecated. The accurate framing is that 2.1 is saturated for frontier models and has been superseded as a ceiling test.

The 3.0 page shows why. Using lab-and-maintainer-run numbers from tbench.ai, Fable 5 scored 83.8% on 2.1 and 33.8% on 3.0, and Opus 4.8 scored 78.9% and 21.1%. The 4.9 point gap between those two models on 2.1 widened to 12.7 points on 3.0. A benchmark that bunches frontier models within five points has little left to discriminate.

How do scores compare across 2.1, 3.0 and 4.0?

Treat the table below as an illustration of the drop, not a clean longitudinal series. The 2.1 and 3.0 columns come from the tbench.ai 3.0 page; the 4.0 column comes from Snorkel's leaderboard. Sources, dates and some harnesses differ, and the 4.0 task set is not the 3.0 task set.

Per BenchLM, the breaking changes "required fresh trials, so 4.0 scores are not directly comparable with 3.0 or 2.x". Several models land close to their 3.0 numbers, which suggests the 4.0 task set is of similar difficulty to 3.0. That is an inference from these rows, not a claim any source makes.

What is the 4.0 leaderboard, and who measured it?

The table captures sources as of 12 October 2026 unless noted. Snorkel and Artificial Analysis give no capture date on the page; BenchLM labels its snapshot 7 October 2026.

All rows are 4.0 (the Snorkel run command specifies terminal-bench@4.0.0). Snorkel's page says its effort column is "the harness's own reasoning or agent effort setting", not a Terminal-Bench parameter.

Why do lab-reported and independent scores disagree?

Mostly because the harness and the settings differ. Artificial Analysis runs all 66 tasks with mini-swe-agent and reports pass@1 averaged over three repeats. Snorkel and BenchLM list Claude Code and Codex rows, and BenchLM states 5 trials per task. The leader flips between sources: Sonnet 5.5 tops Artificial Analysis, Opus 5.5 tops Snorkel.

The 66.4% from Anthropic comes with caveats it states itself. It is the xhigh result, which the page calls the model's highest score. Anthropic ran with production safeguards on, and says that when they intervened, fallback models (Opus 4.8 for cybersecurity, Opus 5 for biology and frontier LLM development tasks) completed the task, which it says "likely reduces" Opus 5.5's performance on these benchmarks. For calibration, Anthropic reports its own setup reproduced the public leaderboard's Opus 5 figure (52.3% against 51.8%) "within noise".

The gap from 66.4% to 59.6% is 6.8 points. Anthropic's infrastructure noise study of Terminal-Bench 2.0 found a 6 point spread between the most and least resourced setups (p < 0.01) and advised skepticism toward leaderboard differences under 3 points until configurations are documented and matched. That is a different version of the benchmark, so it supports the direction of the argument, not an explanation of this specific gap.

Reported numbers across labs also disagree on third-party models. DeepSeek's model card lists GPT-5.6 Sol at 39.9 on Terminal-Bench 4.0, while Anthropic and Snorkel both list 37.3%. The causes are not documented.

Who runs Terminal-Bench 4.0, and are there conflicts of interest?

Snorkel describes the benchmark as hosted by Harbor and the Laude Institute and supported by Snorkel AI. The 4.0 page says leaderboard experiments are funded by grants from OpenAI, Anthropic, Z.ai, SpaceX AI and the Laude Institute. The labs whose models top the board are therefore also funders of the leaderboard runs. The sources do not allege any influence on results, and the verifier and tasks are open. But a reader should know the funding structure when weighing it.

Resource calibration follows the method in Anthropic's noise post, which the maintainers cite. The page gives no CPU or memory numbers.

What are the weaknesses?

Several are visible from the sources alone.

• Harness confound. Each leaderboard row is a model plus an agent. Claude models run in Claude Code and OpenAI models in Codex on Snorkel's board, so vendor harness and model effects are entangled.
• Token spend varies widely. The 4.0 page says Sonnet 5 used 21.6B tokens on its leaderboard run against 6.5B for Opus 5, hit timeouts and output-token-exceeded errors, and showed large variance in execution time. Its 12.4% may partly reflect configuration, such as the 128k max-output-tokens setting that was not enabled.
• Remaining errors are not all capability. The page says leftover errors are largely model refusals and output-token limits.
• Unclear statistics. Snorkel does not state the confidence level of its margins or the trial count. Artificial Analysis shows no intervals in the text of its page.
• Saturation risk is already visible. The 4.0 definition removes tasks every current-generation model solves, which is a ratchet: each release removes the easy end, so scores can fall for reasons unrelated to model quality.
• Verifier hardening is pending. The page lists tamper-resistant verifiers as a 4.1 item, which implies current verifiers are not yet hardened.

Should you trust it, and for what?

Use it as a relative signal for terminal agent capability within one version, harness and resource configuration. Do not use it to track progress across versions. My assessment, scored 1 to 5 from the sources above:

What would a better version look like, and what's still unclear?

Every reported score would carry the version string, harness, effort, trial count, resource multiplier and run date. The maintainers' own roadmap points the same way, with verifier hardening in 4.1. Leaderboards would also flag who ran each row, which Snorkel's page currently does not.

Still unclear:

• Official release date. The 4.0 page carries none; Snorkel says August 2026.
• Total task count. The page does not print 66, and Snorkel counts 20 revised tasks against the page's 19.
• Whether labs have standardised on 4.0. Anthropic and DeepSeek report 4.0 figures, but no source documents what every lab reports, and the pages do not retire 2.1.
• Anthropic's harness and trial count for the 66.4% figure are not stated on its launch page.
• Why third-party model scores differ between a model card and the leaderboard, for example GPT-5.6 Sol at 39.9 against 37.3.
• The weakest point of the argument above: the cross-version table mixes sources, dates and some harnesses, so it shows direction only.

Disclosure: Model Report Card is affiliated with Humanbit, a data labelling and evaluation company.