AI Benchmark Saturation 2026: The Tests Frontier Models Have Already Maxed Out

Stanford's 2026 AI Index says evaluations meant to stay hard for years are now saturated in months. The sharper picture is uneven. On SWE-bench Verified the top five coding models sit 1.3 points apart in one preprint's data, while on Humanity's Last Exam (HLE) the top model leads the field by more than eight points on Scale's leaderboard as of 13 October 2026.

Saturation is not one thing. A benchmark can be maxed out by score, by lost discrimination, by contamination or by bad labels. This post sorts the 2026 evidence by which of those is actually happening.

What does it mean for a benchmark to be saturated?

A benchmark is saturated when it can no longer rank the models you care about. Krakauer's April 2026 preprint formalizes this as discriminability: the standard deviation of scores among contemporaneous models. When it approaches zero, the benchmark is saturated. The paper also shades scores above 90% as the region where discrimination is lost.

That definition matters because it separates three failure modes that get lumped together:

• Ceiling: scores approach 100%, so there is no room left.
• Compression: the leaders bunch together, so differences fall inside noise.
• Invalidity: contamination or wrong labels mean the score no longer measures what it claims.

The same preprint fits curves to the best score over time. For MMLU it finds a logistic fit with an asymptote of 94%, and for GPQA Diamond a logistic fit with an asymptote of 92% (n=7 and n=8 frontier points, data to December 2025). The author calls the conclusions provisional because of the small sample. HumanEval's fit has an asymptote of 104%, which the paper reads as no saturation yet.

Which benchmarks are saturated, and by what test?

The table below sorts benchmarks by the kind of saturation the evidence supports. Every row is limited to what the cited source measured, and nothing is mixed across benchmark versions.

The 1.3, 9.1 and 26.4 point spreads come from Amin's May 2026 preprint, which used 34 models from 10 labs released between June 2024 and March 2026. Amin ran no evaluations and used scores from public sources, which the paper itself says are mostly self-reported. It is a single-author, non-peer-reviewed paper, so read the spreads as a signal about direction and not as a current leaderboard.

Is SWE-bench Verified still a useful benchmark?

Its value for ranking frontier models is shrinking, and the contamination evidence is the stronger half of the story. Chen et al. (August 2026) had experts probe models with only partial issue text. For each model evaluated, more than 65% of instances showed clear leakage evidence, and more than 18% could be recalled at the patch or test level.

They then changed the repository without changing behavior: renaming identifiers, reordering definitions, rewriting code around the fix. Pass@1 on SWE-bench Verified fell as shown below.

All three numbers are from the authors' own runs, with the mini-swe-agent scaffold at temperature 0. The largest single drop came from the identifier renaming step (6.0 to 7.4 points). On a held-out set of 110 newer SWE-rebench tasks, the transformations left Pass@1 unchanged for GPT-5.4-mini, which the authors read as evidence the drop is not just harder tasks.

If you read a coding score, the harness matters as much as the model. The site's Terminal-Bench 4.0 explainer and the Opus 5.5 Terminal-Bench gap post cover the same harness sensitivity on a different test.

Is Humanity's Last Exam saturated?

No, and it is the clearest counterexample in the 2026 data. Scale's HLE leaderboard, with an update dated 17 September 2026 and read on 13 October 2026, has GPT 6 Astra on top at 54.80 ± 1.94. Below it the field is tight.

Scale runs this leaderboard, so these are independent of the labs but not of the benchmark's operator. Ranks two to four differ by less than their confidence intervals, so they are statistically indistinguishable on this sample. The Stanford AI Index says frontier models gained 30 percentage points on HLE in a single year, which is fast progress on a test that is still far from its ceiling.

Scale also notes that its rolling version exists as a migration path once models begin to hit the noise ceiling on the original dataset. The page does not give a number for that ceiling.

Why did ARC-AGI-1 and ARC-AGI-2 lose their value?

According to The Decoder, both are effectively saturated, largely because of scaffolding approaches. That is a saturation-by-engineering story: the benchmark was solved by the system around the model, not only by the model.

ARC-AGI-3 is an interactive reasoning benchmark. At its March 2026 launch, per The Decoder's account of the technical paper, the scores were:

The metric is Relative Human Action Efficiency. Per-level efficiency is the human action count divided by the AI action count, squared, and capped at 1.0, so scores cannot be compared with ARC-AGI-1 or 2. The harness gap is large: a Duke test cited in the article saw Opus 4.6 reach 97.1% on a known environment with a hand-crafted harness and 0% on an unfamiliar one. The site's ARC-AGI-3 harness gap post goes further on that.

Are high self-reported scores inflated?

Not on the evidence Epoch published, though that evidence is older than the current frontier. In a September 2025 analysis, Epoch compared developers' GPQA Diamond figures with its own runs and found them within its confidence intervals. Grok 4 self-reported 87.5 against Epoch's 87.0, and GPT-5 self-reported 85.7 against 84.5.

Epoch's standard errors were about 2 to 3 points. That bounds what any single GPQA score can tell you: a gap of one or two points between models is inside the noise even before contamination is considered. The analysis covered only GPQA Diamond and models released before September 2025.

What replaces a saturated benchmark?

Evaluators are using refreshed versions, harder sets and composites. Epoch says in its 2025 impact report that as individual benchmarks saturate, it has become harder to compare frontier models with any single score. Its Epoch Capabilities Index combines at least four scores per model, from over three dozen benchmarks.

As of 13 October 2026, Epoch's hub lists Claude Opus 5.5 on top of the ECI at 167, narrowly ahead of GPT-6 Astra, across 409 models and 89 benchmarks. That page is live, so the numbers will change. Epoch's 2025 impact report says FrontierMath Tier 4 had only 17 of its 48 private questions solved as of January 2026.

Composites trade one problem for another. They are more stable, but a single index number hides which skill moved. The Stanford AI Index also reports invalid-question rates from 2% on MMLU Math to 42% on GSM8K, a reminder that a harder test only helps if its labels are right. The figure is the precision of invalid-question detection at 50 items across nine benchmarks, not a full audit of each.

What's still unclear?

Several points remain open, and the weakest link in the saturation argument is the data.

• Current top scores: the preprints use data cut off between December 2025 and March 2026. The spreads they report may be different for the models released since.
• Verified-set audit: the contamination probing by Chen et al. used smaller models and Python repositories only. Its behavioral test is indirect: the authors conclude that agents may partly rely on memorized repository cues, which is an inference and not proof of memorization.
• Noise ceiling for HLE: Scale refers to one but does not publish a number.
• Fit uncertainty: the MMLU and GPQA asymptotes rest on seven and eight points, and the preprint says its statistical conclusions are provisional.

Saturation claims are only as good as the leaderboards behind them, and most 2026 leaderboards mix lab-reported and independent numbers. Check which kind a score is before you cite it.