ARC-AGI-3 Was "Beaten" at 99.9%. Change the Harness and It Drops to 62.7%

ARC Prize reports two best-observed ARC-AGI-3 Semi-Private scores for OpenAI's GPT-6 Astra: 62.7% at max reasoning for $26,098 under its Standard harness, and 99.9% at high reasoning for $18,817 under a Provider Adapter harness. Same model, same set, same evaluator, a 37.2 point gap. The only variable ARC Prize names is how the harness handles context between steps.

This post separates what each number measures, how the benchmark is scored, and what the gap does and does not show.

What does the 99.9% vs 62.7% gap actually measure?

It measures the effect of context management, not a change in the model. ARC Prize defines the Standard harness as one that "enables a model to carry forward notes it chooses to keep with it throughout the environment." The Provider Adapter harness "preserves opaque reasoning state between requests and uses compaction for longer conversations, allowing the model to reuse prior work."

In plain terms, under the Standard harness the model sees only what it chose to write down. Under the adapter it keeps its own hidden reasoning state across requests. ARC Prize says the Standard harness "provides all the information required to solve each game, but leaves the model responsible for deciding what to preserve in its visible notes."

The Next Web reports that the adapter uses documented API features any developer can call, so this is not a hidden trick. The issue is comparability, not legitimacy.

What are the full scores under each harness?

Every reasoning level scores far higher under the adapter. This is lab-independent: ARC Prize ran both, on the Semi-Private set, and published them on 3 September 2026 (captured 12 October 2026).

Source: ARC Prize blog table, independent evaluator, ARC-AGI-3 Semi-Private, single benchmark version for all rows. The results page lists the same runs to two decimals, for example 99.95% for adapter high.

Three details stand out:

• With reasoning set to none, the adapter scores 96.7%, which is 34.0 points above the best Standard result (62.7% at max).
• The headline rows are not the same effort level: the 62.7% headline is max, the 99.9% headline is high. Quoting "99.9% at max" is wrong, since adapter max is 98.6%.
• In the Standard column, none (35.2%) beats low (17.5%). ARC Prize does not comment on this.

How is ARC-AGI-3 scored?

By action efficiency against humans, not by pass or fail. The benchmark paper (ARC Prize Foundation, submitted 24 March 2026) calls the metric RHAE, Relative Human Action Efficiency. In v1 of the paper, a level score is min(1.0, h/a) squared, where h is the human baseline action count and a is the AI's. Only actions that change environment state count, so reasoning steps, tool calls and retries are free.

The paper's own example: a human baseline of 10 actions against 100 AI actions scores (10/100)² = 1%. Level scores are combined into an environment score with a linear weighting that favours later levels, and the total is the mean over environments. The paper lists 25 Public Demo, 55 Semi-Private and 55 Fully Private environments.

Two caveats. This describes v1; a v2 revision followed on 17 April 2026, and the formulas may differ there. And the human baseline is defined as the second-best human action count in the paper, while ARC Prize's Astra post describes it as the median action count among players who completed each level.

Squaring matters for this story. Because efficiency is squared, a model that wastes actions on re-exploration is punished sharply. A harness that lets the model reuse prior work cuts exactly that waste. ARC Prize says that in the adapter runs Astra (max) used fewer actions than the human baseline on 96.0% of levels, and 51.7% fewer actions per level on average. Those figures apply to the 98.6% max run, not the 99.9% high run.

Who runs it, and are there conflicts of interest?

ARC Prize Foundation designed the benchmark, wrote the standard harness and ran both evaluations.

OpenAI told Fortune, as relayed by The Next Web, that most evaluations carry noise of a few percentage points depending on checkpoint, scaffold and run. That is plausible, but it cannot explain a 37 point gap between harnesses.

ARC Prize's testing policy and leaderboard notes add constraints: only systems that cost less than $10,000 to run are shown, incomplete outputs are marked incorrect, and results marked "preview" are unofficial.

Which comparisons are fair?

Only same-harness comparisons. The Next Web reports that a widely shared comparison paired Astra's Provider Adapter score with a rival model's Standard harness score, which mixes two different conditions. A fair comparison sets Astra's 62.7% Standard score against other models' Standard scores.

ARC Prize says it will "report both Standard harness and Provider Adapter harness results on the ARC-AGI leaderboard, with each evaluation condition clearly labeled." Its framing is that the two harnesses answer different questions. Standard asks how models compare "under the same minimal, provider-neutral interface." Provider Adapter asks how well a model does "when it can use the context-management features its provider designed for it."

What are the weaknesses of reading either number?

Neither number is the model's capability in isolation, and ARC Prize says so about a third configuration too. A separate run used the PRO-LONG harness with a code-execution sandbox, and ARC Prize says its results "should be understood as the combined performance of the model and its tools." The same logic applies to both headline numbers.

Other limits:

• Bounded task. ARC Prize notes ARC-AGI-3 has "deterministic, closed-ended mechanics and goals" and does not represent real-world open-endedness.
• Cost spread. Standard-harness costs run $26,098 to $49,791 across effort levels. Adapter costs run $17,332 to $23,457. The adapter is cheaper at every level, and the high adapter run is about 28% cheaper than the best Standard run.
• Non-monotonic effort. The Standard column does not rise smoothly with effort (none beats low), and the adapter's best score is at high, not max. With no confidence intervals published, small differences between adjacent rows should not be over-read.
• Human baseline. ARC Prize says humans can solve 100% of environments, using about 500 members of the public. The baseline definition differs between documents, as noted above.

Does 99.9% mean ARC-AGI-3 is solved or that Astra is AGI?

ARC Prize says no. It writes: "while we believe Astra represents meaningful progress towards generalization, we are not claiming that it is AGI." It also says it believes "a future AGI should be able to solve ARC-AGI-3" under the Standard harness conditions, where Astra's best observed score is 62.7%. The Next Web quotes ARC Prize co-founder Mike Knoop: "we lack evidence to call this AGI yet."

Should you trust it, and for what?

Trust the pair of numbers, not either one alone. A rough credibility scorecard for the ARC-AGI-3 Astra results, my own assessment:

Use the Standard number to compare models on a neutral interface. Use the adapter number to estimate what a developer gets from a provider's own context features. Do not use a cross-harness gap to claim one model is better than another.

What's still unclear?

• What drives the gap. ARC Prize attributes it to context handling, but no ablation separates preserved reasoning state from compaction.
• Which scoring revision applies. The formulas above are from v1 of the paper; the later revision may differ.
• Whether other labs have adapter equivalents. The Standard harness is provider-neutral by design. How much other providers' context features would lift their scores is not published.
• The weakest point of this analysis. The 37 point gap is real, but a single model on a single benchmark. It shows harness sensitivity is large here, not how large it is elsewhere.