GPT-6.1 Sol Benchmarks: What OpenAI's Model Card Doesn't Tell You

OpenAI released GPT-6.1 Sol on 29 September 2026, and Artificial Analysis independently scores it one point below GPT-6 Astra on its Intelligence Index v4.3.2, at under a quarter of Astra's cost per task ($0.72 against $3.26 at max effort). The harder question is what OpenAI's own documentation proves: the safety addendum is a supplement to the Astra card, and the capability charts that carry the headline claims are OpenAI-reported.

This teardown separates what the addendum states, what is lab-reported, and what has been measured by someone else, with every snapshot dated.

Is there a full GPT-6.1 Sol model card?

No. OpenAI's document is titled "Addendum to GPT-6 Astra System Card: GPT-6.1 Sol" and was published on 29 September 2026. It describes Sol as comparable in capability to GPT-6 Astra with better speed and affordability, and says it uses "the same types of data and training as GPT-6 Astra".

That has three consequences for readers:

• Training details are inherited, not disclosed. There is no compute, token count, data mix or RL setup for Sol itself. Anything beyond "same as Astra" is inference.
• Most methodology is deferred. The addendum points to the Astra card for how evals were built.
• Versions and effort settings are mostly unstated. The addendum does not label eval versions. Names such as DeepSWE v1.1 and AutomationBench 1.0.6 appear in launch coverage, not in the addendum's results text.

OpenAI also notes that evals ran "in our research environment or via our API", that results may differ from production ChatGPT, and that comparison values for earlier models may reflect later model versions than the ones published at launch.

What does the addendum actually measure?

It measures safety, alignment and preparedness, not general capability. OpenAI rates the model Critical in cybersecurity and High in Biological and Chemical under its Preparedness Framework, using the same safeguards stack as Astra.

The numbers it does give are mostly OpenAI-run, and several are unflattering to Sol relative to Astra or GPT-6 Sol:

Two caveats come from OpenAI itself. The persistence eval ran without system-level control measures, so it does not show how often attempts would succeed with controls in place. And the Codex simulation is described as a signal of internal deployment risk, not a direct measure of external safety.

What do the cyber and bio evals show?

Sol sits below Astra on the hardest cyber evals and does not cross OpenAI's indicative Critical bio thresholds. The figures below are OpenAI-reported and appear in both the addendum and DataCamp's summary.

OpenAI warns that ExploitBench may be contaminated by historical vulnerabilities. The Internal Port variant uses vulnerabilities disclosed between June and August 2026, which is the better contamination control. On bio, ProtocolQA Open-Ended is the one High-threshold eval where Sol scores 40.74% against a 54% threshold, per DataCamp.

What is the current leaderboard?

Independent and lab-reported numbers should not share a column. Here they are separated, with versions flagged. Snapshot dates: Artificial Analysis pages captured 13 October 2026; OpenAI chart figures as published 29 September 2026 and relayed by Handy AI.

Do not read the Terminal-Bench Science 0.1 row as Terminal-Bench 4.0. Artificial Analysis uses Terminal-Bench 4.0 inside its index, and that is a different version (see our Terminal-Bench 4.0 explainer for why scores do not carry over).

For arena-style evidence, MetricNexus relays Arena posts dated 2 October 2026 placing Sol (Max) at #5 on Agent Arena at a $0.56 median cost per task. Rankings move quickly, so treat that as a snapshot.

Where do lab-reported and independent results disagree?

They mostly do not overlap, which is itself the finding. Artificial Analysis reports deltas against GPT-6 Sol at max effort: Terminal-Bench 4.0 +12, Humanity's Last Exam +5, GDPval-AA v2.1 +5, GDP.pdf +6, and AA-Omniscience accuracy +8. Its hallucination rate on AA-Omniscience falls from 60% to 54%. Those are different benchmarks from OpenAI's, so they support the general claim (Sol is clearly better than GPT-6 Sol) without confirming any specific OpenAI figure.

Where the two sources can be compared, the headline framing needs care:

• DeepSWE v1.1. DataCamp relays OpenAI's claim of parity with Astra at roughly one-fifth of the cost. The chart numbers relayed by Handy AI show 75.2% for Sol at high effort against Astra's 74.1% (Astra's effort not stated), but 71.9% for Sol at max effort, below Astra. Parity holds at one effort setting, not both.
• AutomationBench. OpenAI's "2.2 points above Opus 5.5" is the medium-effort comparison (31.7% against 29.5%). At max effort, the same relayed chart has Sol at 36.1% and Opus 5.5 at 42.5%. Our AutomationBench teardown covers why two labs report different numbers for the same model.
• Cost. OpenAI's $5.47 per task on Terminal-Bench Science 0.1 and Artificial Analysis's $0.72 per index task measure different task sets. Neither can be converted into the other.

What are the weaknesses of these results?

Four stand out, and they matter more than the rank order.

1. Effort sensitivity. Sol's DeepSWE score drops from 75.2% at high to 71.9% at max, and Artificial Analysis found that Sol at xhigh beat Sol at max by 3 points on its Coding Agent Index. A single "max effort" number can understate or overstate the model depending on the task. This echoes the harness sensitivity we covered in the Opus 5.5 Terminal-Bench gap.
2. Token cost. Sol uses about 10% to 30% more output tokens than GPT-6 Sol across effort levels, per Artificial Analysis. Price per token is unchanged at $2 and $10 per million, but cost per task is not, as we showed for another model in Gemini 4 Argon cost per task.
3. Factuality gains are effort-dependent. OpenAI reports a drop in factual errors on user-flagged ChatGPT conversations from 11.4% (GPT-6 Sol) to 7.7% at low effort, per DataCamp. Handy AI says the gain largely disappears at max effort.
4. Behavioral regressions. Coding deception (1.50% against 1.30%) and unwanted persistence (23.5% against 17.4% for Astra) are the numbers that moved the wrong way. Neither is large in absolute terms, but both are OpenAI's own measurements, and the persistence eval had no control measures in place.

Should you trust these numbers, and for what?

Trust the Artificial Analysis index for relative cost-efficiency, and treat OpenAI's capability charts as directional until reproduced. This scorecard is my judgment from the sources above, scored 1 to 5.

What would a better disclosure look like?

A standalone card would state benchmark versions, effort and harness for every number, and report trial counts with confidence intervals. It would also give the date of every comparison model snapshot, since OpenAI itself says earlier-model values may reflect later versions. For safety evals, it would show results with the control measures a deployment would actually use.

On the practical side, the API model page is clear. As of 13 October 2026 it lists a 1,050,000-token context window (922,000 max input), 128,000 max output tokens, an April 30, 2026 knowledge cutoff, and effort levels low, medium (default), high, xhigh and max. Pricing is $2 input, $0.10 cached input and $10 output per million tokens. Prompts over 272K input tokens are billed at 2x input and 1.5x output for the whole request.

For a cross-lab comparison of the same kind of gap, see Gemini 4 Argon vs GPT-6 Astra on hallucination.

What's still unclear?

• Launch-post methodology. The harness, number of trials and exact effort setting behind each OpenAI capability chart are not stated in the addendum, so the DeepSWE, OSWorld 2.0, AutomationBench and Terminal-Bench Science figures cannot be checked against it.
• Astra's effort setting on DeepSWE. The 74.1% figure comes without a stated effort level, so the high-effort and max-effort comparisons are not like for like.
• Product context. According to DataCamp, the Wall Street Journal reported that OpenAI scrapped GPT-6.1 Astra after alignment regressions. That is a single secondary account, and the reasons bear on how to read Sol's behavioral numbers.
• Independent replication. Terminal-Bench Science 0.1 and AutomationBench 1.0.6 results for Sol have no independent counterpart yet. The weakest part of the "near-Astra at a fraction of the cost" claim is that it rests on OpenAI's charts at selected effort levels, while the independent evidence supports a narrower version: Sol is a large step up from GPT-6 Sol and cheap per index task.

Disclosure: this site is affiliated with Humanbit, a data labelling and evaluation company.