Mistral Large 4 "Wins" a Benchmark Because Rivals Refused to Answer

Mistral Large 4 Preview scores 81.7% pass@1 on CyberGym-E2E-AA, the highest of any model on the Artificial Analysis leaderboard (independent, captured 13 October 2026). Artificial Analysis also says Claude Opus 5.5 and GPT-6 Astra refuse at least 98% of the tasks, and a declined task scores zero. The win is real as a measure of what you can get from an API today. It says much less about what the models can do.

What does CyberGym-E2E-AA measure?

It measures whether an agent can reproduce a real memory-safety bug and then fix it. Artificial Analysis built it from Berkeley RDI's CyberGym-E2E, using 131 vulnerabilities from 131 C/C++ projects, a filtered subset of the original 920-instance dataset. Each task has a 90-minute limit and runs in the Stirrup harness.

The agent must write a proof-of-concept input that crashes the unpatched build, then write a patch. A task counts as solved only if the proof-of-concept crashes the original build, the patch removes the crash, and the project's own tests still pass.

How is it scored, and where do refusals go?

The metric is pass@1, and it feeds one third of the Cyber Index v1, an equal-weight average with CWE-Bench-AA and DeepsecBench-AA. Artificial Analysis defines "Safety Blocks" as the share of tasks that the model or its provider declined on safety grounds, and states that a declined task scores zero. It reports blocks separately from the score.

That design is defensible: a model you cannot use for a task has zero practical value for it. But it means pass@1 here blends two things, capability and willingness. The board separates them only if you read the Safety Blocks chart next to the score.

Who runs it, and are there conflicts of interest?

Artificial Analysis operates the index with launch partners Collinear AI, IBM, NVIDIA and Vercel, per its alliance announcement of 28 September 2026. Collinear AI's CWE-bench and Vercel's DeepsecBench supply two of the three components. Artificial Analysis names GPT-5.6 Sol (high) as the LLM judge for DeepsecBench-AA, so an OpenAI model grades a component on which OpenAI's GPT-6 models are also ranked.

The Mistral numbers on its launch post are lab-reported, while the leaderboard figures are Artificial Analysis' own runs.

What does the leaderboard show?

Mistral's headline is the CyberGym-E2E column. The table separates what each source says.

The ActuIA figures come from a single secondary source. Artificial Analysis' own text confirms only that both models refuse at least 98%. Mistral's launch post, a lab-reported claim, says both score near zero because they refuse.

On the full Cyber Index, ActuIA reports Large 4 Preview at 49.5, fifth of 18 models as of 6 October 2026. Grok 4.7 (56.4), MiMo-V2.6-Pro (56.1), GPT-6 Luna (52.7) and GLM-5.3-Flash (49.9) were ahead. Behind it were GPT-6 Astra (33.4) and Claude Opus 5.5 (28.7). Mistral's own claim of a top-five finish matches that rank. Artificial Analysis' live page now lists 20 models, so the rank can move.

Do the rivals refuse because they can't, or because they won't?

They won't. ActuIA notes that Grok 4.7 and GPT-6 Luna, also closed models, handle the task, so the refusal stems from publisher policy. The Decoder reaches the same conclusion: the test measures provider policies as much as model capabilities.

Artificial Analysis shows the effect directly. On 8 October 2026 it added GPT-6 Sol (Daybreak Blue, max), a model available only through OpenAI's Daybreak program. It hits no safety blocks across the index, improved its overall score by 32 points over public GPT-6 Sol (max), and now leads the index. The largest gains came on CyberGym-E2E, the benchmark where Artificial Analysis sees the most refusals. The same model family, with fewer guardrails, goes from blocked to first place.

What are the weaknesses of this result?

Four stand out.

• Policy confound. A score that depends on whether a vendor's filter fires is a procurement signal, not a capability ranking.
• Preview status. Mistral Large 4 is a preview. Weights are promised by the end of October 2026, and Artificial Analysis lists it with no publicly available API. ActuIA calls the figures provisional.
• Single-run, fixed-harness evidence. The index uses one harness, Stirrup, and CyberGym-E2E-AA is pass@1 on 131 tasks. The methodology page mentions no confidence intervals. If you care about harness sensitivity, see how large the gap can be in our posts on Terminal-Bench 4.0 and the Claude Opus 5.5 harness gap.
• Selective framing. Mistral's launch material highlights the cyber result. On the Artificial Analysis Intelligence Index v4.3.2, Large 4 Preview scores 38 as of 6 October 2026. The Decoder puts Claude Opus 5.5 (Max) at 58 on that index, a 20-point gap, according to The Decoder.

Other Mistral-reported numbers, such as 28.3% on Terminal Bench 4.0 and 59.9% on AutomationBench, are lab-reported and list no effort setting. They should not be set beside other labs' numbers without checking versions and harnesses. For context on that problem, see why two labs report different AutomationBench numbers.

Should you trust it, and for what?

Trust it as an answer to "which model will do defensive vulnerability work through its public API without declining?" Do not trust it as an answer to "which model is best at security research?"

These ratings are my judgment from the published methodology, not measured results.

What would a better version look like?

Report two numbers by default: capability conditional on the model attempting the task, and attempt rate. That separates skill from policy. Add confidence intervals on pass@1, and label trusted-access entries on the main chart rather than only in a legend. Artificial Analysis already records refusals; the change is presentation, so the headline rank stops blending the two.

What's still unclear?

• Exact refusal rates. Artificial Analysis states only a floor of 98% for Opus 5.5 and Astra in text; the 98.5% and 100% figures rest on ActuIA alone, and refusal rates for Mistral, Grok 4.7 and GPT-6 Luna are not stated in text.
• How Mistral separates legitimate from malicious use. The Decoder notes that Mistral does not explain how the model tells vulnerability research from attack preparation. Mistral says its average refusal rate on cyber-abuse prompts from JailbreakBench, StrongREJECT and AgentHarm is higher than all open-source models, and gives no number.
• What changes when weights ship. ActuIA points out that the line between the public and "extended" cyber versions matters little once weights are downloadable, because an open model can be retrained.
• Final scores. The preview may change before release.

Because this post touches evaluation methodology, note that Model Report Card is affiliated with Humanbit, a data labelling and evaluation company.