Best AI Coding Benchmark 2026: FrontierCode vs DeepSWE vs SWE-Bench Pro

No coding benchmark is best on every axis in 2026, and the three newest each have a documented weak point. FrontierCode's top Main score is 54.6% (Opus 5.5, Medium effort, per Mercor's republication of Cognition's leaderboard), DeepSWE v1.1 tops out at 74% Pass@1 (Datacurve, updated 22 September 2026), and the SWE-Bench Pro public board leads at 61.50% (Scale AI). Those numbers use different graders, task sets and harnesses, so they cannot be ranked against each other.

This post compares what each benchmark measures, who runs it, and where it breaks. All scores below are lab-reported or operator-reported unless stated. Leaderboards were captured on 13 October 2026 unless a page carries its own date.

What does each benchmark actually measure?

FrontierCode measures whether a patch would survive maintainer review, DeepSWE measures behavior on original long tasks, and SWE-Bench Pro measures whether fail-to-pass tests turn green without breaking pass-to-pass tests. The designs differ enough that a model can rank differently on each.

Sources: Cognition, Datacurve, Scale AI paper.

Patch size separates them too. Datacurve reports mean reference solutions of 668 added lines for DeepSWE against 120 for SWE-Bench Pro. Cognition says FrontierCode has smaller patches than DeepSWE but is harder for agents, and that it scales difficulty with quality rubrics rather than patch size.

How is each one scored, and who judges?

FrontierCode is the only one of the three that grades quality, and it uses an LLM for part of that. Cognition describes an ensemble: injected tests, shell commands, reverse tests that must fail on the base commit, scope checks, an adaptive test tool that uses an LLM, and an LLM reviewing the diff against a prompt. A patch passes only if it clears every blocker criterion. The score is a weighted total, and failing a blocker scores 0. The post names no judge model.

DeepSWE avoids an LLM in scoring. Per Datacurve, verifiers are hand-written, run three times during authoring to catch flakiness, and also run regression checks on every trial. SWE-Bench Pro uses the classic SWE-bench rule: a task counts as resolved only if the fail-to-pass tests pass and all pass-to-pass tests keep passing (Scale AI paper).

Rubric grading buys sensitivity to style and scope, and costs objectivity. Cognition says in its own post that rubric grading is subjective and that hardening a rubric is harder than hardening unit tests.

Who runs it, and are there conflicts of interest?

Every operator has a stake. Cognition builds Devin and runs FrontierCode on private tasks. The post contains no conflict statement, and it says model creators can request an evaluation from Cognition. Epoch AI states it sources FrontierCode results from Cognition's leaderboard, so Epoch and Mercor are republishers, not independent reruns.

Scale AI built SWE-Bench Pro, runs its leaderboard, rewrote problem statements with its annotators, and purchased the commercial codebases. Datacurve wrote DeepSWE and then audited SWE-Bench Pro's verifier in the same post that launched DeepSWE.

The competitive claims between them deserve equal skepticism:

• Cognition claims FrontierCode has an 81% lower false positive rate than SWE-Bench Pro, based on trajectory analysis. The tasks are private, so outsiders cannot check it.
• Datacurve reports SWE-Bench Pro false positives of 8.5% and false negatives of 24.0%, against 0.3% and 1.1% for DeepSWE. The audit used 30 random tasks per benchmark, 3 rollouts each, and an LLM analyzer, not humans. Datacurve itself says some verdicts will be wrong.

Neither claim is an independent measurement. Both come from the author of the challenger benchmark.

What are the current leaderboards?

The three boards share few models, so each needs its own table. Do not read across them.

FrontierCode 1.1, Main, score view (Mercor, republishing Cognition, page undated, captured 13 October 2026; Cognition's own changelog shows its latest entry on 7 October 2026). The page shows every confidence interval as ±0.0%, so treat the intervals as absent. No harness is shown.

Cognition runs each model 5 times at each reasoning effort and reports the best effort. The top five span 2.5 points with no usable intervals, so the order is not established.

DeepSWE v1.1, 113 tasks, mini-swe-agent, Pass@1 (Datacurve, updated 22 September 2026):

The cost column is the most useful part: three models tie at 74%, and their cost per task ranges from $2.36 to $11.84. For more on why cost per task matters, see our Gemini 4 Argon cost-per-task analysis.

SWE-Bench Pro, public set, Pass@1 (Scale AI, no date on page, captured 13 October 2026). Rows marked * were run with mini-swe-agent; the others used SWE-Agent.

This board has none of the Opus 5 or GPT-6 generation models that lead the other two. It is a snapshot of an older model set, and the page's own prose still says top models score around 23%, which contradicts its table.

Does the harness change the answer?

Yes, and none of the three controls it the same way. DeepSWE runs every model through mini-swe-agent with a single bash tool. Scale's board mixes SWE-Agent and mini-swe-agent rows and uses different turn limits: grayed-out models were capped at 50 turns and a cost limit, others ran uncapped with 250 turns. Epoch notes that FrontierCode data uses Claude Code, Codex, mini-SWE-agent and Devin depending on the model.

Datacurve's own pilot shows the size of the effect, with a small sample. On 10 SWE-Bench Pro tasks, Claude Opus 4.7 scored 50% in mini-swe-agent and 40% in Claude Code, and Gemini 3.1 Pro scored 40% against 20% in Gemini CLI. GPT-5.5 was 40% in both. Datacurve notes that part of mini-swe-agent's edge may come from prompt tuning. For the same pattern on another benchmark, see Terminal-Bench 4.0 Explained and the Opus 5.5 lab-versus-independent gap.

What are the weaknesses of each?

FrontierCode. Tasks are private and cannot be rerun, so contamination is controlled by secrecy rather than by evidence anyone can check. Version 1.1 (dated 7 July 2026) added a fair-internet-use prompt and a verifier that zeroes flagged runs, demoted 75 overly strict blockers after auditing 1,000+ criteria, and deprecated the Diamond subset as noisy. Cognition says absolute scores changed but relative performance did not substantially change. The earlier Diamond headline, Claude Opus 4.8 at 13.4%, applies to a retired subset of version 1.0 and should not be compared with current Main scores.

DeepSWE. Epoch AI reviewed v1.1 on 7 September 2026 and rated it Flawed: issues in at least 23 of 113 tasks, over 20.3%, all false negatives. For 18 of the 23, models edited existing tests that the verifier discards or collides with, through duplicate symbols or removed helpers. Epoch calls it a partial review and says it stopped after crossing its 20% threshold, so false positives were not examined. Epoch's page did not link a response from Datacurve. DeepSWE also covers only five languages, mostly TypeScript, Go and Python, and Datacurve lists the single-harness design as a limitation.

SWE-Bench Pro. Its contamination controls are legal and access barriers (copyleft repos, private splits), which the paper says reduce risk rather than eliminate it. The paper lists limited language coverage and dependence on the test suite as limits, since a correct solution can fail tests it did not anticipate. Human-written problem statements also carry a lot of weight: GPT-5 (high) scored 25.9% with the augmentation and 8.40% without.

Should you trust each one, and for what?

These 1 to 5 scores are my editorial judgments from the documented facts above, not a vendor or third-party rating.

Use FrontierCode when the question is code quality and mergeability. Use DeepSWE for cost-aware comparison, but treat any one-point gap as noise until the false negatives are fixed. Use SWE-Bench Pro for continuity with older results, not for ranking current frontier models.

What would a better coding benchmark look like?

It would publish one score with three harnesses, not one harness. It would report confidence intervals on every row and date every snapshot. It would let a third party rerun the grader, and it would release a held-out split to outsiders on request instead of only to model creators. Datacurve lists its single-harness design as a limitation, and Cognition's v1.1 audit of its own blockers is a step toward the grader transparency the field needs.

What's still unclear?

• Whether FrontierCode's top five are separable. Mercor's ±0.0% intervals carry no information, and Cognition's launch post published no intervals.
• Whether Datacurve will fix the test-collision errors Epoch found, and how many scores move when it does. Datacurve's v1.1 post says ordering stays close to v1, but that predates Epoch's review.
• When Scale's SWE-Bench Pro board was last updated, and whether newer models will be added.
• Whether the rival false-positive and false-negative claims between Cognition, Datacurve and Scale hold under a neutral audit. None has been done in public.
• The FrontierCode and SWE-Bench Pro release pages use ambiguous date formats or no date. The FrontierCode launch post is labeled 06.08.26, which could mean 8 June or 6 August 2026. This post uses 8 June, matching the June 2026 coverage of its release.