DeepSeek V4.1 Flash Benchmarks Look Too Good to Be True. Here's Why

DeepSeek reports that DeepSeek-V4.1-Flash scores 90.6 on Terminal-Bench 2.1 and 74.2 on DeepSWE v1.1, ahead of every model in its own comparison table, according to the model card published 10 September 2026. Every one of those numbers is lab-reported. On the harder Terminal-Bench 4.0 the same table gives it 31.2.

The numbers are probably not fabricated. They are best read as a ceiling: the best scaffold, the highest effort setting, a benchmark version that has partly saturated.

What does DeepSeek actually claim for V4.1 Flash?

DeepSeek lists 19 benchmark scores in its changelog entry dated 10 September 2026, and says the model gives "benchmark results ahead of flagship models, including DeepSeek-V4-Pro" in its launch post. The post also says "Tests by multiple parties put V4.1-Flash ahead of V4-Pro," without naming those parties.

The model is a mixture-of-experts with a 552B backbone, 8B active parameters at prefill and 16B at decode, a 1M-token context and an MIT licence, per the model card. Everything below is lab-reported unless marked otherwise.

Snapshot captured 13 October 2026 from DeepSeek's changelog and model card. Pass@1 at maximum reasoning effort (DeepSWE is reported as resolved rate).

How does it compare with Opus-5.0 and GPT-5.6 Sol?

In DeepSeek's own table, V4.1 Flash leads on Terminal-Bench 2.1 and DeepSWE v1.1 but trails Opus-5.0 and GPT-5.6 Sol on Terminal-Bench 3.0 and 4.0. The comparison models were run at max reasoning effort, and the card does not say who ran them, so treat the competitor columns as DeepSeek-reported too.

All values lab-reported by DeepSeek, captured 13 October 2026. Columns are different benchmark versions: Terminal-Bench 2.1, 3.0 and 4.0 are not interchangeable, and scores do not carry over between them (see our explainer on Terminal-Bench 4.0).

Two things stand out. On TB 2.1, six models sit within 2.7 points of each other (87.9 to 90.6), which is a compressed range. On TB 4.0, the same models span 12.4 to 51.8.

Why do the scores collapse between Terminal-Bench 2.1 and 4.0?

Because they are different tests. Terminal-Bench 2.1 (6 May 2026) revised 2.0 by fixing 28 of 89 tasks. Terminal-Bench 3.0 (30 July 2026) moved to 74 tasks across 7 domains, with open internet, multi-container networks, GPUs and separate agent and verifier containers.

Terminal-Bench 4.0 is a version update of the 3.0 dataset. It sets a flat 8-hour agent timeout, fixes 19 tasks and removes 8 (two saturated, two refusals, two with public solutions, two with quality or platform issues).

The drop is not specific to DeepSeek. Kimi K3 falls from 88.3 to 12.6 in the same table. The relevant point for V4.1 Flash is relative: it improved far more on 4.0 over V4-Flash (7.0 to 31.2, +24.2) than on 2.1 (82.7 to 90.6, +7.9), but remains about 20 points behind Opus-5.0 on 4.0.

How much does the harness move the number?

By up to 8.7 points, and the headline numbers are the top of the range. DeepSeek's card includes a scaffold comparison at max effort, with 8 samples per task on DeepSWE, 3 on Terminal-Bench 2.1, a 1M context limit and 500 maximum steps. All figures are lab-reported.

The headline DeepSWE figure (74.2) is the mini-SWE run and the headline Terminal-Bench 2.1 figure (90.6) is the DSH Minimal run. Each is the highest value in its column. Published openly, this is a useful disclosure. It also means the cross-model table compares DeepSeek's best scaffold against competitors whose scaffold the card does not specify.

This is the same harness sensitivity discussed in our write-ups of Opus 5.5 on Terminal-Bench and ARC-AGI-3.

The sample counts matter too. Three samples per task on a benchmark with 89 tasks gives no confidence intervals on the card, so small gaps between rows are not interpretable.

Can anyone reproduce the harness?

Not confirmed. The V4-Flash-0731 model card said agent tasks used "the minimal mode of DeepSeek Harness (to be released)". The V4.1 card again names DeepSeek Harness minimal mode for the Terminal-Bench, DeepSWE, NL2Repo-Bench and ProgramBench runs, and does not state a release status. Without the harness, an outside team cannot rerun the headline configuration.

Other settings are disclosed: reasoning_effort=100, temperature 1.0, top_p 0.95. Agents' Last Exam and AutomationBench used their official scaffolds, which makes those two rows easier to check than the rest.

What does independent evidence say?

Only a composite measure exists. Artificial Analysis scores DeepSeek V4.1 Flash (Max) at 39 on its Intelligence Index v4.3.2, ranked 7th of 116 models, against an open-weight peer median of 18 (independent, captured 13 October 2026). Terminal-Bench 4.0 is one of the Index's ten components, but the Index is a composite, not a Terminal-Bench score.

The same page flags verbosity: about 250M output tokens to run the Index, against a 140M median, ranked last of 40 models compared. It lists 197.3 output tokens per second and 1.06 seconds to first token through DeepSeek's API.

That verbosity matters for cost. DeepSeek's pricing page lists deepseek-flash (DeepSeek-V4.1-Flash) as of 13 October 2026:

Peak hours are 01:00 to 04:00 and 06:00 to 10:00 UTC, Monday to Friday, excluding Chinese public holidays. Cheap tokens do not make a cheap task if the model emits nearly twice the median output. For the method, see our piece on cost per task.

Should you trust the DeepSeek V4.1 Flash numbers?

Trust them as an upper bound from the lab, not as a ranking. A scorecard for the headline agentic claims, 1 to 5:

Versioning deserves credit: DeepSeek reports the harder Terminal-Bench 3.0 and 4.0 results alongside 2.1, which a lab chasing headlines would not have to do. Scores are also disclosed per scaffold, which most launches omit.

What's still unclear?

• Harness availability. Whether DeepSeek Harness minimal mode is publicly released, and whether third parties can reproduce 90.6.
• Who ran the competitor rows. The card does not say whether Opus-5.0, GPT-5.6 Sol, K3 and GLM-5.3 were run by DeepSeek or taken from their labs.
• "Multiple parties." The launch post cites unnamed tests placing V4.1 Flash ahead of V4-Pro. DeepSeek later said it would keep serving V4 Pro after 14 September 2026 in the changelog, reversing the phase-out stated in the launch post.
• Internal inconsistency. NL2Repo-Bench is 65.4 in the changelog and 64.0 in the card table.
• Parameter count. The card says 552B backbone parameters; the Hugging Face page header shows 763B, and the card describes a separate 196B sparsely accessed Engram memory.
• Contamination. No contamination analysis is given for the agentic benchmarks.

The weakest point of this critique is that nothing here shows the numbers are wrong. Self-reported is not the same as false, and the model's independent Index score of 39 suggests it is a strong open-weight model. The open issue is how much of the 90.6 transfers to your harness.

What would better evidence look like?

A third-party run on Terminal-Bench 4.0 with a common harness, effort budget and at least 5 samples per task, reported with confidence intervals. Plus the DeepSeek Harness released so the 90.6 can be rerun. Until then, compare V4.1 Flash against other models on one benchmark version at a time. See also why two labs report different AutomationBench numbers for how much setup differences matter.