Claude Code vs Codex CLI: The Same Model Can Cost 5x More Depending on the Harness

The HarnessTax study, released 16 September 2026 by researchers at UC Berkeley and Arena, found that the same model cost up to about 5x more per attempt depending on the harness, while average success moved only about ±2 points on SWE-bench Lite and ±5 on Terminal-Bench 2.0, as reported by Digital Applied. On SWE-bench Lite, Claude Code cost more than both other harnesses tested, at about 1.6x Codex CLI by geometric mean. The sample is small (30 tasks per benchmark), so treat the direction as credible and the exact ratios as soft.

How was the Claude Code vs Codex CLI comparison run?

HarnessTax ran 7 models through 3 harnesses (Claude Code, Codex CLI and Pi, an open-source four-tool harness) on SWE-bench Lite and Terminal-Bench 2.0, per Digital Applied and braindetox. The models were Claude Fable 5, Opus 4.8, Sonnet 4.6, Haiku 4.5, GPT-5.6 Sol, GPT-5.6 Luna and Kimi K3.

Each benchmark used 30 randomly sampled tasks, run 3 times per model and harness pair. Digital Applied reports a 100-turn cap, each harness at its high-effort setting, and 95% bootstrap intervals over the 30 task averages.

Cost is token usage priced at one fixed API list price dated 1 September 2026, applied identically to a model in every harness. It does not model subscription plans. Benchmark versions are given only as "SWE-bench Lite" and "Terminal-Bench 2.0", and the two are never merged in one column below.

How much more does Claude Code cost than Codex CLI?

On geometric-mean cost ratios across shared models, Claude Code ran about 2.0x Pi and 1.6x Codex CLI on SWE-bench Lite, and about 1.5x Pi on Terminal-Bench 2.0 (Digital Applied, braindetox). The spread is much wider for individual models.

The table below uses the per-attempt figures published by Digital Applied from the study's chart data (study-reported, independent of any model lab, API list prices as of 1 September 2026). The ratio column is my division of the two published costs.

Claude Code was not always the costlier option. Sonnet 4.6 and Kimi K3 were slightly cheaper in Claude Code on SWE-bench Lite. Three things stand out: the biggest gaps sit with the GPT-5.6 models, which Claude Code is not built around; Fable 5, Claude Code's home model family, shows a more modest 1.5x to 1.6x; and Opus 4.8 on Terminal-Bench 2.0 is close to a tie.

Where does the "5x" come from?

The headline is a Claude Code vs Pi number, not Claude Code vs Codex CLI. GPT-5.6 Luna on SWE-bench Lite cost $0.152 per attempt in Claude Code and $0.030 in Pi, with success of 55.6% and 53.3%, per Digital Applied. That is about 5.1x by division. Against Codex CLI, the largest gap in the published table is the 4.3x on the same model and benchmark.

Luna is also a cheap model, so the absolute gap is $0.12 per attempt. The larger dollar gap is GPT-5.6 Sol on SWE-bench Lite: $1.540 in Claude Code against $0.561 in Codex CLI, roughly $1 per attempt for a 4.5 point higher observed success rate that sits well inside the noise.

Does the more expensive harness win more tasks?

No, not reliably. The study's average harness effect on success is about ±2 points on SWE-bench Lite and ±5 on Terminal-Bench 2.0, and across the six Anthropic and OpenAI models a non-vendor harness gave the highest observed success in 9 of 12 comparisons (braindetox).

GPT-5.6 Sol on Terminal-Bench 2.0 is the clearest case: 71.1% in Claude Code at $1.355, 78.9% in Codex CLI at $0.761, and 83.3% in Pi at $0.421. Fable 5 on SWE-bench Lite points the other way, with Claude Code's 97.8% edging Codex CLI's 96.7%. Neither gap is statistically meaningful at 30 tasks. Digital Applied notes that most success intervals span 25 to 35 points, so small rank differences between rows are not findings.

Why would the same model use so many more tokens?

The study reports that Claude Code's first-call context is more than 10x Pi's (Digital Applied). That points to longer system instructions and larger tool schemas sent on every call. It is a measured fact about the first call, not a full explanation of the bill.

Turn counts do not explain it either: Fable 5 on SWE-bench Lite averaged 15.4 turns in Pi and 15.3 in Claude Code. Caching, generated tokens and later calls also contribute, and the study authors say turn definitions differ between harnesses. What the harness actually sends and caches is the hidden variable, which is the same lesson as in our look at cost per task beyond the headline benchmark score.

Do other studies find the same thing?

Partly. A private-suite study, arXiv 2609.11987, compared vendor SDKs (claude-agent-sdk with Opus 4.8, openai-codex with GPT-5.5) against a neutral harness on the same models. These are the vendor SDKs, not the Claude Code and Codex CLI products, though the Codex cell pins a CLI version.

It found success statistically indistinguishable: native minus neutral was -1.25 points for Opus 4.8 and +1.25 points for GPT-5.5, with intervals spanning several points either side of zero. On cost the sign flipped relative to HarnessTax: the neutral harness cost 1.3 to 1.6 times as much per solved task on Opus 4.8 and 1.2 times as much on GPT-5.5. The authors caution that 58 Opus runs ended without a usage record, and allocating that spend to either side could move the Opus ratio anywhere between 0.7 and 2.3.

GitHub, which sells a competing harness, reported in June 2026 that Copilot CLI matched Claude Code (Sonnet 4.6, Opus 4.7) and Codex CLI (GPT-5.4, GPT-5.5) on completion rates with lower token consumption in most configurations, per its own post. That is a vendor-reported result, the figures are in charts, and it reports the best of five runs on benchmarks under 100 instances.

What's still unclear?

Several things limit how far these numbers travel.

• Sample size. 30 tasks per benchmark and 3 runs per pair give wide intervals. The cost ratios are likelier to hold than the success rankings, but cost also varies by task.
• Contamination. Both benchmarks are public, so models may have seen them in training (Digital Applied).
• Pricing basis. Costs are API list prices as of 1 September 2026. Most people run Claude Code and Codex CLI on subscriptions, where per-attempt cost is not what you pay.
• Conflicting cost direction. HarnessTax finds the lightweight harness cheaper; arXiv 2609.11987 finds the neutral harness costlier. The studies differ in harnesses, models, task sets and pricing method, so neither settles the question.
• Unusual pairings. Running GPT-5.6 models inside Claude Code is a stress test of the harness, and may overstate the gap you would see with each vendor's own model.

How should you evaluate a harness for your own team?

Measure cost per solved task on your own repositories, with the harness and model as separate variables. Fix the model, fix the turn cap and effort setting, run at least three repeats, and price tokens at one list price so harnesses are comparable.

Report the harness name and version beside every score, as with the Opus 5.5 Terminal-Bench gap and the ARC-AGI-3 harness gap. Scores on Terminal-Bench 4.0 do not carry over from earlier versions either, so version your own harness tests the same way.