Benchmark Cost Per Task and Fallback Routing: Why the Fable 5.1 Number Moved From $3.76 to $7.63

Artificial Analysis reports that Claude Fable 5.1 scored 66 at max effort at $3.76 per Intelligence Index task on 1 September 2026. Its live model page, read on 13 October 2026 and labelled Index v4.3.2, shows 53 and $7.63 for the same model. Both numbers come from a run that used Anthropic's "default" server-side fallback, which silently hands some requests to a different model.

The documented problem is narrower than any claim of misstated costs, and easier to miss: a cost-per-task figure can describe a blend of models, and the label rarely says so.

What is server-side fallback, and why does it affect benchmark cost?

Server-side fallback re-runs a request on a second model when a safety classifier declines it on the first. Anthropic says that when Fable's classifiers detect cybersecurity, biology and chemistry, or distillation requests, the response is handled by Claude Opus 4.8 instead. The company says this happens in less than 5% of sessions on average (company claim).

In the API it is opt-in. Per the refusals and fallback docs, customers set fallbacks to "default" or to an explicit list, with a beta header. Only a safety classifier decline triggers it. A rate limit or server error is returned as is.

For a benchmark, this matters because the model name on the leaderboard row is the requested model, not necessarily the one that answered every task.

How did Artificial Analysis use fallback for Fable 5.1?

Artificial Analysis says it evaluated Fable 5.1 with Anthropic's "default" server-side fallback. It says flagged requests routed to Claude Opus 4.8 or Claude Opus 5, and that fallback served about 4% of output tokens across the Intelligence Index.

Two details limit what that figure tells you:

• It is a share of output tokens, not of tasks. A fallback that fires on a few long tasks and one that fires on many short ones can give the same 4%.
• The article does not say how the split between Opus 4.8 and Opus 5 went.

The live page carries the label "Claude Fable 5.1 (Max, Default Fallback)", so the configuration is visible there. It gives no fallback share.

Why do the two Artificial Analysis pages disagree?

The numbers differ because the pages appear to measure different things, and the pages do not say exactly how. The comparison below is lab-independent (Artificial Analysis is a third-party evaluator), and the two rows are not on a shared index version as far as we can tell.

The live page lists 10 evaluations, including Terminal-Bench 4.0, GDPval-AA v2.1 and AA-Omniscience. The article does not name a version, so we infer a version change explains the gap. That is inference. Do not put the two rows in one chart.

The cost gap is larger than the score gap in relative terms: $7.63 is about 2.0 times $3.76. The live page lists 190M output tokens for the Index run. The article says Fable 5.1 uses about 1.7 times the output tokens of Fable 5. Token volume, not list price, drives the cost.

How is benchmark cost computed in the first place?

Artificial Analysis says it uses token counts reported by each model's API provider where available. It combines them with live measurements of the model's typical cache hit rate, rather than the one-off measurement from when the evaluation ran.

That design has a consequence. Pricing changes show up in cost per task without a re-run. The article says Anthropic cut cache-read pricing from $1 to $0.25 per million tokens, which saves about $1.40 per task, and that Fable 5.1 would cost about $5.16 per task without the cut. The listed prices are $10 input and $50 output per million tokens.

So a cost-per-task number moves for at least four reasons: token volume, cache hit assumptions, price changes and which model served each attempt. Only the last one is hidden by default.

How does the Claude API report which model answered?

The API reports it, if the harness reads the right fields. The docs say the top-level model field reports the model that produced the returned message. A fallback content block marks the handoff, and usage.iterations lists each attempt, with the serving model shown as fallback_message.

Billing has a trap for cost accounting. Each attempt is billed at the rates of the model that ran it, and every attempt that produced output is billed separately, including one that declined partway through. The top-level usage counts describe only the attempt that produced the returned message. A harness that reads only that field undercounts the declined attempt.

The docs also note that fallback is not supported on the Message Batches API and is not available on Amazon Bedrock, Google Cloud or Microsoft Foundry. A benchmark run through a cloud provider and one run through the Claude API can differ in fallback behavior for that reason.

Can fallback silently change a benchmark result?

Yes, and one independent paper documents it. RuBench, a single-author coding benchmark by Evgeny Shilov, reports that on 5 of 25 tasks (20%) an official safeguard fallback re-routed routine HTTP-protocol fixes from Fable 5 to Opus 4.8 in Claude Code 2.1.201. The author found it by auditing trajectories, where a model_refusal_fallback record named the original and fallback models.

The author's response was to add a rule that the executing model must match the requested model, and to exclude the affected cells. The paper states that the deployed product, not the model, is the unit actually being measured. It also says this is not a Fable 5 quality result.

Anthropic's own redeployment post acknowledges the mechanism. It says the new classifier comes at the cost of flagging benign requests more often during routine coding and debugging tasks (company statement). That matches the kind of tasks RuBench saw rerouted.

This is a single-author benchmark with 25 tasks, so treat the 20% as an observation about one harness, not a general rate.

What triggers fallback in practice?

Anthropic's help center says classifiers run on every user request and on everything the model reads, including web search results and files. A block can therefore be triggered by content the user did not type. For offensive cybersecurity requests it says users should expect high fallback rates.

For agentic benchmarks, that means a task that touches security tooling, or a repository that contains it, is more likely to leave the requested model. The help center names Opus 5 as the fallback for biology, chemistry and life sciences requests and Opus 4.8 for offensive cybersecurity technique requests (company statement, undated page).

Cost implications follow. The fallback model may be cheaper or more expensive than the requested one, and mid-stream declines bill twice. The direction of the error is not fixed, which is why a single "cost per task" cannot be corrected by a constant.

Is there an OpenAI equivalent?

OpenAI's misalignment monitoring guide describes a different behavior: a blocked request returns HTTP 403 with code misalignment_policy_violation, and the API does not provide a general way to resume the stopped conversation.

A stop and a reroute distort benchmarks differently. A stop shows up as a failed or missing task. A reroute can show up as a pass by another model. Neither lab's mechanism is a cost fudge, but the second is harder to see in a leaderboard row.

What's still unclear?

• Task-level fallback share. Artificial Analysis gives about 4% of output tokens, not the share of tasks, and not the split between Opus 4.8 and Opus 5.
• Why the article and live page differ. An index version change is plausible and unconfirmed.
• Whether other vendors' runs used any routing. The OpenAI page cited above documents a stop, not a reroute, and says nothing either way about routing in benchmark runs.
• Weakest point of this post. The central evidence for hidden model substitution is one 25-task independent paper plus Artificial Analysis's own disclosure. Neither shows that a published ranking changed because of fallback.

What should a benchmark report about fallback?

Based on the fields the Claude API exposes, a defensible minimum is five items per row:

1. Whether fallback was enabled, and in which mode (default or an explicit list).
2. The share of tasks, not only tokens, where a fallback fired.
3. The executing model per attempt, read from usage.iterations.
4. Cost with and without the fallback attempts, billed at each model's rates.
5. The index or harness version and the capture date.

Artificial Analysis already does the first and part of the second. RuBench's executing-model rule covers the third. Readers should treat a cost-per-task figure without these as a property of a product configuration, not of a model.

Disclosure: Model Report Card is affiliated with Humanbit, a data labelling and evaluation company.