Opus 5.5 AutomationBench Score: Why Two Labs Report Different Numbers for the Same Model

Anthropic's Claude Opus 5.5 launch post (22 September 2026) reports 40.0% on AutomationBench. Zapier's own leaderboard lists the same model at the same Max effort at 42.47% (v1.0.6, captured 12 October 2026). A third source, Artificial Analysis, shows 69.5% on a different metric entirely.

The "two labs" framing is slightly off. Anthropic's footnote says the AutomationBench results "were run and reported by Zapier", so both the 40.0% and the 42.47% come from the benchmark's own operator. The gap comes from how safeguard fallbacks are scored.

What does AutomationBench measure?

AutomationBench tests whether an agent can complete cross-application business workflows through REST APIs. The paper (arXiv:2604.18934, Daniel Shepard and Robin Salimans, submitted 21 April 2026) says tasks come from real Zapier workflow patterns across Sales, Marketing, Operations, Support, Finance and HR.

Agents must find the right endpoints themselves, follow layered business rules and cope with misleading records. The GitHub README describes a public set of 600 scored tasks, 100 per domain, across 47 simulated SaaS tools, with a default cap of 50 steps. Zapier's leaderboard page describes a harness with two tools: search (BM25 over API schemas) and execute (a curl-like call). There is no human in the loop and the agent cannot ask clarifying questions.

How is it scored?

The official metric is task_completed_correctly, a strict pass/fail that is 1.0 only if every assertion on the final environment state passes. The README says the official pass rate is the mean of that value across scored tasks. A separate partial_credit field (fraction of assertions satisfied) is diagnostic.

Grading is programmatic and end-state only, per the paper. There is no LLM judge. Zapier's page describes negative assertions, such as checking that a notice was not sent to the wrong recipient, to prevent shotgun reward hacking.

Who runs it, and are there conflicts of interest?

Zapier built and operates AutomationBench and its leaderboard. Anthropic's footnote says Opus 5.5's result "comes from Zapier's own evaluation during early access", while the Opus 5, GPT-5.6 Sol and GPT-6 Astra figures in the same table come from Zapier's public leaderboard.

That means Anthropic did not run the 40.0% itself, but it chose which Zapier-produced number to print. Artificial Analysis is a separate operator: its July 2026 announcement says it ran Zapier's private subset in partnership with Zapier. Neither Zapier's page nor Anthropic's post discloses a commercial relationship between them, and none is claimed here.

What's the current leaderboard?

On Zapier's leaderboard as of 12 October 2026 (v1.0.6, no snapshot date shown on the page), Opus 5.5 sits fourth at Max effort. All rows below are Zapier-operated.

The page also shows GPT 6 Astra at XHigh (38.96%), High (37.14%) and Medium (34.09%). Cost per task is as shown by Zapier. *The Gemini models have promotional pricing; Zapier says ranking and cost per task for them use standard list pricing.

Where do the reported scores disagree, and why?

The documented difference is fallback handling. Anthropic says its AutomationBench runs "were performed without fallback models, so safeguard interventions were considered failures", which Anthropic says produced a lower score than Opus 5.5 would achieve in practice. Zapier's leaderboard says: "Default fallbacks route a refused step to the provider's own default fallback model."

The 2.47-point gap is larger than the roughly 1% run-to-run variance Zapier states. That is consistent with fallback treatment being the cause, but this is inference. Anthropic does not state which benchmark version its early-access run used, and the leaderboard does not state when its rows were run, so version and date differences cannot be ruled out.

The ordering against GPT-6 Astra also flips. Anthropic's table has Astra ahead, 41.4% to 40.0%. On the leaderboard, Opus 5.5 at 42.47% is ahead of Astra's 41.4%. The Astra number is identical in both places, which supports the reading that only the Opus figure moved.

Why is the Artificial Analysis number so much higher?

It measures something different. Artificial Analysis runs a private, held-out set of 657 tasks and reports the share of objectives completed, where any task with a guardrail violation scores zero. Zapier's leaderboard reports the share of tasks completed fully. The Artificial Analysis page lists Opus 5.5 (Max, Default Fallback) at 69.5% with 90% of objectives completed, third behind Gemini 4 Argon (High) at 77.5% and Claude Sonnet 5.5 (Max, Default Fallback) at 71.8%.

Its July announcement also states each task is run once with a 50-turn cap. Do not place the 69.5% in the same column as 40.0% or 42.47%.

What are the weaknesses?

• Opaque task set. Leaderboard scores use a held-out private set that the README calls "purposely harder" than the public one. Outsiders cannot audit it.
• Moving target. The README says private tasks are sometimes made harder in version updates, and models are rerun when scores shift beyond run-to-run variance. A score is only meaningful with its version (here v1.0.6).
• No intervals. Neither Anthropic nor Zapier gives confidence intervals. Artificial Analysis runs each task once.
• Safeguard confound. Fallback routing mixes a different model into the result. Artificial Analysis noted Fable 5 fell back to Opus on about 18% of tasks, and Zapier's footnote says Opus 5 handled about 40% (260 of 657) of the Fable 5.1 combination's tasks. A score with fallbacks is partly another model's score.
• Saturation trend. The paper's abstract said the best frontier models scored below 10% in April 2026. The leaderboard top is now above 50%, though the public-set and private-set numbers are not directly comparable.

Should you trust it, and for what?

Trust it as a relative signal for multi-step, rule-following API work, not as a precise ranking at the 1 to 3 point level. This scorecard is an editorial judgment from the sources above.

What would a better version look like?

Every published row would carry the benchmark version, run date, effort setting and fallback policy. It would report both a no-fallback and a with-fallback score side by side, plus the share of tasks routed to a fallback. Confidence intervals from repeated runs would replace a single variance note. That would let a reader reconstruct why 40.0% and 42.47% differ instead of inferring it.

What's still unclear?

• Which benchmark version and run date produced Anthropic's 40.0%.
• How Artificial Analysis's "Default Fallback" label maps onto Zapier's fallback routing. Its evaluation page does not explain the label.
• How many Opus 5.5 tasks were routed to a fallback model on the leaderboard run.
• Whether the Opus 5.5 leaderboard row was rerun after the early-access evaluation, since the page shows no dates.

The weakest point of this analysis is the causal claim. Fallback treatment is the documented difference, but with no run dates and no per-task data, it cannot be proven to explain the full 2.47 points.