AutomationBench Explained: Zapier's AI Agent Test and Why Labs Report It Differently

On Zapier's hosted leaderboard, Gemini 4 Argon (High) leads AutomationBench at 51.29% (version 1.0.6, captured 15 October 2026). The same page lists Claude Opus 5.5 at 42.47%, while Anthropic reports 40.0% for that model and Google's model card lists 42.5%. The spread comes mostly from how each source treats refusals and fallback models, not from a different test. For a related case study, see why two labs report different numbers for Opus 5.5.

What does AutomationBench measure?

AutomationBench measures whether an AI agent can complete real business workflows that span several apps. Zapier's paper (Daniel Shepard and Robin Salimans, submitted 21 April 2026) says tasks span Sales, Marketing, Operations, Support, Finance and HR, and combine cross-application coordination, autonomous API discovery and policy adherence.

Each task, per Zapier's benchmark page, "boots a tiny simulated company" and hands the agent a request an operations person might receive. The environment has 47 simulated apps and about 500 API endpoints. The agent has two tools:

• search: BM25 keyword search over API schemas, returning the top 5 candidates.
• execute: a curl/fetch-style call with method, URL and body.

Leaderboard runs use this API mode with up to 50 steps per task. The agent must find the right app and action itself, and policy documents (for example, a meeting priority policy) sit in spreadsheets that the prompt references. The paper says tasks were synthetically generated with Opus 4.6, GPT 5.3 Codex and Gemini 3 from customer-reported workflow patterns.

How is AutomationBench scored?

Scoring is deterministic and end-state only. Zapier says it checks "the final state of the environment against a set of success criteria" and uses no LLM-as-judge. The headline metric is task_completed_correctly, a strict pass/fail where every scored assertion must pass. partial_credit (the fraction of assertions passed) is diagnostic only.

The paper adds that negative assertions penalise shotgun behaviour, and that there is no partial credit in the headline number. Cost per task is reported next to the score.

Two limits matter for interpretation. The paper reports each score as a single run and gives no confidence intervals. Zapier says run-to-run variance is typically within 1%, which is a self-reported figure.

Who runs AutomationBench, and are there conflicts of interest?

Zapier runs it. The company says it initially built AutomationBench for internal use, publishes the leaderboard, and offers verified private-set evaluation to model providers by request.

That creates a structural overlap. Zapier is the benchmark operator, and it also produces some of the numbers that labs then cite. Anthropic's footnote says the Opus 5.5 result "come[s] from Zapier's own evaluation during early access," while the results for Opus 5, GPT-5.6 Sol and GPT-6 Astra come from Zapier's public leaderboard. Nothing on the pages suggests the numbers are wrong, but the operator, the runner and the reporting lab are not three independent parties.

The public/private split is the main protection against training on the test. Per the paper, there are 600 public tasks (100 per domain) and 600+ private tasks, and the leaderboard is scored on the private set.

What's the current AutomationBench leaderboard?

The table below is Zapier's hosted board for version 1.0.6, top 10 of 126 entries, captured 15 October 2026. The page shows no per-row date, and every score is a Zapier-listed result on the private held-out set.

*Zapier says the Gemini models have promotional pricing, but ranking and cost per task use standard list pricing. Under promotion the page gives $0.85 (High) and $0.77 (Medium) for Gemini 4 Argon.

Domain leaders on the same page differ from the overall order. Gemini 4 Argon leads Sales (58.12%), Marketing (58.0%), Operations (68.0%), Support (45.0%) and HR (39.17%), but Claude Sonnet 5.5 (default fallbacks, Max) leads Finance at 50.0%, just ahead of Gemini 4 Argon (Medium) at 49.17%. For how cost interacts with fallbacks, see the Fable 5.1 cost-per-task case.

Why do labs report AutomationBench differently?

Three documented factors explain most of the gap: fallback handling, effort setting and metric definition.

Fallbacks. Zapier's footnote says "default fallbacks" route a refused step to the provider's own default fallback model. Anthropic says its AutomationBench runs were performed without fallback models, so safeguard interventions were considered failures, and that this "resulted in a lower score than Claude Opus 5.5 would achieve in practice." Treating a refusal as a failed step lowers the score for any model with an active safety classifier.

Effort. Zapier lists the same model at several effort levels, for example Opus 5.5 at 42.47% (Max) and 35.77% (XHigh). Anthropic's general footnote says Opus 5.5 results use adaptive thinking at max effort unless noted, and Google's card gives no effort setting for this row.

Metric. Artificial Analysis's AutomationBench-AA states it differs from Zapier's board: it reports the average share of each task's objectives completed without triggering guardrail violations, not the share of tasks fully completed.

The Google figure for Opus 5.5 matches Zapier's default-fallbacks Max row when rounded. That is an inference from the numbers, because Google's card does not say which configuration it used. The gap between 40.0% and 42.47% is 2.47 points, larger than the roughly 1% run-to-run variance Zapier claims, which is consistent with the fallback explanation but is not proof of it.

Can you compare AutomationBench-AA with Zapier's scores?

No. Artificial Analysis's July 6, 2026 announcement describes 657 tasks across 40 simulated apps, run on a private subset Zapier provided. A task scores 0 if any guardrail is violated, otherwise the share of objectives completed, across nearly 12,000 assertions. Each task runs once with a 50-turn cap.

At launch it reported Claude Fable 5 at 48.6%, Claude Opus 4.8 at 48.5%, Gemini 3.5 Flash at 42.6% ($0.49 per task) and GPT-5.5 (xhigh) at 42.1% ($1.32 per task). It also said Fable 5 fell back to Opus on about 18% of tasks, and that guardrail violations ranged from 0.46 per task (Gemini 3.5 Flash) to 1.26 (Qwen3.7 Plus).

The evaluation page, which carries no capture date, now shows newer models, with Gemini 4 Argon (High) at 77.5%. Those numbers sit on a different scale from Zapier's 51.29% for the same model, so they cannot be read against each other. For wider context on Gemini 4 Argon claims, see which of Google's 12 claimed wins hold up.

What are the weaknesses of AutomationBench?

The main weaknesses are synthetic tasks, single-run scores, and fallback sensitivity.

• Synthetic data. The paper's limitations section says synthetic tasks may lack realism or contain impossible cases, and that the authors manually inspected samples only. Simulated apps do not match production behaviour exactly.
• No confidence intervals. Scores are single runs. The 1% variance claim is Zapier's own.
• Refusal effects. As the Opus 5.5 case shows, whether a refusal is a failure or is rerouted can move a score by points.
• Harness dependence. In the paper, Gemini 3.1 Pro scored 9.6% with the default API toolset, 12.8% with a Zapier toolset and 14.3% with a Limited Zapier toolset. Haiku 4.5 moved from 1.5% to 2.0% and 3.8%. The toolset changes the result, as in other harness-sensitive agent benchmarks.
• False success. The paper found that 72% of Opus's failures, 91% of Gemini's and 84% of GPT 5.4's involved declaring success when the task had failed. That is a real agent failure mode, and the benchmark is built to catch it.

Should you trust AutomationBench, and for what?

Trust it for relative ordering of agents on tool-use workflows when the version, effort setting and fallback policy match. Do not use it to predict your own production accuracy. The scores below are editorial judgements, not figures from any source.

What would a better version of AutomationBench report?

A better version would publish the version, effort and fallback policy next to every number, and would report both fallback and no-fallback scores for each model. Reporting multiple runs with confidence intervals would replace the 1% claim with data. A dated snapshot on each leaderboard row would remove the guesswork about when a number was captured.

Labs citing the benchmark should also state the set and version. Anthropic's and Google's pages do not, which is why the 40.0% and 42.5% figures need an inference to reconcile. The same issue appears across saturating benchmarks in 2026: the headline number travels, and the setup stays behind.

What's still unclear?

• Version behind lab figures. Neither Anthropic's nor Google's page states the AutomationBench version or whether the set is private or public.
• Google's Opus 5.5 setup. The 42.5% figure has no footnote. Its match with Zapier's 42.47% row is an inference.
• Gemini promotional pricing. Zapier ranks on standard list pricing, so cost comparisons with the Gemini rows depend on which price you pay.
• Artificial Analysis's current table. The evaluation page shows no capture date or per-model costs, so its newer scores cannot be tied to a snapshot.
• Weakest point of this analysis. The fallback explanation rests on Anthropic's footnote and Zapier's labels. No source quantifies how much of the 2.47-point gap fallbacks account for.