tau-bench Explained: Why Agent Pass^k Falls as You Repeat the Run

tau-bench tests whether a tool-using agent can follow domain policies while talking to a simulated user. Its reliability metric, pass^k, counts a task as solved only if the agent succeeds on all k independent trials. In the 2024 paper, gpt-4o succeeded on under 50% of tasks, and its pass^8 in the retail domain was under 25% (arXiv).

What does tau-bench measure?

tau-bench (also written τ-bench) emulates a conversation between a user, simulated by a language model, and a language agent. The agent gets domain-specific API tools and policy guidelines. The paper argues that existing benchmarks do not test agents on their interaction with human users or on their ability to follow domain-specific rules (arXiv).

The authors set three requirements: agents must interact with both humans and programmatic APIs, follow complex policies accurately, and stay consistent and reliable at scale (OpenReview). The benchmark covers two domains, a retail store and an airline booking system (alphaXiv).

The paper's authors are Shunyu Yao, Noah Shinn, Pedram Razavi and Karthik Narasimhan, and it was posted to arXiv on 17 June 2024 (Hugging Face). The code and data are published under Sierra Research's GitHub organisation (GitHub).

Is the original tau-bench task set still current?

No. The repository README warns that its airline and retail tasks are outdated. It directs users to τ³-bench for the latest fixed tasks and new domains (GitHub).

That changes how a score should be labelled. A result on the original task set is a different test from a τ³-bench result, so any comparison table should flag the version in its header. The same problem appears across agent benchmarks, as covered in Terminal-Bench 4.0 Explained: What Changed From 2.1 and 3.0, and Why Scores Do Not Carry Over.

What is pass^k, and how is it different from pass@k?

pass^k counts a task as solved only when the agent succeeds on all k independent attempts at that task. The metric was introduced to measure the reliability of agent behavior over multiple trials (arXiv). pass@k, the metric most readers know from coding benchmarks, counts a task as solved if at least one of k attempts succeeds.

If each trial succeeds independently with probability p, pass^k equals p^k. That is arithmetic, not a measured result, but it shows the shape of the curve. Any p below 1 makes the score fall as k rises, and a lower p makes the fall steeper. The metric works because it does not give credit for a lucky run.

Why does a repeated run expose inconsistency?

A single success can hide inconsistency. The paper's framing is that real reliability means an agent succeeds across multiple, slightly different conversational paths for the same task. An agent that succeeds once may still behave inconsistently when the same intent arrives through a different conversation (alphaXiv).

The paper's headline result follows from that framing. It reports that even state-of-the-art function-calling agents, with gpt-4o named, "succeed on <50% of the tasks, and are quite inconsistent (pass^8 <25% in retail)" (arXiv). These are 2024 results for one model in the paper's setup. They describe gpt-4o in 2024, not the models shipping today.

The repeated-trial design is the useful part for readers. Two agents with the same single-run success rate can produce very different pass^k curves, so a single number hides the difference that pass^k is built to show.

Read the paper's two figures together. The first, under 50%, is the share of tasks the agent succeeded on. The second, under 25% at k=8, is the share of tasks it solved on all eight trials. Both are upper bounds in the paper's wording, so the exact gap between them is not stated.

How much should you trust a tau-bench score?

Treat a tau-bench score as a measure of one simulated setup, not as a general reliability rating. The benchmark's own description makes the main limitation visible: the user is a language model, not a customer. Results therefore depend on the simulator as much as on the agent under test.

Five checks come before a score is cited.

When you cite a tau-bench result, record five things alongside it: the task version (original or τ³-bench), the value of k, the model that simulated the user, the number of tasks, and the date of the run. Without them, a pass^k figure cannot be checked or compared.

The benchmark tells you how an agent behaved in two simulated domains. It does not tell you how the same agent will behave on your own users or data. For how the test harness alone can move an agent's score, see Your AI Coding Agent Benchmark Is Wrong If You Ignore the Harness.

What's still unclear?

• Current scores. The only figures here are the 2024 gpt-4o results. No current tau-bench leaderboard is included, so this post does not rank models.
• Other values of k and other models. No pass^k figures are given for k other than 8, or for models other than gpt-4o.
• τ³-bench results. τ³-bench exists, and its maintainers say it fixes the original tasks. This post gives none of its scores.
• The weakest point of the argument. The headline finding rests on one model in one 2024 paper. The metric is only as informative as the independence of its repeated trials, and how a given harness controls that is not covered here. For how to read scores that come with ranks and intervals, see How to Read AI Leaderboards: Elo, Confidence Intervals and the Ranks That Mean Nothing.