OSWorld Explained: Why Computer-Use Agent Scores Don't Match Across Labs

OSWorld-Verified scores are quoted by many labs, and the numbers do not line up. As of 15 October 2026, BenchLM lists 37 models and says only 3 of 37 rows come from the benchmark owner or an independent run. The other 34 are provider self-reports, and Anthropic's own system card says it had been underreporting OSWorld across its model families until it changed its harness.

This post covers what the benchmark measures, how it is scored, who runs it, and which settings move the number. All scores below are lab-reported unless marked otherwise.

What does OSWorld measure?

OSWorld measures whether a multimodal agent can finish open-ended tasks on a real computer. The original paper (arXiv 2404.07972, submitted 11 April 2024) describes 369 tasks across real web and desktop apps, OS file I/O, and workflows that span several applications. Each task has an initial-state setup and a custom execution-based evaluation script.

The agent works in a live virtual machine through mouse and keyboard. In the original baselines the screenshot was 1920 x 1080, and agents could also receive an accessibility tree or Set-of-Mark overlays. The project page lists 134 execution-based evaluation functions.

Early results were low: the best model in the paper scored 12.24%, against a human result of over 72.36%.

How is OSWorld scored?

The metric is task success rate, checked by scripts that inspect the final machine state. The original paper's baselines used a max step limit of 15, temperature 1.0 and top-p 0.9. Those are not the settings labs use now.

Verified changed some scoring too. XLANG Lab says it added fuzzy 0-to-1 scoring for some document comparisons instead of strictly binary checks, and added alternative valid solution paths to the gold answers. It says it mostly edited evaluators rather than task text, to keep score continuity.

Two reporting choices matter on top of that:

• Task count. The project page says 8 Google Drive tasks can hit setup problems, and that running 361 tasks instead of 369 is officially permitted. Anthropic's figures are on 361 tasks.
• Run policy. Anthropic reports first-attempt success rate averaged over five runs. Xiaomi's published number comes with no stated step budget or run count, per Steel.dev.

No confidence intervals accompany the leaderboard rows. The weaknesses section gives a rough estimate.

Who runs OSWorld, and are there conflicts of interest?

OSWorld comes from XLANG Lab at the University of Hong Kong, with co-authors at Salesforce Research, Carnegie Mellon and the University of Waterloo, per the project page. XLANG announced OSWorld-Verified on 28 July 2025 and says a roughly 10-person team spent about two months fixing 300+ reported issues.

The conflict question is about the feedback loop. XLANG's acknowledgements thank Moonshot AI, HUD, OpenAI, ByteDance Seed TARS, Anthropic, Simular and the HKU Data Intelligence Lab for feedback and for taking part in fixes. That is normal for a community benchmark, but the labs that report scores also helped shape the evaluators.

The leaderboard process is semi-open. To be listed, XLANG says submitters schedule a meeting so the maintainers can run the agent code on their side. Submitters must allow the agent implementation to be disclosed and provide a report, though they may keep the model API private. Trusted institutions may share trajectories instead. Verified trajectories are hosted on Hugging Face.

The site separates general models, specialized models and agentic frameworks. That matters because a framework (for example a planner plus a grounder) is a different object from a base model.

What does the OSWorld-Verified leaderboard look like in October 2026?

The top rows sit at 85% to 86.1%, and nearly all are lab-reported. The table mixes provenance on purpose, so read the last two columns first. Rows follow Steel.dev (captured 11 October 2026) unless noted. The GPT-5.5 row is the figure Anthropic cites in its system card table.

BenchLM says the top three are within 1.1 points of each other, and cautions that a provider self-report is displayable evidence, not an independent run. This post does not reproduce the official OSWorld-Verified table, which the maintainers host on the project page. Check it directly for owner-run numbers.

Anthropic's table also gives Gemini 3.1 Pro at 76.2% and Gemini 3.5 Flash at 78.4%. Anthropic says competitor figures come from the developers' published system cards or leaderboards, so these are not matched runs.

Why do OSWorld scores differ across labs?

Because the harness is part of the score. Step budget, tokens per turn, tool bugs, run count and task count all change the percentage, and labs rarely hold them constant. The clearest evidence is labs restating their own numbers.

The Opus 4.7 row is the cleanest case. Anthropic's system card says the changes were made to reflect real-world performance better, and that re-running earlier models showed it had been underreporting OSWorld performance across its model families. The same model moved from 78.0% to roughly 82% on a changed harness. That shift is several points, on the order of the gaps between rival models in the table above.

Vellum, an outside analyst, notes the restatement makes the Opus 4.7 to 4.8 delta look smaller than it did under the original scoring. That is Vellum's reading, not an Anthropic claim.

Step budget is the other big lever. XLANG's rerun found o3 ranged from 9.1% to 23.0% depending on step budget. OpenAI's January 2025 Computer-Using Agent post reported 38.1% on OSWorld and said performance improved when more steps were allowed. Steel.dev lists the same original CUA at 32.6% under a 50-step setting taken from the Agent S2 paper's baseline table. The two figures use different step budgets, so neither is wrong.

For scale, the original paper used 15 steps, and Anthropic's current runs use 100. A score without its step budget is not a complete score.

Where do lab-reported and independent scores disagree?

No clean matched pair is documented for the 2026 frontier models: no owner-run score and lab score for the same model, effort level and step budget. What is documented is a within-lab gap between old and new harnesses, and a leaderboard where independence is the exception.

The disagreement that can be documented is between harness versions, not between lab and referee. Anthropic's old and new numbers for one model differ by several points. Steel.dev notes that self-reported rows can use different max steps, OS images and tool permissions.

XLANG states the underlying problem in its own words: when everyone runs a private copy, some people modify tasks to score higher, and results become opaque and incomparable. That is the stated reason for the Verified public evaluation platform.

Anthropic's system card says its run followed the default 1080p setting and a 100-step maximum. Its changes were a zoom tool fix and a larger token limit per turn. Steel.dev labels these rows as running on Anthropic's revised harness, which is the right way to read them: same benchmark name, different scaffold.

What are the weaknesses of OSWorld?

The main ones are sample size, saturation, public tasks and harness sensitivity. None is unusual for an agent benchmark, but together they limit what a one-point gap can mean.

Sample size. With 361 tasks and a score near 85%, the standard error from task sampling alone is about 1.9 points, so a 95% interval is roughly plus or minus 3.7 points. This is my back-of-envelope binomial calculation, not a published interval. Averaging five runs reduces run-to-run noise but not task-sampling noise. A top-three spread of 1.1 points is inside that band.

Saturation. The original human figure was 72.36%, and OpenAI's post rounds it to 72.4%. Top self-reported scores of 85% to 86.1% now exceed it. The human number predates Verified's task fixes, so the comparison is loose, but it suggests the benchmark no longer separates frontier models well.

Public tasks. The tasks are public, and BenchLM labels OSWorld-Verified a public benchmark set. In 2025 XLANG said the success of UI-TARS and OpenCUA stems primarily from extensive human computer-use trajectory data. That is a training-data observation, not proof of contamination, and no contamination measurement is cited here.

Environment drift. XLANG documents anti-crawling pages, CAPTCHAs, IP blocks and changed site layouts that broke tasks. Live-web dependence means a task can change after a score is published.

Scope. BenchLM notes that fixed applications and configured start states do not reproduce production logins, permissions, network failures or app updates.

Should you trust an OSWorld score, and for what?

Trust it for rough capability tiers and for within-harness comparisons, not for ranking models within a few points. The scorecard below is my editorial judgment from the evidence above, scored 1 (weak) to 5 (strong).

Use a lab's number only with its step budget, task count, run count, effort level and tool changes. If any are missing, treat the figure as marketing-grade.

Is OSWorld 2.0 a replacement?

No. It is a separate, much harder protocol, and its scores cannot be read against OSWorld-Verified. The OSWorld project page announced OSWorld 2.0 on 26 June 2026.

The paper describes 108 long-horizon workflows. Each takes humans a median of about 1.6 hours, and requires an average of 318 tool calls with Claude Opus 4.7 at maximum thinking, compared with about 30 in OSWorld 1.0. Under its primary binary-completion metric at 500 steps, the authors report Claude Opus 4.8 with maximum thinking and batched tool calls at 20.6% completion, with a 54.8% partial score. GPT-5.5 plateaus near 13%. These are paper-reported results, not independent ones.

BenchLM says plainly that a score change between the two pages is not a like-for-like comparison. A better OSWorld report would state the protocol, give step budgets, include confidence intervals, and publish the harness diff whenever a lab restates a number.

What's still unclear?

• Which Opus 4.7 restatement is right. Anthropic's news-page footnote says 82.3%, while its system card table and Vellum say 82.8%. Both are Anthropic documents, and neither explains the 0.5-point difference.
• How Alibaba and Xiaomi ran their evals. Steel.dev marks Qwen3.8 Max harness code as uncertain and says Xiaomi did not state step budget, run count or harness settings.
• Whether other labs have the same underreporting. Anthropic attributes its low earlier numbers to a zoom tool bug and a 16K per-turn token cap. Whether rivals' harnesses carry comparable issues is not documented.
• Owner-run scores for the newest models. The official table is the right referee, but a matched owner-run versus lab-run comparison for the same 2026 model is not documented.
• Secondary-source figures. Several leaderboard details (Mythos Preview 85.4%, Sonnet 4.6 78.5%, Mythos 5 and Fable 5 at 85.0%) are as reported by Steel.dev's aggregation of Anthropic system cards. Treat them as according to Steel.dev.