Humanity's Last Exam vs Agents' Last Exam: Which "Hardest AI Test" Matters?

Humanity's Last Exam (HLE) and Agents' Last Exam (ALE) are not rivals for the same job. HLE is 2,500 closed-ended expert questions scored in a single turn. ALE hands an agent a professional task in a virtual machine and grades the deliverable. As of 15 October 2026, the top ALE row on Snorkel's leaderboard is Claude Opus 5.5 (Max effort, Claude Code harness) at a 38.2% pass rate, while the organisers' HLE-Diamond no-tools leader is GPT-6 Astra at 59.9%.

Those two numbers answer different questions, and the sections below explain which one to trust for what. Both benchmarks are run by organisations that sell data or evaluation services, so this post covers conflicts of interest explicitly.

What does Humanity's Last Exam measure?

HLE measures expert-level knowledge and reasoning on short, verifiable answers. The arXiv abstract says it "consists of 2,500 questions across dozens of subjects," and lastexam.ai records the set as finalized on 3 April 2025.

According to the v7 paper, about 14% of questions need both text and an image, 24% are multiple-choice, and the rest are exact-match. Questions came from nearly 1,000 subject experts at over 500 institutions in 50 countries, with a $500,000 prize pool.

The adversarial filter matters for interpretation. Each question was first tested against several frontier models of the time and was kept only if they failed. The paper reports over 70,000 attempts that produced about 13,000 questions that stumped the models and went to expert review. So HLE difficulty is defined relative to 2024-era models, not to humans or to current models.

How is Humanity's Last Exam scored?

Accuracy is the headline metric, with RMS calibration error as a second metric. The v7 paper says it uses o3-mini as the judge to check answers against model predictions while accounting for equivalent formats, and models are asked to state a confidence from 0% to 100%.

There is a small inconsistency in the paper: one appendix passage refers to a GPT-4o judge, while the main text and the project page name o3-mini. The project page also says a private held-out set exists "to assess model overfitting," with no size given.

HLE-Diamond, released on 22 September 2026, is a 1,000-question refined subset: 500 reasoning and 500 knowledge questions, produced after what the organisers call a year-long cleaning process. Its models are evaluated at high reasoning effort.

What does Agents' Last Exam measure?

ALE measures whether an agent system, meaning a model plus harness, tools and memory, can finish long professional tasks with checkable outputs. The paper covers 55 sub fields grouped into 13 industry clusters, restricted to non-physical work and defined with reference to O*NET and SOC 2018. It credits 250+ industry experts and describes 1K+ tasks.

Each task runs in a remote virtual machine, Linux or Windows, with input, software, output and reference directories, according to the full text. The paper publicly releases only 150 of 1,490 task instances (about 10%) and holds the rest in a private pool.

Counts differ by source. Snorkel's pages say 147 public tasks and 1,500+ in the corpus. Snorkel cites 300+ experts where the paper says 250+. The paper's tier table (67, 55 and 38 tasks) sums to 160, not 150.

How is Agents' Last Exam scored?

Mostly with deterministic checks, not a judge model. The paper says ALE "deliberately avoids LLM-as-judge wherever a deterministic alternative exists." It uses deliverable-based or milestone-based checks against expert references and rubrics.

A common pattern is gate-and-score: if a required gate fails, the task score is forced to zero regardless of partial progress. That is why the leaderboard reports two numbers, a strict pass rate and a mean score. For Claude Opus 5.5 the same row reads 38.2% pass and 63.2% score.

Runs are expensive. Leaderboard rows list estimated cost and runtime totals, and the paper notes that, because of compute budgets, only a subset of configurations include repeated runs. The leaderboard shows no confidence intervals, so small gaps between rows should not be read as rankings. For why harness choice moves agent results, see our harness-effect analysis and the Claude Code vs Codex CLI cost comparison.

What do the current leaderboards show?

The two leaderboards are not comparable, and neither states a capture date for every row. The tables below are snapshots taken on 15 October 2026 from Snorkel and the HLE-Diamond blog, using the first (headline) table Snorkel shows. All figures are published by the benchmark organisers or hosts. Neither page says who ran the evaluations.

ALE (Snorkel leaderboard, accessed 15 October 2026; selected rows. Grok 4.3 is ranked 43rd of 43 in that view, and the effort cell for some rows lists several levels on the page)

HLE-Diamond, no tools, high reasoning (organisers' blog, 22 September 2026; same figures on BenchLM, checked 5 October 2026)

The ordering flips between the two. GPT-6 Astra leads HLE-Diamond, while Claude Opus 5.5 leads ALE. The ALE rows also differ in harness and effort, so the model is not the only variable. The Diamond blog additionally reports web-and-code results, with GPT-6 Astra at 82.9% and Claude Opus 5.5 at 73.9%, which should not be placed next to the no-tools column.

One more caution: the Snorkel page shows several views of the data, and the same model can carry different numbers in different views. The table above uses the first view only. For how a single model's score can shift with harness choice, see ARC-AGI-3 and the harness gap.

Who runs each benchmark, and are there conflicts of interest?

Both have a commercial adjacency to data and evaluation work. HLE is organised by the Center for AI Safety and Scale AI, per the paper; Scale AI sells data and evaluation services. ALE comes from UC Berkeley RDI with Snorkel AI. Snorkel's pages offer to help teams "label, refine, and evaluate high-quality, domain-specific datasets," and the paper credits Snorkel for financial and credit support.

Neither conflict means the numbers are wrong. It does mean the scores come from the benchmark owners' own pipelines, and neither source states a third-party replication. HLE's own authors list also includes researchers affiliated with Anthropic, OpenAI, Google DeepMind and Microsoft Research, according to lastexam.ai, which is relevant because those labs' models sit on the leaderboard.

What are the weaknesses of each?

HLE's main weaknesses are label noise, contamination and saturation pressure. FutureHouse reported on 23 July 2025 that 29 ± 3.7% (95% CI) of text-only chemistry and biology questions had answers with directly conflicting evidence in peer-reviewed literature, from a 321-question subset. The HLE team's three-expert re-review found about 18% problematic in a bio/chem subset, with at least one reviewer disagreeing 25% of the time, per FutureHouse's update. The public set has also circulated since early 2025, and the organisers keep a private set partly for that reason. HLE-Diamond exists to provide a cleaner migration path before frontier models reach the noise ceiling, in the organisers' words.

Saturation is already visible. According to BenchLM, fifteen models had passed 50% on the original HLE as of July 2026, with the top tool-assisted score at 64.7%. See our benchmark saturation review for the wider pattern.

ALE's weaknesses are different. Public and private counts are inconsistent across sources. The hardest-tier average full pass rate is "below 1%" in the arXiv abstract but 2.6% on Snorkel's research page. The leaderboard shows models absent from the paper's own table, and the paper's top overall pass rate is 24.0% (Codex with GPT-5.5), well below the leaderboard's 38.2%. Cost and runtime vary by an order of magnitude between rows, so a pass rate without its cost column is incomplete.

Which one matters, and for what?

Use HLE-Diamond for questions about expert-level knowledge and reasoning on short answers, and ALE for questions about long, tool-using professional work. A rough credibility read, which is a judgment, not a measured score:

For agent buying decisions, ALE is closer to the work, but its harness and effort columns mean you are ranking configurations, not models. For a coding-specific view, see Terminal-Bench 4.0 explained.

What's still unclear?

Several points are open as of 15 October 2026.

• Neither leaderboard page states whether its scores were produced by the organisers, by the labs or by third parties, and ALE rows carry no per-row capture date.
• ALE's public task count is 147, 150 or 160 depending on which source and table you use.
• The HLE-Diamond blog reports no confidence intervals, and the size of HLE's private held-out set is not disclosed.
• The two ALE pass-rate figures for the hardest tier (below 1% and 2.6%) conflict, possibly because of different paper versions.
• The weakest part of any "which matters" answer is that nobody has published the two benchmarks run on the same models under the same harness and effort. The comparison here is across leaderboards, not within one experiment.

Disclosure: Model Report Card is affiliated with Humanbit, a data labelling and evaluation company, and this post covers benchmarks built by organisations in that market.