How to Read AI Leaderboards: Elo, Confidence Intervals and the Ranks That Mean Nothing

On the Arena text leaderboard as of 15 October 2026 (page date, 8,834,211 votes across 416 models), the model in rank 1 has a rank spread of 1 to 3, and the model in rank 40 has a spread of 23 to 52. A rank is a point estimate. The spread is the claim you can defend.

This post explains how Arena scores are built, how confidence intervals turn into rank spreads, and which ranks in a table carry no information. It also covers what sits underneath the number: style effects, private testing and sample size. The reading rules apply to most pairwise-preference and benchmark tables.

What does an Arena score actually measure?

It measures how often a model wins blind, pairwise human votes, fitted with a Bradley-Terry model. The Chatbot Arena paper describes anonymous, randomized battles in a crowdsourced setting, with over 240K votes at the time of writing (submitted 7 March 2024).

Arena started with online Elo. In December 2023 it moved to a Bradley-Terry model, which fits all votes in one batch regression instead of updating ratings game by game. Arena says it reports the coefficients multiplied by 400 and shifted by 1000, so they land in the same range as chess Elo, and calls the transformation purely cosmetic.

Two consequences follow. A score has no meaning outside its own vote pool, task mix and date. And "Elo" on this leaderboard is shorthand for a regression coefficient, so the uncertainty comes from the vote count and the fit, not from game order.

How big is a rating gap in win probability?

Smaller than most readers assume. Under the classic Elo convention, a player 100 points higher is expected to score 64%, according to LumiChess. Arena's 2023 write-up states that its Elo formula uses the logistic curve with base 10.

At the top of the 15 October 2026 table, rank 1 leads rank 2 by 15 points (1524 vs 1509), and the two intervals overlap. Arena does not publish a win-probability conversion for its current scores, so this post does not put a percentage on that gap.

How do confidence intervals and rank spreads work?

Arena reports a score with a ± interval, and since 14 November 2025 it also reports a raw rank and a rank spread. Per Arena's ranking method post, the raw rank is the rank of the score with no ties, and the spread is the best and worst rank a model could have given all models' confidence intervals.

The definitions are exact:

• Best rank = 1 + the number of models whose lower CI endpoint is above this model's upper endpoint.
• Worst rank = 1 + the number of models whose upper CI endpoint is above this model's lower endpoint.
• Tie rule: two models are tied if their rank spreads overlap.

Before that change, Arena says it reported only the optimistic rank, which favoured models with large intervals such as new models. The underlying idea comes from the original paper, which defines an approximate ranking from the confidence set using the same infimum and supremum logic. The paper chose sandwich standard errors over the pivot bootstrap for the intervals, citing their smaller size in large samples.

What does the leaderboard look like right now?

The snapshot below is from Arena's overall text leaderboard, page dated 15 October 2026. It is an independent, crowd-preference ranking operated by Arena, with no task harness and no benchmark version: the unit is a blind human vote. Rank spread is shown as low to high.

Read the top rows as a group. If the ± values are symmetric half-widths, rank 1 spans about 1515 to 1533 and rank 2 spans about 1498 to 1520, so the intervals overlap. Every spread in the top seven rows includes both rank 2 and rank 3. Rank 1 is the best point estimate, not a separated winner.

Why do raw ranks mean nothing outside the top few?

Because many models sit within a few score points of each other, so their intervals overlap heavily. Ranks 39 and 40 share a score of 1474, yet their spreads run from 16 to 61 and from 23 to 52. Position 40 on this page is compatible with any true rank between roughly the mid-twenties and low fifties.

Vote count drives the width. The rank 39 row has 3,934 votes and a ±10 interval. The rank 40 row has 65,172 votes and ±4, with the same score. The rank 5 row (81,394 votes, ±3) is the tightest in the table above.

Two reading rules follow:

1. Compare spreads, not raw ranks. Overlapping spreads mean the data cannot separate the models.
2. Check the vote count and any Preliminary label before quoting a rank. The 15 October 2026 page labels both Gemini 4 Argon (High) and Gemini 3.8 Flash (High) as Preliminary.

The leaderboard page does not define "Preliminary" in its table text. In 2025, Arena said it planned to mark pre-release-tested scores provisional until another 2,000 fresh votes when more than 10 models were tested in parallel. Do not assume the two labels mean the same thing.

What biases sit underneath the score?

Three are documented: style, selection and data access. Each shifts scores without a visible interval.

Style. Arena's style control post added answer token length and counts of markdown headers, bold elements and lists as features in the regression. Rankings changed: GPT-4o-mini and Grok-2-mini fell below most frontier models, while Claude 3.5 Sonnet, Opus and Llama-3.1-405B rose. The authors say the analysis is observational and that confounders remain, for example chain-of-thought raising both length and quality. The leaderboard page lists Style Control among its adjustments, and it is not clear which setting the table above uses.

Selection. The Leaderboard Illusion (Cohere Labs and academic coauthors, data as of 23 April 2025) reports 27 private variants tested by Meta before Llama-4. It argues that private testing with the option to retract scores lets providers pick the best variant, which biases published scores.

Data access. The same paper estimates that Google and OpenAI received 19.2% and 20.4% of arena data, against 29.7% for a combined 83 open-weight models. It also reports that 205 of 243 public models were silently deprecated, versus 47 officially listed, and argues deprecation can violate Bradley-Terry assumptions.

Who runs it, and do the critics have a stake?

Arena operates the leaderboard, and it decides which models are tested, retired and labelled, so it is both referee and platform. The Leaderboard Illusion's authors are listed at Cohere Labs and Cohere, and Arena says it helped Cohere evaluate pre-release models, so neither side is a disinterested party.

Arena disputes parts of that paper. These are party claims on both sides:

I cannot adjudicate the numerical disagreement from public documents. What is not in dispute is that private pre-release variants exist and that Arena planned to state its multiple-variant policy and add provisional marking.

How should you read benchmarks that are not Arena?

Most benchmark tables have a point estimate and no interval, which is the opposite problem. Anthropic's Evan Miller noted in his error-bars paper that error bars remain noticeably absent from traditional question-and-answer evals, while Chatbot Arena popularised confidence intervals.

His recommendations, in order:

1. Compute the standard error of the mean with the Central Limit Theorem, and report it beside the mean.
2. Use clustered standard errors when questions come in related groups.
3. Reduce variance by resampling answers and analysing next-token probabilities.
4. When comparing two models, test question-level paired differences rather than two summary scores.
5. Run a power analysis to see whether the eval can detect the difference you care about.

A 95% interval is the mean plus or minus 1.96 times the standard error, per the paper. Benchmark scores also depend on version and harness, which intervals do not capture. See Terminal-Bench 4.0 Explained on version drift, Your AI Coding Agent Benchmark Is Wrong If You Ignore the Harness on harness effects, and our post on LLM judge self-preference bias for judge bias. For saturation, see AI Benchmark Saturation 2026.

What's still unclear?

• The CI level. The leaderboard shows ± values but does not state the confidence level behind them. The rank-spread logic works at any level, but the width does not.
• Which view the table uses. Whether style control is applied to the 15 October 2026 snapshot is not clear from the page.
• What "Preliminary" means on the page. Arena's 2025 provisional rule is a different mechanism, and the page does not define the label.
• Selection bias size. The Leaderboard Illusion and Arena disagree on the open-model share and on the size of the pre-release boost. Both are interested readings of the same system.
• Weakest point of this post. It reads one leaderboard on one date. Spreads move as votes accumulate, so the 1 to 3 and 23 to 52 figures are a snapshot, not a constant.