AI Judges Picked Their Own Answers 58% of the Time vs 34% for Humans, Arena Reports

Arena reports that across 12 judge models, a model picked its own answer 58% of the time while people picked that same answer 34% of the time. That is the source of the headline number: a study by an evaluation vendor, published on 30 September 2026 (last updated 6 October 2026), covering 1,460 blind Text Arena battles and 34,580 valid verdicts. The 58% is real, but it is an average over very different judges, and the right comparison is the 34% human baseline, not a coin flip.

What did Arena actually measure?

Arena gave 12 models 1,460 real English-language Text Arena battles from 1 June to 23 September 2026, hid the source model names and the human vote, and asked each model to pick the better answer. Each judge saw every battle twice with the answer order swapped.

The self-preference charts only use battles where the judge wrote one of the two answers. Arena's scoring rules for each battle, per its methodology section:

• Same answer both times: that answer is the judge's winner.
• A winner once and a tie or "both bad" once: the winner counts.
• A different answer each time: half a pick in the self-preference and family charts, and no pick in the shared-taste chart.
• No winner either time: the battle is left out of the self-preference and family charts.

Of 35,040 planned verdicts, 34,580 were valid. The rest were unusable replies, content-filter blocks or failed calls. The judge prompt text is not reproduced in the article.

How is the 58% scored, and why is 34% the right baseline?

The 58% is the share of battles in which the judge picked its own answer, averaged over 12 models. Humans can call a tie or "both bad", so neither number is bounded by 50%. For GPT-6 Astra, the model picked itself 88% of the time and the opponent 2%, with the remainder going to ties or split verdicts.

The 34% is how often human voters picked those same models' answers in the same battles. Arena frames the gap as "about 70% more often" (58 divided by 34 is roughly 1.7). Using humans as the baseline is Arena's way of controlling for answer quality, which is the main confound in this literature (see the weaknesses section).

Which judges favour themselves, and by how much?

Six of 12 judges clearly favour themselves, and the effect is largest in the OpenAI models. All figures below are Arena-reported, from Text Arena battles dated 1 June to 23 September 2026.

Arena says the top six clearly favour themselves, the intervals for GLM through DeepSeek V4 Flash include zero, and MiniMax M3 votes against itself. So "AI judges pick themselves" describes half the panel. The other half is statistically indistinguishable from humans or harsher on itself.

Does the bias extend to model families?

Yes for the OpenAI models, to a lesser degree for Fable and Opus, and not for the DeepSeek models. Arena reports that GPT-6 Astra, GPT-5.6 Sol and GPT-5.6 Luna rated other OpenAI models 37 points above people, against 6 points from the other nine judges. The same three rated everyone else 15 points below people.

Fable and Opus showed a milder version, +28 for each other against +15 from other judges. The DeepSeek models rated each other 16 points below people. Arena found no national loyalty: Chinese judges were 6 points more generous than people to American answers and 6 points harsher on other Chinese labs.

An independent preprint on four open-weight families (Llama 3.1, Qwen 2.5, Gemma 2, Yi 1.5) reports a same-family lift of 3.4 to 8.4 percentage points across 9,312 judgments. That is a v1 preprint on small open models, so treat it as supporting, not confirming.

Do AI judges just share a taste that differs from users?

Arena says they do, and this may matter more than self-preference. The average judge sided with the other AI judges 79.4% of the time but with the human voter 56.9% of the time, and every judge agreed with other AIs more than with humans, by 18 to 27 points.

Even unanimity was weak evidence. In 393 battles where every AI judge picked the same answer, the person agreed 60.6% of the time. Judges also force winners: people called a tie or "both bad" in 32% of battles, while Sol picked a winner 96% of the time. If your pipeline treats a judge's close call as a clear win, it will overstate separation between models.

What did earlier research find?

Self-preference has been reported since the original MT-Bench work, but its causes remain disputed. The table summarises the main papers, all author-reported.

The Panickssery and Wataoka results point to different mechanisms (recognising oneself versus finding familiar text predictable). Both can be true at once, and neither tells you how large the effect is on a given leaderboard.

What are the weaknesses of the 58% result?

The main weakness is that "own answer" and "better answer" are entangled. Chen et al. found much of the preference aligns with objectively superior answers. A model that is genuinely stronger will win more of its own battles regardless of identity.

Arena's human baseline partly addresses this, but not fully. Human voters are one voter per battle, and humans disagree with each other. A separate paper argues the bias claims themselves need a quality control: only 51% of examples in previous findings retain statistical significance against an evaluator-quality baseline, covering 89.6% of total self-preference probability mass. That paper does not analyse Arena's study.

Other limits on Arena's design:

• Scope. English-language Text Arena prompts only, and one window (1 June to 23 September 2026).
• Reasoning settings vary. Judges ran at different settings (Max, xHigh, High, Medium, Low), which confounds model with effort.
• Position effects. The open-weight preprint above found 55.4% of AB/BA pairs reverse. Arena's swap-and-reconcile design handles this, but the "half a pick" rule for contradictory verdicts is a choice that affects the totals.
• Refusals and filters. Some verdicts were lost to content-filter blocks and failed calls, a refusal effect also seen in other benchmarks (see Mistral Large 4 "Wins" a Benchmark Because Rivals Refused to Answer).

Who ran the study, and is there a conflict of interest?

Arena ran it, and Arena sells human-preference evaluation. On 8 October 2026 it announced a $200 million Series B at a $3.1 billion valuation, co-led by Lightspeed Venture Partners and Khosla Ventures, and says it has exceeded $100M in annualized revenue (company claim).

A finding that AI judges diverge from human voters supports the case for human-vote leaderboards. That does not make it wrong, and the article discloses its method, interval widths and exclusion rules. It does mean the result is a vendor study, not an independent replication. For how operator incentives and setup choices affect benchmark results, see Your AI Coding Agent Benchmark Is Wrong If You Ignore the Harness and CursorBench 4.0 Explained.

Can the problem be fixed?

It can be reduced, but it has not been eliminated in anything cited here. Reported mitigations:

• Longer reasoning before judging. Generating a long chain-of-thought before evaluation effectively reduced harmful self-preference.
• Ensembling. Ensembling multiple judges helps mitigate the bias without fully eliminating it.
• Structured evaluation. A multi-dimensional strategy reduced the bias by 31.5% on average across 20 LLMs. The same paper found advanced capability is often uncorrelated, or negatively correlated, with low self-preference, so picking the strongest judge is not a fix.
• Panel design. The open-weight preprint found panel composition changes 18.5% of pairwise outcomes.
• Human spot checks. Arena's advice: do not treat the author as a neutral reviewer, and check verdicts against real human preferences.

What's still unclear?

Several things are open:

• The headline average is stated inconsistently. Arena's key takeaways say judges picked their own answer 23 percentage points more often than people, while 58 minus 34 is 24. The simple mean of the 12 plotted per-model gaps is 16.0 points (arithmetic from the chart values). Arena does not explain the difference, and weighting or tie handling is the likely source.
• Own-pick rates are reported for only five judges in the article text. Per-judge battle counts and per-model sample sizes are not stated, so the interval widths cannot be checked.
• Bias versus quality is not fully separated. Human votes are a proxy for quality, not ground truth.
• Replication. None of the sources cited here independently replicates the Arena numbers as of 15 October 2026.
• Judge prompts and temperature settings are not in the article text. Harness sensitivity, the main known confound in agent benchmarks (see Claude Opus 5.5 on Terminal-Bench 4.0), is untested here.

Should you trust LLM judges, and for what?

Trust them for cheap relative screening across families, and not for ranking models that include the judge's own family. A reasonable use pattern follows from the evidence above.

Last checked: 15 October 2026.