ElevenLabs Says 75% of Listeners Prefer v4. Here's What That Test Hides

ElevenLabs says about 75% of listeners preferred Eleven v4 in blind head-to-head tests against competing models. That is a company-run result, and the post does not publish a sample size, grader pool or confidence interval. The independent Artificial Analysis arena also ranks Eleven v4 first, but its Elo gaps imply a much smaller preference margin, closer to 56% than 75% (our calculation, below).

What exactly did ElevenLabs claim?

ElevenLabs claims Eleven v4 was "preferred by ~75% of listeners in blind head-to-head tests over competing models," per the launch post published 28 September 2026 and last updated 12 October 2026. The claim is company-reported (lab-reported), not independent.

The footnote is the only methodology on the page. It says the testing was done in September 2026 against four systems: Cartesia Sonic 3.6, Inworld TTS-2, Google Gemini 3.8 Flash-Lite TTS and Google Gemini 3.8 Flash TTS. For each pair, graders heard the same line from Eleven v4 and one competitor, blind, and judged which was more expressive and which sounded more natural. Ties counted as half.

A chart on the same page carries the alt text "Eleven v4 wins 65%–81% of blind head-to-head TTS preference tests." So the headline 75% sits inside a 65% to 81% spread across pairs. The post does not say whether 75% is a pooled rate or an average of per-competitor rates, and it does not map any number to a named competitor in text.

What does the test not disclose?

The page gives no sample size, no grader pool and no confidence intervals. Tech Times made the same point on 30 September 2026, writing that no sample size, methodology or external peer review had been published alongside the figure.

Here is what is missing, and why each gap matters for a preference test:

To show why n matters, take a 75% win rate. Under a simple normal approximation, the 95% interval is about ±8.5 points at n=100, ±4.2 at n=400 and ±2.7 at n=1,000. Those are illustrative figures, not ElevenLabs data. Without n, a reader cannot tell which of those regimes applies.

Who picked the competitors, and who was left out?

ElevenLabs picked the comparison set, and that choice shapes the headline. The four named competitors are not the strongest rivals on the independent board. On Artificial Analysis's Provider Voice leaderboard, captured 16 October 2026, Qwen-Audio-3.1-TTS-Plus ranks third at 1,296 Elo, ahead of Sonic 3.6 and both Gemini models. It is not in ElevenLabs's list.

This is not evidence of cherry-picking. ElevenLabs may have tested models available to it, or it may have prioritised Western incumbents. The point is narrower: "75% over competing models" describes four chosen systems, not the field. On Controlled Voice, Artificial Analysis reported on 28 September 2026 that Qwen-Audio-3.1-TTS-Plus (1,178 Elo) was ahead of Eleven v4 (1,157 Elo).

The latency claims use a different comparison set again. The launch post's latency footnote lists Cartesia Sonic 3.6, xAI TTS, Gemini 3.8 Flash-Lite TTS and OpenAI GPT-4o mini TTS, with network latency measured and removed. That is lab-reported, and it is a separate test from the preference study.

What does the independent leaderboard say?

Artificial Analysis ranks Eleven v4 Turbo and Eleven v4 at the top of its Provider Voice Arena. The arena uses blind pairwise votes from listeners who hear the same text and pick which sounds more natural. This is independent of ElevenLabs, though the pages reviewed do not describe who runs the voting pool or how voters are recruited.

Snapshot from the Artificial Analysis Provider Voice leaderboard, captured 16 October 2026. Metric: Elo from blind pairwise votes, English (US and UK accents), 8 voices per model for the rows below. All rows independent.

Two cautions on reading this table. First, the board moves as votes accrue: at launch, Artificial Analysis reported Eleven v4 at 1,319 (+19/-19) across 1,674 appearances, versus 1,329 across 2,121 samples now. Second, Eleven v4 and v4 Turbo have fully overlapping intervals, so the order between them is not resolved. The v4 interval (1312 to 1346) meets the Qwen interval (1280 to 1312) only at its edge.

How does an Elo gap compare with a 75% win rate?

Under standard Elo scaling, the gaps on the independent board imply preference rates in the mid-50s, not 75%. Using the usual 400-point logistic formula, a 44-point gap (v4 at 1,329 vs Sonic 3.6 at 1,285) gives about a 56% expected win rate. A 54-point gap over Gemini 3.8 Flash TTS (1,275) gives about 58%. These conversions are our calculation and assume Artificial Analysis uses that scale, which the reviewed pages do not state.

The two numbers are not strictly comparable, which is the point:

• ElevenLabs's graders judged two questions (expressive and natural), while the Artificial Analysis voters judge one (more natural).
• ElevenLabs scored ties as half, and the arena treatment of ties is not described on the pages reviewed.
• The ElevenLabs test used four selected competitors. The arena aggregates votes across many pairings, and Elo is a fitted rating, not a raw win rate against one rival.
• The arena covers English with US and UK accents, using eight voices per model. The ElevenLabs post claims 90+ languages but the preference test does not say which languages were used.

A gap of 56% versus 75% could be explained by criteria (expressiveness favours v4's design), by line selection, or by the rival set. With the information published, it cannot be attributed to any one of them.

Does v4 still win when the voice is held constant?

Not consistently. The Provider Voice arena lets each vendor supply its own native voices, so a model can rank high partly because of voice choice. Artificial Analysis's Controlled Voice arena standardises this using the same eight cloned voices (US and UK English).

On that board, Eleven v4 ranked second at launch: 1,157 Elo (+16/-16) across 1,483 appearances, behind Qwen-Audio-3.1-TTS-Plus at 1,178. By 9 October 2026, Artificial Analysis reported HeyGen Voice first at 1,201, then Qwen-Audio-3.1-TTS-Plus at 1,182, Eleven v4 Turbo at 1,166 and Eleven v4 at 1,162. The cloned-voice gap between first and fourth is about 39 points.

This matters for the 75% claim because ElevenLabs's own post emphasises voice cloning, and the claim does not say whether the preference test used native or cloned voices. On the evidence that is public, the independent lead for v4 is largest on native voices and smaller or absent when voices are controlled.

Should you trust the 75% number, and for what?

Treat it as a marketing signal that v4 beat four chosen rivals in expressiveness and naturalness, and not as a measured population preference. The independent arena supports the direction (v4 is at or near the top) but not the size.

Credibility scorecard (our assessment, 1 to 5, where 5 is best):

A practical use: if you are choosing a TTS model for an expressive use case, use the Elo ranking to shortlist and run your own blind test on your scripts, voices and languages.

What would a better version of this test look like?

A defensible preference study would publish the following, in a table or appendix:

1. The number of comparisons per competitor, with win rates and 95% intervals for each pair.
2. Who graded (employees, crowd, trained listeners) and how they were screened.
3. The script set, how lines were chosen, and whether lines were chosen before seeing outputs.
4. The voices used per system, and whether native or cloned voices were compared.
5. Whether v4 audio tags were used, and whether rivals received equivalent direction.
6. Languages covered, since the claim of 90+ languages is not matched by disclosed test coverage.
7. A separate result for each criterion (expressive, natural) rather than a combined one.
8. Raw judgments or at least a pre-registered analysis plan.

What's still unclear?

• Sample size and graders. The post does not say how many comparisons were run or who judged them.
• How 75% was computed. The text says about 75% while the chart says 65% to 81%. Pooled versus averaged is not stated.
• Which competitor produced which rate. Not published in text.
• Whether the arena Elo scale maps to win rates as assumed. Our 56% to 58% estimate depends on standard 400-point scaling.
• Arena voter pool. The Artificial Analysis pages reviewed describe blind pairwise voting but not who the voters are.
• Drift. Both boards changed within three weeks of launch: v4 moved from 1,319 to 1,329 on Provider Voice and fell from second to fourth on Controlled Voice (by Artificial Analysis's 9 October post).
• Weakest point of this analysis. The comparison of 75% with about 56% uses different criteria, different pools and an assumed Elo scale. It shows the claim is hard to verify, not that it is false.

For context on the company: ElevenLabs raised $500M at an $11 billion valuation on 4 February 2026. A well-funded vendor publishing its own preference test is normal practice in the sector, and it is the reason independent arenas exist.