Artificial Analysis STT Leaderboard: Can a Benchmark Built on Its Own Dataset Be Fair?

Artificial Analysis (AA) has made half of its speech-to-text score depend on a dataset it developed and verified itself. Since AA-WER v2.0 on 17 February 2026, AA-AgentTalk (469 samples, about 250 minutes) carries 50% of the index. As of 16 October 2026, the leaderboard lists two models tied at 1.7% AA-WER. This teardown looks at whether that design is fair, using only what AA and third parties have published.

What does AA-WER measure?

AA-WER measures word error rate for speech-to-text models, combined across three datasets. According to AA's v2.0 article, the index covers about 8 hours of audio, weighted 50% AA-AgentTalk, 25% VoxPopuli-Cleaned-AA and 25% Earnings22-Cleaned-AA.

Source: AA-WER v2.0 article and AA methodology page. All figures are AA's own description.

By simple arithmetic on those durations (484 minutes in total), AgentTalk is about 52% of the audio and carries 50% of the weight. The weights roughly track audio time, so this is not an arbitrary upweighting of the in-house set. It does mean that one dataset AA controls decides half the outcome.

AgentTalk's construction matters. AA says samples are scripted prompts read aloud by participants across 6 content categories, 17 accent groups and 8 speaking styles. The mix is voice agents and call centers (40%), AI agent interaction (20%), industry jargon (15%), meetings (10%), consumer and personal (10%) and media (5%). Participants recorded on their own devices, and AA says about 70% of clips were recorded in quiet indoor settings.

How is AA-WER scored?

Scoring is standard WER: substitutions plus insertions plus deletions, divided by words in the reference. Per AA's methodology page, both the reference and the model output are normalized with OpenAI's Whisper normalizer plus AA's own additions before comparison.

Results within each dataset are a time-weighted average, then combined with the 50/25/25 weights. AA tests API endpoints directly and, where a model accepts a prompt, supplies a fixed verbatim-transcription instruction. For long Earnings22 calls, AA transcribes the full audio by default and chunks (about 9 minutes, then about 30 seconds if needed) only for models with input limits, then rejoins chunks per call before computing WER.

The normalizer is where much of the scoring judgment sits. AA's version history shows v2.1 (April to May 2026) and v2.2 (May 2026 onward) each widened the set of variants treated as equivalent, such as phone-number grouping, spelled-out letters, URL and email forms, and unit abbreviations. That is defensible, but it means a "v2" score from February and one from October are not strictly the same measurement.

The methodology page and the leaderboard page do not publish confidence intervals, bootstrap ranges or run-to-run variance for AA-WER. AA's co-founders said in a Latent Space interview that its LLM Intelligence Index reports 95% confidence intervals from repeated runs, so the practice exists elsewhere in the company.

Who runs it, and are there conflicts of interest?

AA runs it, and AA also created the main dataset. The company describes itself on its About page as "the independent benchmarking company for AI". The same page lists individual backers, including Nat Friedman, Daniel Gross, Andrew Ng, Adam D'Angelo and Clem Delangue, rather than model vendors.

On revenue, the co-founders said in a Latent Space interview that no one pays to be on the public website. They described two customer groups: enterprises buying benchmarking and insight subscriptions, and companies across the AI stack buying private benchmarking. They said private benchmarking can include work for AI companies that built models. This is the company's own account of its model.

The conflict risk is structural, not proven. A firm that sells private evaluation to model builders and also publishes public rankings has an incentive to keep both sides satisfied. Nothing in the sources reviewed shows any ranking being influenced. They also contain no external audit that would let a reader rule it out.

Vendors also use AA's numbers in marketing. StepFun's launch post says its ASR "reaches 1.7% WER, matching the best result on the leaderboard." That figure is AA's measurement quoted by the vendor, not a separate lab-reported test.

What does the Artificial Analysis STT leaderboard show today?

As of 16 October 2026, the non-streaming leaderboard puts Fun-Realtime-ASR-preview and StepAudio 3 ASR level at 1.7%. Every row below is AA-measured through provider APIs, so "independent" here means independent of the model vendor, not independent of AA.

Prices are AA's normalized estimates of USD per 1,000 minutes of audio, as of 16 October 2026. The leaderboard header reads "AA-WER v2" without a minor version, while the methodology page lists v2.2 as current.

The top of the table is tight. Gaps between ranks one to five are 0.0 to 0.3 percentage points, with no published intervals. AA's own 22 September post calls the top two "effectively tied", and says StepAudio 3 ASR scores 1.4% on AgentTalk but 2.8% on Earnings22, where Fun-Realtime-ASR-preview scores 1.8%. That split shows how much the dataset mix drives the headline rank.

Where do vendor-cited and AA-measured scores differ, and why?

For STT, vendors mostly cite AA's number rather than publishing a competing one, so the gaps appear between AA snapshots and between providers. Three patterns show up in the sources.

Scores drift between snapshots. At the v2.0 launch, AA reported ElevenLabs Scribe v2 at 2.3% and Mistral's Voxtral Small at 3.0%. The October leaderboard lists Scribe v2 at 2.2% and Voxtral Small at 2.8%. Possible causes include normalizer updates and re-runs, but the pages reviewed do not attribute the changes.

Secondary reports differ in detail. The Decoder's coverage of the same launch gave Gemini 3 Flash 3.1%, where AA's article gives 3.2%. Cite AA's page, not a rewrite, and record the date.

The same model varies by provider. AA lists Whisper Large v3 at 4.1% on fal.ai, 4.5% on Together AI and 10.1% on Replicate, as of 16 October 2026. A leaderboard row is therefore a model-provider pair, not a pure model score. The page does not explain the Replicate gap.

Readers comparing coding benchmarks will recognise the pattern. Harness and serving differences move scores, as covered in Your AI Coding Agent Benchmark Is Wrong If You Ignore the Harness.

Can a benchmark built on its own dataset be fair?

It can be fairer than a public one on contamination, and weaker on auditability. Both halves of that answer have evidence.

The case for a held-out set. AA says public test sets carry a growing risk of data contamination and contain reference errors. Hume AI researchers reported on Hugging Face (21 August 2026) that several top-scoring open-source ASR models, out of 11 tested, reproduced benchmark transcripts from VoxPopuli English and LibriSpeech even when audio contradicted them. In one VoxPopuli clip with an audible "Thank you, Mr. President" missing from the reference, 6 of 11 models reproduced the error. Hume is a voice-AI company, and its study covers open models and public sets, not AgentTalk. AA's correction work also moved scores: AA reports that cleaning cut average WER by 3.5 percentage points on VoxPopuli and 5.6 on Earnings22.

The case against.

• No outside audit. AgentTalk is proprietary, so nobody else can re-run it or check the transcripts. The cleaned public sets are on Hugging Face and can be inspected.
• Selection by the operator. AA narrowed about 1,300 recordings to 469 where transcripts matched the audio. This is my inference: that filter may favor clearer recordings, but AA does not publish what was excluded.
• Scripted speech. Read-aloud prompts do not carry the disfluency of live agent calls. AA lets participants edit transcripts to reflect what they actually said, which helps ground truth but does not make the speech spontaneous.
• Small effective samples. Earnings22 is 6 calls carrying 25% of the score. If the 469 AgentTalk clips were spread evenly over 17 accent groups, each group would have about 28 clips (my arithmetic; AA shows the real distribution only as a chart).
• Operator-chosen rules. Weights, normalizer and model shortlist are AA's decisions. The inclusion test is an "industry significance" and competitive performance test that AA says it is still refining.

Neither side proves bias. A private set lowers one risk (training on the test) and raises another (nobody can check the test).

Should you trust it, and for what?

Trust it as a coarse screen for English voice-agent-style speech, not as a ranking of near-tied models. The scorecard below is my editorial assessment from the published material, not a measured quantity.

Use it to shortlist models, then test on your own audio. The VoxPopuli subset in the index is English, and AgentTalk is built around voice-agent prompts read aloud, so spontaneous call audio, dictation in specialist domains or multilingual use sit outside what these numbers directly support. AA says its cleaned sets and normalizer exist to avoid penalizing correct transcription, which makes AA-WER better suited to relative comparison than to predicting your production WER.

For the broader pattern of benchmarks that saturate or get optimized against, see AI Benchmark Saturation 2026. For a comparable question about a benchmark built by a party with a stake, see CursorBench 4.0 Explained.

What would a better version look like?

A better AA-WER would keep the held-out design and add the checks that make it verifiable. These are suggestions, not claims about AA's plans.

1. Publish intervals. Bootstrap over clips (and over the 6 Earnings22 calls) and show ranges, so ties read as ties.
2. Report per-dataset and per-accent scores by default. The 1.4% versus 2.8% split for one model shows why a single index hides structure.
3. Escrow the held-out set. Let an outside party run a fixed sample of audio and transcripts to confirm ground truth without releasing the set.
4. Rotate AgentTalk. Retire a slice each cycle and release it, so the public sees what the private set looked like.
5. Publish a normalizer sensitivity table. Show the leaderboard under plain Whisper normalization and under v2.2, so readers see how much rules move ranks.
6. Add spontaneous speech. Real recorded agent calls, with consent, would test the disfluency that read prompts miss.
7. Disclose commercial overlap per modality. State whether any listed vendor has bought private benchmarking in the same modality.

What's still unclear?

Several points cannot be settled from published material.

• Whether AgentTalk is clean. AA calls it held-out, and no public evidence of leakage appears in the sources. Equally, nothing lets outsiders confirm it.
• Why scores moved after launch. Scribe v2 and Voxtral Small changed between February and October, and the pages do not attribute each change to normalizer updates, re-runs or model updates.
• The Replicate Whisper gap. 10.1% versus 4.1% for the same named model is unexplained on the leaderboard page.
• Version labeling. The methodology page lists v2.0 as running from February 2026 to the present and v2.2 from May 2026, while the leaderboard says "AA-WER v2".
• Model counts. The leaderboard's FAQ text refers to 58 models, while a filter on the same page refers to 72.
• Commercial overlap in STT. AA says nobody pays to be listed and that private work exists. Whether any vendor on this board has bought private STT benchmarking is not disclosed.
• Weakest point of this analysis. The scorecard is judgment. The evidence for contamination comes from a vendor study of open models on public data, which is indirect for AgentTalk.