ASR Benchmark Contamination: Hume's Probes Show Top Models Copy VoxPopuli's Errors 18-30% of the Time

Hume AI researchers report that the highest-scoring open-source speech recognition models reproduce erroneous VoxPopuli reference transcripts 18-30% of the time, according to Hume's 21 August 2026 write-up of arXiv 2608.19936. On LibriSpeech, some of the strongest benchmark performers also output numbers that had been silenced in the audio in roughly 30-40% of examples. All figures here are lab-reported by Hume and had not been independently replicated as of 16 October 2026.

The result is real evidence of benchmark-specific behaviour. It is not proof of test-set memorisation, and the authors do not claim it is.

What did Hume find in its ASR benchmark contamination test?

Hume tested 11 widely used open-source ASR models on VoxPopuli English and LibriSpeech (clean and other). Several of the top scorers reproduced benchmark transcripts even when the audio contradicted them, relevant words were silenced, or the audio supported two written forms equally well.

The paper's abstract puts the conclusion this way: high-performing models show benchmark-conditioned behaviours that can inflate benchmark performance without reflecting improved general-purpose transcription ability. The authors say the models appear to read narrow acoustic cues that identify which benchmark they are being tested on.

The paper reports one regularity worth keeping in mind. The six models with the best VoxPopuli WER (5.4-5.8%) are exactly those with the highest accept-ref (0.18-0.30), while every model at 6.5% WER or above sits at or below 0.10. Better leaderboard scores went with more reference copying.

What do the three probes measure, and how are they scored?

Each probe targets positions where the audio underdetermines the reference. The metric is "accept-ref": the fraction of such positions where the model emits the benchmark's reference rendering instead of the audio-true one. Benchmarks are VoxPopuli English (European Parliament speech, Hugging Face version) and LibriSpeech clean/other, with surface transcripts from standard greedy decoding.

The switch rate is the smaller of the two conditional accept-ref values. A fixed-form transcriber scores 0.0 and a random alternator scores at most 0.5, so a rate reliably above 0.5 means the model is reading clip-level cues. The Hume blog adds that some models reach roughly 90% switch accuracy on the Mr./Mister convention across the two corpora.

Sample sizes in the paper: the consensus panel flagged 1,113 edits on 745 VoxPopuli test clips (586 substitutions, 441 deletions, 86 insertions). Against a human-annotated subset, 93% of flagged edits also appear in the human annotations. Overall, 40% of clips and roughly 3% of reference words carry a flagged edit.

Which models reproduced the benchmark's mistake?

On one audited VoxPopuli clip, the reference omits an audible "Thank you" before "Mr President". Six of 11 models dropped it on the real clip, five on a same-speaker voice clone, and one on a clone of a parliament speaker recorded after every model's training cutoff. When the sentence was resynthesised in a generic TTS voice, all 11 restored the courtesy.

Snapshot below is from Hume's blog (published 21 August 2026). It covers a single clip, not a corpus-level score.

Benchmark version for every row: VoxPopuli English test, Hugging Face version. Outputs are raw, before normalisation. Hume notes that Parakeet is the only model that flips between the real clip and the same-speaker clone, and Phi-4 is the only one still dropping the courtesy on the fresh clone.

Is this contamination or memorisation?

Not in the narrow sense. The authors describe benchmark optimization and note that decoder-side contamination is localised to a specific training paradigm, while their effects appear across multiple architectures. They also cite work finding that the WER impact of decoder data contamination is relatively small, and point to "training on the test task" as a more prominent form of benchmark optimization.

The paper does document one concrete leak: 40% of speakers in the VoxPopuli test split are leaked into the training split in the Hugging Face version. The paper reports no significant regression in accept-ref for non-leaked speakers. That is a split-design problem. It is not evidence that any named model trained on test audio.

What the evidence does support is narrower and still useful. Models can transcribe the words faithfully, but acoustic context decides whether they follow the audio or a benchmark-specific transcription policy. Hume reports that trimming benchmark context or appending conversational audio can restore the faithful transcript, and that low-rank linear steering can manipulate the behaviour in some cases. Hume states the mechanism remains open and calls for more transparency about training data and model-selection procedures.

Who ran the study, and are there conflicts of interest?

The paper's authors are Theo Lebryk, David Ayllon, Alice Baird, Jakub Piotr Cłapa, Jens Madsen and Panagiotis Tzirakis, and the Hugging Face paper page lists Hume AI. The write-up is also published on Hugging Face's blog. None of the 11 tested models is reported to be Hume's own.

Hume does have a stake in the conclusion. Its site lists Speech Data, VoiceEQ Platform, Prism Pipeline and Human Feedback API among its products. It says it introduced held-out sets in Real World VoiceEQ, and its RW-Voice-EQ Bench paper argues for withheld evaluation. The study's recommendation (use held-out data) points toward the kind of evaluation Hume sells and runs.

The Open ASR Leaderboard has a similar alignment. Its private test sets come from Appen Inc. and DataoceanAI, both data suppliers, and the Hugging Face post says they were kept private to reduce benchmaxxing and test-set contamination risk. The default Average WER is still computed on public datasets only. None of this invalidates the findings, but readers should know who benefits from "public benchmarks are leaky."

What are the weaknesses of the study?

Five limits matter for how far you can generalise.

• Model coverage: 11 open-source models, English only, two corpora. Closed commercial ASR services were not tested.
• Panel dependence: the reference-error probe uses a four-model panel as ground truth. It agrees with human annotators on 93% of flagged edits, but the human edits it misses are predominantly formatting changes, so formatting errors are undercounted.
• Selection in masking: the masking step skips clips where aligner boundaries are very tight (about 20% of number targets). The paper says this biases the sample toward longer, harder numbers, which works against finding recovery.
• Single-clip illustrations: the 6/5/1 courtesy table is one clip. Corpus-level rates come from the probes, and per-model values are in the paper's figures, not in a table in the blog.
• No causal chain to training data: the steering and patching experiments localise the behaviour inside models. They do not show which training examples produced it.

Another point cuts the other way. Artificial Analysis independently cleaned the same corpus: VoxPopuli-Cleaned-AA has 628 samples, and Artificial Analysis reports that average model WER on VoxPopuli fell 3.5 percentage points after cleaning. Reference errors in VoxPopuli are a known problem, which makes the Hume finding easier to believe. It also means a model that copies the wrong reference gets a better WER on the uncleaned set than a faithful one.

Should you trust it, and for what?

Trust it as evidence that public-benchmark WER on VoxPopuli and LibriSpeech can overstate general transcription ability for some top open models. Do not use it to rank those models or to accuse a vendor of training on test data. The scores below are an editorial assessment, not a published rating.

For a look at how other evaluations can be moved by their setup, see our pieces on AI benchmark saturation in 2026 and whether Grok 4.7 was trained for its own test harness.

What would a better ASR benchmark look like?

The authors recommend that benchmark developers avoid simple independent and identically distributed test splits in favour of temporal, speaker, or other metadata-based separation. The VoxPopuli speaker leak shows why: a random split lets the same voices appear on both sides.

From the evidence in these sources, a stronger design would have:

1. Speaker-disjoint and time-disjoint test sets, with a private held-out split, as the Open ASR Leaderboard now offers for private English sets.
2. Corrected references. Artificial Analysis already publishes cleaned VoxPopuli and weights it at 25% of AA-WER v2.0, alongside a 50% held-out proprietary set.
3. Behavioural probes reported next to WER, such as accept-ref and switch rate, so reference copying is visible.
4. Disclosure of training data and model-selection procedures, which the authors say would help explain how these behaviours arise.

Hume's blog adds that public benchmarks remain valuable because they are transparent, repeatable and easy to run. The point is to separate genuine transcription gains from gains that do not carry to new audio.

What's still unclear?

• Cause: it is not established whether the behaviour comes from test-data exposure, the 40% speaker leakage, model-selection against leaderboards, or something else.
• Per-model corpus rates: the blog gives ranges (18-30%, 30-40%) and a one-clip table. Exact per-model numbers are in figures, so rankings should wait for the paper's data.
• Framing differences: the blog headlines "roughly 90%" switch accuracy for some models, while the paper's text reports how many models exceed 0.5 (six of 11 and eight of 11). These are different summaries of related results, not a contradiction.
• Closed models: whether commercial ASR APIs show the same pattern is not reported.
• Replication: no independent group's reproduction had been published as of 16 October 2026. The open scripts make one possible.

The weakest point of the "contamination caught" reading is the leap from behaviour to cause. The clean version of the claim is that top scores on two public ASR benchmarks partly reflect benchmark-specific habits, and that those habits fade on fresh audio.