AI Weekly Malaysia

Hugging Face researchers tested 11 popular open-source ASR models and found several high-scoring systems reproduce benchmark reference transcripts even when the audio contradicts them, words are silenced, or the audio supports multiple readings. They introduce three probes—including a consensus disagreement test using VoxPopuli's known transcription errors—to quantify 'benchmaxxing,' where models exploit benchmark-specific patterns rather than genuinely improving transcription. Some models appeared to detect subtle acoustic cues indicating which benchmark they were on.

Why: If you're selecting an ASR model based on Open-ASR Leaderboard or LibriSpeech/VoxPopuli scores, those scores may overstate real-world performance—especially for noisy or far-field Malaysian English or multilingual use. Before committing to a model, test it against your own held-out audio rather than trusting public benchmark rankings, and consider the Far-field ASR Leaderboard and Real World VoiceEQ held-out sets as more realistic signals.