Introducing Real World VoiceEQ: Measuring the human quality of voice AI

Hugging Face explore speech model leaderboards with audio samples A broader benchmark for voice AI Key findings from Real World VoiceEQ Progress in voice AI is becoming increasingly specialized. Voice models have become better at speaking than actually listening.

Traditional benchmarks increasingly overestimate real-world performance. Human evaluation remains essential. Why Voice AI needs a new measurement layer Existing benchmarks suggest voice AI is nearing human-level performance but real-world conversations tell a different story. Voice is rapidly becoming AI’s primary interface. From customer support and healthcare to education, entertainment, and personal assistants, speech is increasingly replacing text as the way people interact with AI. Over the last few years, voice models have improved dramatically. Word error rates continue to fall, latency has reached conversational speeds, and many established benchmarks are approaching saturation. Yet anyone who regularly uses voice AI knows something still feels off. Voice models can sound like different people over the course of a conversation, miss hesitation or uncertainty, and struggle with accents, noise, or emotional speech. Those shortcomings are easy to miss in benchmarks focused on latency and word error rate. People care whether a voice system can truly listen, respond appropriately, and remain natural and reliable in real conversations. To measure those qualities, we built Real World VoiceEQ—a benchmark designed to evaluate the human quality of voice interaction. It assesses whether voice systems can recognize, produce, and respond to the acoustic information transcripts leave out, from tone and emotion to speaker identity and background context. Real World VoiceEQ evaluates more than 40 leading proprietary and open-source voice models across 15+ key evaluation dimensions and more than 60 metrics spanning Automatic Speech Recognition (ASR), Text-to-Speech (TTS), Speech-to-Speech (S2S), and Speech Understanding. Real World VoiceEQ was developed from more than 1 million individual human ratings collected across different demographics, speaking styles, and acoustic environments.