**Know Who Spoke When: Build Real-Time, Multi-Speaker AI with NVIDIA Nemotron 3 Diarization**

Hugging Face turn overlapping conversations into speaker-aware data with one open-weight, 100M-parameter model – ranked #1 in Voice Arena’s initial Diarization-Bench results. Why speaker diarization matters How Nemotron 3 Diarization works One model for offline and streaming conversations From audio to speaker activity Diarization and Speaker Attributed ASR (Multi-speaker ASR) are different tasks Balance latency and accuracy Benchmark results: Ranked #1 in Voice Arena’s initial Diarization-Bench Measuring diarization accuracy An average 40% relative DER reduction at 1.04-second latency Improvements increase in higher-speaker-count conditions Accuracy and throughput See Nemotron 3 Diarization in action Bring real-time, speaker-aware transcription on device Get started with NVIDIA NeMo Speech Install the dependencies Implement streaming diarization with streaming ASR (multi-speaker ASR) in real-world scenarios Run an offline diarization on a recording Choose the latency-quality operating point Combine diarization with Offline ASR Deployment considerations Resources and next steps Turn overlapping conversations into speaker-aware data with one open-weight, 100M-parameter model – ranked #1 in Voice Arena’s initial Diarization-Bench results.

Every conversation carries two layers of information: what was said and who said it. Speech recognition captures and transcribes the words. Speaker diarization classifies who spoke when, helping applications connect what was said to the right participant. Consider a transcript from a meeting, customer call, or podcast in which every sentence is correct but none is attributed to a speaker. You can read the words, but you cannot reliably tell who made a commitment, who raised an objection, or which participant interrupted. Search, summaries, action items, conversation analytics, and voice-agent memory all become less useful. Speaker diarization identifies the time intervals during which each speaker is active, including intervals where people talk over one another. Those speaker timestamps can then be combined with automatic speech recognition (ASR) to create a speaker-attributed transcript. NVIDIA Nemotron 3 Diarization is an open-weight, 100M-parameter model that ranks #1 on Voice Arena’s Diarization-Bench leaderboard with a 14.72% Diarization Error Rate (DER). Supporting up to eight speakers across live and recorded conversations, it handles overlapping speech, chunked processing for flexible recording lengths, and customizable streaming latency. Earlier models like NVIDIA Streaming Sortformer established this approach for four-speaker diarization, including the streaming diar_streaming_sortformer_4spk-v2.1 checkpoint used as the baseline below. Nemotron 3 Diarization expands support to eight speakers and improves the accuracy and throughput measured in the following evaluations. Diarization systems must solve two related problems. First, they must detect speech and assign it to the correct speaker. Second, they must preserve that assignment throughout the conversation, even after silence, interruptions, or long gaps between a speaker’s turns.