NVIDIA Releases Nemotron 3 Diarization, a 100M-Parameter Open-Weight Model Ranked #1 on Voice Arena's Diarization-Bench

What launched

NVIDIA released Nemotron 3 Diarization, an open-weight, 100-million-parameter model that determines who spoke when in an audio recording — a task known as speaker diarization. According to NVIDIA, the model ranked #1 among 12 systems and 17 total configurations in Voice Arena's initial Diarization-Bench evaluation, posting a 14.72% Diarization Error Rate (DER), compared with 19.3% for the next-best system. The benchmark covered 139 English-language conversations totaling roughly 22 hours, with overlapping speech scored, system-generated speech activity detection, and no boundary collar applied.

Diarization is distinct from automatic speech recognition (ASR). ASR transcribes words; diarization assigns time intervals to speakers. Combining both produces a speaker-attributed transcript — the format needed for meeting summaries, call analytics, and voice-agent memory.

Technical specifications

Nemotron 3 Diarization supports up to eight speakers across both offline (recorded) and streaming (live) audio. Key architecture details, according to NVIDIA:

• Accepts 16 kHz, single-channel audio, converted to Mel-spectrogram features with a 10 ms frame step
• Features are stacked by a factor of 8 into 80 ms frames fed to a 31-layer Transformer encoder with rotary positional embeddings (RoPE)
• Output is a [T, 8] tensor representing per-frame probability that each of eight speaker channels is active, natively handling overlapping speech
• Speakers are labeled by arrival order (first voice detected becomes channel one, etc.), following the Sortformer approach used in NVIDIA's earlier Streaming Sortformer models, avoiding the need to resolve speaker permutations across audio chunks
• Streaming inference relies on two memory mechanisms: an Arrival-Order Speaker Cache (AOSC) that retains information about previously observed speakers, and a FIFO queue supplying recent frame context
• Supports four recommended input-buffer latencies: 30.4s, 1.04s, 0.64s, and 0.32s, trading off responsiveness against accuracy; an 80 ms buffer is technically possible but not recommended

The model was trained on public and licensed speech data, including multispeaker-annotated real-world conversations and simulated mixtures spanning 21 languages licensed from David AI. NVIDIA states that adding David AI data reduced compound DER by 0.77 absolute points, from 11.19% to 10.42%, at both offline and ultra-low-latency operating points.

Output labels are anonymous (e.g., "speaker_2") rather than identity-linked; applications must separately map channels to named individuals using metadata or voice verification models.

Predecessor and scope

Nemotron 3 Diarization builds on NVIDIA's earlier Streaming Sortformer line, including the four-speaker diar_streaming_sortformer_4spk-v2.1 checkpoint used as a baseline in NVIDIA's own comparisons. The new model doubles maximum speaker support to eight and, according to NVIDIA, improves both accuracy and throughput over that baseline. Chunked processing removes any fixed maximum audio duration, though NVIDIA notes performance can still degrade on long recordings or audio with heavy noise, reverberation, far-field capture, or domain mismatch.

Weights are open and available via Hugging Face.

What this means

Diarization has historically lagged ASR in accuracy and open tooling, leaving most "who said what" logic to closed, proprietary pipelines from vendors like Otter.ai or enterprise call-center platforms. An open-weight model claiming state-of-the-art results on a third-party leaderboard gives developers building meeting assistants, call analytics, and voice agents a self-hostable alternative — relevant for privacy-sensitive deployments where sending audio to a third-party diarization API is a non-starter. The eight-speaker ceiling and configurable latency (down to 320ms) also target real-time voice-agent use cases, not just offline transcription. The DER improvement from licensed David AI data underscores that diarization quality remains heavily data-dependent, meaning gains may not transfer identically to languages or acoustic conditions outside the training distribution — a caveat NVIDIA itself flags for noisy, far-field, or reverberant audio.