NVIDIA Releases Nemotron 3 Diarization, an Open-Weight Speaker ID Model Supporting Up to 8 Speakers

NVIDIA has released Nemotron 3 Diarization, an open-weight speaker diarization model designed to determine "who spoke when" in real-world audio. The model, published on Hugging Face, supports both streaming and offline inference and handles up to eight speakers per audio input.

Technical Details

Nemotron 3 Diarization builds on NVIDIA's Sortformer architecture, resolving speaker permutation by ordering output channels according to each speaker's first arrival in the input audio. For streaming use cases, the model implements the Arrival-Order Speaker Cache (AOSC) and a FIFO queue, techniques introduced in NVIDIA's earlier Streaming Sortformer research. The AOSC retains speaker information from previous audio chunks to preserve speaker identity over time, while the FIFO queue supplies recent frame context for each processing step.

A single checkpoint supports a range of latency configurations. According to NVIDIA, input buffer latency can go as low as 80 milliseconds, though the company recommends a minimum practical configuration of 0.32 seconds for ultra-low-latency applications. An offline-style configuration uses a 30.4-second input buffer for maximum accuracy. Output frame resolution is configurable in multiples of 10 milliseconds, and with chunked inference the model places no upper limit on total audio duration.

NVIDIA lists four recommended streaming configurations, ranging from "very high latency" (offline, 30.4s buffer) down to "ultra-low latency" (0.32s buffer), each defined by five parameters: speaker cache length, FIFO queue length, chunk length, right context frames, and cache update period. Latency is calculated as (chunk length + right context) × 80 milliseconds and does not include compute processing time.

Access and Integration

The model is available through NVIDIA's NeMo Speech toolkit, requiring Python 3.12 or later and a recent PyTorch installation. It is also natively supported in Hugging Face's Transformers library (installed from source) via the AutoModelForAudioFrameClassification class, and can be run locally through NeMo-Speech.cpp, a lightweight C++ runtime that also supports adding word-level speaker tags to transcriptions via a combined transcribe-and-diarize command.

Input formats include individual WAV files, lists of file paths, raw numpy arrays with a specified sample rate, and line-delimited JSON manifests for batch processing with offset and duration fields.

NVIDIA has not published specific Diarization Error Rate (DER) benchmark scores in the released materials, though it provides an evaluation script (e2e_diarize_speech.py) for users to test accuracy and speed across different parameter settings on their own datasets.

The model is released for both commercial and non-commercial use, and NVIDIA has published a live demo on Hugging Face Spaces pairing the diarization model with streaming ASR (automatic speech recognition).

What This Means

Speaker diarization is a persistent bottleneck for meeting transcription, call center analytics, and multi-speaker voice applications, where systems must not only transcribe speech but attribute it correctly to individual speakers in real time. NVIDIA's release targets that gap directly with a single checkpoint that scales from offline batch processing down to 80-millisecond streaming latency, a range that could reduce the need for separate models tuned to different deployment scenarios.

The open-weight release, paired with native Hugging Face Transformers support and a lightweight C++ runtime, lowers the barrier for developers to integrate diarization into local or edge applications without depending on cloud APIs. The lack of published DER benchmarks means engineers evaluating the model for production use will need to run their own accuracy tests against existing systems like pyannote or commercial diarization APIs before making adoption decisions.