NVIDIA Releases Nemotron VoiceChat 11B, an Open Full-Duplex Speech Model with Live Tool Calling

NVIDIA Ships Open Full-Duplex Voice Model With Live Tool Calling

NVIDIA has released NemotronLabs VoiceChat 11B, an 11-billion-parameter end-to-end speech model designed for real-time, full-duplex conversational AI. According to NVIDIA, the model is the first open full-duplex system to support live tool calling while maintaining natural conversation flow during tool execution. The model card lists a release date of August 3, 2026, and is available on Hugging Face under the OpenMDW License Agreement version 1.1, restricted to research use.

Architecture: One Model Instead of Three

Traditional voice assistants chain three separate systems: automatic speech recognition (ASR), a language model, and text-to-speech (TTS). NemotronLabs VoiceChat replaces that cascade with a single hybrid Mamba/Transformer architecture. Audio input passes through a Fast Conformer speech encoder (derived from Nemotron-Speech-Streaming-En-0.6b), producing audio tokens fed into a Nemotron Nano V2 9B LLM backbone. The backbone predicts text tokens that go to a TTS decoder for speech generation, while a separate output channel handles tool-calling scripts in parallel.

Input accepts text prompts and 16kHz audio; output includes agent text, 22.05kHz agent speech, and user transcription text. NVIDIA recommends the vLLM runtime engine and lists supported hardware including A100, H100, H200, B100, B200, and RTX-6000 GPUs, running on Linux.

Benchmark Results

According to NVIDIA, the model ranks #2 among open full-duplex models on VoiceBench and #2 among open models on Full-Duplex-Bench 1.0. Reported turn-taking latency is approximately 450 milliseconds. Specific Full-Duplex-Bench 1.0 scores include:

• Pause Handling (Synthetic) TOR: 0.153 (lower is better)
• Pause Handling (Candor) TOR: 0.255 (lower is better)
• Smooth Turn Taking TOR: 0.82 (higher is better), latency 448 ms
• User Interruption TOR: 1.0, latency 480 ms, GPT-4o rating 4.33

On the AU Harness BFCL-v3 tool-calling benchmark (spoken conversions of textual tool-use instructions), the model scored: Simple 58.5%, Multiple 62.5%, Parallel 42.5%, Parallel Multiple 27.5%, Irrelevance 89.6%, and an Average of 56.1%. On Full-Duplex-Bench v3, a human-collected benchmark for multi-step tool use under naturalistic speech, the model scored 82.5% tool selection accuracy, 44.2% argument accuracy, and 33% Pass@1. NVIDIA describes this as "competitive with frontier models" on tool selection specifically — the lower argument accuracy and Pass@1 scores indicate meaningful gaps remain in execution reliability.

Training Data

NVIDIA trained the model on roughly 550,000 hours of audio and text data, blending real speech recordings (Fisher, LibriVox, LibriTTS, HiFi-TTS, VCTK, Voxmovies, JL-Corpus) with synthetic speech generated via internal TTS systems and text corpora including Nemotron 5.5 pre-training/SFT data and Ultrachat. Data collection is described as a hybrid of human, synthetic, and automated methods, with automated labeling throughout.

Pricing is not yet disclosed, as this is a research-only release rather than a hosted commercial product.

What This Means

This release targets a specific gap in open-source voice AI: unified full-duplex architectures that avoid ASR-LLM-TTS handoff latency while still supporting agentic tool calls. NVIDIA's own benchmarks place the model second, not first, among open full-duplex systems on both VoiceBench and Full-Duplex-Bench 1.0 — a candid admission that leaves room for competitors. The tool-calling numbers are more telling: strong tool-selection accuracy (82.5%) paired with weak argument accuracy (44.2%) and Pass@1 (33%) suggests the model can often identify which tool to call but frequently fails to execute it correctly, a gap enterprise developers building voice agents will need to account for before production use. With a research-only license and no pricing structure, this is a capability preview rather than a deployable product, aimed at establishing NVIDIA's technical position in full-duplex speech ahead of a likely commercial follow-up.