NVIDIA Releases Audex-30B-A3B: Unified Audio-Text Model With 1M Token Context and Speech Generation
NVIDIA released Audex-30B-A3B, a unified audio-text model built on the Nemotron-Cascade-2-30B-A3B backbone. The model handles audio understanding, speech recognition and translation, text-to-speech, audio generation, and speech-to-speech while supporting up to 1M token context length.
NVIDIA Releases Audex-30B-A3B: Unified Audio-Text Model With 1M Token Context and Speech Generation
NVIDIA released Audex-30B-A3B, a unified audio-text language model that handles speech and audio tasks while maintaining text reasoning capabilities. The model is built on Nemotron-Cascade-2-30B-A3B, a 30B parameter mixture-of-experts (MoE) architecture with 3B activated parameters.
Core Specifications
- Architecture: 30B MoE with 3B active parameters
- Context window: 1M tokens
- Modalities: Text, speech, and general audio input/output
- Reasoning modes: Thinking and instruct (non-thinking) modes
- License: NVIDIA Oneway Noncommercial License
- Released: June 8, 2026 (according to model card)
- Base model: Nemotron-Cascade-2-30B-A3B
Technical Architecture
Audex-30B-A3B extends its text-only backbone with discrete audio token vocabularies for speech and audio outputs, plus an audio encoder for inputs. The model uses the ChatML template and encloses reasoning content within <think> and </think> tags. To activate non-thinking mode, the system prepends <think></think> to assistant responses.
Inference requires vLLM 0.20.0 or transformers >= 4.53.0. Audio functionality needs additional packages (mamba-ssm, causal-conv1d) and audio codecs not bundled in the base vLLM container.
Capabilities
The model handles six primary audio tasks:
- Audio understanding: Question-answering about audio content (top_p=0.9, temperature=0.7)
- Speech recognition: Transcription using greedy sampling
- Speech translation: Cross-lingual speech conversion
- Text-to-speech (TTS): Uses standalone Audex causal speech decoder or XCodec2
- Text-to-audio (TTA): General audio generation using XCodec1 with optional 48kHz enhancement VAE
- Speech-to-speech: Direct audio-to-audio conversion with text reasoning
NVIDIA claims the model preserves the "reasoning, alignment, knowledge, long-context, and agentic capabilities" of its text-only backbone with "marginal or no regression."
Inference Options
The model supports multiple inference paths:
- vLLM (recommended): Offline batch processing and OpenAI-compatible server with audio_url support
- Hugging Face/transformers: Requires transformers >= 4.53.0
- Text-only reasoning: Can convert checkpoint to pure text by removing audio vocabularies
Audio understanding and speech recognition use transformers 4.53.3 or Megatron-LM's native inference for benchmark results. Text-to-audio uses XCodec1 with enhancement VAE, while text-to-speech uses the original non-streaming XCodec2 decoder.
What This Means
NVIDIA's release adds another entry to the growing unified multimodal model category, following similar efforts from other labs. The 1M token context window matches recent long-context releases, while the MoE architecture with only 3B active parameters should enable efficient inference. The noncommercial license restricts commercial deployment, limiting this to research and development use cases. The model's ability to maintain text reasoning while adding audio capabilities addresses a common trade-off in multimodal models, though independent verification of the "no regression" claim on text tasks would be needed. The requirement for multiple inference setups depending on task type adds implementation complexity compared to unified inference pipelines.
Related Articles
Nvidia Research: Custom Harness Pushes Claude Opus 5 from 30% to 100% on ARC-AGI-3
Nvidia researchers report that a custom agent harness with a supervisory component drove Claude Opus 5 from a 30% baseline score to a perfect 100% on the ARC-AGI-3 benchmark. The findings add to mounting evidence that scaffolding, not the underlying model, is the dominant factor in long-horizon agentic performance.
DeepSeek Releases Experimental V4-Flash-Vision-Exp, Claims Near-Parity With Opus 4.8 on Agent Benchmarks
DeepSeek has released V4-Flash-Vision-Exp, an experimental multimodal extension of V4-Flash that adds image understanding while preserving text reasoning capabilities. The company claims the model approaches or beats Anthropic's Opus 4.8 on its internal multimodal agent benchmarks.
DeepSeek Releases V4 Flash Vision Exp, an Experimental Multimodal MoE Model with 1M Context
DeepSeek has released V4 Flash Vision Exp, an experimental vision-enabled variant of DeepSeek V4 Flash 0731 that adds image understanding while matching the base model's text performance. The sparse mixture-of-experts model uses 13B active parameters out of 284B total and supports a 1M token context window.
NVIDIA Nemotron 3.5 Lightning Arrives on Amazon SageMaker JumpStart, Targets High-Volume Agentic Workloads
NVIDIA's Nemotron 3.5 Lightning, a 30B-parameter hybrid Mixture-of-Experts model with only 3B active parameters, is now available for one-click deployment on Amazon SageMaker JumpStart. NVIDIA claims up to 4x higher throughput and 30% faster task completion for high-volume agentic workloads compared to larger frontier models.
Comments
Loading...