NVIDIA Releases Audex-30B-A3B: Unified Audio-Text Model With 1M Token Context and Speech Generation
NVIDIA released Audex-30B-A3B, a unified audio-text model built on the Nemotron-Cascade-2-30B-A3B backbone. The model handles audio understanding, speech recognition and translation, text-to-speech, audio generation, and speech-to-speech while supporting up to 1M token context length.
NVIDIA Releases Audex-30B-A3B: Unified Audio-Text Model With 1M Token Context and Speech Generation
NVIDIA released Audex-30B-A3B, a unified audio-text language model that handles speech and audio tasks while maintaining text reasoning capabilities. The model is built on Nemotron-Cascade-2-30B-A3B, a 30B parameter mixture-of-experts (MoE) architecture with 3B activated parameters.
Core Specifications
- Architecture: 30B MoE with 3B active parameters
- Context window: 1M tokens
- Modalities: Text, speech, and general audio input/output
- Reasoning modes: Thinking and instruct (non-thinking) modes
- License: NVIDIA Oneway Noncommercial License
- Released: June 8, 2026 (according to model card)
- Base model: Nemotron-Cascade-2-30B-A3B
Technical Architecture
Audex-30B-A3B extends its text-only backbone with discrete audio token vocabularies for speech and audio outputs, plus an audio encoder for inputs. The model uses the ChatML template and encloses reasoning content within <think> and </think> tags. To activate non-thinking mode, the system prepends <think></think> to assistant responses.
Inference requires vLLM 0.20.0 or transformers >= 4.53.0. Audio functionality needs additional packages (mamba-ssm, causal-conv1d) and audio codecs not bundled in the base vLLM container.
Capabilities
The model handles six primary audio tasks:
- Audio understanding: Question-answering about audio content (top_p=0.9, temperature=0.7)
- Speech recognition: Transcription using greedy sampling
- Speech translation: Cross-lingual speech conversion
- Text-to-speech (TTS): Uses standalone Audex causal speech decoder or XCodec2
- Text-to-audio (TTA): General audio generation using XCodec1 with optional 48kHz enhancement VAE
- Speech-to-speech: Direct audio-to-audio conversion with text reasoning
NVIDIA claims the model preserves the "reasoning, alignment, knowledge, long-context, and agentic capabilities" of its text-only backbone with "marginal or no regression."
Inference Options
The model supports multiple inference paths:
- vLLM (recommended): Offline batch processing and OpenAI-compatible server with audio_url support
- Hugging Face/transformers: Requires transformers >= 4.53.0
- Text-only reasoning: Can convert checkpoint to pure text by removing audio vocabularies
Audio understanding and speech recognition use transformers 4.53.3 or Megatron-LM's native inference for benchmark results. Text-to-audio uses XCodec1 with enhancement VAE, while text-to-speech uses the original non-streaming XCodec2 decoder.
What This Means
NVIDIA's release adds another entry to the growing unified multimodal model category, following similar efforts from other labs. The 1M token context window matches recent long-context releases, while the MoE architecture with only 3B active parameters should enable efficient inference. The noncommercial license restricts commercial deployment, limiting this to research and development use cases. The model's ability to maintain text reasoning while adding audio capabilities addresses a common trade-off in multimodal models, though independent verification of the "no regression" claim on text tasks would be needed. The requirement for multiple inference setups depending on task type adds implementation complexity compared to unified inference pipelines.
Related Articles
Nvidia team: huge context windows often overkill, as summaries and lookups matched them in many tests
An Nvidia research team argues that very large context windows are often overkill and expensive, according to The Decoder. In many tests, periodic summaries and targeted lookups worked about as well.
OpenAI releases GPT-Live-1.1 in the API, with standby mode billing idle WebRTC time at $0.02 per minute
OpenAI has released GPT-Live-1.1 in the API. The full-duplex voice model improves multilingual accents and transcription, and a new standby mode bills idle WebRTC time at $0.02 per minute instead of $0.05.
Thinking Machines Lab releases Inkling 2 Mini: 26B-parameter looped MoE with 3B active and 1M-token context
Thinking Machines Lab has released Inkling 2 Mini, a multimodal Mixture-of-Experts model with about 26B physical total parameters and 3B active per token. It repeats its middle layers three times, supports a 1M-token context, and ships with a DSpark drafter. Hugging Face reports it leads 32 of 48 benchmarks against four comparable models.
NVIDIA releases Nemotron-Labs-Sigma: 14 open 3B and 8B continuous diffusion language model checkpoints
NVIDIA released Nemotron-Labs-Sigma, a set of 14 open continuous diffusion language model checkpoints at 3B and 8B parameters. According to MarkTechPost, Sigma denoises text in a learned 16-D embedding space, distills to few-step samplers, and decodes up to 4.20x faster than autoregressive decoding.
Comments
Loading...