NVIDIA Releases Nemotron-Labs Diffusion Models With 6.4× Faster Token Generation Than Autoregressive Decoding
NVIDIA has released Nemotron-Labs Diffusion, a family of diffusion language models at 3B, 8B, and 14B scales that generate multiple tokens in parallel rather than one at a time. The 8B model achieves 6.4× higher tokens per forward pass than autoregressive models in self-speculation mode while maintaining comparable accuracy.
NVIDIA Releases Nemotron-Labs Diffusion Models With 6.4× Faster Token Generation
NVIDIA has released Nemotron-Labs Diffusion, a family of diffusion language models that generate tokens in parallel rather than sequentially, achieving up to 6.4× higher tokens per forward pass than traditional autoregressive models.
Model Specifications
The Nemotron-Labs Diffusion family includes:
- Text models: 3B, 8B, and 14B parameter versions
- Vision-language model: 8B parameter multimodal variant
- License: NVIDIA Nemotron Open Model License (text models), NVIDIA Source Code License (VLM)
- Training data: 1.3T tokens for pretraining, 45B tokens for supervised fine-tuning
- Release includes: Base models, instruction-tuned chat variants, and training code via NVIDIA Megatron Bridge framework
Performance Benchmarks
According to NVIDIA, the 8B model demonstrates:
- 1.2% higher average accuracy than Qwen3 8B
- 2.6× tokens per forward pass in diffusion mode versus autoregressive models
- 6× TPF with linear self-speculation
- 6.4× TPF with quadratic self-speculation
- ~865 tokens/sec on B200 GPU in self-speculation mode (approximately 4× autoregressive baseline)
Tokens per forward pass (TPF) measures decoding efficiency independent of specific hardware configurations.
Three Generation Modes
Nemotron-Labs Diffusion supports three inference modes in a single model:
- Autoregressive mode: Standard left-to-right generation for compatibility with existing workflows
- Diffusion mode: Generates 32-token blocks in parallel, iteratively refining tokens across multiple denoising steps
- Self-speculation mode: Uses diffusion to draft candidate tokens, then verifies them autoregressively
Developers can switch between modes at deployment time without application-level changes.
Technical Architecture
The models build on recent research showing pretrained autoregressive models can be converted to diffusion language models through continued pretraining. Key design elements:
- Block-wise attention mechanism enables KV-cache compatibility
- Joint AR and diffusion training objective preserves original autoregressive capabilities
- Confidence thresholding determines when generated tokens are committed
- Built-in inference budget control through adjustable refinement steps
Unlike autoregressive models that finalize each token immediately, diffusion models can revise previously generated tokens, making them suitable for text editing and fill-in-the-middle tasks.
Deployment
Inference support is available through SGLang, with integration currently accessible via GitHub issue tracker. The same checkpoint can serve all three generation modes through a single configuration parameter (ar_mode).
NVIDIA reports self-speculation mode produces lossless output versus autoregressive decoding at temperature 0, maintaining deterministic compatibility.
What This Means
Nemotron-Labs Diffusion addresses a fundamental bottleneck in language model inference: memory bandwidth. Traditional autoregressive models spend most GPU time on memory operations rather than computation, particularly at small batch sizes. By generating and refining multiple tokens in parallel, diffusion models better utilize modern GPU architectures.
The ability to switch between AR and diffusion modes in the same model is the practical innovation here. Developers can deploy autoregressive mode for maximum compatibility, diffusion for throughput-critical workloads, or self-speculation when both speed and deterministic output matter. The 6.4× speedup claims remain to be independently verified, but if confirmed, this represents a meaningful shift in how inference-optimized models might be designed.
Related Articles
Nvidia team: huge context windows often overkill, as summaries and lookups matched them in many tests
An Nvidia research team argues that very large context windows are often overkill and expensive, according to The Decoder. In many tests, periodic summaries and targeted lookups worked about as well.
NVIDIA releases Nemotron-Labs-Sigma: 14 open 3B and 8B continuous diffusion language model checkpoints
NVIDIA released Nemotron-Labs-Sigma, a set of 14 open continuous diffusion language model checkpoints at 3B and 8B parameters. According to MarkTechPost, Sigma denoises text in a learned 16-D embedding space, distills to few-step samplers, and decodes up to 4.20x faster than autoregressive decoding.
NVIDIA: Sparse attention indexer and Top-k take 47% of a DeepSeek V3.2 prefill layer at 128K tokens
NVIDIA published a co-design primer for model-level sparse attention. It reports that the indexer and Top-k take 47% of a DeepSeek V3.2 prefill layer at 128K tokens, and that reusing the prior decode step's selection speeds up Top-k by 2.1x to 2.4x.
Bytedance-led team argues three training steps are enough for AI agents, releases competitive open model
A team from Bytedance and several US universities argues in a new paper that three training steps are enough to build capable AI agents. The authors say stacked, multi-phase pipelines make it hard to tell which step helps and make reproduction expensive. According to The Decoder, the work is backed by a competitive open model.
Comments
Loading...