NVIDIA releases Nemotron-Labs-TwoTower-30B: block-wise diffusion model claims 2.42× faster generation at 98.7% baseline
NVIDIA released Nemotron-Labs-TwoTower-30B-A3B-Base-BF16, a block-wise diffusion language model that generates text by denoising blocks of tokens in parallel rather than sequentially. According to NVIDIA, the model achieves 2.42× the wall-clock generation throughput of its autoregressive baseline while retaining 98.7% of aggregate benchmark quality.
How Nemotron-Labs-TwoTower-30B-A3B-Base-BF16 ranks now
| Benchmark | Score | Rank | Source |
|---|---|---|---|
| MMLU-Pro | 60.9% | 70 of 76 | Hugging Face model card, Jul 2026 ↗ |
Rank is against every model in our database with a sourced score on that benchmark, each counted once with its best score.
NVIDIA Releases Nemotron-Labs-TwoTower-30B: Block-Wise Diffusion Model Claims 2.42× Faster Generation
NVIDIA released Nemotron-Labs-TwoTower-30B-A3B-Base-BF16, a block-wise diffusion language model that generates text by denoising blocks of tokens in parallel rather than sequentially. According to NVIDIA, the model achieves 2.42× the wall-clock generation throughput of its autoregressive baseline while retaining 98.7% of aggregate benchmark quality.
Architecture: Two Frozen and Trainable Towers
The model uses a dual-tower architecture built on the NVIDIA-Nemotron-3-Nano-30B-A3B-Base-BF16 backbone:
- Context tower (AR/context): Frozen causal autoregressive tower that processes the prompt and previously committed tokens
- Denoiser tower (diffusion/denoiser): Trainable tower that generates blocks of up to 16 tokens at a time via mask diffusion
Both towers consist of 52 layers combining Mamba-2, self-attention, and MoE components. Total model parameters: ~60B (30B per tower). Active parameters per token: ~3B per tower, with 128 routable experts of which 6 are activated plus 2 shared experts.
The denoiser tower uses bidirectional in-block attention, layer-aligned cross-attention to the context tower, and context-seeded Mamba-2 states. Time conditioning is handled via adaLN-single modulation (PixArt-α style).
Training and Data
Training occurred in two stages:
- Backbone pre-training: The single-tower baseline was pre-trained from scratch on ~25T tokens using next-token prediction
- Denoiser training: Only the diffusion/denoiser tower was trained (context tower frozen) using a masked-diffusion objective over ~2.1T tokens
Data cutoff: June 25, 2025. Model development: September 2025 – April 2026.
Benchmark Performance
Default configuration: confidence threshold γ = 0.8, block size 16, BF16 on 2×H100 GPUs.
Key results (diffusion vs. AR baseline):
- MMLU (5-shot): 78.24 vs. 78.56
- HumanEval (0-shot): 75.58 vs. 79.27
- GSM8K (8-shot): 90.14 vs. 92.49
- MATH-500 (4-shot): 80.60 vs. 84.40
- ARC-Challenge (25-shot): 92.66 vs. 91.72
According to NVIDIA, the model retains 98.7% of aggregate baseline quality while delivering 2.42× throughput. Lowering the confidence threshold increases throughput but reduces quality.
Generation Modes
Three generation modes are available:
- Mask Diffusion: Block-wise iterative denoising (up to block_size tokens per step)
- Mock-AR: Two-tower autoregressive mode (1 token per step)
- AR: Standard autoregressive using context tower only (1 token per step)
Mask diffusion works by initializing blocks as all [MASK] tokens, then iteratively denoising over multiple steps. High-confidence positions are committed based on a confidence threshold, with remaining positions re-masked until the block is complete.
Availability
The model is released under the NVIDIA Nemotron Open Model License Agreement and is ready for commercial use. Released on Hugging Face as nvidia/Nemotron-Labs-TwoTower-30B-A3B-Base-BF16.
What This Means
This release represents NVIDIA's attempt to accelerate inference through parallel token generation rather than sequential decoding. The 2.42× speedup claim comes with a measured quality trade-off—the 3.8-point drop on HumanEval and 3.8-point drop on MATH-500 indicate meaningful degradation on technical tasks, despite the "98.7% retained quality" aggregate figure. The architecture's complexity (dual towers, cross-attention, Mamba-2 states) may limit adoption compared to simpler speculative decoding approaches that achieve similar speedups. Pricing and availability through NVIDIA's API endpoints have not been disclosed.
Related Articles
NVIDIA releases Nemotron-Labs-Sigma: 14 open 3B and 8B continuous diffusion language model checkpoints
NVIDIA released Nemotron-Labs-Sigma, a set of 14 open continuous diffusion language model checkpoints at 3B and 8B parameters. According to MarkTechPost, Sigma denoises text in a learned 16-D embedding space, distills to few-step samplers, and decodes up to 4.20x faster than autoregressive decoding.
Nvidia team: huge context windows often overkill, as summaries and lookups matched them in many tests
An Nvidia research team argues that very large context windows are often overkill and expensive, according to The Decoder. In many tests, periodic summaries and targeted lookups worked about as well.
Tencent releases Hy4 flagship weights: 770B-parameter MoE, 1,048,576-token context, conditional license
Tencent released downloadable weights for its flagship Hy4 on November 20, with a System Card. The multimodal MoE has a 770-billion-parameter language backbone, 49 billion active parameters per text token and a 1,048,576-token context window, under a license with commercial conditions for model-as-a-service operators.
Paradigm Intelligence launches Phai-A1 language model and Phai-M1 video model, trained on Huawei Ascend
Paradigm Intelligence (范式智能, formerly 第四范式) has unveiled two in-house models: Phai-A1, a language model that runs on a single RTX 3090, and Phai-M1, a video model with synchronized audio. The company says both were trained on Huawei Ascend hardware, with open weights coming soon.
Comments
Loading...