NVIDIA releases Nemotron-Labs-3-Puzzle-75B, compressed from 120B to 75B parameters with 2× throughput
NVIDIA has released Nemotron-Labs-3-Puzzle-75B-A9B, a compressed variant of Nemotron-3-Super that reduces the model from 120.7B total/12.8B active parameters to 75.3B total/9.3B active parameters. According to NVIDIA, the model achieves approximately 2× higher server throughput on a single 8×B200 node and increases sustainable 1M-token single-H100 concurrency from 1 request to 8 requests while maintaining strong accuracy across benchmarks.
NVIDIA Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4 at a glance
How NVIDIA Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4 ranks now
| Benchmark | Score | Rank | Source |
|---|---|---|---|
| GPQA | 78.0% | 118 of 165 | Hugging Face model card, Jul 2026 ↗ |
| MMLU-Pro | 82.2% | 40 of 76 | Hugging Face model card, Jul 2026 ↗ |
| LiveCodeBench | 79.9% | 22 of 48 | Hugging Face model card, Jul 2026 ↗ |
Rank is against every model in our database with a sourced score on that benchmark, each counted once with its best score.
NVIDIA Releases Compressed Nemotron-Labs-3-Puzzle-75B with 2× Throughput Gains
NVIDIA has released Nemotron-Labs-3-Puzzle-75B-A9B, a deployment-optimized large language model that reduces the parent Nemotron-3-Super from 120.7B total/12.8B active parameters to 75.3B total/9.3B active parameters. The model is available immediately on Hugging Face under the OpenMDW License Agreement version 1.1 for commercial use.
Architecture and Compression Approach
The model employs a hybrid MoE architecture with interleaved Mamba, MoE, and Attention layers, and supports Multi-Token Prediction (MTP) for faster text generation. The compression was achieved through NVIDIA's Iterative Puzzle framework, which applied three key optimizations:
- Heterogeneous MoE Channel Pruning: Expert intermediate dimensions reduced from 2688 to 1280-2688 depending on layer sensitivity
- Active Expert Reduction: Number of activated routed experts per token reduced from 22 to 4-18 across layers
- Mamba SSM State Pruning: State size reduced from 128 to 96 channels
The compression pipeline involved three stages of iterative pruning and recovery, followed by long-context knowledge distillation at 128K and 512K sequence lengths, using up to 100B training tokens per phase.
Performance Claims
According to NVIDIA, compared to Nemotron-3-Super, Puzzle-75B achieves:
- Approximately 2× higher server throughput on a single 8×B200 node at matched user-throughput constraints
- 8× increase in sustainable 1M-token single-H100 concurrency (from 1 to 8 requests)
- Maintained accuracy across reasoning, coding, multilingual, long-context, and agentic benchmarks
Benchmark Results
The NVFP4 quantized variant scores 82.2 on MMLU-Pro, 89.9 on AIME25 (no tools), and 92.9 on HMMT Feb25 (no tools). For long-context tasks, RULER scores reach 95.3% at 256K tokens, 94.8% at 512K, and 93.2% at 1M tokens.
Coding benchmarks show 79.9 on LiveCodeBench (v5, 2024-07 to 2024-12) and 40.3 on SciCode subtasks. Agentic performance includes 23.4 on Terminal Bench hard subset and an average of 59.9 across TauBench V2 tasks.
Multilingual capabilities span English, French, German, Italian, Japanese, Spanish, and Chinese, with an average MMLU-ProX score of 76.5 and WMT24++ (en→xx) score of 85.1.
Training Details
The model was produced through a multi-stage pipeline combining knowledge distillation from the parent model, with training performed at sequence lengths up to 512K tokens using a global batch size of 16M tokens. The final model underwent reinforcement learning recovery, post-training quantization, and continued MTP training.
What This Means
This release demonstrates a practical approach to reducing inference costs while maintaining model quality through heterogeneous compression—pruning different layers differently based on their impact on downstream tasks. The 8× increase in sustainable concurrency per H100 GPU at 1M-token contexts directly addresses the economics of deploying long-context models at scale. NVIDIA's decision to use layer-specific compression rates rather than uniform pruning suggests that not all transformer layers contribute equally to model performance, a finding that could inform future model design and compression research.
Related Articles
NVIDIA releases Nemotron-Labs-Sigma: 14 open 3B and 8B continuous diffusion language model checkpoints
NVIDIA released Nemotron-Labs-Sigma, a set of 14 open continuous diffusion language model checkpoints at 3B and 8B parameters. According to MarkTechPost, Sigma denoises text in a learned 16-D embedding space, distills to few-step samplers, and decodes up to 4.20x faster than autoregressive decoding.
Nvidia team: huge context windows often overkill, as summaries and lookups matched them in many tests
An Nvidia research team argues that very large context windows are often overkill and expensive, according to The Decoder. In many tests, periodic summaries and targeted lookups worked about as well.
OpenAI releases GPT-Live-1.1 in the API, with standby mode billing idle WebRTC time at $0.02 per minute
OpenAI has released GPT-Live-1.1 in the API. The full-duplex voice model improves multilingual accents and transcription, and a new standby mode bills idle WebRTC time at $0.02 per minute instead of $0.05.
Baseten releases open-weight bison-1 and groundhog-1 decision models; bison-1 scores 76 on AA Decision Index v2.1
Baseten released bison-1 and groundhog-1, two open-weight decision models built on Gemma 4. Bison-1 scores 76 on the AA Decision Index v2.1, and both models support 128k context and image input.
Comments
Loading...