NVIDIA Releases Compressed Nemotron-Labs-3-Puzzle-75B with 2× Throughput Gains

NVIDIA has released Nemotron-Labs-3-Puzzle-75B-A9B, a deployment-optimized large language model that reduces the parent Nemotron-3-Super from 120.7B total/12.8B active parameters to 75.3B total/9.3B active parameters. The model is available immediately on Hugging Face under the OpenMDW License Agreement version 1.1 for commercial use.

Architecture and Compression Approach

The model employs a hybrid MoE architecture with interleaved Mamba, MoE, and Attention layers, and supports Multi-Token Prediction (MTP) for faster text generation. The compression was achieved through NVIDIA's Iterative Puzzle framework, which applied three key optimizations:

• Heterogeneous MoE Channel Pruning: Expert intermediate dimensions reduced from 2688 to 1280-2688 depending on layer sensitivity
• Active Expert Reduction: Number of activated routed experts per token reduced from 22 to 4-18 across layers
• Mamba SSM State Pruning: State size reduced from 128 to 96 channels

The compression pipeline involved three stages of iterative pruning and recovery, followed by long-context knowledge distillation at 128K and 512K sequence lengths, using up to 100B training tokens per phase.

Performance Claims

According to NVIDIA, compared to Nemotron-3-Super, Puzzle-75B achieves:

• Approximately 2× higher server throughput on a single 8×B200 node at matched user-throughput constraints
• 8× increase in sustainable 1M-token single-H100 concurrency (from 1 to 8 requests)
• Maintained accuracy across reasoning, coding, multilingual, long-context, and agentic benchmarks

Benchmark Results

The NVFP4 quantized variant scores 82.2 on MMLU-Pro, 89.9 on AIME25 (no tools), and 92.9 on HMMT Feb25 (no tools). For long-context tasks, RULER scores reach 95.3% at 256K tokens, 94.8% at 512K, and 93.2% at 1M tokens.

Coding benchmarks show 79.9 on LiveCodeBench (v5, 2024-07 to 2024-12) and 40.3 on SciCode subtasks. Agentic performance includes 23.4 on Terminal Bench hard subset and an average of 59.9 across TauBench V2 tasks.

Multilingual capabilities span English, French, German, Italian, Japanese, Spanish, and Chinese, with an average MMLU-ProX score of 76.5 and WMT24++ (en→xx) score of 85.1.

Training Details

The model was produced through a multi-stage pipeline combining knowledge distillation from the parent model, with training performed at sequence lengths up to 512K tokens using a global batch size of 16M tokens. The final model underwent reinforcement learning recovery, post-training quantization, and continued MTP training.

What This Means

This release demonstrates a practical approach to reducing inference costs while maintaining model quality through heterogeneous compression—pruning different layers differently based on their impact on downstream tasks. The 8× increase in sustainable concurrency per H100 GPU at 1M-token contexts directly addresses the economics of deploying long-context models at scale. NVIDIA's decision to use layer-specific compression rates rather than uniform pruning suggests that not all transformer layers contribute equally to model performance, a finding that could inform future model design and compression research.