model releaseNVIDIA

NVIDIA releases Nemotron-Labs-3-Puzzle-75B, compressed from 120B to 75B parameters with 2× throughput

TL;DR

NVIDIA has released Nemotron-Labs-3-Puzzle-75B-A9B, a compressed variant of Nemotron-3-Super that reduces the model from 120.7B total/12.8B active parameters to 75.3B total/9.3B active parameters. According to NVIDIA, the model achieves approximately 2× higher server throughput on a single 8×B200 node and increases sustainable 1M-token single-H100 concurrency from 1 request to 8 requests while maintaining strong accuracy across benchmarks.

2 min read
0

How NVIDIA Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4 ranks now

BenchmarkScoreRankSource
GPQA78.0%118 of 165Hugging Face model card, Jul 2026 ↗
MMLU-Pro82.2%40 of 76Hugging Face model card, Jul 2026 ↗
LiveCodeBench79.9%22 of 48Hugging Face model card, Jul 2026 ↗

Rank is against every model in our database with a sourced score on that benchmark, each counted once with its best score.

All NVIDIA Nemotron-Labs-3-Puzzle-75B-A9B-NVFP4 specs and scores →

NVIDIA Releases Compressed Nemotron-Labs-3-Puzzle-75B with 2× Throughput Gains

NVIDIA has released Nemotron-Labs-3-Puzzle-75B-A9B, a deployment-optimized large language model that reduces the parent Nemotron-3-Super from 120.7B total/12.8B active parameters to 75.3B total/9.3B active parameters. The model is available immediately on Hugging Face under the OpenMDW License Agreement version 1.1 for commercial use.

Architecture and Compression Approach

The model employs a hybrid MoE architecture with interleaved Mamba, MoE, and Attention layers, and supports Multi-Token Prediction (MTP) for faster text generation. The compression was achieved through NVIDIA's Iterative Puzzle framework, which applied three key optimizations:

  • Heterogeneous MoE Channel Pruning: Expert intermediate dimensions reduced from 2688 to 1280-2688 depending on layer sensitivity
  • Active Expert Reduction: Number of activated routed experts per token reduced from 22 to 4-18 across layers
  • Mamba SSM State Pruning: State size reduced from 128 to 96 channels

The compression pipeline involved three stages of iterative pruning and recovery, followed by long-context knowledge distillation at 128K and 512K sequence lengths, using up to 100B training tokens per phase.

Performance Claims

According to NVIDIA, compared to Nemotron-3-Super, Puzzle-75B achieves:

  • Approximately 2× higher server throughput on a single 8×B200 node at matched user-throughput constraints
  • 8× increase in sustainable 1M-token single-H100 concurrency (from 1 to 8 requests)
  • Maintained accuracy across reasoning, coding, multilingual, long-context, and agentic benchmarks

Benchmark Results

The NVFP4 quantized variant scores 82.2 on MMLU-Pro, 89.9 on AIME25 (no tools), and 92.9 on HMMT Feb25 (no tools). For long-context tasks, RULER scores reach 95.3% at 256K tokens, 94.8% at 512K, and 93.2% at 1M tokens.

Coding benchmarks show 79.9 on LiveCodeBench (v5, 2024-07 to 2024-12) and 40.3 on SciCode subtasks. Agentic performance includes 23.4 on Terminal Bench hard subset and an average of 59.9 across TauBench V2 tasks.

Multilingual capabilities span English, French, German, Italian, Japanese, Spanish, and Chinese, with an average MMLU-ProX score of 76.5 and WMT24++ (en→xx) score of 85.1.

Training Details

The model was produced through a multi-stage pipeline combining knowledge distillation from the parent model, with training performed at sequence lengths up to 512K tokens using a global batch size of 16M tokens. The final model underwent reinforcement learning recovery, post-training quantization, and continued MTP training.

What This Means

This release demonstrates a practical approach to reducing inference costs while maintaining model quality through heterogeneous compression—pruning different layers differently based on their impact on downstream tasks. The 8× increase in sustainable concurrency per H100 GPU at 1M-token contexts directly addresses the economics of deploying long-context models at scale. NVIDIA's decision to use layer-specific compression rates rather than uniform pruning suggests that not all transformer layers contribute equally to model performance, a finding that could inform future model design and compression research.

Comments

Loading...