NVIDIA releases Nemotron 3.5 Super VL NVFP4: 120B-parameter, 12B-active vision-language model with 1M context

NVIDIA has released Nemotron 3.5 Super VL 120B-A12B in NVFP4 precision, a vision-language model with 120B total parameters, 12B active per token, and a context window of up to 1M tokens. The weights are on Hugging Face under the OpenMDW-1.1 license, which NVIDIA says permits commercial and non-commercial use. The Hugging Face card lists the release date as October 15, 2026.

Architecture and specs

• Parameters: 120B total, 12B active
• Design: Hybrid of interleaved Mamba-2, Mixture-of-Experts and attention layers, with 512 routed experts per MoE layer
• Vision: C-RADIO vision encoder plus a vision-language connector
• Inputs/outputs: Text, images and video in; text out
• Context: Up to 1M tokens. NVIDIA notes that individual deployment recipes may use less to fit GPU memory and concurrency.
• Decoding: A shared Multi-Token Prediction (MTP) head enables native speculative decoding
• Reasoning: Configurable through the chat template (enable_thinking=True/False), with levels of off, on (medium) and on. NVIDIA says medium generates fewer tokens but may reduce accuracy.
• Pricing: Not applicable to the open weights. No hosted API pricing has been disclosed.

The NVFP4 checkpoint was produced with quantization-aware distillation from the BF16 model. NVIDIA lists deployment targets of 1× DGX Spark, 1× H200, 2× H100 or 1× GB200.

Training

According to the model card, the language backbone is initialized from Nemotron 3 Super Base. A shared MTP head is trained for 100B tokens with the backbone frozen, followed by 1T tokens of distillation from Nemotron 3 Ultra Base. Later stages include 100B tokens of Indic continued pre-training. The source text is truncated after that point, and the training data cutoff date is not stated in the available material.

Benchmarks (NVIDIA-reported)

All figures below are NVIDIA's own, collected via its NeMo Gym tooling. NVIDIA says BF16 and NVFP4 runs used identical configs and harnesses. Scores are NVFP4, with BF16 in parentheses.

• GPQA-Diamond: 89.90 (89.52)
• LiveCodeBench v6: 87.41 (88.22)
• SWE-bench Verified: 71.07 (71.10)
• SWE-bench Pro: 51.30 (51.78)
• Terminal-Bench 2.1: 62.78 (60.67)
• OSWorld-Verified: 52.3 (51.14)
• BrowseComp: 77.5 (78)
• HLE, text-only: 31.44 (31.79)
• MMLU-ProX: 73.76 (76.91)
• CharXiv reasoning: 79.88 (80.82)
• Video-MME v2: 32.54 (32.85)

Quantization loss is small on most tests, but not uniform. MMLU-ProX drops about 3 points. In the offline agentic set, WebVoyager falls from 77.72 to 66.31 and DeepSWE from 24.65 to 18.2. Several benchmarks (BabyVision, CharXiv-RQ, MathVision, MMLongBench-Doc, OCR-Reasoning, OCRBench-v2, V*, Video-MME-v2) use official or internal harnesses that NVIDIA says it plans to open-source later. No third-party replication has been published.

Deployment notes

The quick-start targets DGX Spark (GB10) with vLLM, using a companion draft model (...-NVFP4-LiLiCorr) with seven speculative tokens per draft and a --video-pruning-rate of 0.5. NVIDIA recommends temperature 1.0 and top_p 0.95 for text, and temperature 0.6 and top_p 0.95 for vision tasks. SGLang is also supported.

What this means

The notable point is the footprint. A 120B-parameter model with 1M context and video input, running on a single DGX Spark or one H200, moves long-context multimodal agents onto much smaller hardware. The 12B active parameter count and native MTP are aimed at keeping latency workable.

If NVIDIA's numbers hold, NVFP4 costs little on coding, reasoning and document tasks, so most teams can skip BF16. The exceptions are browser-agent workloads like WebVoyager and DeepSWE, where the reported drops are large enough to test before committing. Because the evaluation uses NVIDIA's own harnesses, independent results will matter, especially for the 1M-context claim in practice, since the card says recipes often use shorter windows.