model releaseNVIDIA

NVIDIA releases Nemotron-Labs-Diffusion-14B with tri-mode decoding achieving 3.3x speed-up on GB200

TL;DR

NVIDIA released Nemotron-Labs-Diffusion-14B, a 14-billion parameter language model that supports three decoding modes by switching attention patterns during inference. The model achieves 850 tokens per second on GB200 hardware at concurrency 1, representing a 3.3x speed-up over standard autoregressive decoding and outperforming Qwen3-8B-Eagle3 by 2.2x in self-speculation mode.

2 min read
0

NVIDIA Releases Nemotron-Labs-Diffusion-14B with Tri-Mode Decoding

NVIDIA released Nemotron-Labs-Diffusion-14B, a 14-billion parameter language model that switches between autoregressive (AR), diffusion-based parallel decoding, and self-speculation modes by changing attention patterns during inference. According to NVIDIA, the model achieves 850 tokens per second on GB200 hardware at concurrency 1, representing a 3.3x speed-up compared to 253 tok/sec in standard AR mode.

Technical Architecture

The model family includes 3B, 8B, and 14B variants in base, instruct, and vision-language configurations. The architecture enables what NVIDIA calls "self-speculation": the same model performs diffusion-based parallel drafting and AR verification with shared KV cache. This approach shifts generation from memory-bound to compute-bound by loading model weights once and reusing them to compute multiple tokens.

The 8B variant shows 5.9x tokens per forward pass compared to Qwen3-8B without multi-token prediction, maintaining the same accuracy. In self-speculation mode, NVIDIA claims 3x higher acceptance length and 2.2x speed-up versus Qwen3-8B-Eagle3 in SGLang.

Performance Benchmarks

On DGX Spark hardware (8B model, concurrency 1), the model achieves 112 tok/sec using w4a16 quantization, representing 2.7x speed-up over AR's 41.8 tok/sec. On GB200, the 8B model reaches 850 tok/sec in self-speculation mode versus 360 tok/sec with Eagle3. Custom CUDA kernels push performance to 1,015 tok/sec, a 4x improvement over baseline AR.

NVIDIA's "speedup-of-light analysis" suggests throughput could double current best performance for single-user scenarios with improved sampling algorithms.

Implementation Details

The model supports three inference modes through simple API calls:

  • ar_generate() for standard autoregressive decoding
  • generate() for diffusion mode with configurable block length and threshold
  • linear_spec_generate() for self-speculation with optional LoRA adapter

An optional LoRA adapter can be applied to the diffusion drafter in linear self-speculation mode to increase acceptance length. The model requires transformers>=5.0.0 and runs on bfloat16 precision.

Availability

The model is available on Hugging Face under the NVIDIA Nemotron Open Model License. The release includes base model weights and a linear_spec LoRA adapter subfolder. NVIDIA provides example code for all three decoding modes with chat template support.

What This Means

This release represents a architectural shift in how language models handle inference efficiency. By enabling multiple decoding strategies within a single model through attention pattern switching, NVIDIA eliminates the need to deploy separate models for different latency-throughput tradeoffs. The self-speculation approach delivers substantial speed gains without external draft models, potentially reducing deployment complexity for organizations operating at varying concurrency levels. However, real-world performance will depend on workload characteristics and whether the compute-bound regime benefits materialize across diverse use cases.

Related Articles

model release

Google releases Gemini 4 Neon at $3/$15 per 1M tokens with a 1M-token context window

Google released Gemini 4 Neon, available today through the API at $3 per million input tokens and $15 per million output tokens. It is the fourth new Gemini 4 model in four weeks, with a context window of about one million tokens.

model release

Tencent releases Hy4 flagship weights: 770B-parameter MoE, 1,048,576-token context, conditional license

Tencent released downloadable weights for its flagship Hy4 on November 20, with a System Card. The multimodal MoE has a 770-billion-parameter language backbone, 49 billion active parameters per text token and a 1,048,576-token context window, under a license with commercial conditions for model-as-a-service operators.

model release

Resolve AI launches SRE-1, a 27B-parameter model trained to find root causes of production alerts

Resolve AI has launched SRE-1, a 27 billion-parameter model built on open weights and trained with reinforcement learning to find root causes of production alerts. The company also made Primary On-call generally available, putting its AI agents on call ahead of human engineers.

model release

Microsoft launches seven models at Ignite; claims MAI Code 2 matches 96% of Claude Opus 5.5 at 5% of price

Microsoft unveiled seven AI models at Ignite on Nov. 17, aimed at coding, cybersecurity and other enterprise tasks. The company claims MAI Code 2 delivered 96% of Claude Opus 5.5's performance across four coding benchmarks at 5% of the price, using a promotional rate.

Comments

Loading...