NVIDIA: Sparse attention indexer and Top-k take 47% of a DeepSeek V3.2 prefill layer at 128K tokens
NVIDIA published a co-design primer for model-level sparse attention. It reports that the indexer and Top-k take 47% of a DeepSeek V3.2 prefill layer at 128K tokens, and that reusing the prior decode step's selection speeds up Top-k by 2.1x to 2.4x.
NVIDIA published a practical guide to designing sparse attention for its GPUs on November 17, 2026, reporting that the indexer and Top-k stages account for 47% of per-layer prefill time in DeepSeek V3.2 at an input sequence length of 128K tokens. The post, by Timmy Liu, Nidhi Bhatia, Samkit Jain, Makesh Tarun Chandran and Bita Darvish Rouhani, appeared on the NVIDIA Developer blog.
Where the cost goes
According to NVIDIA, dense attention scores every query against every key, so cost per query grows with context length N. At 128K tokens, attention can take most of the prefill time in a layer. In NVIDIA's Figure 1, attention is 91% of per-layer prefill time for DeepSeek-R1 (dense MLA).
For DeepSeek V3.2 (DSA), the same figure splits prefill time as follows:
- Indexer and Top-k: 47%
- Core attention: 24%
- FFN: 25%
Sparse attention has three stages: an indexer scores every key, Top-k keeps the k highest scores, and core attention runs softmax attention over those k keys and values. The last two stages scale with k, while the indexer still scales with N. NVIDIA's framing is that the cost does not disappear but moves into the indexer.
Scope and method
The post covers model-level sparse attention, where sparsity is trained into the model. It does not cover post-training drop-in methods such as BLASST, MInference or Quest. NVIDIA used a production FP8 kernel to sweep the indexer's design knobs: query heads, key heads, head dimension, compression ratio and speculative-decoding depth. It also measured how Top-k cost scales and compared token-level with block-level core attention.
Design guidelines
- Group size and speculation: The indexer runs as a batched matmul with a fused epilogue. Its arithmetic intensity scales with group size G = QH/KH and speculative draft length D+1, which should fit within one Tensor Core tile of about 200 rows.
- Key heads: Using KH = 1 with a cross-head reduction writes one logit per key. This avoids the linear decode runtime growth seen when key heads are increased.
- Head dimension and precision: An indexer head dimension of 128 or 256 balances Tensor Core utilization against decode memory traffic. Running the epilogue and accumulation in FP16 improves throughput further.
- Fewer keys scored: Compression, hierarchical indexing or index sharing cut work in both the indexer and Top-k selection without changing arithmetic intensity.
- Fused Top-k: Fusing Top-k into the indexer kernel eliminates the logit write and read-back. Reusing the previous decode step's selection as a guess speeds up Top-k by 2.1x to 2.4x.
- Block-level selection: Large blocks let core attention fetch keys in bulk and share them across queries. A large core attention group size increases block reuse across query heads.
Code and papers
NVIDIA points readers to the Guess-Verify-Refine Top-k paper for the decode-time selection optimization, the TensorRT LLM repository for the fused Top-k implementation, and the selective Top-k path in FlashInfer for an open-source fused indexer and Top-k kernel.
What this means
The 47% figure shows that in a sparse-attention model, the indexer and Top-k can take a larger share of prefill time than core attention (24%) or FFN (25%) in DeepSeek V3.2 at 128K tokens. Model designers therefore have to treat the indexer as a first-class cost, not an afterthought. NVIDIA's guidance ties architecture choices such as head counts, head dimension and block size to Tensor Core tile limits and memory traffic, so these decisions can be made before training. The 2.1x to 2.4x Top-k speedup from reusing the previous decode step's selection is an NVIDIA-reported result from its own kernel sweeps. Teams can check it against the open-source FlashInfer path and the TensorRT LLM implementation.
Related Articles
NVIDIA launches Nemotron for Science with Nemotron 3.5 Super VL and a chemistry dataset of nearly 50,000 questions
NVIDIA introduced Nemotron for Science on November 16, 2026. It pairs the open Nemotron 3.5 Super VL model with scientific evaluations, post-training recipes and LitmusBench v0.2, a chemistry dataset of nearly 50,000 questions.
NVIDIA recipe: RL post-training lifts Nemotron 3.5 Super VL accuracy a further 17% on drug-metabolism task
NVIDIA has published a recipe for post-training Nemotron 3.5 Super VL on scientific tasks using NeMo Gym and NeMo RL. In a drug-metabolism example, supervised fine-tuning improved accuracy by 12% and RL post-training lifted it a further 17% at a 16,384-token output budget.
NVIDIA Vera Rubin NVL72 posts over 10x GB300 NVL72 throughput per kW on MLPerf Endpoints v1.0
Provisional MLPerf Endpoints v1.0 results show NVIDIA Vera Rubin NVL72 delivering over 10x the throughput per kilowatt of GB300 NVL72 at a shared minimum interactivity of 50 tokens per second per user. The Vera Rubin results are pending MLCommons verification; the GB300 results are verified.
NVIDIA Vera Rubin NVL72 posts fastest time to train on six of seven MLPerf Training v6.1 benchmarks
In its first MLPerf Training appearance, NVIDIA's Vera Rubin NVL72 delivered the fastest time to train on six of seven benchmarks, with up to 2x the per-GPU performance of an optimized GB300 NVL72 baseline, according to NVIDIA. On DeepSeek-V3 671B it trained in 4.14 minutes on 1,024 GPUs versus 8.03 minutes for GB300 NVL72.
Comments
Loading...