research

Gemma 4, DeepSeek V4, and ZAYA1 Deploy KV Cache Compression to Cut Long-Context Memory Costs

TL;DR

Recent open-weight LLM releases from Google, DeepSeek, and others are adopting architectural techniques that reduce KV cache size by approximately 50% at long contexts. These include cross-layer KV sharing in Gemma 4, which saves 2.7 GB at 128K context for the E2B model, and compressed convolutional attention in ZAYA1-8B.

3 min read
0

Gemma 4, DeepSeek V4, and ZAYA1 Deploy KV Cache Compression to Cut Long-Context Memory Costs

Multiple open-weight LLM releases in April and May 2026 have adopted architectural techniques specifically designed to reduce KV cache size and memory traffic at long contexts, according to a technical analysis by Sebastian Raschka.

Cross-Layer KV Sharing in Gemma 4

Google's Gemma 4 suite, released in early April, implements cross-layer KV sharing in its E2B and E4B variants. Instead of computing separate key-value projections in each transformer layer, later layers reuse KV tensors from earlier layers while still computing their own query projections.

The Gemma 4 E2B model has 35 transformer layers but only 15 compute their own KV projections—the final 20 layers reuse KV tensors from previous layers. According to Raschka's calculations, this saves approximately 2.7 GB of memory at 128K context length (bfloat16 precision) for the E2B model. The E4B variant, with 42 layers (24 computing KV, 18 sharing), saves approximately 6 GB at 128K context.

The technique is not exclusive to Gemma 4. The cross-layer attention approach was described in Brandon et al.'s "Reducing Transformer Key-Value Cache Size with Cross-Layer Attention" (NeurIPS 2024), but Gemma 4 represents the first major open-weight implementation.

Additional Architecture Techniques

Beyond Gemma 4, other recent releases have implemented complementary approaches:

  • ZAYA1-8B: Uses compressed convolutional attention to reduce memory footprint
  • Laguna XS.2: Implements layer-wise attention budgeting to allocate attention compute selectively
  • DeepSeek V4: Combines multi-head compressed (mHC) attention with additional compression techniques

All three models also use Grouped Query Attention (GQA), which shares key-value heads across multiple query heads—a now-standard technique for KV cache reduction.

Model Variants and Target Use Cases

The Gemma 4 release includes three categories:

  1. E2B and E4B models: Optimized for mobile and embedded devices (IoT)
  2. 26B mixture-of-experts (MoE): Designed for efficient local inference
  3. 31B dense model: Optimized for maximum quality and easier fine-tuning

The E2B and E4B variants combine cross-layer KV sharing with a 4:1 pattern of regular GQA and sliding window attention. Specifically, E2B uses MQA (the single-KV-head special case of GQA) rather than full GQA.

Technical Implementation Details

In cross-layer KV sharing, sliding-window attention layers share KV with previous sliding-window layers, while full-attention layers share with previous full-attention layers. Each layer still computes its own query projections, allowing distinct attention patterns while eliminating redundant KV cache storage.

The memory savings scale with context length. At very long contexts (128K+), the KV cache becomes the dominant memory consumer, making these optimizations critical for reasoning models and agent workflows that maintain extended conversation history.

What This Means

The convergence on KV cache reduction techniques across multiple independent releases signals that long-context efficiency has become a primary architectural constraint. With reasoning models and agent workflows keeping more tokens active for longer periods, memory traffic and attention costs now dominate over pure compute. The ~50% KV cache reduction achieved through cross-layer sharing makes 128K+ context windows practical on consumer hardware. Expect these techniques to become standard in future model releases, particularly for edge deployment and long-context applications.

Related Articles

research

Microsoft Research's TokenEconomizer plugin cuts coding agent costs 18.4% to 30.9% on SWE-Bench

TokenEconomizer, an open-source plugin from UC Berkeley, Microsoft Research and Microsoft Security, learns from an agent's past sessions in a repository to cut token costs. Across 277 SWE-Bench tasks and five agent and model combinations, mean cost per task fell 18.4% to 30.9%, according to the authors.

research

Together AI says CUDA Graphs and PDL recover 98% of megakernel performance without custom kernels

Together AI reports that CUDA Graphs and PDL on top of default Torch Inductor kernels recover 98% of its earlier megakernel performance. With "multi-kernels" and agent loops, it claims up to 1.7x over its old megakernels, reaching over 1100 TPS for Llama 1B on H100. The work led to Together Compiler, a framework for continual AI inference optimization.

research

Google's Tunix hits bitwise-identical sampler and trainer logprobs for on-policy RL on 64 TPU v5p chips

Google engineers report a zero train-inference mismatch (Zero-TIM) implementation in Tunix, in which sampler and trainer assign bitwise-identical logprobs to the same token at the same weights. It was verified at multi-host scale across 64 TPU v5p chips (DP8×TP8). The recipe combines fixed-order Pallas kernels, explicit cross-shard reductions, a shared logprob path and custom differentiation rules.

research

NVIDIA recipe: RL post-training lifts Nemotron 3.5 Super VL accuracy a further 17% on drug-metabolism task

NVIDIA has published a recipe for post-training Nemotron 3.5 Super VL on scientific tasks using NeMo Gym and NeMo RL. In a drug-metabolism example, supervised fine-tuning improved accuracy by 12% and RL post-training lifted it a further 17% at a 16,384-token output budget.

Comments

Loading...