EdgeRock-4B holds KV cache at 72 MiB up to 256K context, with 12,124 tokens/s on one 80GB GPU

EdgeRock-4B, an open-source model that keeps its attention KV cache fixed at 72 MiB at 256K context, was released by Rock Matrix (岩芯矩阵), a unit of Yanshan Technology (岩山科技), together with HKUST and Carnegie Mellon University, according to a report by Chinese outlet QbitAI (量子位), published in Chinese on 2026-10-26. The team says this is a 99.1% reduction in KV cache versus its source model, Qwen3.5-4B.

Throughput and memory

The throughput test used 32K input and 32K output on a single Hopper 80GB GPU, with each model run at its fastest measured batch size.

• EdgeRock-4B: 512 concurrent sequences, 12,124 tokens/s output, which the team puts at 6.42x the source model.
• Qwen3.5-4B: best configuration at batch size 64, 1,889 tokens/s.
• Gemma 4 E4B, MiniCPM5-2B, LFM2.5-2.6B: between 1,700 and 2,200 tokens/s.

Qwen3.5-4B adds 32 KiB of KV cache per token, which is 2 GiB per sequence at 32K input plus 32K output. EdgeRock-4B stores a fixed 97.76 MiB of total history state per sequence in batched runs. The report states that running 512 such sequences on Qwen3.5-4B would need about 1 TiB of KV cache at the end of generation, roughly the memory of 13 80GB cards. EdgeRock peaks at 61.26 GiB including model weights.

Across 16 workloads combining 4K to 32K input and output lengths, raising batch size from 32 to 512 lifted weighted decode throughput from 4,128 to 12,497 tokens/s without reaching the single-card memory limit. At 128K input followed by 128K generated tokens, EdgeRock held 233.7 tokens/s throughout, while Qwen3.5-4B fell from about 169 to 151.3 tokens/s.

For the same workload on the same card, EdgeRock needs 24.9% of Qwen3.5-4B's GPU time at 4K input and 15.6% at 32K. Averaged over four input lengths, its throughput is 4.94x Qwen3.5-4B and 7.18x LFM2.5-2.6B.

Reasoning benchmarks

The team compared EdgeRock-4B with 16 bounded-state models (GDN, RWKV, Mamba and others), three of them at 14B scale.

• AIME 2024-2026 average pass@1: 82.9% for EdgeRock-4B. The best bounded-state baseline, RWKV-7 G1j 13.3B, scored 18.8%. AHN-GDN Qwen2.5 14B scored 11.4%.
• 29 text benchmarks: first among bounded-state models on 24. The equal-weighted mean is 23.8 points above the strongest baseline, AHN-GDN Qwen2.5 14B.
• Versus Qwen3.5-4B: AIME average pass@1 is 96.3% of the source model's, with MATH-500 reaching 97.5%. EdgeRock is ahead on 8 of 29 text benchmarks, including MMLU-Pro, SuperGPQA, HumanEval and MBPP.
• Vision: 84.9% average across six benchmarks, retaining 95.3% of Qwen3.5-4B's score.

How it works

The core is Rock Memory Anchors (RMA). Each sequence's history is split into three parts: a recurrent state summarizing evolving context, a local window covering the latest 1,024 tokens, and an anchor bank holding the most valuable details from earlier history.

A writer produces a candidate record per token and a scoring network rates it. After every 256 candidates, they are compared with existing anchors and only the top 1,024 are kept. This runs independently in each of four KV groups.

The architecture keeps 24 Gated DeltaNet layers from Qwen3.5-4B, replaces 8 full-attention layers with 1,024-token sliding-window attention, and adds separate RMA modules. The model was converted from Qwen3.5-4B in stages, without pretraining from scratch.

In an ablation on 256 segments of 16K to 130K tokens (4 questions each, 1,024 total), the model answered 964 correctly (94.1%) with RMA on and 4 correctly (0.4%) with all eight RMA modules off.

Availability

Weights and inference code are open-sourced on Hugging Face and GitHub, and the model supports text and image reasoning. The team says commercial use is allowed and that larger EdgeRock sizes are coming. Pricing is not applicable to the open weights.

What this means

Because per-sequence memory is fixed, the memory a request needs can be calculated before inference starts, regardless of how long the model reasons. The report says this allows capacity planning by request count and one-time memory allocation on edge devices. The ablation result indicates the anchor bank, not the recurrent state or local window, carries long-range recall in this design. The throughput and benchmark figures come from the developers' own report, and the comparison base is a 4B model converted from Qwen3.5-4B.