Ai2's StateBench: Frontier models fall to 13% or lower on hardest long-input state tracking

Ai2 has introduced StateBench, a benchmark for long-context state tracking, and reports that four frontier models (Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna) fall to 13% accuracy or lower on the longest general finite problems. The release includes a tech report, data and code. The announcement was published November 3, 2026 on the Hugging Face blog.

What StateBench measures

State tracking is how well a model follows information about entities as it changes across a long document. Examples include variables reassigned in code or objects changing hands in a story. Ai2 says existing tests tend to focus on the hardest cases, while StateBench is built to show which kinds of state tracking each architecture can handle.

The benchmark has four difficulty levels, framed as types of variable assignment and grounded in Krohn-Rhodes theory:

• R-trivial: immutable assignment. The model recovers the value a variable was first set to (let x = 4).
• Aperiodic: reassignment by value. The model finds the most recent value (x = 4, later x = 2).
• Soluble: self-referential updates, where each update builds on the current value (x = x + 5).
• General finite: assignment by reference, where one variable takes another's current value (x = y, or swaps). The S5 problem, tracking a ball as five cups are repeatedly swapped, belongs to this level.

All four problems derive from one general finite sequence. They share the same number of variables and distinct values, and assignments fall at the same points in the sequence. Only the assignment type changes. StateBench also tests the same underlying problem in several formats.

Results

Open pretrained models. Ai2 tested two 7-billion-parameter models, Olmo 3 (a transformer) and Olmo Hybrid (which adds Gated DeltaNet layers), plus smaller models with 275 million and 2.7 billion parameters. All were pretrained on the same data, and none saw contexts longer than 64,000 tokens in training. Almost all performed well on r-trivial problems, lost accuracy on aperiodic ones as inputs grew, and stayed at chance on general finite problems. Format mattered: the 275-million-parameter hybrid and Gated DeltaNet models scored near the ceiling on RULER-style r-trivial problems but near chance on the same problems framed as spreadsheet edits.

Frontier models. Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna were asked to read each sequence of assignments and generate the final value of every variable. All four performed well on r-trivial and aperiodic problems. On soluble and general finite problems, accuracy fell sharply as inputs grew longer, reaching 13% or lower on the longest general finite problems. Ai2 says one likely factor is sensitivity: at the harder levels, many more steps can affect the final answer, leaving less room to miss an update.

Models trained directly on the task. Ai2 trained several 275-million-parameter models from scratch: two transformer variants, two hybrids, and two Gated DeltaNet variants with different memory-update constraints. They trained on sequences of up to 256 assignments and were tested on sequences as long as 64,000 assignments, or 250 times the maximum training length. Gated DeltaNet and hybrid models generalized almost perfectly on r-trivial, aperiodic and soluble problems out to the longest sequences tested, tens of thousands of assignments. Transformers learned the easier levels but degraded on longer sequences.

What this means

The frontier-model results show a split by problem type rather than a uniform weakness. The four models handled finding the latest value but lost accuracy once answers depended on carrying changes forward or following references between variables, and the drop grew with input length.

The from-scratch experiments address whether the cause is architecture or training data. When trained directly on StateBench, recurrent and hybrid models generalized far beyond their training length, while transformers degraded. Ai2 frames this as evidence about the architecture and optimizer rather than the pretraining data.

The format gap is a practical caution for evaluation. Models near the ceiling on RULER-style problems were near chance on the same problem framed as spreadsheet edits, so a single-format test can overstate or understate state-tracking ability. That is why StateBench uses several formats.