state-tracking

Ai2 released StateBench, a benchmark with four difficulty levels for how language models track entity state changes in long documents. Four frontier models scored well on the two easier levels but fell to 13% or lower on the longest general finite problems. Recurrent and hybrid models trained directly on the task generalized much farther than transformers.