Language models lose track of changing facts in long texts, Ai2 says
In Ai2's own tests, leading language models handled the easy levels of its new StateBench test but struggled on the hard ones. The test has four levels, from fixed values to swapped items, and its data and code are published.
Language models, the AI systems that read and write text, lose accuracy when the facts in a long document keep changing, according to a test from Ai2 (the Allen Institute for AI). Ai2 announced the test, StateBench, on 3 November 2026.
StateBench has four difficulty levels, each allowing more complex kinds of updates. Ai2 explains the easier ones with everyday pictures: a fixed birth date, a thermostat where the latest setting wins, and a running game score. It explains the hardest with a magician who hides a ball under one of five cups, then swaps the cups again and again.
Ai2 also gives a reading example: in a mystery novel, a reader must know that a knife passed from one character to another just before a murder changes each character's situation.
In Ai2's own tests, four of the most capable models, Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna, did well on the two easier levels. On the two harder levels, accuracy fell sharply as inputs grew longer, dropping to 13% or lower on the longest hardest problems.
Ai2 suggests a likely reason: at the harder levels many more steps affect the final answer, so missing one update is more likely to make the answer wrong.
Open pretrained models that Ai2 tested, meaning models already trained before the test, almost all did well on the easiest level. They lost accuracy on the second level as inputs grew, and did no better than chance on the hardest.
Presentation mattered too. Some small models scored near the top on one version of the easiest problems but near chance on the same problems framed as spreadsheet edits.
In the accompanying paper, Jackson Petty, William Merrill and Ashish Sabharwal, all affiliated with Ai2, write that none of the pretrained models, open or closed, reliably solved even the third level. They present that as a direction for future improvement.
Ai2 also trained small models directly on StateBench, on sequences of up to 256 updates. It tested them on sequences up to 64,000 updates long, 250 times the longest training length.
Some newer designs, such as Gated DeltaNet and hybrid models, carry a running memory forward as they read. When trained directly, they handled the first three levels almost perfectly out to the longest sequences tested.
Transformers, the design these newer ones are offered as an alternative to, never learned the hardest level, even when trained directly.
Ai2 names a limit of its test: StateBench assumes a fixed number of things to track, while real tasks often bring in new people and objects over time.
Ai2 has published a technical report, the data and the code, and says it hopes StateBench helps researchers find where models break down.
Sources
- Statebench · allenai.org
- arxiv.org · arxiv.org