Contents

StateBench shows that reading long documents and correctly tracking changes within them are different abilities. In reported tests, frontier models performed well on simple retrieval, but accuracy fell to 13% or below on the longest problems requiring chains of dependent updates.

Key Takeaways#

  • StateBench divides state tracking into four difficulty levels, from retrieving unchanged values to tracking updates that depend on other variables.
  • Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol, and GPT-6 Luna struggled with difficult tracking problems as sequences grew longer.
  • Directly trained recurrent and hybrid models generalized far beyond their training lengths at some difficulty levels, while Transformers showed stronger limitations.
  • Evaluate the update patterns your application uses. A large context window alone does not guarantee reliable tracking.

What StateBench Measures#

The StateBench announcement, published on November 3, 2026, introduces a benchmark for tracking changing information within documents. The central distinction is whether an answer can be found in a single assignment or must be computed from multiple updates.

The benchmark has four levels.

| Level | Allowed Updates | What the Model Must Do | |---|---|---| | R-trivial | Assign a value that never changes afterward | Retrieve the original value | | Aperiodic | Replace a value directly | Find the latest assignment | | Soluble | Update a value using its current value | Account for changes throughout the sequence | | General finite | Copy another variable’s current value or swap values | Track dependencies between variables |

These distinctions matter in evaluation. Finding a thermostat’s latest setting is easier than reconstructing a score from a history of point additions. Tracking an object repeatedly passed between people requires yet another kind of tracking.

Where Model Accuracy Fell#

The researchers tested open pretrained models, including Olmo 3, a 7-billion-parameter Transformer, and Olmo Hybrid, which includes Gated DeltaNet layers. The models being compared shared pretraining data and had not been trained on contexts longer than 64,000 tokens.

Most models performed well at retrieving unchanged values. Accuracy on problems asking for the latest value declined as inputs grew longer, while performance on General finite problems remained at chance level. Presentation format also affected results. Smaller hybrid and Gated DeltaNet models scored nearly perfectly on RULER-style retrieval problems, but approached chance level when equivalent problems were presented as spreadsheet edits.

The four frontier models performed well on the two easier levels. On Soluble and General finite tasks, accuracy dropped sharply as sequences grew longer, reaching 13% or below on the longest General finite problems. Because earlier updates affect multiple later operations, missing a single step can derail the final result.

What Direct Training Changed#

To distinguish gaps in pretraining from architectural limitations, the researchers trained six 275-million-parameter models from scratch: two Transformers, two hybrids, and two Gated DeltaNet variants.

Training sequences contained up to 256 assignments. Evaluation extended to 64,000 assignments, 250 times the maximum training length. Gated DeltaNet and hybrid models generalized nearly perfectly on the first three levels through the longest sequences tested. Transformers learned the easier levels, but performance declined with length, and they failed to learn the hardest level in these experiments.

On the hardest problems, the lengths models could handle were more limited. A Gated DeltaNet variant with more flexible memory updates maintained nearly perfect accuracy up to about 10 times the maximum training length. This supports further investigation of recurrent memory, but does not demonstrate reliable General finite tracking across the full 250-fold extension.

How to Apply the Findings#

Use StateBench’s categories to design evaluations around the changes your system must track. A document assistant that retrieves fixed facts needs different tests from an agent that tracks ownership, balances, or changing task states.

A practical evaluation can follow these four steps.

1. Identify the required update types: fixed facts, replacements, cumulative changes, and dependencies between entities. 2. Create sequences of varying lengths, including the longest expected in use, and independently compute the correct final answers. 3. Present equivalent sequences in formats users actually use, such as spreadsheet edits or conversation messages. 4. Measure final-state accuracy by update type and length, and examine where errors begin.

For applications where final-state errors have serious consequences, keep the authoritative state in a deterministic system such as a database. Have the model propose structured updates, then validate them and compute the resulting state outside the model.

What These Results Do Not Prove#

StateBench assumes a fixed number of variables. Real conversations and workflows introduce new people, objects, and records, so the current framework cannot cover every tracking requirement.

The findings also require careful interpretation. Direct-training results concern specific architectures and experimental conditions; they do not guarantee the behavior of deployed assistants. Questions also remain about the theoretical capabilities of constrained Gated DeltaNet variants, including the effects of numerical precision and attention layers.

The practical conclusion is narrower: test state tracking directly. Strong retrieval performance, an advertised large context window, and success on short examples are insufficient to establish that a model can maintain dependent updates throughout a long document.

Next Step#

Build one evaluation from a real workflow. Take the longest expected update sequence, independently compute its final state, and check whether your chosen model correctly returns every tracked value.

Source: https://huggingface.co/blog/allenai/statebench

Contents