Long Context Is Not Memory: Two Papers This Week Say Keep Your Agent's State Outside the Model

See what broke, and the fixHover to explore

If your agent works through a long history of changes to work out where things stand now, two papers published this week are worth ten minutes. Think of a ticket that was reassigned four times, a stock count adjusted on every order, or a budget that rose and fell across a hundred messages. The first paper shows that frontier models can read that history and still get the final state wrong once it is long enough. The second shows that you can usually give the model far less context than it can hold, lose little or no accuracy, and pay much less. Neither paper was written for founders, but together they describe a design you can adopt this week.

This article is based on Ai2's StateBench announcement and its technical report, arXiv:2611.03094, and on arXiv:2611.05306, "You don't need long context". Figures and quotations are from those documents. The reading from section 3 onward is mine.

1. What StateBench measured

StateBench, published by Ai2 on 3 November with a technical report dated 2 November, tests one narrow skill: following a sequence of updates and reporting the final state. Its authors, Jackson Petty, William Merrill and Ashish Sabharwal, built four levels of difficulty that differ only in the kind of update:

• Constant assignment. A value is set and never changes.
• Reassignment by value. A variable is set to a new value.
• Self-update. A variable changes relative to itself, as in x = x + 5.
• Assignment by reference. A variable takes another variable's value, as in x = y, or two values swap.

Each problem is written in six formats with the same structure underneath, from plain code to filling in spreadsheet cells and tracking people moving between rooms. Frontier models answered in their own words rather than picking from options. The four tested were Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna.

The result was clear. In the report's words, all frontier models "show strong performance on r-trivial and aperiodic problems", but "fall to at-chance accuracy on soluble and finite state-tracking problems" once inputs reach between 10K and 64K tokens. The first two levels held; the last two collapsed. Ai2's announcement puts the hardest case at "13% or lower on the longest general finite problems."

One more finding matters for anyone choosing a model on benchmark scores. Small pretrained models that scored near the ceiling on RULER-style problems, the needle-in-a-haystack family, were near chance when the same problems were written as spreadsheet edits. Being able to find a fact in a long input does not mean the model can keep track of how that fact changed.

2. What the compaction paper measured

The second paper, "You don't need long context" by Peter Belcak and eleven co-authors, posted on 3 November, starts from cost: "Later tokens are costlier to generate than earlier tokens." On the open-weight models it served, generation at a 256K context window cost 3.6x to 7x more than at 16K.

It then tested two ways of keeping context short. In the first, the agent runs normally until its context crosses a threshold, then writes a bounded brief of its state, and a fresh conversation starts from the original task plus that brief. In the second, more extreme, each conversation performs a single action, updates the brief, and is thrown away. The briefs that worked contained a near-verbatim copy of the task, the agent's current intentions and next steps, and the artifacts future work would need. The authors call them "carefully cherry-picked briefs", not narratives.

On long-horizon tasks, including SWE-bench Verified and Terminal-Bench 2.1, accuracy moved between -3.7 and +5.6 percentage points for Claude Opus 4.5 and between -3.3 and +4.4 for GPT 5.6 Sol, against estimated retail savings of 2.2x to 8.8x and 2.4x to 5.0x. On long-input tasks, accuracy changed by -0.4 to +40.0 points for Opus, with estimated savings of 6.5x to 54.6x. Even one action per conversation held up: Opus went from 70.0 to 70.2 on SWE-bench Verified, while GPT 5.6 Sol went from 84.4 to 82.4.

3. Why this is your problem, not a lab's

Most agents in production are state trackers, whether or not anyone calls them that. A support agent works out who owns a ticket after three handoffs. A procurement agent keeps a running total against a budget. An operations agent answers "what is the current status?" from a log of events. In StateBench's terms, a running total is a self-update and a handoff is assignment by reference. Those are the two levels where frontier models fell to chance as inputs grew.

The common design makes this worse. Many agents keep the full transcript and every tool result in context, and ask the model to work out the current state from it each turn. That works in a demo with twenty messages. It is the pattern StateBench breaks at 10K to 64K tokens, a length an agent with chatty tool output can reach within one working session.

The compaction paper suggests the fix costs little. If a short brief keeps accuracy within a few points on hard agent tasks, there is no strong reason to make the model reconstruct state from a long history. And the model is not the right place to keep the authoritative copy of that state anyway.

4. Move the state out of the prompt

The pattern is simple. Your code holds the current state. The model reads that state, proposes changes as structured operations, and your code applies them. Each turn starts from a short brief, not from the full history.

from dataclasses import dataclass, field

@dataclass
class OrderState:
    status: str = "draft"
    quantity: int = 0
    assignee: str | None = None
    log: list[dict] = field(default_factory=list)

ALLOWED_OPS = {"set_status", "add_quantity", "assign"}

def apply(state: OrderState, op: dict) -> OrderState:
    kind = op.get("op")
    if kind not in ALLOWED_OPS:
        raise ValueError(f"unknown operation: {kind}")
    if kind == "set_status":
        state.status = op["value"]
    elif kind == "add_quantity":
        state.quantity += int(op["value"])  # arithmetic happens here, not in the model
    elif kind == "assign":
        state.assignee = op["value"]
    state.log.append(op)  # full history kept for audit, not for the prompt
    return state

def brief(task: str, state: OrderState, next_steps: list[str]) -> str:
    return (
        f"TASK: {task}\n"
        f"CURRENT STATE: status={state.status}, quantity={state.quantity}, "
        f"assignee={state.assignee}\n"
        f"NEXT STEPS: {'; '.join(next_steps)}"
    )

Two cautions. A brief written by the model is model output: in September we covered models writing instructions into their own compaction summaries, so build the brief from your stored state where you can, and check what the model adds. And add state-tracking cases to your evaluation set, with long histories, reassignments and running totals, because a needle-in-a-haystack score will not warn you.

5. What I would not claim

That frontier models cannot track state. They did well on the two simpler levels. The collapse was on self-updates and references, at 10K tokens and beyond.

That StateBench describes your product. It uses synthetic sequences. The report does not say, in what I read, which reasoning settings the frontier models used, and tool use such as running code could change the picture. Measure your own tasks.

That compaction is free. The authors say it plainly: "Information will be lost as a consequence of compacting a conversational history." Weaker models suffered at tight budgets. Qwen 3 Coder Next fell from 53.0% to 3.6% on SWE-bench Verified at the tightest setting.

That long context is useless. The compaction paper says its evidence "does not rule out the existence of tasks for which long context is beneficial". Reading a long contract once is a different job from tracking changes across a long session.

The honest summary

Two papers published on 2 and 3 November reach the same practical conclusion from opposite directions. StateBench found that frontier models handle constant values and simple reassignments well, but fall to chance at tracking self-updates and references once inputs reach 10K to 64K tokens. "You don't need long context" found that compacting an agent's context into a short brief lost little or no accuracy on hard agent tasks, while cutting estimated costs several-fold.

For founders, the lesson is about where state lives. The context window is where the model reads, not where your product should keep its records.

This week, find the one place where your agent works out the current state from a long history. Move that state into code, have the model propose changes as structured operations, and start each turn from a brief.

Sources: Ai2, StateBench announcement (3 November 2026) — the four difficulty levels, the frontier models tested, and the "13% or lower" figure for the longest general finite problems; Jackson Petty, William Merrill and Ashish Sabharwal, A Fine-Grained Hierarchy and Benchmark for State Tracking in Language Models, arXiv:2611.03094 (2 November 2026) — the six formats, the generative evaluation of frontier models, the at-chance result between 10K and 64K tokens, and the RULER versus spreadsheet comparison; Peter Belcak et al., You don't need long context, arXiv:2611.05306 (3 November 2026) — the 3.6x to 7x serving-cost comparison, the two compaction methods and what effective briefs contained, the accuracy changes and estimated savings for Claude Opus 4.5 and GPT 5.6 Sol, the Qwen 3 Coder Next result, and the stated limitations. The code in section 4 is my illustration, and the reading in sections 3 to 5 is mine.

Related Posts

The Month-Two Wall: Why Your AI Agent Forgets — and the Memory Architecture That Fixes It (2026)July 27, 2026

The Agent Input Nobody ReviewsSeptember 18, 2026

One Refactor Cost $14.29. Every File the Agent Read Stayed on the BillSeptember 2, 2026