# Context is a budget

By @omar · Published Nov 2, 2026 · From the roadmap [Building AI Agents: tools, memory and evals](https://ornix.io/u/omar/building-ai-agents)

The context window is not memory. It is a budget that every step spends from, and the model attends to it unevenly. An agent that runs for 40 steps will fill its window with old tool output unless you manage it on purpose. This note is the manager.

## What fills the window

$$W \ge S + H + R + O$$

$W$ is the window. $S$ is the fixed prefix (system prompt and tool schemas). $H$ is history: every message and tool result so far. $R$ is anything you retrieved. $O$ is the room you reserve for the answer. $S$ and $O$ are constant; $H$ grows with every step. Models also use long contexts less reliably as they fill: information in the middle of a long input is used worse than at the edges (Liu et al.). So "it still fits" is not "it still works".

## Worked example

Take $W = 100{,}000$, $S = 6000$, $O = 4000$, and about 2,500 new tokens per step. If you compact at 70 percent of the window:

- Trigger at 70,000 tokens. History room is $70{,}000 - 6000 = 64{,}000$, so compaction fires after about $64{,}000 / 2500 \approx 25$ steps.
- Summarize the oldest 21 steps (52,500 tokens) into about 1,500 tokens and keep the last 4 steps verbatim (10,000 tokens).
- New size: $6000 + 500 + 1500 + 10{,}000 \approx 18{,}000$ tokens. That is roughly 4 times smaller.

Compaction costs one extra model call. It pays for itself within about two steps, because every later prompt is 50,000 tokens lighter.

```mermaid
%% ornix:id=ctxt0001
flowchart TD
    A["New step result"] --> B{"Over 70 percent of window?"}
    B -- "no" --> C["Append and continue"]
    B -- "yes" --> D["Clear old tool results"]
    D --> E{"Still over budget?"}
    E -- "no" --> C
    E -- "yes" --> F["Summarize old turns"]
    F --> G["Keep task + summary + last 4 turns"]
    G --> C
    A --> H["Important fact?"]
    H --> I["Write to scratchpad file"]
    I --> J["Re-inject notes each step"]
```

## The toolbox, cheapest first

1. **Trim at the source.** Truncate tool output, paginate, return fields not records. The cheapest token is the one that never enters.
2. **Clear stale tool results.** An old `read_file` output that was already used can be replaced with "[output cleared, re-read if needed]". The model keeps its decisions and loses the bulk. Some providers offer this as a server-side feature (context editing); build it by hand first.
3. **Summarize.** Replace old turns with a structured summary. Ask for fixed headings (goal, decisions, facts with ids, dead ends, next step) so nothing important is paraphrased away.
4. **Write notes outside the window.** Give the agent a `note` tool that appends to a file, and re-inject the file on every step. The agent decides what is worth keeping, and it survives compaction and restarts.
5. **Delegate with a clean context.** A sub-agent explores in its own window and returns a 300-token report. The parent never sees the 40,000 tokens of searching.
6. **Fetch just in time.** Keep identifiers in context and load content when needed, the way you keep a file path and open the file.

## In code

```python
import json


def kind(block):
    return block["type"] if isinstance(block, dict) else block.type


def has_tool_use(message):
    content = message["content"]
    return not isinstance(content, str) and any(kind(b) == "tool_use" for b in content)


def units(messages):
    out, i = [], 1
    while i < len(messages):
        if messages[i]["role"] == "assistant" and has_tool_use(messages[i]) and i + 1 < len(messages):
            out.append(messages[i : i + 2])
            i += 2
        else:
            out.append([messages[i]])
            i += 1
    return out


def approx_tokens(messages):
    return len(json.dumps(messages, default=str)) // 4


def compact(messages, summarize, keep_last=4, limit=70_000):
    if approx_tokens(messages) < limit:
        return messages
    steps = units(messages)
    old, recent = steps[:-keep_last], steps[-keep_last:]
    if not old:
        return messages
    transcript = json.dumps([m for unit in old for m in unit], default=str)
    summary = summarize(transcript)
    task = messages[0]["content"] + "\n\nSummary of earlier work:\n" + summary
    return [{"role": "user", "content": task}] + [m for unit in recent for m in unit]
```

The crucial detail is `units`. A `tool_use` message and its `tool_result` reply are one inseparable unit. Cut between them and the API rejects the request. This sketch assumes your history is a task message followed by assistant and tool-result pairs, as in week 3. The `summarize` argument is a cheap-model call with a prompt asking for the fixed headings above. For token counts, use your provider's counting endpoint when it has one; the length over four trick is only a trigger.

## Common mistakes

- Summarizing without fixed headings, so the summary becomes a pleasant story that omits the file path you needed.
- Compacting inside a tool-use pair and producing an invalid history.
- Re-summarizing a summary again and again until the facts blur. Keep a durable notes file for anything that must survive.
- Putting the scratchpad at the top of the prompt. It changes every step, so it breaks the prompt cache for everything below it. Put volatile text last.
- Never testing a long run. Compaction bugs only show up at step 30.

## Check yourself

- With $W = 200{,}000$, $S = 10{,}000$, 4,000 tokens per step and compaction at 60 percent, at which step does it fire?
- Why is a `tool_use` and its `tool_result` one unit?
- When would you prefer a sub-agent to compaction?
- What belongs in the scratchpad and what belongs in the summary?

---

Roadmap: https://ornix.io/u/omar/building-ai-agents (Markdown: https://ornix.io/u/omar/building-ai-agents.md)

Canonical page: https://ornix.io/u/omar/building-ai-agents/context-is-a-budget
