The Context Window
By the end of this module
- Describe the window as a shared budget rather than a length limit
- Say who decides what gets dropped when the budget runs short
- Explain why position inside the window changes how well something is used
Definition
A context window is usually explained as how much text a model can read at once, which makes it sound like a property of the conversation. It is better understood as a budget with several claimants, all of them competing, most of them invisible to the person typing.
The agent loop module noted that the prompt is rebuilt on every iteration and only grows. This is where that ends.
The Budget
Below is one window and the things claiming space in it. Move both sliders up.
Everything fits, with 14,450 tokens spare. Note that the tool definitions cost 3,400 on every single call, whether or not any tool is used.
Three things in that panel surprise people. The tool definitions cost their 3,400 tokens on every call whether or not a tool is used, which is the cost side of the registry argument from the tools module. The answer needs budget too, so a window packed to the edge with input leaves the model no room to reply. And history and retrieval starve each other: every document retrieved is a turn of conversation that no longer fits.
What Falls Out
When the budget runs short something has to go, and the model has no opinion about what. A policy somebody wrote decides, and the usual policies are these.
| Policy | What it does | What it costs |
|---|---|---|
| Drop the oldest turns | Keeps a sliding window of recent conversation. | Loses the opening of the session, which is where people state constraints and goals. |
| Summarise older turns | Replaces old history with a compressed account of it. | Keeps the shape and loses specifics. Names, numbers and exact wording go first. |
| Drop retrieved material | Protects the conversation at the expense of supporting documents. | The model answers from memory where it should have answered from a source. |
| Pin and then truncate | Marks some items as never-droppable and trims the rest around them. | Requires deciding in advance what matters, which is the work people avoid by not choosing. |
| Fail the request | Refuses rather than silently answering on partial input. | Honest, unpopular, and the right choice more often than it is chosen. |
Whichever is chosen, the failure is silent. Nothing errors, the model answers confidently from what remains, and the answer is shaped by material that was dropped without anyone being told. An agent that forgets a constraint the user gave twelve turns ago has usually not forgotten anything. The constraint is simply not in the prompt any more.
Position
Fitting is necessary and not sufficient. Where something sits inside the window changes how reliably it gets used.
Material at the start and at the end is used more reliably than material in the middle, and the effect gets stronger as the window fills. This is why an instruction placed after a long block of retrieved context is followed more often than the same instruction placed before it, and why doubling a window rarely doubles how much a model actually attends to.
The practical consequence is that a large window is not a substitute for choosing what goes in it. Putting everything in because it fits moves the problem rather than solving it: the material that matters is now competing with everything else for attention, in the weakest position.
A merchant tells an agent 'do not contact my accountant' at the start of a long session. Forty turns later the agent drafts an email to the accountant. What is the most likely explanation?
Discussion
For a long-running assistant your team uses, ask what happens to the first thing a user said once the conversation gets long. Then ask whether anyone could tell you, from the logs, what got dropped on a given run.