Memory Architecture
By the end of this module
- Separate the three scopes state can live at, and say what each survives
- Explain why making session state explicit beats relying on the conversation
- Decide what is worth storing, and what storing it costs later
Definition
A model has no memory. Every call is answered from the text in front of it, and the sense that an assistant remembers you is produced entirely by something outside the model putting the right text back in front of it each time.
Memory architecture is the set of decisions about what that something keeps, for how long, and how it gets back into the prompt. The previous module showed why this cannot be left to the conversation history: history is the first thing truncated, and it is truncated silently.
Three Scopes
| Scope | Lives for | What belongs there |
|---|---|---|
| Turn | One model call, or one loop iteration. | Scratch reasoning, a tool result being worked on. Nothing that needs to outlive the step. |
| Session | The conversation, across many turns. | The goal, the constraints the user stated, the facts established so far. The things a person would expect it to still know in ten minutes. |
| Persistent | Across sessions, indefinitely. | Stable facts about the person or the account. Things that would still be true next month. |
Most systems have the first and the third and nothing in between, which is why they feel forgetful in a long session and overconfident across sessions. The middle scope is the one that makes a conversation coherent.
Session State as Cards
One way to hold session state is to extract it into a small set of named cards and re-prepend them every turn. Edit anything below and watch the panel underneath.
Anchor: Work out why this merchant's payouts keep failing Priority: Merchant m_4471, failures from 14 October Priority: Do not promise a resolution time Aside: They have already tried reconnecting the bank account twice
0 cards edited by hand. Change any line above and it is kept exactly as you wrote it.
Two properties come out of this shape. The cards survive truncation, because they are not part of the history competing for budget; they are re-added each turn at a fixed cost. And they are legible, so a person can see what the agent thinks the conversation is about.
The second is the bigger win, and it is a design idea rather than a technical one. An agent that has quietly misunderstood the goal will keep misunderstanding it, and in an ordinary chat interface there is no surface where that is visible before it produces a wrong answer. Cards put the misunderstanding on screen while it is still cheap, and let someone correct it directly rather than by arguing with the agent.
What Is Worth Storing
Persistent memory is where teams overreach. Storing everything is easy, and every stored item is something that will later be retrieved into a prompt, possibly wrongly.
| Candidate | Worth keeping |
|---|---|
| A stated preference, such as wanting short answers | Yes. Stable, cheap to hold, and noticeably good when it is remembered. |
| An account or entity identifier | Yes. It is a key rather than a fact, and saves asking again. |
| A decision that was reached and acted on | Usually. Record the decision and when it was made, not the deliberation. |
| Current circumstance, such as where someone is this week | No. It will be retrieved with full confidence long after it stopped being true. |
| Everything the user said, in case it is useful | No. It is a retrieval problem being created on purpose, and the noise competes with the signal. |
| The agent's own conclusions about the person | Rarely, and only with the evidence attached. An inference stored as a fact cannot be challenged later. |
The question that settles most of these is whether the fact would still be true next month. A stable preference is worth storing and a transient circumstance is not, because the transient one will be retrieved with full confidence long after it stopped being true, and nothing in the retrieval will mark it as stale.
An assistant remembers that a user prefers short answers, and also that they were travelling in Portugal. Six months later it mentions Portugal. Which storage decision was wrong?
Discussion
For an assistant your team runs, ask where session state actually lives. If the answer is the conversation history, work out what happens to it at turn fifty, and whether anyone would be able to see what the agent currently believes the session is about.