%%{init: {"theme": "base", "themeVariables": {"primaryColor": "#E8ECFF", "primaryBorderColor": "#4054B2", "primaryTextColor": "#16171B", "lineColor": "#3B4351", "edgeLabelBackground": "#FAF7F0", "clusterBkg": "#EFE9DC", "clusterBorder": "#766F65"}}}%%
flowchart TB
M(["Model --- stateless,<br/>reads only the window"])
subgraph WM["Working memory"]
CW["Context window: task,<br/>recent exchange, retrieved<br/>facts, scratchpad"]
end
subgraph LTM["Long-term memory"]
EP[("Episodic store:<br/>past events,<br/>actions")]
SEM[("Semantic store:<br/>documents,<br/>knowledge base")]
end
X["✗ gone for good"]
M <-->|"read / write"| CW
CW -->|"retain / consolidate"| EP
CW ~~~ SEM
EP -->|"recall"| CW
SEM -->|"retrieve"| CW
CW -.->|"evict / forget"| X
classDef world fill:#EFE9DC,stroke:#766F65,color:#16171B
class EP,SEM world
7 Memory, Context, and Retrieval
Memory is the faculty an agent must build most completely for itself, because it is the one the substrate supplies not at all. Notice the gradient the previous chapters trace. The model reasons, however unreliably, out of its own resources; when it acts, it at least chooses the move and leaves the harness to carry it out; but it neither keeps a memory nor knows that it has anything to keep. It is stateless (Section 3.3): each call begins from nothing but the text it is handed, and ends having retained none of it. An agent’s memory is therefore not a faculty latent in the model and waiting to be drawn out, as reasoning was, but a piece of apparatus assembled wholly around a thing that forgets everything the instant it has spoken — and, being apparatus, it falls to the harness, exactly as action did. If reasoning is the model’s and action is shared, memory is the harness’s almost entirely: the purest case in the book of a faculty built around the substrate rather than coaxed from it. This chapter is about that apparatus — how to give a creature with no memory of its own both the appearance and the function of remembering.
Memory, too, is among the most thoroughly studied subjects there is, and from three directions at once. Cognitive psychology mapped its kinds — the fleeting working memory in which we hold a thought, the episodic memory of particular past events, the semantic memory of general facts shorn of any occasion of learning them. Classical artificial intelligence built the machinery of knowledge — ontologies, semantic networks, the great hand-built knowledge bases — and worried at length over how to represent what an agent knows. And computer science has spent its whole history on the problem in its barest form: caches, indexes, and databases, the art of keeping the useful near to hand and the rest within reach. The agent-builder inherits all three traditions. The context window is a working memory; a vector store is an episodic one; a knowledge base is a semantic one; and the discipline of choosing what to load into the window at each moment — context engineering — is a new name for the oldest question memory poses: what to keep in mind, and what to leave on the shelf until it is wanted.
One fact governs the whole undertaking, and it is worth fixing before we begin. The model sees only its context window, and the window is finite — while the agent’s history, and the world’s knowledge, are not. Memory for these agents is therefore not, at bottom, a problem of storage, which is cheap and easy, but of recall: of getting the few things that matter now into the small space the model can actually attend to, and — since it attends even to that space unevenly — of getting them there in a form it will not overlook. Everything in this chapter is, one way or another, a strategy for that: deciding what to retain, what to recall, and what to let go.
We proceed from the constraint to the machinery to the knowledge it carries. We look first at the shape memory must take when the thing doing the remembering has no memory of its own, then at the classical taxonomy — working, episodic, semantic — that maps the territory. We take the context window as the agent’s working memory, and the craft of managing it; retrieval as the agent’s long-term memory, and the act of recall; and context engineering as the practice that unifies them, the deliberate assembly of what the model is shown on each call. We close on knowledge itself — ontologies, knowledge graphs, and the structured representations through which agents come to understand each other — which is also the threshold of Part III, for a single agent’s memory is one thing, and the shared memory of several agents, who must establish common ground before they can so much as disagree, is quite another. This chapter is the book’s one home for memory; the multi-agent complications of shared state are left, deliberately, to the chapters where they belong.
7.1 The Stateless Substrate and the Shape of Memory
The single most important thing to grasp about an agent’s memory is that none of it is in the model. We settled the substrate’s statelessness back in Part I: the model is a function from a context to a continuation, and it begins every call from nothing but the text it is handed. So when an agent appears to remember — recalls what was said an hour ago, holds to a persona, pursues a goal it set itself earlier — the remembering is happening entirely outside the model, in the apparatus that gathers the relevant past into text and lays it before the model on the next call. The model does not remember; it is re-told. Memory, for these agents, is a property not of the mind but of the machinery around it.
The consequence is slightly unsettling, and worth stating without flinching: there is no continuous self in there. Each call is answered by a model that has, strictly, never seen any of the others; the impression of one persistent agent — patient, consistent, carrying a thread of purpose — is manufactured turn by turn, by feeding back a curated record of what came before. The agent is, in this sense, a flipbook: a sequence of stateless snapshots that the harness animates into the semblance of a single continuous creature. This is not a defect awaiting a patch but the medium one is obliged to build in. Get the feeding-back right and the illusion is seamless and genuinely useful; get it wrong and the agent forgets your name mid-conversation, or flatly contradicts what it said two turns ago, because in the only sense that counts it never knew. And it makes no difference to any of this that a platform may now keep the flipbook’s pages for you — storing the conversation on its own servers and replaying it on request, rather than leaving it in the harness to shuffle (Section 7.5) — because wherever the pages are kept, each remains a stateless snapshot, and the animating of them into one continuous creature is still something done to the model from outside it, never by it.
If memory is the machinery of feeding the past back in, then building it reduces to three questions — the same three a librarian, a database, or a human mind must answer. What to retain: of everything that has happened, what is worth keeping at all. What to recall: of everything kept, what to bring back before the model’s eyes now. And what to forget: what to let go, deliberately, because keeping it costs and surfacing it distracts. The three are not independent. You cannot recall what you never retained; you cannot retain everything; and so forgetting is not an option but an obligation. What welds the three puzzles into one is the criterion they share, and it faces forward rather than back: the test of a memory is not whether it faithfully records what happened but whether it will earn its keep next time — memory as the compression of past experience into readiness for future action, not an archive maintained for its own sake. Seen so, forgetting stops looking like a leak to be plugged and becomes part of the design; cognitive scientists make the same case for human memory, where the active shedding of detail is what keeps what remains flexible and general rather than brittle and overfitted (Richards & Frankland, 2017). The whole of this chapter is an extended answer to these three questions in their modern, mechanical form.
One of the three is far harder than it looks, and it is not the one most people would name. Storage is easy — disk is cheap, and an agent can keep a verbatim transcript of every word it has ever exchanged for a pittance; retention, crudely understood, is very nearly free. The difficulty is recall, because the model cannot look at the store. It can attend only to what sits in its context window, a small and finite space, so to “recall” a thing is to physically copy it into that window, in competition with everything else that might have gone there instead. And the window is not merely small but unevenly used: the model attends to its middle markedly less reliably than to its ends — the lost-in-the-middle effect (Liu et al., 2024) — so that even a fact duly recalled can, for practical purposes, be missed. Memory is therefore a problem of selection under scarcity: choosing, at each moment, the few things worth their place in the precious window, and putting them where the model will actually look. Storing is remembering everything; recall is remembering the right thing at the right time; and only the second is hard.
This reframing sets the agenda. Because storage is cheap and recall dear, an agent’s memory takes the shape of a hierarchy: a small, fast, scarce store the model reads directly — its working memory, the context window — backed by large, slow, cheap stores it reaches into only on demand — its long-term memory, the archives it can retrieve from. The craft lies in the traffic between them: what to promote into the window, what to evict from it, what to compress, what to discard. It is, structurally, the oldest pattern in computing, the memory hierarchy of register, cache, and disk, rediscovered for an agent whose “register” is a few thousand tokens of fickle attention. To manage that traffic we first need a vocabulary for the kinds of memory an agent keeps — and that vocabulary, conveniently, was worked out by psychologists and AI researchers decades before there was an agent to apply it to. We borrow it next.
7.2 Working, Episodic, and Semantic Memory
Psychology has spent a century carving memory at its joints, and the joints it found turn out to fit an agent almost too neatly. The broadest cut is by duration and role. On one side sits a small, fast, transient store holding whatever is being thought about at this instant — working memory, the few items a mind keeps live while it does something with them: the digits of a phone number while you dial it, the first half of a sentence while you parse the second. Baddeley and Hitch gave it its modern form and its name (1974), refining the older notion of a short-term memory — a passive store that merely holds — into a working one that operates on what it keeps; the two terms are often used interchangeably, but the later one earns its keep by insisting on that difference. Its defining traits are that it is fast, that it is where active reasoning actually happens, and that it is famously, frustratingly small. On the other side sits a vast, durable long-term memory holding everything else, consulted only when something is needed from it. The first is the desk you work at; the second is the filing cabinet behind you.
Long-term memory then divides again, this time by what it holds, in a distinction we owe to Tulving (1972). Episodic memory is the autobiographical record: particular experiences, each tagged with its occasion — when it happened, where, in what circumstance. Semantic memory is the store of general knowledge, abstracted away from any occasion of its learning: facts, concepts, meanings, held as simply true rather than as things that happened to you. The difference is the difference between remembering that Paris is the capital of France — semantic, a bare fact — and remembering that you were rained on in Paris last spring — episodic, an event with a date and a place. You almost certainly cannot recall the day you learned the former; that you cannot is precisely what makes it semantic.
These are not loose metaphors but among the most durable findings in the study of the mind, and they map onto an agent’s machinery with very little forcing. Working memory is the context window: the small, active buffer the model reasons over, holding the task at hand, the recent exchange, and the scratchpad of the model’s own thinking. Episodic memory is a store of the agent’s past — the conversations it has had, the actions it took, the results that came back — kept outside the window and fetched in when relevant; “what did the user ask me last week?” is an episodic query. Semantic memory lives, for an agent, in two places at once: the bulk of it is baked into the model’s weights, the compressed knowledge of Section 3.2, and the remainder sits in external knowledge bases — documents, databases, knowledge graphs — that the agent retrieves from. There is a fourth kind worth naming in passing, procedural memory, the knowledge of how rather than that — skills and habits rather than facts.1 An agent’s procedural memory is mostly its tools (Section 6.2) and its standing instructions: capability encoded not as recallable text but as machinery wired into the harness.
A caution before we lean on the mapping too hard: it is an analogy that earns its keep, not an identity to be pressed. A vector store is not literally a hippocampus, and nothing is gained by pretending it is; the cognitive vocabulary is useful here because it is a vocabulary, a set of named distinctions that tell an agent-builder which kind of memory a given need calls for — a fact to be looked up (semantic), an experience to be recalled (episodic), a thing to be held while reasoning (working), a skill to be exercised (procedural) — and which machinery therefore to reach for. That is also why the chapter is shaped as it is. We take the kinds in the order an agent meets them, beginning with the only one the model can touch directly: working memory, the context window.
| Cognitive kind | What it holds | The agent’s machinery | Example |
|---|---|---|---|
| Working | Whatever is being thought about now — fast, small, where reasoning happens | The context window | The task at hand and the model’s own scratchpad |
| Episodic | Particular past experiences, each tagged with its occasion | An external store of the agent’s past, fetched in on demand | “What did the user ask me last week?” |
| Semantic | General facts and concepts, shorn of any occasion of learning | The model’s weights, plus external knowledge bases | “Paris is the capital of France”; a retrieved document |
| Procedural | Skills and habits — knowing how rather than that | The agent’s tools and standing instructions | Calling a tool; following a standing instruction |
7.3 The Context Window as Working Memory
One kind of memory is the model’s directly: working memory. The context window is the whole of what the model can attend to at a given moment; nothing outside it exists, for the model, in that instant. Into it go the task, the recent exchange, whatever facts have been fetched for the occasion, and the model’s own unfolding thought, and over all of that — and only that — the next continuation is computed. The correspondence with human working memory is close and not accidental: the window is fast, since the model reads it directly; it is where the reasoning actually happens; and it is, decisively, small. Whatever the headline figure on a model’s context length, the window is always finite, and always a great deal smaller than the agent’s full history and the world’s available knowledge.
One use of the window makes the working-memory analogy almost literal: the model writes into its own context. The chain of thought we met in Section 5.3 is exactly this — the model emits intermediate reasoning that becomes part of the text it then reads back, using the window as a scratchpad to hold partial results it has nowhere else to keep, having no head to keep them in. Nye and colleagues gave the practice its name, a scratchpad for intermediate computation (2021), and it is working memory in the textbook sense: somewhere to hold the digits while you carry the one. It is also why a model that reasons in steps outperforms one made to answer in a single leap — the steps furnish, on the page, the working memory the model has nowhere on the inside.
The window is scarce, though, in two distinct ways, and both bear on how it must be managed. The first is plain finitude: it fills, and once full, something has to give. An hour-long conversation, a large repository, a long document — each outgrows the window, and the agent cannot simply hold all of it at once. The second is subtler and worse: the lost-in-the-middle effect again, which bites before the window is even full — the model attends to the middle of a long context markedly less reliably than to its ends. The window is therefore not a uniform shelf on which anything placed is equally to hand; it has warm regions and cold ones, and a fact lodged in the cold middle can be present and yet, to all practical purposes, unread. Scarcity and uneven attention together mean one cannot merely cram the window: what goes in, and where, both matter.
Managing it is, in consequence, an active and unending discipline, and its central move is compaction — as the history grows, replacing the verbatim record with a précis, distilling twenty exchanges into a paragraph of what mattered and reclaiming the space — and, not incidentally, the cost, for a history carried in full is re-read, and re-paid for in tokens, on every single call. Compaction is itself a memory operation: a deciding of what to retain and what to let go, performed continuously and on the fly. Done well, it keeps an agent coherent across an interaction of any length on a window of fixed size; done badly, it summarises away the one detail that turns out to matter and proceeds, with perfect confidence, without it. Around it cluster the lesser crafts: placing the most important material at the ends to dodge the cold middle, pruning stale tool output, and maintaining a compact running state the agent updates rather than reconstructs each turn. All of it is the traffic-management of the memory hierarchy: the unceasing question of what earns a place in the small, fast store.
There is a hard floor to how far this can go, and it sets up everything that follows. Compaction is lossy by nature: every summary discards something, and what it discards is gone for good unless a copy was kept elsewhere. Working memory, however artfully managed, cannot therefore be an agent’s whole memory — it is too small, and the very act of keeping it small destroys information. For anything an agent must be able to recover in full, and exactly, long after the window that once held it has been compacted away — a fact, an earlier decision, a document — the answer is not to cram the window harder but to put the thing in a larger and more patient store, and fetch it back only when it is wanted. That store is long-term memory, and the fetching is retrieval, to which we now turn.
7.4 Retrieval and Long-Term Memory
The way past the window’s limits is the way computing has always got past a small fast store: back it with a large slow one, and move things between the two on demand. Whatever will not fit in the window — the older history, the unread documents, the facts the agent has accumulated — goes into an external store, a database or an index or a file, and when something there bears on the moment, it is fetched back into the window. This is an agent’s long-term memory, and the fetching is retrieval. We have met the mechanism already, from the other side: RAG — which Section 6.2 treated as a tool that hands the model text it was never trained on — is exactly this seen from the standpoint of memory. Retrieval is recall — the act of bringing some relevant fragment of the past back before the model’s eyes, into the working memory of Section 7.3 where alone it can be put to use.
7.4.1 Finding the Few Things That Matter
The whole difficulty of retrieval is relevance: out of everything in the store, finding the few items that bear on the present moment. The dominant answer is to search by meaning, over embeddings. Each stored item is turned into a vector — a point in a high-dimensional space, positioned so that items close in meaning lie close together, the compressed representations of Section 3.2 put to a fresh use; the query is embedded the same way; and the items whose vectors fall nearest the query’s are returned. This is dense retrieval (Karpukhin et al., 2020), and it lets an agent find “the conversation where we discussed the deployment bug” by what it means rather than which words it happened to use. A store, an embedding model, and a nearest-neighbour index together make a vector database, the unglamorous workhorse on which most agent memory is built.
Similarity alone, though, is a crude ranking: the most semantically apt memory is not always the one worth surfacing. An influential refinement scores each candidate by three things at once — its relevance to the query, its recency, and its importance, the last fixed when the memory is first laid down — so that a salient or a recent memory may outrank a merely similar one. It comes from the Generative Agents of Park and colleagues (2023), whose simulated townsfolk recalled their lives by just this weighted blend, and it carries a sound intuition: what to recall turns not only on what a memory is about, but on how much it mattered and how lately it was formed. In practice the three are folded into a weighted sum: with m a candidate memory, q the query, and each component normalised to [0,1],
\mathrm{score}(m) \;=\; w_{\mathrm{rel}} \cdot \mathrm{rel}(m, q) \;+\; w_{\mathrm{rec}} \cdot \mathrm{rec}(m) \;+\; w_{\mathrm{imp}} \cdot \mathrm{imp}(m),
and the weights are not universal — a coder agent hunting the one function that matters wants relevance to dominate, while a long-running assistant wants recency, lest it dredge up last month’s resolved bug. Tune them against your own recall failures, not Park’s townsfolk.
The rule is small enough to run as written. Give each memory its three components as plain numbers — in a live store, rel would come from embedding similarity and rec from a decaying clock, but the ranking neither knows nor cares — and recall is a sort:
from dataclasses import dataclass
@dataclass
class Memory:
text: str
rel: float # components precomputed and normalised to [0, 1]
rec: float
imp: float
def score(m: Memory, w_rel: float = 1.0,
w_rec: float = 0.5, w_imp: float = 0.5) -> float:
return w_rel * m.rel + w_rec * m.rec + w_imp * m.imp
store = [Memory("deploy broke after schema change", 0.9, 0.2, 0.6),
Memory("user prefers terse replies", 0.3, 0.5, 0.9),
Memory("staging failed this morning", 0.7, 1.0, 0.4)]
[m.text for m in sorted(store, key=score, reverse=True)[:2]]
# -> ['staging failed this morning', 'deploy broke after schema change']Even with relevance weighted twice as heavily as recency and importance, the fresher memory takes first place from the more similar one — the blend doing precisely what it was hired to do, and precisely the ordering to inspect when it starts surfacing the wrong things.
The same machinery serves both halves of long-term memory from Section 7.2. An episodic store holds the agent’s own past — the conversations it has had, the actions it took, how they turned out — each entry an event it can later recall; “what did we decide about the schema last week?” is a query against it. A semantic store holds knowledge rather than experience — documents, references, a knowledge base — and answers “what is true of X?”, the classic use of RAG. In practice the line smudges, a remembered conversation being itself full of facts; but the distinction still earns its place in design, because episodic memory prizes recency and specificity while semantic memory prizes coverage and correctness, and the two are tuned differently.
7.4.2 A Memory’s Life Cycle
Storing and fetching verbatim is the easy case; doing it well across a long agent lifetime asks for more. What is worth keeping must be extracted from the raw stream — a rambling exchange reduced to a few crisp memories; consolidated as new information arrives, so that a corrected fact supersedes the old one rather than the two sitting contradictorily side by side — the failure of drift, where the world changes but the memory does not, and the obsolete version is recalled as though still true; and finally allowed to be forgotten, lest the store silt up with stale residue. These are, not by coincidence, close to the operations by which human memory is thought to consolidate in sleep.
They are the province of agent memory architectures: Packer and colleagues cast the problem as an operating system — MemGPT (2023), with the context window as main memory and the external store as disk, and the agent paging information between them as need arises — which is the register–cache–disk hierarchy restated almost to the letter. MemGPT’s quieter contribution was to hand the paging to the agent itself: the model is given tools to search its own archive and rewrite its own core memory, so that when to remember becomes a decision the agent makes rather than a rule the developer wires in advance. A growing shelf of memory frameworks now packages the extract–consolidate–retrieve cycle for the agent-builder.2
Generative Agents put a second device to work, deeper than its retrieval formula: reflection. Rather than only tidying and merging what arrives, the agent periodically pauses, gathers a batch of raw memories, and has the model synthesise from them a higher-level observation — “the user keeps returning to deployment, so reliability is their real concern” — which is written back into the store as a new memory, open to later recall and to further reflection in its turn. It is the closest thing yet to episodic-to-semantic consolidation made mechanical: particular experiences distilled, over time, into stable general knowledge. It is also the subtlest way memory can betray an agent. A reflection is the model reasoning over its own memories, heir to every failure of model reasoning, and a mistaken insight written back with the same confidence as a sound one becomes a false premise the agent will recall, reason from, and reflect upon again, the error compounding quietly with each pass. Memory that can improve itself can also corrupt itself, and nothing in the loop is charged with telling the two apart.
7.4.3 When Recall Misleads
If reflection is memory corrupting itself on the way in, retrieval is the matching hazard on the way out: it extends an agent’s memory without limit, but smuggles in a failure the window did not have. The retriever can fetch the wrong thing — a plausible but irrelevant memory, a fact since superseded, a passage that shares the query’s words while answering a different question — and a wrongly retrieved memory is worse than none at all, for it does not merely fail to help but actively misleads, dropping relevant-looking falsehood into the context where the model will take it as established (the very irrelevant-context hazard that degraded reasoning in Section 5.3). Recall, in agents as in people, is reconstructive and fallible; it can confabulate. So the standing caution of this book applies once more: what memory hands back — whether dredged up by retrieval or distilled by reflection — is a candidate to be checked, not a fact to be trusted on arrival. With working memory and long-term memory now both in hand, what is left is the discipline that governs the traffic between them — deciding, turn by turn, what the model is actually shown. That is context engineering, and we turn to it now.
7.5 Context Engineering
On every call, an agent faces a single decision afresh: what, precisely, to place in the context window this time. That decision is context engineering,3 the deliberate assembly, on each call, of the context the model is shown: the system instructions that fix its role, the task at hand, the memories worth retrieving for it, the latest results from its tools, and the compacted history of what has happened, arranged in some order and trimmed to some length. We named it in Section 3.3 and deferred it to here; this is the chapter it was waiting for. Working memory and long-term memory, for all their apparatus, are in the end just inputs to this one assembly step, which is where they finally meet.
7.5.1 The Engineering of Perception
The deepest way to see context engineering is as the engineering of the agent’s perception. We made the point in Section 6.1, and it pays its dividend here: a language-model agent does not perceive the world directly but reads a report of it that the harness has composed, and context engineering is the composing. The agent sees the context and nothing besides; whatever is left out of it, for that turn, the agent is blind to. To engineer the context well, then, is to give the agent good eyes; to engineer it badly is to feed it a partial or distorted picture and then marvel at its mistakes — a misleading context being, exactly as a misleading observation was in Section 6.1, indistinguishable to the agent from the truth. This lifts the activity from a knack for prompting to a first-class part of building an agent: you are constructing its perception of its own situation, turn by turn.
It is also what prompt engineering grew into. Prompt engineering was the craft of wording a single request well — the phrasing, the worked examples, the output format that coaxed a good answer from the model. The moment agents arrived, the unit that mattered stopped being the one prompt and became the whole, freshly assembled context of a long interaction; the question shifted from how do I phrase this question? to what ought to be in the window right now, given all that has happened? The skill broadened to match, from authoring a fixed string to managing a working memory through time, and the change of name from prompt to context marks a real change in where the difficulty lies.
It is worth being exact about the term’s reach, because the neighbouring crafts blur into it. Context engineering is the assembly of what the model is shown on each call — its input, no more and no less. It is not the whole of building the harness (Section 3.4): the loop that decides when to call the model and when to stop, the tools that let it act (Section 6.2), the safeguards that keep it within bounds (Section 6.6) belong to the harness too, and each has had, or will have, its own chapter. What makes context engineering this chapter’s business is that it is where memory comes due — every store and stratagem of the sections before exists, in the end, to get the right few things into that single assembly. It is the input-facing face of the harness, and the place the whole apparatus of memory finally meets the model.
7.5.2 Who Assembles the Window
A second shift is worth marking, because it changes who does the assembling. In the original idiom the context was a flat transcript of text that you, in your harness, rebuilt and resent on every call. The newer model APIs reframe it as a list of typed items — a user message, a tool call, a tool result, a span of the model’s own reasoning — and will, if you let them, keep that list on their servers and extend it by reference, assembling each turn’s context for you rather than leaving it in your hands (Open Responses, 2026). This is at once a real convenience and a real surrender. The platform now performs some of the context engineering on your behalf — choosing what to carry forward, what to compact, what to discard — and performs it where you cannot fully see. The distinction to keep is between client-managed context, which you compose and can therefore inspect, tune, and be held to account for, and server-managed context, which is composed for you and is correspondingly opaque. Neither is the wrong choice; but the whole contention of this chapter is that these are the decisions in which an agent’s quality quietly lives, and a decision you have delegated to a platform is one you can no longer audit when it goes wrong. What you cannot see, you cannot curate.
7.5.3 The Craft of Curation
What makes it an art rather than a checklist is the constraint the chapter keeps returning to: the window is finite and unevenly attended (Section 7.3), so the goal is to get the fewest, most relevant things into it, in a form the model will actually use. Two failures stand on either side. Supply too little and the agent wants for a fact it needed, and flounders or invents one. Supply too much — the entire history, every retrieved document, every tool result, on the comfortable theory that more cannot hurt — and the agent drowns, for more context is emphatically not better context. Irrelevant material does not lie there inertly; it dilutes the model’s attention and measurably degrades its reasoning (the irrelevant-clause effect again), while the one genuinely useful fact, buried in the heap, is read straight past. The aim is therefore not to maximise the context but to curate it — the right things, well placed, and nothing else — which is harder and a great deal less glamorous than it sounds, and is where a surprising share of an agent’s real quality is quietly won or lost.
The coder agent shows the whole discipline in a single turn. Before each step its harness builds a context: the standing instructions that give it its role and its limits; the issue it has been asked to fix; the few files that bear on this change, retrieved rather than the whole repository tipped in wholesale; the last commands it ran and what they printed; and a compacted note of what it has already tried and ruled out. Every item there is a context-engineering decision — what to admit, what to omit, what to compress, where to place it — and the agent’s competence on that turn is, to a first approximation, the quality of that assembly. Give it the right files and it is pointed and effective; give it the entire repository and it loses the thread; leave out the failing test’s output and it confidently mends the wrong thing.
Throughout, context engineering has handled the knowledge it retrieves as plain text — passages, documents, remembered exchanges. But knowledge can also be given structure — sorted into entities and relations, ontologies and graphs — and structure changes what can be retrieved, what can be reasoned over, and, above all, what can be shared. That structuring is the chapter’s last subject, and the bridge from a single agent’s memory to the common ground through which several agents come to understand one another.
7.6 Knowledge, Ontologies, and Understanding Each Other
So far, an agent’s knowledge has lain flat — a heap of text, documents and passages and remembered exchanges, to be retrieved and re-read. For a single agent that is often enough. But knowledge can be given shape, and the discipline of giving it shape is among the oldest in artificial intelligence. This closing section returns to semantic memory and looks at it not as something stored but as something represented — and in doing so revives a programme that long predates the language model, and that turns out to matter most precisely when one agent becomes two.
7.6.1 From Text to Structure
To represent knowledge is to commit to a structure. A semantic network draws concepts as nodes and relations as labelled edges — a canary is-a bird, a bird can fly — so that meaning lives in the connections, not the words alone. An ontology makes the commitment explicit: in Gruber’s much-quoted phrase, it is a specification of a conceptualisation (1993), an agreed account of what kinds of thing exist in some domain and how they may relate. Scaled up and populated with millions of facts, the same idea is today called a knowledge graph (Hogan et al., 2021): entities as nodes, relations as edges, a structure a machine can traverse. The contrast with the vector store of Section 7.4 is the whole point of the exercise. Embeddings find what is similar; a graph encodes what is connected, and lets you follow the connection — from a function to its callers, from a caller to its module, from the module to the service it belongs to — a chain of reasoning that similarity alone can neither represent nor walk.
7.6.2 The Sharing Programme, Then and Now
None of this is new, and the part of its history that concerns us is the part the field has half-forgotten. In the early 1990s a substantial effort went into making knowledge not merely representable but shareable — so that what one system knew, another could use, without the two having been built by the same hands or even in the same decade. The DARPA Knowledge Sharing Effort (Neches et al., 1991) set out to supply the common machinery: a shared representation language, the Knowledge Interchange Format (KIF) (Genesereth & Fikes, 1992), in which one program’s assertions carried the same meaning to another, and a companion protocol, the Knowledge Query and Manipulation Language (KQML) (Finin et al., 1994), for the asking and the telling. The dream was interoperable knowledge, and it largely faded — the ontologies proved laborious to build and harder still to agree upon. But notice what the effort was for. It was the field’s first serious attempt at the problem this book turns to next: not how one system represents what it knows, but how two of them can come to mean the same thing by it.
The programme has quietly returned, under new names and on new foundations. Structured retrieval now builds a graph over a corpus and reasons across it: GraphRAG (Edge et al., 2024) constructs a knowledge graph from a document collection so that a model can answer the broad, synthesising questions — “what are the main themes here?” — that flat retrieval, fetching a handful of similar passages, simply cannot reach. Portable knowledge formats long associated with the semantic web carry structured facts between systems much as KIF once hoped to. And the old programme’s deepest difficulty is worth naming, because the modern setting transforms it: a classical knowledge base was a closed world of symbols that meant only what other symbols said they meant — the symbol-grounding problem (Harnad, 1990) — whereas a language model arrives already saturated in the very text those symbols were meant to summarise, and can read a graph and the world it describes in something like the same breath. Structure and statistics, having spent thirty years as rivals, turn out to be complements.
7.6.3 Ontology Goes Operational
Industry has revived the word more loudly still, and bent the classical idea toward action. Palantir, more than any other company, has put ontology back into commercial currency — meaning by it not a logician’s axiomatisation but an operational model of an organisation: its objects and their properties, the links among them, and, crucially, the actions that may be taken upon them, a digital twin through which software, and now language-model agents, both read the world and change it.4 This is ontology in the engineering sense rather than the philosophical — the term and its organising impulse revived more than its formal apparatus — but it makes the chapter’s case in the most concrete way there is: a structure you can traverse is also one you can act on, and grounding an agent in an explicit model of its world is a sturdier footing than turning it loose on unstructured text (Section 6.5).
7.6.4 From Memory to Common Ground
This is the threshold we have been walking toward. For a lone agent, structure is a convenience; it reads its own notes, and plain text will usually serve. The instant there are two, structure becomes a necessity, because shared memory needs an agreed shape — a common vocabulary, a representation both parties read the same way — before it can be shared at all. Our coder team has felt the want of it already: a knowledge graph of the codebase, with entities for modules and functions and edges for what calls and imports what, is precisely the sort of common ground every member could consult and trust. And common ground — the knowledge two parties not only hold but know that they hold together — is the hinge on which all cooperation turns; structured, portable, shared knowledge is how it gets built.
And so Part II closes. We have taken a single agent apart — its architecture and its reasoning, its tools and its actions, its memory and its knowledge — and assembled a working picture of one mind built upon a stateless machine. One division ran through the whole dissection: each faculty sorted into what the substrate already supplies and what the harness must build around it, and the balance shifted steadily as we went. Reasoning the model nearly manages alone (Section 5.3); action it only shares, choosing the move but leaving the harness to perform it (Section 6.2); memory it supplies not at all, leaving the harness to build the whole of it (Section 7.1). The more a faculty must outlast the single call, the less the model gives and the more the harness must build — which is the anatomy of a single agent in one line: a stateless mind, and the apparatus wrapped around it to make it persist. Each faculty, moreover, we described quietly in the singular. The moment we admit a second agent, every one of them takes on a social dimension it did not have alone: reasoning becomes negotiation, action becomes coordination, and memory becomes the common ground on which a collective either understands itself or dissolves into confusion. That step — from the agent to the agents — is the business of Part III.
7.7 Summary
- The substrate has no memory, so an agent’s memory is entirely apparatus. The model is stateless; every thread of continuity is sustained from outside, by feeding the relevant past back in as text. Memory is the one faculty that must be built around the model rather than coaxed out of it.
- Memory is a problem of recall, not storage. Storage is cheap; the difficulty is getting the few things that matter into a finite, unevenly attended context window at the moment they are wanted. The standing questions are what to retain, what to recall, and what to forget — each judged by what will serve the next decision rather than by fidelity to the past.
- The classical taxonomy maps the machinery. Working memory is the context window; episodic memory is a retrievable store of past events; semantic memory is general knowledge, held in the weights and in external knowledge bases. Cognitive psychology drew the map decades before there was an agent to fit to it.
- The context window is working memory, and it is scarce. Finite, expensive, and used unevenly, it must hold everything the model reasons over; the craft is summarisation, compaction, and the steady triage of a bounded window against an unbounded history.
- Retrieval is long-term memory, and it is recall made mechanical. Storing externally and fetching the relevant fragment back into the window — retrieval as remembering — extends memory indefinitely, at the price of a new failure mode: the retriever can fetch the wrong thing, and a confidently irrelevant memory misleads as surely as a hallucination.
- Context engineering is the discipline that unifies them. Assembling, on every call, what the model is shown — instructions, task, retrieved memory, tool results, compacted history — is the engineering of the agent’s perception, and the agent-era successor to prompt engineering.
- Shared, structured knowledge is how agents understand each other. Ontologies and knowledge graphs, the old knowledge-sharing programmes returning under new names, are semantic memory made representable and portable — and the point at which a single agent’s memory becomes the common ground of several, which is where Part III begins.
7.8 Exercises
Several of the following exercises use the retrieval scoring rule of Section 7.4, \mathrm{score}(m) = w_{\mathrm{rel}} \cdot \mathrm{rel}(m, q) + w_{\mathrm{rec}} \cdot \mathrm{rec}(m) + w_{\mathrm{imp}} \cdot \mathrm{imp}(m), with every component normalised to [0, 1]; unless an exercise says otherwise, the weights are the chapter’s defaults, w_{\mathrm{rel}} = 1.0, w_{\mathrm{rec}} = 0.5, w_{\mathrm{imp}} = 0.5. The agents throughout are the book’s software-engineering team.
Exercise 1. The team’s reviewer agent is preparing to review a change to the payments module, and its episodic store offers three candidates, components already computed: m_1, “the payments module has a known race condition in its retry logic”, with \mathrm{rel} = 0.8, \mathrm{rec} = 0.1, \mathrm{imp} = 0.9; m_2, “this morning’s CI run failed two payments tests”, with 0.6, 0.8, 0.5; and m_3, “the maintainer prefers small, single-purpose commits”, with 0.2, 0.5, 0.8. (a) Compute the three scores under the default weights and give the ranking. (b) Holding w_{\mathrm{rel}} = 1.0 and w_{\mathrm{imp}} = 0.5, find the exact value of w_{\mathrm{rec}} above which m_2 overtakes m_1. (c) Determine whether any choice of w_{\mathrm{rec}} \ge 0 and w_{\mathrm{imp}} \ge 0, still with w_{\mathrm{rel}} = 1.0, puts m_3 first. Derive the conditions and either exhibit such weights or prove none exist; then say in a sentence what a builder who shipped those weights would have done to the agent.
Exercise 2. A long-running orchestrator computes recency from a decaying clock: \mathrm{rec}(m) = \lambda^{\Delta t}, where \Delta t is the memory’s age in hours and \lambda = 0.9. Two memories of equal importance compete for one recall slot, so the importance term cancels: a, “the schema migration is still half-applied”, with \mathrm{rel}(a, q) = 0.9, laid down \Delta t hours ago; and b, “the linter configuration changed”, with \mathrm{rel}(b, q) = 0.6, laid down this instant, so \mathrm{rec}(b) = 1. (a) With w_{\mathrm{rel}} = 1.0 and w_{\mathrm{rec}} = 0.5, find the age at which b overtakes a. (b) Recompute with w_{\mathrm{rec}} = 1.0, and say in a sentence what doubling the recency weight did. (c) Show in general that, for a relevance gap g = \mathrm{rel}(a, q) - \mathrm{rel}(b, q) > 0 and a freshly laid-down rival, the crossover age is
\Delta t^{*} \;=\; \frac{\ln \bigl( 1 - w_{\mathrm{rel}} \, g / w_{\mathrm{rec}} \bigr)}{\ln \lambda},
and that no crossover exists at all when w_{\mathrm{rec}} \le w_{\mathrm{rel}} \, g. What does the second finding imply about how much staleness the default weights can ever punish?
Exercise 3. Say that, for the present query q, memory a dominates memory b when \mathrm{rel}(a, q) \ge \mathrm{rel}(b, q), \mathrm{rec}(a) \ge \mathrm{rec}(b), and \mathrm{imp}(a) \ge \mathrm{imp}(b), with at least one inequality strict. (a) Show that if a dominates b, then \mathrm{score}(a) > \mathrm{score}(b) for every strictly positive weight vector. (b) An engineer proposes speeding up recall by discarding every dominated candidate before the sort. Show that for strictly positive weights this never changes which memory is recalled top-1, but can change top-k recall for k \ge 2: with components (\mathrm{rel}, \mathrm{rec}, \mathrm{imp}) of a = (0.9, 0.9, 0.9), b = (0.8, 0.8, 0.8), and c = (0.3, 0.2, 0.1), compute the top-2 under default weights before and after the discard, and say what the agent would now be shown. (c) The engineer next proposes deleting dominated memories from the store outright. Construct a concrete pair of memories and two queries showing that domination under today’s query need not survive tomorrow’s, and identify which of the three components is solely responsible for the reversal, given that all memories age under the same clock.
Exercise 4. Model the lost-in-the-middle effect crudely: an item placed in the opening or closing quarter of the assembled context is actually used by the model with probability 0.9; an item in the middle half, with probability 0.55; reads of distinct items are independent. On this turn the coder agent’s context carries two critical items — the failing test’s output and a note of the API constraint the fix must respect. (a) Compute the probability that both items are used if the harness assembles chronologically, which lands both mid-window, and if it instead places both at the ends. (b) A colleague proposes duplication: leave each item where chronology puts it and repeat it once at an end. Assuming the two reads of the same item are independent, compute the probability that an item is used, and that both are. (c) Now assume the reads are nested rather than independent — whenever the middle copy would be read, the end copy would be too — and recompute. What do (b) and (c) together reveal about where duplication’s advantage over simple end-placement comes from, and what should a builder measure before trusting it?
Exercise 5. The coder agent makes one model call per turn and runs for T = 40 turns. Its context carries a fixed preamble of s = 1{,}000 tokens (instructions and the issue text), and each turn appends h = 500 tokens of history — tool output and the model’s reply. Uncompacted, the context at turn t is s + h(t-1) tokens, and every token of it is paid for again on every call. (a) Compute the total input tokens paid for across the run — the growing-context arithmetic series that Chapter 3’s Exercise 2 derived for the stateless coder, here with the memory chapter’s constants. (b) Now the harness compacts: at the end of turns 10, 20, and 30, the accumulated history — any earlier précis included — is replaced by a précis of c = 800 tokens. Give the context size at turn t under this policy, recompute the total, and express the saving as a percentage. (c) The window holds W = 8{,}000 tokens. If the harness instead compacts at the end of every k-th turn, the context peaks at s + c + h(k-1) tokens; find the largest k that never overflows the window. (d) Give the leading term of the total cost as a function of T under each policy, and state in one sentence what compaction buys asymptotically and what it irrecoverably costs.
Exercise 6. Extend the chapter’s Memory record with two fields: key, the subject an entry is about, and written, the turn on which it was laid down. Against tonight’s query “does the orders table need an index?”, the tester agent’s store holds three entries: key orders-index, “orders table lacks an index on user_id”, with \mathrm{rel} = 0.85, \mathrm{rec} = 0.30, \mathrm{imp} = 1.00, written at turn 3; key orders-index, “index added on orders.user_id in commit f3a2”, with 0.80, 0.90, 0.40, written at turn 9; and key retry-policy, “payments retries use exponential backoff”, with 0.20, 0.60, 0.50, written at turn 7. (a) Rank the unconsolidated store under default weights, and name the life-cycle failure of Section 7.4 that the top-1 recall exhibits. (b) Implement consolidate(store), which keeps only the newest entry per key, and show that it repairs the recall. (c) In a few sentences: for which of the two long-term stores of Section 7.2 is newest-per-key the right rule, and what does it destroy in the other?
Exercise 7. The team keeps a knowledge graph of its repository: nodes are functions, and a directed edge from f to g records that f calls g. The edges are: main calls start_server and check_env; start_server calls load_settings; run_migration calls load_settings; load_settings calls parse_config; check_env calls parse_config; parse_config calls read_file; and render_docs calls format_markdown. The reviewer asks: “what can break if parse_config’s return type changes?” (a) Compute the affected set by hand, and explain why read_file is not in it despite being adjacent to parse_config. (b) A vector store scores each function’s docstring against the query: parse_config 0.93, render_docs 0.72, format_markdown 0.66, load_settings 0.61, read_file 0.58, check_env 0.42, run_migration 0.18, start_server 0.15, main 0.09. Which affected functions does top-4 similarity recall miss, which unaffected ones does it admit, and what property of the query makes similarity so poor a proxy for it? (c) Write a function affected(calls, root) that computes the affected set from the edge dictionary, and check it against your answer to (a).
Exercise 8. A reflection pass in the style of Section 7.4 periodically synthesises new memories: each pass-k reflection reads b memories from pass k-1, their soundness independent, and is unsound if any of its inputs is unsound or — with probability \varepsilon, independently — through fresh misreasoning even from sound inputs. Pass-0 memories are all sound. Let q_k be the probability that a pass-k memory is unsound. (a) Derive the recurrence
q_k \;=\; 1 - (1 - \varepsilon)\,(1 - q_{k-1})^{b},
and compute q_1, q_2, and q_3 for \varepsilon = 0.05 and b = 3. (b) Show that q_k is strictly increasing and converges to 1 for every \varepsilon > 0 and b \ge 1. (Hint: (1-q)^{b} \le 1-q on [0, 1].) (c) Find the smallest k with q_k > 0.9; then say in two sentences which assumption of the model a repair must break for the limit to change, and what machinery would break it.
Exercise 9. The coder agent’s harness has been assembling the following context every turn, against a window of 32,000 tokens: standing instructions, 1,200 tokens; the full text of every repository file, 48,000; the verbatim transcript of all thirty-five earlier turns, 21,000; the forty memories the retriever returned — everything above a similarity floor of 0.2 — 8,000; the complete CI log, 12,000, of which the failing test’s output is the final 300; the issue text, 400; and the output of the last command run, 900. It does not fit, and when the harness trims blindly from the tail the agent mends the wrong thing. (a) Dismantle the assembly: produce a revised one of at most 12,000 tokens, giving each item’s budget, justifying every cut, compression, or retention by a principle from this chapter, and identifying the one item that must survive verbatim. (b) State where in the window you would place the two most critical items, and why. (c) The platform now offers to keep the transcript server-side and assemble each turn’s context for you (Section 7.5). Which parts of your plan in (a) could you no longer perform or verify, and what is the trade?
Exercise 10 (project). Starting from foundations/algorithms/retrieval.py in the companion repository — its Memory, score, and top_k — build a recall benchmark for the team and tune the weights against it, as Section 7.4 instructs. (a) Construct a store of at least eight memories, each with an age in hours and an importance, and at least six queries, each naming one gold memory; include one query whose gold memory is older than every distractor, and one phrased so that it shares no informative word with its gold. (b) Implement \mathrm{rel} as Jaccard overlap of lowercased word sets, and \mathrm{rec} = 0.9^{\Delta t} with \Delta t the age in hours; measure top-1 accuracy under the default weights and list the failures. (c) Grid-search (w_{\mathrm{rel}}, w_{\mathrm{rec}}, w_{\mathrm{imp}}) over \{0, 0.25, \dots, 1.5\}^{3}; report the best accuracy and characterise where the maximising weights lie. (d) Diagnose every residual failure as either a weight failure — the blend misweights sound components — or a component failure, where a component itself is the wrong proxy; and relate what you find to the chapter’s advice on which agents want which weights.
Exercise 11 (lab). Test the lost-in-the-middle finding of Liu and colleagues (2024) on a model you can call; Appendix C describes the set-up. (a) Synthesise a context of about sixty invented key–value facts of the form “the deploy code for service-17 is QRV-482”; plant one probe fact at relative depths 0, 0.25, 0.5, 0.75, and 1.0; ask the model for the probe’s value; run at least twenty trials per depth with fresh fillers each trial; and tabulate accuracy by depth. Describe the shape you find. (b) Repeat with the probe duplicated at the end of the context, as in Exercise 4(b), and estimate whether the two placements fail independently by comparing the observed joint failure rate with the product of the marginal failure rates. What do your measurements say about the duplication trick, and about the independence assumption Exercise 4 rested on? Record the model identifier and the date beside the table.
The distinction between knowing how and knowing that is Gilbert Ryle’s (1949); procedural memory is the home of the former.↩︎
One such is LangMem, the LangChain family’s memory SDK: https://langchain-ai.github.io/langmem/. The concept is durable; the particular tool, as ever in this corner of the field, less reliably so.↩︎
A practitioners’ term; Anthropic’s Effective Context Engineering for AI Agents (https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) is a representative treatment.↩︎
Palantir’s own vocabulary splits the ontology into semantic elements — objects, properties, links — and kinetic ones — actions and functions — the latter being what lets an agent not merely read the model but alter what it represents. The branding is the company’s; the underlying move is general.↩︎