%%{init: {"theme": "base", "themeVariables": {"primaryColor": "#E8ECFF", "primaryBorderColor": "#4054B2", "primaryTextColor": "#16171B", "lineColor": "#3B4351", "edgeLabelBackground": "#FAF7F0", "clusterBkg": "#EFE9DC", "clusterBorder": "#766F65"}}}%%
flowchart TB
subgraph AG["Agent"]
direction TB
M["Model<br/>decides"]
H["Harness<br/>perceives and acts"]
end
E["Environment"]
M -->|"tool call: proposes"| H
H -->|"composed perception"| M
H -->|"action: disposes"| E
E -->|"result / world state"| H
classDef world fill:#EFE9DC,stroke:#766F65,color:#16171B
class E world
4 Tools, Actions, Memory, and Context
Every remedy Section 3.7 proposed ran on credit, borrowed against a power that chapter never examined: the agent’s power to act. Left to itself, the foundation model only produces text about a world it cannot affect (Section 2.3); an agent acts, and the difference is tools, the apparatus through which the model’s words become deeds and the world’s reply becomes its next perception. The frame is the oldest in the study of agents — a thing coupled to an environment, perceiving and acting in a loop — and into it the foundation model arrives with a peculiar profile: every capacity to decide what to do, no native ability to do any of it. Tools close the gap; the modern vocabulary of function calling and computer use is largely the classical loop wearing an API. This chapter is about how an agent reaches out of the closed world of text and lays hands on the world, and what it costs to let it.
Action wears two faces. The first is grounding: the substrate inhabits a closed world of symbols — knowing the word stove and never once having been burned — and tools are how those symbols get fastened, however imperfectly, to the things they are about. The second is danger — some changes cannot be changed back — and the engineering of action is largely that of letting an agent do enough to be useful and not so much that a single confident mistake is unrecoverable.
Then the chapter’s second movement begins: memory — the one faculty the substrate supplies no mutable part of, and so the purest case of apparatus — ending at the structured knowledge through which agents come to understand one another.
4.1 The Perception–Action Loop
In Russell and Norvig’s standard framing, an agent perceives through sensors and acts through actuators, joined by an agent function mapping perceptual history to next action (2021); the ends close into the perception–action loop — perceive, decide, act; the world changes; perceive again — around which the study of agents is built. The foundation-model agent sits in the picture exactly: its perception is the context it is handed — task, conversation, the results of whatever it last did — and its action is the text it emits that the harness is built to read as a move upon the world, a tool call, a message, a fragment of code. The agent function is split across Section 3.6’s two layers — the model decides, the harness perceives and acts — and the arrow that section labelled act is what this chapter is about: how a string of tokens becomes a change in the world.
One difference from the classical robot shapes everything downstream: a robot’s perception is raw at the sensor; a language-model agent’s is composed all the way down, nothing reaching the model except as text the harness chose to assemble. The agent does not see the world; it reads a report something else wrote, and its grip on reality is only as good as that report. A misleading observation — a truncated error message, a stale file listing — is easy to produce by accident, easier on purpose: the attack Section 20.9 treats as prompt injection. Deciding what an agent perceives is itself a discipline — the context engineering of Section 4.13: perception here is not given but built.
The frame teaches one more lesson, easily skipped: the line between agent and environment is drawn, not found. Is the file system part of the agent or of its world? The interpreter? Nothing in nature settles these: the boundary is a modelling choice, fixing what counts as perception and what as action — draw the interpreter inside the agent and running code is private deliberation; draw it outside and the same act becomes an action upon an environment. For multi-agent systems the choice turns sharp, because one agent’s action is another’s perception, and the environment of the first agent contains the second — why a boundary that is an afterthought for a lone agent becomes a central design question the instant there are two (Part III returns to it). Figure 4.1 draws the loop with the harness at its hinge; what it leaves unexplained is how, concretely, the model’s text becomes a deed. That mechanism is function calling.
4.2 Tools and Function Calling
How does a string of tokens become a deed? The model does not act; it asks. It emits a structured request — a tool’s name and arguments — and stops; the harness performs the operation and returns the result as the next observation. The model proposes; the harness disposes. This is function calling, or tool use: the agent’s actuator is its own output, taken at its word. The separation is the whole point: the model never touches the file system or the network, so every action passes through one chokepoint the engineer, not the model, controls — what makes the closed world of text safe to wire to a consequential one, makes tools the actuators of Section 4.1, and puts permissions and safeguards, as Section 4.7 will press, in the harness rather than the model’s good intentions. The category is deliberately broad — any operation the harness can perform on request, from reading a file to delegating a subtask to fetching from memory, the second movement’s case: Retrieval-Augmented Generation (RAG) (Lewis et al., 2020) is, from this vantage, a tool that returns relevant text.
The mechanism arrived by stages: models were first coaxed into it, in Section 3.6’s ReAct manner; Toolformer then showed a model could be taught, self-supervised, to decide which interface to call, when, with what arguments (Schick et al., 2023); the laboratories productised the idea as function calling — declare your tools in a schema, a post-trained model emits conforming calls — and the interfaces were standardised, most consequentially by the Model Context Protocol (Anthropic, 2024b), whose common wire format lets one tool serve any compliant agent (Chapter 19). The platforms now run hosted tools — web search, retrieval, code execution — on the model’s behalf: one fewer thing to build, one fewer you can see inside.
Nor is a protocol the oldest tool surface. The Command Line Interface (CLI) already gave every program a uniform invocation — arguments in, text out — so one run-a-shell-command tool puts the whole command-line ecosystem, grep and git and curl among thousands, at an agent’s disposal, no integration written for any of them. The protocol offers a typed, discoverable interface; the shell, the accumulated inheritance of decades — a real choice, and one Section 4.3 returns to.
There is a classical antecedent hiding in the JSON: a tool’s schema — name, plain-language description of what it does and when to use it, typed argument list — is very nearly a STRIPS operator of Section 3.9, read now to choose the next action rather than search for a plan, and shorn of the guarantees (Section 3.11): the model’s checking of preconditions is just more fallible reasoning.
The correspondence is not merely notional: a tool is just data, and the layer that reads it a few lines of ordinary code. A single run-a-test-suite tool, with its dispatcher, comes to the following — note that the malformed call returns an observation, an error dictionary rather than a raised exception, because the model’s next perception must be readable and retriable, not a stack trace. A toy, deliberately — the harness of Part VI will dispose at industrial scale — but the skeleton is all here.
from collections.abc import Callable
run_tests_tool = {
"name": "run_tests",
"description": "Run the suite at a path; use to check the code.",
"parameters": {"type": "object", "required": ["path"],
"properties": {"path": {"type": "string"}}}}
suite = {"tests/test_auth.py": {"passed": 5, "failed": 0}}
types = {"string": str, "integer": int, "boolean": bool}
def run_tests(path: str) -> dict:
return suite[path] # a real runner would shell out here
def dispatch(tool: dict, fn: Callable[..., dict], args: dict) -> dict:
for field, spec in tool["parameters"]["properties"].items():
if field in tool["parameters"]["required"] and field not in args:
return {"error": f"missing field: {field}"}
if field in args and not isinstance(args[field], types[spec["type"]]):
return {"error": f"{field} must be {spec['type']}"}
return fn(**args)
dispatch(run_tests_tool, run_tests, {"path": "tests/test_auth.py"})
# -> {'passed': 5, 'failed': 0}
dispatch(run_tests_tool, run_tests, {"path": 42})
# -> {'error': 'path must be string'}Anything you can wrap in a function, you can hand an agent as an action — which raises the next section’s question: not what one tool is, but which tools, and how many.
4.3 Action Spaces
Which tools should an agent be given, and how many? The set of moves available to it is its action space: in a Markov Decision Process (MDP) — Bellman’s model of sequential decision-making (1957) — the actions selectable in each state; for a language-model agent, precisely the tools you elect to give it. The name imports its lessons: the action space bounds what the agent can achieve outright, and every added move widens the branching factor. Designing it is among the most consequential decisions in building an agent and among the least examined — an architectural decision in Section 3.1’s exact sense, fixing reach, reliability, safety (an action never granted is one the agent cannot take — the cheapest safeguard there is, as Section 4.7 will urge), and legibility (named, typed tools are far easier to audit).
The tension at its heart is granularity against number: too coarse or too few and the agent has no move that does the thing; too fine or too many and it drowns, reaching for the wrong tool, stringing ten calls where one would serve — an expressiveness-against-controllability trade-off Section 4.4 meets again at the extreme. Our running coder agent sits near the sensible middle: a handful of well-chosen tools — read a file, edit a file, run the tests, run a shell command — each doing one legible thing, works far better than a do-everything tool or a hundred fiddly micro-tools; the discipline of the small, coherent interface that engineers apply to code applies to actions unchanged.
The UNIX philosophy — each program doing one thing well, composing through pipes, everything as text “because that is a universal interface” — is this discipline’s classical statement (McIlroy et al., 1978), and the command line of Section 4.2 is the original well-designed action space. McIlroy’s closing clause is the uncanny one: text as the universal interface — the very property Section 2.2 found in language — fits the command line to an agent that thinks in text. The field designed the agent’s ideal action space half a century before there was an agent to use it, and called it the shell.
The shell points to a more radical answer still: stop enumerating moves altogether and make the action space a programming language — let the agent write and run code, a whole procedure in a single action. The CodeAct study found agents given code as their interface outperformed those confined to a fixed list of JSON tool calls (Wang et al., 2024), for the plain reason that code is the universal action space: anything computable lies one action away, composition comes free, and the model is unusually good at writing it (Section 2.2). The generality is bought at the standard rate — code that can do anything can do anything wrong, and the wider the space, the harder to bound, sandbox, predict, and trust. And code is only the first of two bids for a genuinely universal action surface; the second is to let the agent operate a computer as a person does.
4.4 Computer Use and the Universal Action Surface
If executable code is one bid for a universal action surface, the second is more literal: give the agent the surface a person already uses. A human at a computer needs no API — only a screen, a pointer, and keys — and an agent equipped the same way, vision in and synthetic clicks out, can in principle do anything a person can do at a computer. This is computer use, or the GUI agent; Anthropic shipped an early version to developers in late 2024 (Anthropic, 2024a), the other laboratories following. The aspiration is not new — Robotic Process Automation (RPA) automated this surface years earlier with scripted clicks pinned to coordinates, breaking the instant a layout shifts (van der Aalst et al., 2018) — and what computer use changes is the how: the agent reads the screen afresh and works out what to do. Generality without integration — no schema, no library, no cooperation from the target — one brittleness exchanged for another.
It remains the hardest action surface to make reliable, stacking every hard problem at once: read a noisy screen, hit small targets, chain dozens of brittle steps without a misstep — slowly, at a screenshot and a model call per click. Benchmarks built to measure the difficulty — OSWorld, WebArena (Xie et al., 2024; Zhou et al., 2024) — record a steeply climbing trajectory — in OSWorld’s case, a tenth of tasks to a clear majority inside a year; any particular figure will be stale as you read it — a waypoint, not a verdict. Most gains have come from refusing the difficulty head-on — accessibility trees rather than raw pixels, specialised perception and execution — and driving a human interface pixel by pixel remains the slowest, costliest, least predictable way to act: the durable reason to reach for it last.
The two bids trade against each other, then, and Table 4.1 sets the three surfaces side by side. The working rule writes itself: prefer the narrowest action surface that still reaches the task. Use a specific tool where one exists; drop to code when you need to compose; reach for computer use only when nothing narrower can touch the target, because every step towards generality is a step away from reliability. It is Section 4.3’s expressiveness-against-controllability trade-off, enlarged to the scale of the whole machine.
| Action surface | Reach | Speed and reliability | When to use it |
|---|---|---|---|
| Specific tool | Only the operation its declared schema (name, description, typed parameters) describes — the narrowest surface | Most reliable and auditable; each does one legible thing | Where one already exists |
| Executable code | Anything computable with a library or interface to call; blind to software with no programmatic handle | Fast, composable, exact; but harder to bound, sandbox, and trust | When you need to compose operations |
| Computer use (GUI agent) | All software, the API-less included — anything a human can do with a screen, pointer, and keys, no cooperation from the target required | Slowest, costliest, least predictable | Only when nothing narrower can touch the target |
A deeper reason computer use is so hard: the screen is designed for human eyes and hands — the agent acts in an environment fitted to somebody else’s body. Which sharpens the next section’s question: what does it mean for an agent’s action to connect to the world at all? That is grounding.
4.5 Grounding, Environments, and Feedback
Recall the symbol grounding problem of Section 2.3 — Harnad’s observation that symbols manipulated only in relation to other symbols never quite fasten to the things they are about (1990), the model fluent about a stove it has never lit. Tools are the practical answer, incomplete but real: the claim “the tests pass” can be checked by running them; the agent does not say the file exists, it reads it. Grounding, in the working sense, is this loop between symbol and world that tools keep turning. The philosophical gap does not close; it relocates: a tool’s result is itself a symbol the model must interpret — in Bender and Koller’s sense its symbols remain form in want of meaning (2020) — but the agent now has a channel to check them against an authority outside itself, plenty for engineering if not for philosophy. Grounding, honestly described, is managed rather than solved.
The thing an agent acts upon and perceives is its environment, which furnishes what the agent cannot manufacture: a source of truth. The lesson of Section 3.11 returns in general form — the cheapest verifier is the world, which does not argue, hallucinate, or flatter — and it is why grounded reflection succeeds where ungrounded self-assessment fails (Section 3.12). Environments differ, and the classical theory supplies the axes (Russell & Norvig, 2021): fully or partially observable, deterministic or stochastic, episodic or sequential, static or dynamic, discrete or continuous, and — the axis this book exists for — single-agent or multi-agent. The running team’s world scores awkwardly on nearly every one: partially observable, stochastic wherever a model or a flaky test sits, sequential, dynamic — CI runs finish, colleagues commit — and multi-agent by construction: the honest reason so much of this book is about coping rather than optimising. Partial observability earns its machinery in Chapter 6 and Chapter 14; plurality, nearly everything else.
One difference the classical axes do not headline matters most day to day: whether the feedback is real. A real environment — the production database, the open web — returns true feedback at true risk; a sandbox — a copy, a simulation — reduced risk, paid for in fidelity. Choosing between them is the action-shaped version of safe exploration, reinforcement learning’s problem of learning from real consequences without suffering the worst of them; dry-runs, staging, and read-only modes buy feedback while bounding risk (Chapter 20). And the boundary-drawing of the loop returns in practical dress: what counts as the environment, and how faithfully it stands in for reality, is itself something the designer chooses.
Whatever the environment returns is an observation: the loop’s perception, complete at last. But mind the assumption that has quietly underwritten the chapter: that an action, once taken, can be checked, corrected, and retried. For a large class of actions it holds; for another it does not: the environment will confirm that the email was sent; it will not let the agent unsend it. That class — the irreversible — is where the weight of action comes down hardest, and the chapter closes with it. A debt falls due first: every surface so far has moved bits, the frame was invented for agents that move matter, and the preface promised that robots remain essential. What becomes of tools, action spaces, and grounding when the environment is physical deserves its own section — not least because there the irreversible becomes the default.
4.6 Acting on Matter: Embodied Agents
The robot is where the chapter’s frame came from, and returning it to its owner changes almost every term. A manipulator’s moves are not a menu of named tools but a vector of joint torques re-issued dozens of times a second: the action space is continuous, dimensions rather than items. The granularity trade-off of Section 4.3 becomes one of abstraction level, and robotics’ settled answer is the skill, or motion primitive — a grasp, a push, a footstep: continuous control bundled into a unit coarse enough to choose among and fine enough to compose. A skill library is a tool menu whose implementations happen to be controllers.
The second term that changes is time: every surface so far has waited for the model, and a body gets no such courtesy. Balance and collision run on control loops cycling hundreds of times a second; a deliberating model decides in seconds, orders of magnitude too slow to sit in the loop, so it must sit above one. The architecture this forces is Section 3.4’s layers, the arbitration that section called the hard part settled by the clock: the visuomotor policy is the reactive layer, the language model the deliberator, the skill library their interface — a hybrid by physical necessity, System 1 and System 2 not as metaphor but as timing diagram.
On the deliberative floor, the chapter’s stories replay with the nouns swapped. SayCan, first to seat a language model in a real robot’s planning chair, met Section 3.11’s problem — plausible steps proposed in language, no idea what is feasible from here, with these hands — and repaired it by scoring each step by the robot’s learned estimate that the skill would succeed, its affordances: what the model wants, weighted by what the body can do (Ichter et al., 2023), the missing verifier rebuilt from value functions. Code as Policies made Section 4.3’s most radical move for bodies: the model writes a program over perception routines and motion primitives, and the program is the policy (Liang et al., 2023). Inner Monologue closed the grounding loop by narrating the world back into the context — success detectors, a human’s correction — so the plan is revised against what happened (Huang et al., 2023). Tools, action spaces, grounded feedback: each of this chapter’s ideas, wearing a gripper.
The frontier since then melts the layers’ boundary: a Vision–Language–Action (VLA) model is a foundation model post-trained on robot trajectories to emit motor actions directly, action as one more token vocabulary. RT-2 named the class, web knowledge surfacing as sensible behaviour toward objects the robot had never handled (Zitkovich et al., 2023), and open models have made the recipe reproducible (Kim et al., 2025): the most direct assault yet on symbol grounding, symbols fastened to referents by training through a body rather than by a tool’s report. Treat any named model as a waypoint; the durable lesson is structural: as the model absorbs the acting, the harness’s share of the disposing (Section 4.2) shrinks toward the motor driver, and the engineer’s chokepoint must be re-established elsewhere.
Embodiment completes a sentence Section 4.4 left one word short: the screen, we said, is an environment fitted to somebody else’s body; the built world is that sentence without the metaphor. Door handles and light switches are designed to a specification — the human hand, the human eye-height — and Table 4.1 extends to them row for row, from manufacturing’s fixtured cell through the programmed skill library to the humanoid: a machine shaped to the one specification every human environment was built against, paying, as the GUI agent pays, for every step of that generality. The working rule transfers verbatim: prefer the narrowest actuator that reaches the task, and reach for the general-purpose body last.
Last, the weight. Physical action is the canonical irreversible class — the dropped glass does not unshatter — and matter is where every mitigation the next section catalogues was invented first, in steel: the workspace limit and torque bound are least privilege; the simulator is the sandbox, its fidelity bill (Section 4.5) named the sim-to-real gap; compliant hardware — joints built to yield — is reversibility by construction; and the emergency stop, which pre-empts the software rather than petitioning it, is human oversight’s limiting case (Chapter 20 will pause on it). The embodied engineer has done this engineering for decades, under the one condition that concentrates the mind: no retry was ever on offer.
4.7 Irreversibility and the Weight of Action
The chapter has leaned on an assumption it can now name and qualify: that an action, once taken, can be checked and retried. Where it holds, action is forgiving — the loop absorbs the brittle reasoning of Section 3.10 and the unverified plan of Section 3.11 by trying again; where it fails, the whole weight of the agent’s fallibility comes to rest on one move. This is the asymmetry that earns action its wariness: a reasoning error costs tokens and is corrected next turn, while an action error can cost the data, the money, or the trust — and, for an action that cannot be undone, there is no next turn. The model may be wrong cheaply; the agent acting on it may not.
The irreversible actions are a category apart — send the email, drop the table, wire the funds: once done, the world offers no path back. This is exactly where the grounding loop quietly fails — the environment will faithfully report what happened, but reporting is not reversing — and the agent has one attempt, backed by reasoning two chapters have taught us not to trust. It is the loaded foot-gun of Section 3.2 made literal: an impulse, reflexive or confidently deliberated, wired to an effect that cannot be taken back, with nothing interposed between the deciding and the doing.
The classical literature saw the difficulty coming: planning’s qualification problem — McCarthy’s name for the impossibility of stating in advance every way an action might misfire, cousin to the frame problem of Section 3.3 — says an action model is always incomplete and the world always holds one more surprise; and the safety literature’s catalogue of concrete hazards — Amodei and colleagues’ negative side effects, the safe exploration of Section 4.5 in its sharpest form — speaks directly to the case at hand (2016). The modern agent inherits both worries sharpened, wielding real tools quickly on judgement we have already learned to distrust.
The engineering of action is therefore largely that of containing it, and the apparatus is mostly unglamorous and borrowed. Least privilege, from security: grant only the actions the task requires, since an action never granted is one the agent cannot botch. Sandboxing and dry-runs: act first against a copy (Section 4.5), against the real world only once checked. Human-in-the-loop approval: for the moves that matter most, the agent proposes, a human disposes. And reversibility by construction: designs in which actions can be undone — soft deletes, transactions, staged commits — keeping the irreversible class small. None alone makes an agent safe; together they make a confident mistake recoverable more often than not — the realistic goal.
Beneath the techniques lies the hinge between this part of the book and the parts to come. To give an agent tools is to give it the power to act, and the power to act is, inseparably, the power to do harm: usefulness and danger enter through the same door, the action interface this chapter has spent itself building. Making what passes through that door dependable, secure, and accountable is parcelled out: dependability to Chapter 20; safety and security under attack to Section 20.9; accountability — who authorised, who could have stopped, who must answer — to Chapter 22. Part II has now assembled very nearly a whole agent. One faculty is still missing — the memory that lets it persist, recall, and carry a thread across the loop’s turns — the last of the anatomy, and this chapter’s remaining business.
4.8 The Last Faculty: Memory
Memory is the faculty an agent must build most completely for itself, the one the substrate supplies no mutable part of: the model carries knowledge frozen into its weights but can lay down nothing new (Section 2.3). Notice the gradient of the anatomy: reasoning the model manages out of its own resources, however unreliably; action it shares; memory is the harness’s almost entirely, the purest case in the book of a faculty built around the substrate rather than coaxed from it. The rest of the chapter is about that apparatus: giving a creature with no memory of its own the appearance and function of remembering.
Memory is also among the most thoroughly studied subjects there is — psychology’s taxonomy of its kinds (Section 4.10), classical AI’s ontologies and knowledge bases, computer science’s caches, indexes, and databases, the art of keeping the useful near to hand. The agent-builder inherits all three, and this chapter — the book’s one home for memory — spends them in turn.
4.9 The Stateless Substrate and the Shape of Memory
None of an agent’s memory is in the model — a function from context to continuation, beginning every call from nothing but the text handed to it (Part I). When an agent appears to remember — what was said an hour ago, a persona, a goal it set itself — the remembering happens entirely outside, in apparatus gathering the relevant past into text and laying it before the model afresh. The model does not remember; it is re-told. Memory, for these agents, is a property not of the mind but of the machinery around it.
The consequence is worth stating plainly: there is no continuous self in there. The agent is a flipbook, a sequence of stateless snapshots animated by the harness into the semblance of a single continuous creature. This is not a defect awaiting a patch but the medium one is obliged to build in: get the feeding-back right and the illusion is seamless and useful; get it wrong and the agent forgets your name mid-conversation — in the only sense that counts, it never knew.
Building the machinery reduces to three questions — what to retain, what to recall before the model’s eyes now, what to forget — and forgetting is an obligation, not an option: you cannot recall what you never retained, and you cannot retain everything. What welds the three into one is a shared, forward-facing criterion: the test of a memory is not whether it faithfully records the past but whether it will earn its keep next time. Cognitive scientists make the same case for human memory, where active shedding of detail keeps what remains flexible rather than brittle and overfitted (Richards & Frankland, 2017).
Storage is easy: disk is cheap, and a verbatim transcript of everything costs a pittance. The difficulty is recall: the model cannot look at the store — it attends only to its context window, so to “recall” a thing is to copy it in. And the window is not merely small but unevenly used — the model attends to its middle markedly less reliably than to its ends, the lost-in-the-middle effect (Liu et al., 2024) — so a fact duly recalled can still be missed. The effect varies by model and has narrowed with newer generations — the exercise bank’s lost-in-the-middle lab measures it on yours — but attention over a long window is not yet uniform. Storage cheap and recall dear, an agent’s memory takes the shape of a hierarchy (Figure 4.2): the small, fast window the model reads directly, backed by large, slow stores reached on demand — register–cache–disk, rediscovered for an agent whose “register” is a few thousand tokens of fickle attention. To manage the traffic between the levels we borrow the classical vocabulary of memory’s kinds.
%%{init: {"theme": "base", "themeVariables": {"primaryColor": "#E8ECFF", "primaryBorderColor": "#4054B2", "primaryTextColor": "#16171B", "lineColor": "#3B4351", "edgeLabelBackground": "#FAF7F0", "clusterBkg": "#EFE9DC", "clusterBorder": "#766F65"}}}%%
flowchart TB
M(["Model --- stateless,<br/>reads only the window"])
subgraph WM["Working memory"]
CW["Context window: task,<br/>recent exchange, retrieved<br/>facts, scratchpad"]
end
subgraph LTM["Long-term memory"]
EP[("Episodic store:<br/>past events,<br/>actions")]
SEM[("Semantic store:<br/>documents,<br/>knowledge base")]
end
X["✗ gone for good"]
M <-->|"read / write"| CW
CW -->|"retain / consolidate"| EP
CW ~~~ SEM
EP -->|"recall"| CW
SEM -->|"retrieve"| CW
CW -.->|"evict / forget"| X
classDef world fill:#EFE9DC,stroke:#766F65,color:#16171B
class EP,SEM world
4.10 Working, Episodic, and Semantic Memory
Psychology has spent a century carving memory at its joints, and they fit an agent almost too neatly. The broadest cut is by duration and role: on one side a small, fast store holding whatever is being thought about now — working memory, Baddeley and Hitch’s refinement of the older, passive short-term memory into a store that operates on what it keeps (1974), fast, where active reasoning happens, and famously, frustratingly small; on the other, a vast, durable long-term memory holding everything else. The first is the desk you work at; the second is the filing cabinet behind you.
Long-term memory divides again, by what it holds, in a distinction we owe to Tulving (1972): episodic memory is the autobiographical record, experiences tagged with their occasions, while semantic memory is general knowledge shorn of any occasion of learning — you cannot recall the day you learned that Paris is the capital of France, which is precisely what makes the fact semantic.
The taxonomy maps onto an agent’s machinery with little forcing; Table 4.2 sets the correspondence out. Working memory is the context window. Episodic memory stores the agent’s past — conversations, actions, results — fetched in when relevant. Semantic memory lives in two places at once: the bulk baked into the model’s weights (Section 2.2), the remainder in external knowledge bases the agent retrieves from. And a fourth kind, procedural memory — the knowledge of how rather than that1 — is mostly the agent’s tools (Section 4.2) and standing instructions, capability wired into the harness rather than recalled as text. The mapping is an analogy, not an identity — a vector store is no hippocampus; the vocabulary’s use is telling a builder which machinery a given need calls for.
| Cognitive kind | What it holds | The agent’s machinery | Example |
|---|---|---|---|
| Working | Whatever is being thought about now — fast, small, where reasoning happens | The context window | The task at hand and the model’s own scratchpad |
| Episodic | Particular past experiences, each tagged with its occasion | An external store of the agent’s past, fetched in on demand | “What did the user ask me last week?” |
| Semantic | General facts and concepts, shorn of any occasion of learning | The model’s weights, plus external knowledge bases | “Paris is the capital of France”; a retrieved document |
| Procedural | Skills and habits — knowing how rather than that | The agent’s tools and standing instructions | Calling a tool; following a standing instruction |
4.11 The Context Window as Working Memory
One kind of memory is the model’s directly: working memory. The context window is all the model can attend to at a given moment; nothing outside it exists, for the model, in that instant. Into it go the task, the recent exchange, whatever has been fetched, and the model’s own unfolding thought; over that alone the next continuation is computed. The window is fast, it is where the reasoning happens, and it is, decisively, small: always finite, always far smaller than the agent’s history and the world’s knowledge.
One use of the window makes the analogy almost literal: the model writes into its own context. The chain of thought of Section 3.10 is exactly this — intermediate reasoning the model reads back, the window used as a scratchpad, the name Nye and colleagues gave the practice (2021): somewhere, on the page, to hold the digits while you carry the one.
The window is scarce in two ways: it fills, and once full, something has to give; and — subtler and worse — the lost-in-the-middle effect bites before it is even full, leaving warm regions at the ends and cold ones in the middle, where a fact can be present yet, to all practical purposes, unread. What goes in, and where, both matter.
Managing it is an unending discipline whose central move is compaction: as the history grows, replacing the verbatim record with a précis — reclaiming both space and cost, for a history carried in full is re-paid for in tokens on every call. Done well, compaction keeps an agent coherent across any length of interaction; done badly, it summarises away the one detail that turns out to matter and proceeds, with perfect confidence, without it. Around it cluster the lesser crafts: material placed at the warm ends, stale tool output pruned, a running state updated rather than reconstructed. But compaction is lossy — every summary discards something, gone for good unless a copy was kept elsewhere — so working memory cannot be an agent’s whole memory: anything that must be recoverable in full belongs in a larger, more patient store, fetched back only when wanted. That store is long-term memory, and the fetching is retrieval.
4.12 Retrieval and Long-Term Memory
The way past the window’s limits is to back the small fast store with a large slow one: whatever will not fit goes into an external store, fetched back when needed. This is an agent’s long-term memory, and the fetching is retrieval — RAG, which Section 4.2 treated as a tool, seen now from the standpoint of memory. Retrieval is recall: bringing a relevant fragment of the past back into the working memory of Section 4.11, where alone it can be used.
The simplest store is the one current practice has converged on: files. Rather than let compaction summarise a bulky thing away — a full test log, the survey of an unfamiliar codebase — the agent writes it out, entire and exact, to a file it can name, keeping only the path and a précis line in the window. This is offloading, compaction’s lossless complement: one shrinks what stays in, the other moves the bulk out, the copy elsewhere complete. The coding harnesses of the day lean on the move heavily, note-taking to files a standard technique (Ji, 2025). And a note written for one’s later self is legible to others: a second agent can read what the first wrote, and the private memory tier becomes a shared one — the germ of the common ground Chapter 6 will build and the trace-reading coordination Chapter 8 will name stigmergy.
The whole difficulty of retrieval is relevance: finding the few items that bear on the present moment. The dominant answer is to search by meaning, over embeddings: each stored item becomes a vector positioned so that closeness in space tracks closeness in meaning — the compressed representations of Section 2.2 put to fresh use — and the query, embedded likewise, returns its nearest neighbours. This is dense retrieval (Karpukhin et al., 2020); store, embedding model, and nearest-neighbour index make a vector database, agent memory’s unglamorous workhorse. Exact identifiers — a function name, an error code — defeat embeddings, though, yielding to plain lexical search, whose standard-bearer BM25 still sets the baseline a dense retriever must beat (Robertson & Zaragoza, 2009). Production retrieval is therefore usually hybrid, blending the rankings — most simply by reciprocal rank fusion, which trusts neither ranker’s scores, only their orderings (Cormack et al., 2009) — and increasingly the retriever is the agent itself, wielding search and grep tools (Section 4.2) and iterating: retrieval as behaviour rather than lookup.
Similarity alone is a crude ranking, though: the most apt memory is not always the one worth surfacing. An influential refinement from the Generative Agents of Park and colleagues (2023) scores each candidate by relevance, recency, and importance at once, so a salient or recent memory may outrank a merely similar one. The three are folded into a weighted sum: with m a candidate memory, q the query, and each component normalised to [0,1],
\mathrm{score}(m) \;=\; w_{\mathrm{rel}} \cdot \mathrm{rel}(m, q) \;+\; w_{\mathrm{rec}} \cdot \mathrm{rec}(m) \;+\; w_{\mathrm{imp}} \cdot \mathrm{imp}(m),
and the rule is small enough to run as written: give each memory its components as plain numbers — in a live store, rel would come from embedding similarity and rec from a decaying clock, but the ranking neither knows nor cares — and recall is a sort:
from dataclasses import dataclass
@dataclass
class Memory:
text: str
rel: float # components precomputed and normalised to [0, 1]
rec: float
imp: float
def score(m: Memory, w_rel: float = 1.0,
w_rec: float = 0.5, w_imp: float = 0.5) -> float:
return w_rel * m.rel + w_rec * m.rec + w_imp * m.imp
store = [Memory("deploy broke after schema change", 0.9, 0.2, 0.6),
Memory("user prefers terse replies", 0.3, 0.5, 0.9),
Memory("staging failed this morning", 0.7, 1.0, 0.4)]
[m.text for m in sorted(store, key=score, reverse=True)[:2]]
# -> ['staging failed this morning', 'deploy broke after schema change']Even with relevance weighted twice as heavily as recency and importance, the fresher memory takes first place from the more similar one — the blend doing exactly what it was hired to do, and exactly the ordering to inspect when it surfaces the wrong things. The weights are not universal — a coder agent wants relevance dominant; a long-running assistant wants recency, lest it dredge up last month’s resolved bug — so tune them against your own recall failures, not Park’s townsfolk.
The same machinery serves both halves of long-term memory from Section 4.10 — the episodic store of the agent’s past (“what did we decide about the schema last week?”) and the semantic store of knowledge, RAG’s classic use. The line smudges in practice, but episodic memory prizes recency and specificity where semantic prizes coverage and correctness; the two are tuned differently.
Storing and fetching verbatim is easy; a long lifetime asks that memories be extracted — a rambling exchange reduced to a few crisp notes; consolidated as new information arrives, a corrected fact superseding the old, else drift, where the world changes but the memory does not; and finally forgotten, lest the store silt up.
These are the province of agent memory architectures. Packer and colleagues cast the problem as an operating system — MemGPT (2023), the context window as main memory, the external store as disk, the agent paging between them. Its quieter contribution was handing the paging to the agent itself — tools to search its own archive and rewrite its own core memory — so when to remember becomes the agent’s decision rather than a wired-in rule; memory frameworks now package the cycle for the builder.
Generative Agents put a second device to work: reflection. Periodically the agent gathers raw memories and has the model synthesise a higher-level observation — “the user keeps returning to deployment, so reliability is their real concern” — written back into the store, open to recall and further reflection: episodic-to-semantic consolidation made mechanical. It is also the subtlest way memory can betray an agent: a reflection is the model reasoning over its own memories, heir to every failure of model reasoning, and a mistaken insight written back with a sound one’s confidence becomes a false premise the agent will recall, reason from, and reflect upon again, the error compounding quietly. Memory that can improve itself can also corrupt itself, and nothing in the loop is charged with telling the two apart.
If reflection is memory corrupting itself on the way in, retrieval is the matching hazard on the way out. The retriever can fetch the wrong thing — a plausible but irrelevant memory, a fact since superseded — and a wrongly retrieved memory is worse than none, for it actively misleads, dropping relevant-looking falsehood into the context where the model will take it as established (the irrelevant-context hazard of Section 3.10). Recall, in agents as in people, is reconstructive; it can confabulate. So the standing caution applies once more: what memory hands back — retrieved or reflected — is a candidate to be checked, not a fact to be trusted on arrival. What remains is the discipline governing the traffic: deciding, turn by turn, what the model is actually shown.
4.13 Context Engineering
Every call, an agent faces one decision afresh: what to place in the context window this time. That decision is context engineering,2 the deliberate assembly, each call, of what the model is shown: system instructions fixing its role, the task, the memories worth retrieving, the latest tool results, the compacted history. Working memory and long-term memory, for all their apparatus, are just inputs to this one assembly step, where they finally meet.
The deepest way to see it is as the engineering of the agent’s perception — the composing of the report that, Section 4.1 said, the agent reads in place of the world. Whatever is left out, the agent is blind to that turn; a misleading context is, from inside, indistinguishable from the truth. To engineer the context well is to give the agent good eyes; to engineer it badly is to feed it a distorted picture and marvel at its mistakes — not a knack for prompting but the construction, turn by turn, of the agent’s perception of its situation.
It is what prompt engineering grew into: the unit that matters is no longer one request’s wording but the whole assembled context of a long interaction — what ought to be in the window right now, given all that has happened? — and the change of name marks a real change in where the difficulty lies. Its reach is worth fixing: context engineering is the assembly of what the model is shown — its input, no more; the loop, the tools (Section 4.2), and the safeguards (Section 4.7) belong to the harness at large (Section 2.4). What makes it this chapter’s business is that here memory comes due: every store and stratagem of the sections before exists to get the right few things into that single assembly.
A second shift changes who does the assembling. Originally you rebuilt and resent a flat transcript every call; the newer model APIs reframe it as a list of typed items — messages, tool calls and results, spans of the model’s own reasoning — and will, if you let them, keep the list server-side and assemble each turn’s context for you (Open Responses, 2026). It is at once a convenience and a surrender. The distinction to keep is client-managed context, which you compose and can therefore inspect, tune, and answer for, against server-managed, composed for you and correspondingly opaque. Neither is wrong; but these are decisions where an agent’s quality quietly lives, and a delegated decision is one you can no longer audit when it goes wrong. What you cannot see, you cannot curate.
What makes it an art is the chapter’s recurring constraint — a finite, unevenly attended window (Section 4.11) — so the goal is the fewest, most relevant things, in a form the model will actually use. Two failures stand on either side. Too little, and the agent wants for a fact it needed, and invents one. Too much, and it drowns: irrelevant material dilutes attention and measurably degrades reasoning (the irrelevant-clause effect again), while the one genuinely useful fact, buried in the heap, is read straight past. The aim is not to maximise the context but to curate it — the right things, well placed, nothing else — and there a surprising share of an agent’s real quality is quietly won or lost.
One small device deserves mention because nearly every coding harness carries it: the agent keeps a running to-do list — the plan of Section 3.11, reduced to ticked and unticked lines — re-written into the warm recent end of the window as it works. The tool that writes it often does nothing else; its value lies in where the words land, reciting the plan each turn to steer attention out of the cold middle (Ji, 2025) — the plan pressed into service not as reasoning, and not yet as Chapter 18’s inspectable artefact, but as context.
The coder agent shows the whole discipline in one turn. Before each step its harness builds a context: standing instructions giving it its role and limits; the issue it must fix; the few files that bear on this change, retrieved rather than the whole repository tipped in wholesale; the last commands it ran and what they printed; a compacted note of what it has already tried and ruled out. Every item is a context-engineering decision; the agent’s competence on the turn is, to a first approximation, the quality of the assembly. Give it the right files and it is pointed and effective; give it the entire repository and it loses the thread; leave out the failing test’s output and it confidently mends the wrong thing. Throughout, the knowledge handled has been plain text; but knowledge can also be given structure — entities and relations, ontologies and graphs — and structure changes what can be retrieved, reasoned over, and, above all, shared: the chapter’s last subject, and the bridge from one agent’s memory to the common ground of several.
4.14 Knowledge, Ontologies, and Understanding Each Other
This closing section looks at semantic memory not as something stored but as something represented — among the oldest disciplines in artificial intelligence, and one that matters most precisely when one agent becomes two.
To represent knowledge is to commit to a structure: concepts as nodes and relations as labelled edges in a semantic network (a canary is-a bird); an ontology making the commitment explicit — in Gruber’s much-quoted phrase, an explicit specification of a conceptualisation (1993); and, scaled to millions of facts, today’s knowledge graph (Hogan et al., 2021), entities and relations a machine can traverse. The contrast with the vector store of Section 4.12 is the whole point: embeddings find what is similar; a graph encodes what is connected, and lets you follow the connection — function to callers, caller to module — a chain of reasoning similarity alone can neither represent nor walk.
The half-forgotten part of the history concerns us most. In the early 1990s the DARPA Knowledge Sharing Effort (Neches et al., 1991) set out to make knowledge not merely representable but shareable: the Knowledge Interchange Format (KIF) (Genesereth & Fikes, 1992), so one program’s assertions carried the same meaning to another, and the Knowledge Query and Manipulation Language (KQML) (Finin et al., 1994), for the asking and the telling. The dream faded — ontologies proved laborious to build and harder to agree upon — but notice what the effort was for: the field’s first serious attempt at how two systems can mean the same thing, the problem this book turns to next.
The programme has quietly returned under new names. GraphRAG (Edge et al., 2024) builds a knowledge graph over a document collection so a model can answer the broad, synthesising questions (“what are the main themes here?”) that flat retrieval, fetching similar passages, cannot reach. And the old programme’s deepest difficulty is transformed: a classical knowledge base was a closed world of symbols meaning only what other symbols said — the symbol-grounding problem (Harnad, 1990) — whereas a language model arrives saturated in the very text those symbols were meant to summarise, and reads a graph and the world it describes in something like the same breath. Structure and statistics, rivals for thirty years, turn out to be complements.
Industry has revived the word and bent it toward action: Palantir above all has put ontology back into commercial currency — meaning not a logician’s axiomatisation but an operational model of an organisation, its objects, their links, and, crucially, the actions that may be taken upon them: a digital twin through which software — and now agents — both read the world and change it.3 It is ontology in the engineering sense, but it makes the chapter’s case concretely: a structure you can traverse is one you can act on, and an explicit model of the world is sturdier footing for an agent than unstructured text (Section 4.5).
For a lone agent, structure is a convenience — plain text usually serves. The instant there are two, it becomes a necessity: shared memory needs an agreed shape — a common vocabulary both parties read the same way — before it can be shared at all. Our coder team has felt the want already: a knowledge graph of the codebase, entities for modules and functions, edges for what calls and imports what, is precisely the common ground every member could consult and trust. And common ground — knowledge two parties not only hold but know they hold together — is the hinge on which all cooperation turns.
And so Part II closes. We have taken a single agent apart and found one gradient running through it: the more a faculty must outlast the single call, the less the model gives and the more the harness must build. Reasoning the model nearly manages alone (Section 3.10); action it only shares (Section 4.2); memory it supplies only as frozen cargo (Section 4.9) — the anatomy of a single agent in one line: a stateless mind, and the apparatus wrapped around it to make it persist. Each faculty, moreover, we described in the singular. Admit a second agent and each takes on a social dimension: reasoning becomes negotiation, action coordination, memory the common ground on which a collective either understands itself or dissolves into confusion. That step — from the agent to the agents — is the business of Part III.
4.15 Summary
- Tools are how an agent reaches out of the closed world of text. The substrate decides and can do none of it; the model emits a structured request, the harness executes it, and a tool’s schema is very nearly a STRIPS operator: the classical account of actions without the classical guarantees.
- The action space is a design decision: prefer the narrowest surface that still reaches the task. Code and the screen buy generality with reliability; use the specific tool where one exists.
- Tools ground the symbols, and grounding is managed, not solved. An agent can check the world rather than assert it, but the gap between symbol and referent persists — and for the embodied agent the chapter runs at full price: the physical world is not an application of its ideas but their origin.
- Action’s second face is danger. A reasoning error costs tokens; an action error can cost the data, the money, or the trust, and some actions cannot be undone — approval, sandboxing, and least privilege contain exactly that asymmetry.
- The substrate has no memory, so an agent’s memory is entirely apparatus, and the problem is recall, not storage. The context window is scarce working memory; retrieval is recall made mechanical, its new failure mode the confidently irrelevant memory that misleads as surely as a hallucination.
- Context engineering unifies the faculty — assembling what the model is shown is the engineering of its perception — and shared, structured knowledge is where one agent’s memory becomes the common ground of several, where Part III begins.
4.16 Exercises
Exercise 1. One turn of the running team’s coder agent produces five events: (1) at the model’s request, the harness reads src/parser.py from the repository; (2) the harness appends the file’s contents to the context; (3) at the model’s request, an interpreter runs a ten-line snippet that computes a statistic over those contents and prints one number, writing nothing; (4) the harness appends the interpreter’s output to the context; (5) the harness appends a message from the reviewer agent: “the empty-input case is wrong”. Section 4.1 holds that the agent–environment boundary is drawn, not found; draw it twice. Under drawing A the file system and the interpreter sit inside the agent and the reviewer outside; under drawing B only the model and the harness are the agent, and everything else is environment. (a) Classify each of the five events, under each drawing, as internal to the agent, an action upon the environment, or a perception of it — ten classifications, each with a phrase of justification. (b) The snippet in event 3 is later replaced by one that deletes a stale cache directory. Say what that act is under each drawing, name one safeguard from Section 4.7 whose applicability the classification decides, and state the failure that drawing A invites. (c) The snippet is instead changed to write profile.json into the repository tree, and the tester agent reads that file on its own next turn. Show, with a two-step trace in the loop’s vocabulary, why drawing A misdescribes this exchange, and redraw the boundary so that one agent’s action becomes the other’s perception.
Exercise 2. The tester agent’s job takes four steps: set up a fixture, run the suite, parse the failures, file a report. Two action-space designs are proposed. Under the fine design each step is its own tool: a call costs c = 300 tokens and succeeds with probability q = 0.92, independently across attempts. Under the coarse design a single tool performs all four steps in one call costing 900 tokens, which succeeds with probability r = 0.7. In both designs the harness detects every failure, and the agent retries the failed call until it succeeds. (a) Compute the expected token cost of the whole job under each design. (b) Find the coarse-tool reliability r^* at which the two designs cost the same in expectation. (c) Both computations rest on an assumption that Section 4.7 spends itself qualifying. Name it, and explain which design the chapter’s reasoning favours when the report-filing step, once executed, cannot be retracted — and why expected token cost is then no longer the deciding criterion.
Exercise 3. Five tasks land on the running team’s desk: (i) bump the version string in pyproject.toml from 2.3.1 to 2.3.2; (ii) find every call site of a deprecated function across the repository and report a count per module; (iii) mark the team’s ticket as done in the company tracker, which humans drive through its web interface but which also publishes a documented REST API; (iv) enter month-end figures into a supplier’s invoicing portal that offers no API and no export and renders only in a browser; (v) restart a flaky CI runner through an internal web dashboard that has no API — though the infrastructure team ships a small command-line utility that performs the same restart. (a) Using the working rule of Section 4.4 and the columns of Table 4.1, assign each task the narrowest action surface that reaches it, with one sentence of justification apiece; two of the five are traps in which the superficially obvious surface is not the right answer — identify both. (b) For the one task that genuinely requires computer use, state what the choice pays in each of the last two columns of Table 4.1, and propose one change to the task’s environment — not to the agent — that would let a narrower surface reach it.
Exercise 4. Model the lost-in-the-middle effect crudely: an item placed in the opening or closing quarter of the assembled context is actually used by the model with probability 0.9; an item in the middle half, with probability 0.55; reads of distinct items are independent. On this turn the coder agent’s context carries two critical items — the failing test’s output and a note of the API constraint the fix must respect. (a) Compute the probability that both items are used if the harness assembles chronologically, which lands both mid-window, and if it instead places both at the ends. (b) A colleague proposes duplication: leave each item where chronology puts it and repeat it once at an end. Assuming the two reads of the same item are independent, compute the probability that an item is used, and that both are. (c) Now assume the reads are nested rather than independent — whenever the middle copy would be read, the end copy would be too — and recompute. What do (b) and (c) together reveal about where duplication’s advantage over simple end-placement comes from, and what should a builder measure before trusting it?
Exercise 5. The team keeps a knowledge graph of its repository: nodes are functions, and a directed edge from f to g records that f calls g. The edges are: main calls start_server and check_env; start_server calls load_settings; run_migration calls load_settings; load_settings calls parse_config; check_env calls parse_config; parse_config calls read_file; and render_docs calls format_markdown. The reviewer asks: “what can break if parse_config’s return type changes?” (a) Compute the affected set by hand, and explain why read_file is not in it despite being adjacent to parse_config. (b) A vector store scores each function’s docstring against the query: parse_config 0.93, render_docs 0.72, format_markdown 0.66, load_settings 0.61, read_file 0.58, check_env 0.42, run_migration 0.18, start_server 0.15, main 0.09. Which affected functions does top-4 similarity recall miss, which unaffected ones does it admit, and what property of the query makes similarity so poor a proxy for it? (c) Write a function affected(calls, root) that computes the affected set from the edge dictionary, and check it against your answer to (a).
Exercise 6. The coder agent’s harness has been assembling the following context every turn, against a window of 32,000 tokens: standing instructions, 1,200 tokens; the full text of every repository file, 48,000; the verbatim transcript of all thirty-five earlier turns, 21,000; the forty memories the retriever returned — everything above a similarity floor of 0.2 — 8,000; the complete CI log, 12,000, of which the failing test’s output is the final 300; the issue text, 400; and the output of the last command run, 900. It does not fit, and when the harness trims blindly from the tail the agent mends the wrong thing. (a) Dismantle the assembly: produce a revised one of at most 12,000 tokens, giving each item’s budget, justifying every cut, compression, or retention by a principle from this chapter, and identifying the one item that must survive verbatim. (b) State where in the window you would place the two most critical items, and why. (c) The platform now offers to keep the transcript server-side and assemble each turn’s context for you (Section 4.13). Which parts of your plan in (a) could you no longer perform or verify, and what is the trade?
Further exercises for this chapter continue in the web edition’s exercise bank.
The distinction between knowing how and knowing that is Gilbert Ryle’s (1949); procedural memory is the home of the former.↩︎
A practitioners’ term; Anthropic’s Effective Context Engineering for AI Agents (https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents) is a representative treatment.↩︎
Palantir’s vocabulary splits the ontology into semantic elements (objects, links) and kinetic ones (actions) (n.d.); the branding is the company’s, the move general.↩︎