3 The Language Model as a Cognitive Substrate
The previous chapter closed with the missing piece arriving: the foundation model, supplying at last the general competence the hollow agent of Section 2.4 had always lacked. This chapter takes a closer, and more sceptical, look at that piece — at what it actually is, what it genuinely gives you, and, in the part the enthusiasm tends to hurry past, what it conspicuously does not. The title gives the conclusion away. The foundation model is a substrate: the raw material from which agents and multi-agent systems are built, and not the finished article. Mistaking the one for the other is where the field’s most expensive failures begin.
A substrate is something other things are built upon — the silicon beneath a circuit, the soil beneath a forest. It is necessary, it is enabling, and it is emphatically not the thing itself. A foundation model is a cognitive substrate in precisely this sense: an extraordinary, general-purpose capacity for producing fluent, knowledgeable, contextually apt text, upon which one can build something that reasons, remembers, acts, and pursues goals — but which, left to itself, does none of those things. The model is the most capable component we have ever had from which to build agents; it is not an agent. We drew that distinction in passing in Section 1.2; this chapter is where we earn it.
The reason to be careful here — to take an inventory of the substrate’s gifts and its gaps with some precision — is that almost every property the model lacks turns out to be the subject of a later part of this book, and almost every difficulty of multi-agent systems lives in that gap. A model that came with memory, goals, grounding, reliability, and a sense of other minds would leave the rest of the field very little to do. It has none of these dependably, which is exactly why the classical theory — of coordination, communication, commitment, and collective choice — becomes newly applicable rather than redundant the moment the substrate appears. The model does not make the old problems go away. It makes them, for the first time, worth solving.
This chapter is deliberately not a tutorial on transformers, attention, or the mechanics of training; those are covered well elsewhere, and the engineering of multi-agent systems depends remarkably little on them. We work at the altitude the rest of the book actually needs: the model as a thing with certain capabilities and certain limits. We characterise the substrate; catalogue what it provides; catalogue, with equal care, what it withholds; and close by showing why the gap between the two is the bridge back to thirty years of theory. A warning in advance: this is the most sceptical chapter in Part I, and intentionally so. Enthusiasm about what these models can do is not in short supply; a clear-eyed account of what they cannot is the more useful contribution, and the safer foundation on which to build.
3.1 The Substrate, Characterised
Before we can say what the substrate provides and withholds, we need a working picture of what it is — not the picture an engineer training one would require, but the leaner one a person building agents from it can actually use. At this altitude the foundation model is almost embarrassingly simple to describe, genuinely mysterious to explain, and remarkably easy to mischaracterise. We take those three in order: what the model does, why it works at all, and what we shall mean by calling it a substrate.
3.1.1 A Stateless Function from Context to Continuation
Strip away the marketing and the model is a function. It takes a sequence of text — the context — and returns a probability distribution over which token should come next. Writing x_{1:t} = (x_1, \dots, x_t) for the context, a sequence of tokens drawn from a vocabulary \mathcal{V}, the model with parameters \theta is the conditional distribution p_\theta(x_{t+1} \mid x_{1:t}); one step of generation samples a token from it and appends,
x_{t+1} \;\sim\; p_\theta(\cdot \mid x_{1:t}),
and repeating — sample, append, sample again on the lengthened context — gives you a system that continues text. At the operational level, that is the whole of it. The dominant way to build such a function is the transformer (Vaswani et al., 2017), an architecture of stacked attention layers; but the architecture is, for our purposes, an implementation detail. A multi-agent system built on transformers and one built on some yet-to-be-invented successor would raise precisely the same coordination problems, and this book is written to outlast the answer.
Two features of this picture matter for everything that follows. The first is that the model is stateless: each call begins from nothing but the context it is handed — the model’s entire access to the past is whatever appears in x_{1:t}. It does not remember the previous call, does not know that a conversation is occurring, and carries no thread of identity from one invocation to the next. The second is that the function is probabilistic: it does not compute the next token so much as sample one, and the same context can yield different continuations on different calls. A great deal of the engineering in later chapters consists of coaxing reliable behaviour out of a component that is, by construction, neither persistent nor deterministic.
3.1.2 The Surprise of Scale
Described that baldly, the foundation model sounds like a glorified autocomplete, and for most of the history of language modelling that is essentially what it was. The surprise — and it earns the word, because almost no one foresaw its extent — is what happens when this modest objective is pursued at enormous scale. Train a sufficiently large model on a sufficiently large fraction of the written record, and the system that emerges does not merely produce plausible text; it acquires, as a by-product, a startling range of competences it was never explicitly taught — translation, arithmetic, summarisation, code, rudimentary reasoning — for the simple reason that predicting the next token well enough turns out to require them.
This is not folklore but a measured regularity. Performance improves smoothly and predictably as a power law in model size, data, and compute, across many orders of magnitude — the scaling laws (Kaplan et al., 2020) — and it was on the strength of these curves that the field placed the expensive bet on scale that produced models such as GPT-3 (Brown et al., 2020). More contentiously, certain abilities appear to switch on abruptly past a threshold of scale, absent in smaller models and present in larger ones: the claim of emergent abilities (Wei, Tay, et al., 2022). That claim is genuinely contested — a pointed rejoinder argues that much of the apparent emergence is an artefact of how an ability is measured, with discontinuous metrics manufacturing sharp transitions where the underlying behaviour in fact improves smoothly (Schaeffer et al., 2023). We need not adjudicate the quarrel here. But we should register, honestly, that the central empirical fact of the field — the relationship between scale and capability — remains only partly understood by the very people producing it. A substrate whose powers are not fully explained even by its makers is one to build upon with a certain humility.
3.1.3 “Cognitive Substrate”, Defined
It is time to make the chapter’s central metaphor precise. The term foundation model was coined to capture the way these systems serve as a common base — trained once, at great expense, then adapted to countless downstream uses (Bommasani et al., 2021). The word foundation is well chosen, and we lean on it: a foundation is something you build on; it is load-bearing, indispensable, and not itself a building.
We will speak throughout of the model as a cognitive substrate: the raw material of cognition rather than cognition itself. The analogy is to the processor beneath a program. A CPU is a remarkable thing — it can, in principle, compute anything computable — yet it has no purpose, no memory of its own beyond a handful of registers, and does nothing at all until a program organises it towards an end. The foundation model stands in the same relation to an agent. It supplies a general capacity for producing apt text; the agent — through memory, tools, a control loop, and a goal — organises that capacity towards actually doing something. The capacity is necessary, and was until recently the missing ingredient; it is not, on its own, remotely sufficient, and the rest of this book is in large part an account of what must be added to it. To mistake the substrate for the system — to point at the processor and call it the program, or at the model and call it the agent — is the error that every later chapter is, in one way or another, written to prevent.
3.2 What the Substrate Provides
What does the substrate actually give you? The list is short, but each item is large, genuinely new, and — this is the point — exactly the sort of thing the hollow agent of the previous chapter could never be handed. We set the gifts down here in their best light; the next section supplies the qualifications, and the reader of a sceptical disposition is asked to hold it for a few pages.
3.2.1 A Compressed Model of Human Knowledge
The first and most obvious gift is knowledge. In learning to predict text drawn from a substantial fraction of the written record, the model comes to encode an enormous amount of what that record contains: facts, conventions, the gist of how things work, the shape of a sonnet and the syntax of Python. It is illuminating to view this through the lens of compression: a predictor that assigns probabilities well is, by the source-coding theorem, an efficient compressor of what it predicts — and a system that compresses the written record this well must have captured a great deal of its regularity and structure to do so (Delétang et al., 2024). The model is, in effect, a lossy, queryable compression of a great deal of human knowledge — available on demand, in natural language, with no database, no schema, and no query language. That one can interrogate a language model for facts as though it were a knowledge base is by now a familiar observation (Petroni et al., 2019), and it is part of what makes the substrate feel, on first contact, like magic.
Two words in that description are load-bearing, and we shall lean on them hard in Section 3.3: lossy and compression. A lossy compression of human knowledge is not human knowledge, and a system that reconstructs plausible-looking facts from a compressed store will, on occasion, reconstruct plausible-looking facts that were never true. For the moment, though, the gift is real: competence on tap, across an extraordinary breadth of subject matter, with no retrieval step and no programming.
3.2.2 Language as a Universal Interface
The second gift is easy to overlook precisely because it is everywhere: the substrate speaks. Fluent command of natural language sounds like table stakes for a language model, but its consequences are out of all proportion to how obvious it appears. Text is the nearest thing computing has to a universal interface. It is how the model takes instruction, how it returns results, how it invokes a tool, how it addresses a human and — decisively for this book — how one agent addresses another. The insight that an enormous range of tasks can be cast uniformly as text-in, text-out predates the current models (Raffel et al., 2020), but the foundation model is what made the casting pay: a single substrate that consumes and emits the universal medium, and can therefore be wired to anything else that does.
This is the quiet structural fact that makes multi-agent systems buildable at all. If every agent, tool, and human shares a common medium, then composing them becomes a matter of routing text to the right place at the right time — which is hard, but tractably hard, and is much of what Part III is about. The classical dream of agent communication languages (Section 2.3) was a shared tongue for autonomous software; the foundation model supplies one, and it happens to be English (or Mandarin, or Python). What is gained and lost by pressing a natural language into that role is the business of Chapter 8.
3.2.3 Programmable by Description
The third gift is that the substrate can be programmed by description. Conventional software does what it is coded to do; a foundation model can be steered by a statement of what you want, in the same natural language it reads and writes. Two mechanisms deserve naming. The first is in-context learning: shown a handful of examples in its context, the model will often infer the pattern and continue it, with no change to its weights — programming by example, at inference time (Brown et al., 2020). The second is instruction-following: a model post-trained to do as it is told will carry out a task set in plain prose, an ability deliberately instilled, most influentially by training against human preferences (Ouyang et al., 2022).
The reader who recalls Section 2.2 will feel a flicker of recognition. Shoham’s agent-oriented programming proposed, in 1993, that we program machines by declaring their mental states and commitments rather than their procedures; it wanted only for a machine that could act on such a declaration. The system prompt is that declaration, and the foundation model is the machine that, at last, acts upon it. Programming by description is agent-oriented programming’s wish granted three decades late — with the catch, taken up in Section 3.3, that a substrate steered by description is also a substrate mis-steered by description, and exquisitely sensitive to exactly how the description is phrased.
3.2.4 Reasoning, of a Kind
The fourth gift is the one that earns the model its cognitive adjective, and the one to handle with the most care: the substrate can reason — after a fashion. Asked not merely for an answer but to work towards one step by step, models of sufficient scale produce intermediate reasoning that markedly improves their performance on arithmetic, logical, and commonsense problems; the technique, chain-of-thought prompting, is as simple as inviting the model to think aloud (Wei, Wang, et al., 2022). Something in the substrate supports the serial, decompositional working-through that we recognise as reasoning, and it can be coaxed out with a phrase.
How much of this is reasoning and how much a convincing performance of reasoning is a genuine and unresolved question — one we take up in earnest, with the sceptical apparatus it deserves, in Chapter 5. For this chapter the modest claim suffices: the substrate provides a usable, if unreliable, capacity to break a problem down and make progress on the parts, and it is this capacity, more than any other, that lets an agent built upon the model deliberate rather than merely respond. That is the deliberative core the hollow agent never had. What it conspicuously still lacks is the subject of the next section.
3.3 What the Substrate Withholds
Now the bill. The gifts are real, but each arrives wrapped around a set of absences so consequential that an agent is, in the end, mostly the machinery built to compensate for them. The pattern is worth noticing in advance: each thing the substrate withholds is the subject of a later chapter, and each is a place where the classical theory turns out to have something to say. We take the four most important in turn — not to deflate the technology, but because one cannot build reliably on a foundation whose weak points one has not located.
3.3.1 No Memory, No State
Begin with the absence we have already met. Because the model is stateless, it has no memory of its own: every apparent thread of continuity — a conversation’s history, a persona, a goal carried across time — is sustained not by the model but from outside it, by gathering the relevant past into text and feeding it back in as context. This is not a temporary limitation awaiting a larger model; it is what the substrate is. The coder agent of our running example that has just written a function has, by its next call, no recollection of having done so: the continuity that carries its own work across the repository is something the harness manufactures, never something the model brings.
The obvious patch — simply put everything in the context — runs into two hard walls. The context is finite, and nowhere near large enough to hold an agent’s full history of any duration; and, more insidiously, the model does not even use the whole of what it is given equally well. Information placed in the middle of a long context is attended to markedly less reliably than information at the beginning or end, so that a fact can be present in the prompt and yet, for practical purposes, be missed (Liu et al., 2024). The context window is therefore not a memory but a small, lossy, and curiously uneven working buffer. Building genuine memory — what to retain, what to recall, what to forget, and how to fit any of it into that buffer — is an engineering problem the substrate hands the agent designer entirely unsolved. The deliberate assembly of the context handed to the model on each call has acquired its own name, context engineering; it, together with memory, is the subject of Chapter 7.
One clarification guards this claim against a fair objection. A reader who meets the substrate through a modern interface may protest that it is plainly not stateless — that the platform recalls the conversation perfectly well from one call to the next. So it may; but the statefulness is the platform’s, not the model’s. Recent model APIs will keep the running transcript on their own servers and let you continue it by reference rather than resending it each time — OpenAI’s Responses API began the fashion, and a vendor-neutral Open Responses specification (2026) has since begun to standardise the shape across providers — a genuine convenience that changes nothing underneath. The model is still handed a context and still begins from nothing but it; only the apparatus that remembers has moved behind the API instead of sitting in front of it. Statelessness, in short, is a property of the substrate, not of the interface you happen to meet it through, and the machinery that supplies the memory — wherever it is hosted — is the whole business of Chapter 7.
3.3.2 No Goals, No Will to Act
The second absence is more fundamental and more easily missed: the substrate has no goals, and no means of pursuing them. It does not want anything. Handed a context, it returns a continuation and falls silent; it does not act, does not persist, does not check whether the world has changed, and does not come back tomorrow to see a task through. There is no loop, and nothing that would turn one.
This is the precise sense in which, as we insisted in Section 1.2, the model is not an agent. Agency — a goal, the initiative to pursue it, and a perception–action cycle in which to do so — is a property not of the model but of the system wrapped around it: the control loop that calls the model repeatedly, the tools through which its words become deeds, the state that lets it pursue something across many calls. The substrate supplies the capacity to judge what a competent response would be; the agent supplies the standing intention to produce one, and to keep producing them until a goal is met. That wrapper is the business of Chapter 4 and Chapter 6. The model is the engine, not the driver — and a foundation model left to idle does precisely nothing, very fluently.
3.3.3 A Closed World of Text
The third absence is that the substrate lives in a closed world of text — where text should be heard broadly, as the trace reality leaves, since the current generation of models reads images and audio too, and the closure survives every such widening of the letterbox. The substrate’s entire acquaintance with reality is that trace: it knows the word stove, and ten thousand sentences in which stoves appear — shown a photograph, it may nowadays even recognise one — but it has never been burned, and it cannot, unaided, turn one on, or check whether the one in front of you is hot. This is the symbol grounding problem in modern dress — the old worry, sharpened over decades, that symbols manipulated only in relation to other symbols are never quite fastened to the things they are about (Harnad, 1990). The contemporary form of the argument is pointed: a system trained only on linguistic form has, in principle, no direct route to meaning, however fluent its output becomes (Bender & Koller, 2020) — and a diet of pixels widens the record without changing its nature. What the substrate lacks is not another medium but a loop: acting on a world that pushes back, and being corrected by it, on the world’s schedule rather than the training set’s.
One may dispute how absolute the limit is, and we need not settle it; what matters for building agents is the practical consequence. Without tools, the model can neither perceive the world nor change it — it can only produce text about a world it cannot see. Connecting it to reality, through tools that let it sense and act, is therefore not an optional enhancement but the very thing that turns a generator of plausible sentences into something that can get anything done. And even once connected, the gap between a symbol and its referent does not close; it merely becomes a hazard to be managed. This is the subject of Chapter 6.
3.3.4 Capable but Not Trustworthy
The fourth absence is the one most likely to do you harm, because it hides behind the substrate’s greatest strength. The model is fluent, and fluency reads, to a human, as competence and confidence; but fluency is neither accuracy nor honesty. A system that generates statistically plausible text will, when the plausible and the true diverge, cheerfully generate the plausible. This is hallucination: the production of confident, well-formed, and wholly fabricated content, and it is not a bug awaiting a patch but a direct consequence of what the substrate does (Ji et al., 2023). The model does not know that it does not know; its expressions of certainty are themselves merely more generated text, tuned to sound right rather than to be right. The coder agent that reports “tests pass” without having run them is the canonical instance: a fluent, confident assertion with nothing behind it, of exactly the kind the rest of this book is built to catch.
The unreliability has further faces. The substrate is brittle in disconcerting ways: the same request, rephrased or merely reformatted, can yield materially different answers, so that performance swings on changes a human would regard as cosmetic (Sclar et al., 2024). Its knowledge is frozen at the moment its training stopped, leaving it confidently out of date about anything since. And it remains, at bottom, a stochastic parrot in the precise sense its critics intended — a system of extraordinary facility with the form of language and no guaranteed purchase on its content (Bender et al., 2021). None of this makes the substrate useless; it makes it a component whose outputs must be checked, bounded, and corroborated rather than trusted, and reliability something the surrounding system has to manufacture. How to do so runs through Chapter 23 and Chapter 25, and a good deal of the book besides.
3.4 Why the Classical Theory Becomes Newly Relevant
We can now collect the chapter’s two halves into the claim the preface promised. Set the substrate’s gifts beside its gaps — a general competence to deliberate, wrapped around a near-total absence of memory, goals, grounding, and reliability — and a striking thing comes into view. The shape of what the foundation model leaves undone is, almost line for line, the shape of the classical theory of agents and multi-agent systems. The arrival of the substrate does not make that theory redundant. It makes it, for the first time, applicable.
3.4.1 The Hollow Agent, Filled
Recall the hollow agent of Section 2.4: classical architectures for deliberation, communication, and commitment, with no general competence to put inside them. The foundation model is precisely the missing filling. Drop a general, promptable competence into the cavity the classical architectures were shaped to hold, and structures that had been elegant but inert — agent-oriented programming, the belief–desire–intention loop, the whole apparatus of Section 2.2 — become, all at once, things one can actually run.
This is why the theory was premature rather than wrong (Section 2.6). Bratman’s intentions, Shoham’s mental-state programming, the planners and the communication languages: none of these was mistaken about how a competent agent should be organised. They were merely waiting for a competent agent to organise. The substrate does not replace that body of work; it completes it, by supplying the one component every part of it assumed and none of it could build.
3.4.2 Every Absence Is a Chapter
Now look again at Section 3.3, but as a reading list. The substrate has no memory — and the management of working, episodic, and semantic memory is a studied subject, the matter of Chapter 7. It has no goals and no loop — and agent architecture, the business of arranging perception, deliberation, and action into something that acts, is the matter of Part II. It is ungrounded — and the connection of symbols to a world through perception and action is the matter of Chapter 6. It is unreliable — and evaluation, dependability, and safety, the disciplines of trusting what an agent does, are the matter of Part VII. Each thing the model withholds names a chapter that supplies it; the book’s table of contents is, in large part, a catalogue of the substrate’s omissions.
One omission towers over the rest, and it is the one this whole book exists for. The substrate knows nothing of other agents. It has no inkling that the text it is shaping might be addressed to, or contested by, or dependent upon another mind pursuing its own ends; no notion of coordination, negotiation, commitment, or collective choice. These are exactly the concerns of the classical multi-agent literature — the canon preserved, against precisely this day, by Wooldridge (2009) and by Shoham and Leyton-Brown (2009). The foundation model fills the single agent; the moment you have two, everything the substrate cannot supply about their interaction is waiting, already worked out, in thirty years of theory. That is the wager of this book, and the substrate is what finally makes it collectable.
3.4.3 Substrate, Not Solution
A closing caution, and the handoff to what follows. The temptation, having grasped how capable the foundation model is, is to treat it as the system rather than the substrate — to assume that a sufficiently good model, suitably prompted, simply is an agent, and that several of them in a group chat simply are a multi-agent system. This is the error the whole chapter has been arming you against. The model is a layer. The agency, the memory, the grounding, the reliability and — above all — the coordination are not in it; they are built on it, and building them is the work, the difficulty, and the subject of the rest of this book. An agent is a foundation model plus the apparatus that makes it act: a loop, tools, memory, a goal (Weng, 2023) — what the practitioner literature has taken to calling the harness, the scaffolding of loop, tools, and context management wrapped around the model,1 in a framing — Agent = Model + Harness — that is becoming common shorthand. A multi-agent system is that, several times over, plus everything that happens between. And before wiring several substrates into a group chat, the question this book keeps returning to — and asks of the software-engineering team most of all — is whether you needed more than one substrate at all.
Forget the distinction and you build the most modern version of the oldest mistake in the field: a hollow agent wearing a foundation model — dazzling in a demonstration, baffling in production, and hollow in exactly the place the classical theory was written to fill. Remember it, and you hold the right picture for everything that follows: a remarkable substrate, with a great deal of careful engineering, and older-than-it-looks theory, stacked on top. With the raw material understood, Part II turns to the first thing we build from it — a single agent.
3.5 Summary
- The foundation model is a substrate, not an agent. It is the raw material of cognition — enabling and necessary, but not the finished mind — and confusing the two is the field’s costliest single error.
- At the book’s level of abstraction it is a stateless function from a context of text to a continuation, trained by prediction at a scale that produced an unexpected, and still only partly understood, general competence.
- What it provides is real and new: a compressed, queryable model of human knowledge; fluent language as a universal interface; programmability by description rather than by code; and a genuine, if limited, capacity to reason.
- What it withholds is just as consequential: no memory between calls, no goals or loop of its own, no grounding in or means of acting on the world, and no reliability — fluent by construction, truthful by none.
- The bridge has two halves: the provision fills the hollow agent’s missing core, and every absence is a problem the classical theory already studied. Memory, goals, grounding, trust, and the very existence of other agents are precisely the field’s subject matter — which is why a thirty-year-old body of theory becomes newly applicable, rather than newly redundant, the moment the substrate arrives.
- The substrate is a layer, not a solution. The agency, the reliability, and the coordination must all be built above it — which is the work of the rest of this book.
3.6 Exercises
Exercise 1. A reviewer agent’s model, handed a context x_{1:t} containing a diff, emits a two-token verdict by the sample-and-append loop of Section 3.1: a judgement drawn from “approve” or “revise” with probabilities 0.7 and 0.3, then a severity code — after “approve”, the code is “clean” with probability 0.8 or “minor” with probability 0.2; after “revise”, it is “minor” or “major” with probability 0.5 each. (a) Compute the probability of each of the four complete verdicts. (b) The orchestrator calls the model twice, independently, on the identical context. What is the probability that the two verdicts agree exactly? (c) The orchestrator treats disagreement between the two calls as evidence that the diff is contentious, and escalates it to a human. What is the probability of an escalation on a perfectly ordinary diff? If the harness switches to greedy decoding — always take the most probable token — the two calls agree with certainty; what information, present under sampling, has the escalation signal now lost?
Exercise 2. Statelessness is paid for in tokens. A coder agent’s session runs to k = 40 model calls. On every call the harness sends a system prompt of s = 1500 tokens plus the entire transcript so far; each turn adds u = 300 fresh tokens of instructions and tool results before the call, and the model replies with r = 200 tokens. (a) Write down the number of input tokens the model processes on call t, and derive a closed form for the total input processed over the session. (b) Evaluate that total, and compare it with the number of distinct tokens the session ever produces — the system prompt, every instruction, every reply. (c) The team moves to a provider that keeps the transcript server-side, in the style noted in Section 3.3, so that the harness transmits only each turn’s fresh material. Which of your quantities changes, which does not, and what exactly has moved? (d) The model’s context window holds 20,000 tokens. On which call does the assembled context first exceed it, and what is the harness forced to begin doing from then on — the business of Chapter 7?
Exercise 3. Five successive generations of a model are benchmarked on emitting one exact ten-token API call; a single wrong token and the call fails. Per-token accuracy improves smoothly across the generations: p = 0.60, 0.75, 0.85, 0.93, 0.98. Treat token errors as independent. (a) Compute each generation’s exact-match success m = p^{10}. (b) The team’s dashboard credits a generation with the ability when m \ge 1/2. Which generations qualify? Describe the shape of the two sequences — per-token and exact-match — and say which of them a reader of the dashboard alone would call emergent. (c) Show that exact match on a k-token answer crosses one half at per-token accuracy p^*(k) = 2^{-1/k}, and evaluate this for k = 10 and k = 30. What happens to the apparent point of emergence as answers grow longer? (d) In the light of (a)–(c), state what you would demand of any claimed discontinuity in ability — the dispute recounted in Section 3.1 — before accepting it as a property of the model rather than of the metric.
Exercise 4. A tester agent’s context is assembled from seven documents, and the probability that the model actually uses a fact depends on where the fact sits: for positions 1 through 7, take the probabilities of use to be 0.95, 0.85, 0.70, 0.55, 0.70, 0.85, 0.95, independently across facts. (a) An oblivious assembler places the one fact the task needs uniformly at random; with what probability is it used? (b) A task needs two facts, both essential. Compare an assembler that lands them in positions 3 and 4 with one that deliberately places them at positions 1 and 7: compute both success probabilities and their ratio. (c) The context must carry one critical instruction — “never push to main” — and six reference documents, two of which you expect to matter far more than the rest. Assign the seven slots, justify the assignment with the numbers, and state in one sentence what your answer implies about the advice to just put everything in the context.
Exercise 5. Under an ideal code, a token to which the model assigns probability p costs -\log_2 p bits. (a) Model A assigns the six tokens of a status report the probabilities 1/2, 1/4, 1/8, 1/2, 1/16, 1/4; model B assigns the same six tokens 1/4, 1/8, 1/8, 1/4, 1/32, 1/8. Compute each model’s cost in bits for the whole report, and explain why A predicts better and A compresses better are the same claim. (b) A harness compacts the tester agent’s report — pass or fail for each of 20 test files, one bit each — into a 12-bit memory record, from which a later call must reconstruct the full status vector. Show that at least two distinct reports must share a record, and conclude that no reconstruction scheme can be right about every test for both. (c) Show further that, whatever the scheme, some report comes back with at least three of its twenty statuses wrong. (Count the reports that must share a single record, and compare with the number of status vectors lying within Hamming distance two of one fixed reconstruction.) (d) Every reconstruction is nonetheless a syntactically perfect twenty-status report. Use (b) and (c) to defend, in two or three sentences, the chapter’s claim that hallucination is a direct consequence of what the substrate does rather than a bug awaiting a patch.
Exercise 6. A vendor’s one-page pitch for Athena, “an autonomous staff engineer”, makes five claims: (1) “Athena remembers every design decision your team has ever made, across sessions and staff changes.” (2) “Athena watches your CI overnight and opens a fix before you wake.” (3) “Athena always knows the current release of every library you depend on.” (4) “Athena’s reviews can be trusted: it explains its reasoning step by step before every verdict.” (5) “Athena has confirmed the patch: all tests pass.” For each claim, identify (i) which withheld property of Section 3.3 the claim silently assumes away, (ii) the machinery a harness would have to supply for the claim to be honest, and (iii) the cheapest experiment a sceptical buyer could run against the live product to test it.
Exercise 7 (project). The chapter’s argument can be run in thirty lines. Take this corpus of status reports:
corpus = [
"the tests pass on linux",
"the tests fail on windows",
"the build passes on windows",
"the build fails on macos",
"the reviewer approves the change",
"the reviewer requests a change",
"the tester reruns the tests",
]- Build a bigram model of it: a table mapping each token to the counts of its successors (pad each sentence with start and end markers), a sampler that draws a successor in proportion to its count, and a generator that samples and appends from the start marker until the end marker arrives. Seed your random generator and print eight continuations. (b) Find, among your outputs, a report that appears nowhere in the corpus and is contradicted by it; trace through the bigram table exactly how the model came to assert it, and compute the probability with which it does so. (c) Wire two copies into a “conversation”: each turn generates a sentence starting from the final token of the previous turn’s sentence, falling back to the start marker when that token admits no continuation. Run six turns and list every place in your program where any state about the exchange lives. (d) Audit the toy against the chapter: which of the four absences of Section 3.3 does this model already exhibit, which of the four gifts of Section 3.2 does it lack, and what does that split tell you about which characteristics of the substrate come from scale and which from the objective itself?
Exercise 8 (lab). The brittleness reported in Section 3.3 — the same request, reformatted, yielding materially different answers — is measurable in an afternoon with an API key (Appendix C describes the setup). Choose a small task with a single mechanically checkable answer, write one canonical prompt for it, and produce seven variants that alter only presentation — separators, ordering of the same instructions, headers, whitespace, quotation style — while holding meaning fixed. (a) State your protocol before running anything: what is held fixed across all calls, and why each such variable would otherwise confound the measurement. (b) Run each of the eight prompts ten times at fixed sampling settings, and tabulate per-variant accuracy and within-variant agreement. (c) Compare the spread in accuracy across variants with the disagreement within a variant, and state what each of the two spreads measures. (d) Write the short note you would send to a team whose system prompt “was tuned until it worked”: what does your table say about how durable that tuning is likely to be? Record the model identifier and the date beside the table.
The term is Anthropic’s; see Effective Harnesses for Long-Running Agents, https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents. It has since acquired an engineering literature of its own: Weng (2026) surveys harness design patterns and the emerging practice of treating the harness itself as the thing one optimises, a development Chapter 17 weighs.↩︎