3  Agent Architectures and Reasoning

Agent = Model + Harness, ran the equation Section 2.4 closed on: the foundation model supplies the raw material, and the apparatus around it makes that material act. Part II is about the apparatus, this chapter about its shape — a problem with three decades of answers behind it, worked out largely by researchers who, possessing no substrate worth the name, drew up blueprints for a harness while waiting for an engine. Russell and Norvig’s tidy formula that an agent is its architecture plus its program (2021) lines up with the equation almost too neatly: the model supplies the program, the harness the architecture — agent architecture, a 1980s term of art, being the older and proper name for the harness.

The families arrange themselves along a single axis — how much the agent thinks between perceiving and acting — and the chapter’s payoff is that the modern observe–reason–act loop is a special case: deliberation run closed-loop. And the two absences the substrate is most embarrassed by (Section 2.3) — no goals, no loop — are exactly what an architecture exists to supply. This chapter is where the idle substrate of Chapter 2 acquires the will to keep going; the second movement then prises open the reasoning step itself: practical reasoning and planning, reflection, and the metacognitive question of when thinking is worth its price.

3.1 What an Architecture Is, and Why a Substrate Needs One

In the agents literature an architecture is more than a structure one could photograph: it is a methodology for decomposition. It fixes what modules an agent is built from — perception, a world model, a planner, behaviours — and, more consequentially, how they are wired together and which gets to decide what the agent does next. That arrangement is the part of an agent its designer commits to early and revises late, because everything else hangs from it.

The architecture–program division earns its keep because the halves have different lifespans. The program — the function from what the agent has perceived to what it does next — is content, supplied afresh on every call by the model and its prompt. The architecture is structure — fixed for the duration, and a good one survives the replacement of nearly everything inside it: you can change the model beneath it — a new release, a different vendor — without redesigning the agent. The harness outlives the model it was built around.

Separate the architecture, too, from the behaviour it happens to produce. Our running coder agent — dissected in Section 3.6 — has one architecture, a loop that reads an issue, proposes a change, edits, tests, and inspects; the frame does not change when the task does. Mistaking the two — taking a clever transcript on one task for a sound architecture — is precisely how a demonstration that dazzles on Tuesday keels over in production on Wednesday.

Why a substrate needs one brings us back to the two absences. The model has no goal and no loop: handed a context it returns a continuation and falls silent. An architecture is exactly the apparatus that makes good both deficiencies — the loop that calls the model again, the state in which a goal can persist, the circuit from decision to action to the following perception. The model judges what a good next move would be; the architecture strings such judgements into something that pursues an end.

That leaves the question every architecture must answer, and the one along which the classical families divide: how much should an agent think between perceiving and acting? The field’s canonical survey lays out the first three families just so (Wooldridge & Jennings, 1995) — reactive at the think-as-little-as-possible end, deliberative at the other, hybrid seeking a principled compromise between them — and to these the book adds a fourth, promoted from the deliberative ranks: the Belief–Desire–Intention model, which furnishes that compromise with something like a theory of mind. The next four sections take them in order — not a museum tour but four settings of a single dial, one that every agent you build has been turned to, whether or not its designer noticed.

3.2 Reactive Architectures: Intelligence Without Deliberation

By the mid-1980s the deliberative orthodoxy’s robots spent so long reasoning over their internal models of the world that the world had moved on by the time they had worked out what to do. Rodney Brooks answered with a heresy whose paper title was the manifesto — Intelligence without representation (1991): the explicit internal model was not the foundation of intelligent behaviour but a costly detour. “The world is its own best model”, ran the slogan that became the movement’s banner (1990); intelligence emerges from the tight coupling of perception and action in a real environment.

Brooks’s constructive answer was the subsumption architecture (1986): not a sense–model–plan–act pipeline but a stack of behaviours, each a near-direct wiring of perception to action, each competent at some simple thing — avoid obstacles, wander, head towards the light. The layers run concurrently; a higher layer subsumes a lower one when circumstances warrant, and nowhere is a single coherent picture of the situation kept, because there is none. Brooks’s robots walked, scuttled, and avoided collisions robustly and in real time, with not a symbol in sight.

The arrangement buys speed, robustness, and graceful degradation — no internal model to drift out of register with reality, nothing to be catastrophically wrong about. The price of carrying no model is carrying no foresight: a purely reactive agent cannot pursue a goal that requires a considered sequence of steps no single moment triggers, cannot explain itself, and proves deceptively hard to engineer for sophisticated behaviour as competences multiply (Wooldridge, 2009). Reactivity scales beautifully with the speed of a task and badly with its depth.

The modern echo is closer than it first appears. A foundation model handed an input and a set of tools, emitting a single action and then falling silent — no loop, no plan, no state carried to the next call — is what Russell and Norvig call a simple reflex agent (2021), the reflex now answered by a substrate which has, in effect, read the manual for almost everything. Routers, guardrails, one-shot function calls: reactive architectures all, and nothing primitive about choosing one. In Brooks’s own field the modern reflex is grander still: the learned visuomotor policy, pixels to motor commands at control rate, is subsumption with the reflex trained rather than written (Section 4.6 returns to it). But a reflex keeps nothing and foresees nothing, however well-read it has become — swift on the routine, helpless the instant a task demands more than one informed reflex can supply. Where a task is fast, bounded, and forgiving, a reflex is the cheapest tool that works; wired straight to an irreversible action — sending the email, merging the branch, spending the money — it is a loaded foot-gun, and we return in Chapter 4 to what it means to let an agent do something it cannot take back.

3.3 Deliberative Architectures: The Sense–Plan–Act Pipeline

If the reactive tradition was a revolt against the internal model, the deliberative tradition is the orthodoxy it revolted against. Its intellectual charter is the physical symbol system hypothesis, Section 1.7’s founding bet (1976): intelligence simply is the manipulation of representations, so where the reactive agent looks again, the deliberative agent looks inward, at a picture it carries and curates.

The canonical embodiment is the sense–plan–act cycle: update a symbolic model from perception, search it for a sequence that reaches the goal, then execute a step at a time. Shakey, the SRI robot, ran exactly this loop, and its planner — Fikes and Nilsson’s STRIPS (1971) — set the field’s template for decades: represent the world as logical predicates, each action as the conditions it requires and the facts it adds and deletes, and reduce “deciding what to do” to a search for a path from start to goal. SOAR later generalised the instinct into a complete account of cognition (Laird et al., 1987). What this buys is everything reactivity could not reach: foresight, and transparency — the plan is an explicit object you can inspect, verify, and interrogate.

The bill comes in three parts: the notorious frame problem — to reason about change in logic, an agent must specify not only what each action alters but the immense quantity it leaves untouched, computationally crushing and philosophically slippery alike (McCarthy & Hayes, 1969); the cost of planning itself — search is combinatorial and slow, and a plan may be obsolete by the time it is ready; and, most fatal in practice, the knowledge bottleneck — the whole edifice rests on a complete, correct, hand-authored symbolic model, and the instant reality strays beyond it the agent is not merely mistaken but blind. This is the hollow agent of Section 1.7, viewed from the side of its architecture: the deliberative tradition designed the cockpit with great care and then waited fifty years for an engine.

And it is the architecture the substrate most directly revives. A foundation model is precisely the queryable world model that sense–plan–act always had to be handed (Section 2.2); and chain-of-thought reasoning (Wei et al., 2022) is its planning step: the plan-then-execute agents of the present day are the pipeline reborn, with a foundation model in the planner’s chair. The substrate dissolves the knowledge bottleneck and takes much of the sting from the frame problem, since natural-language reasoning bends where formal logic snapped. But the older difficulties return in new clothes: deliberation is still slow, now metered in tokens and latency; the plan can be confidently, fluently wrong; and the world still moves while the model thinks. When deliberation is worth its price is the question of Section 3.13.

3.4 Hybrid Architectures: Layering Reflex and Deliberation

Both camps were right about themselves and wrong about each other: a useful agent needs the reflex’s speed for the routine and the deliberator’s foresight for the considered. The hybrid, or layered, architecture builds both into a single agent; Ferguson’s TouringMachines opened with exactly this diagnosis (1992).

Granting both layers, how are they arranged, and which gets to move the actuators? Wooldridge sorts the answers into two styles (2009). In horizontal layering, as in TouringMachines, every layer runs from perception to action in parallel, with a control framework left to referee — flexible, but only as good as a referee who must reconcile a reflex shouting “swerve” with a planner murmuring “stay the course”. In vertical layering, as in Müller’s InteRRaP (1996), the layers are stacked and control passes upward only as needed, from behaviours through local planning to cooperative planning — a top floor that quietly anticipates Part III, a stratification recurring in mobile robotics’ three-layer architectures (Gat, 1998). The genuinely hard part is not having two layers but adjudicating between them; hybrids work, but they are fiddly, tuned by hand and by experience.

Underneath lies an idea the modern field leans on heavily: intelligence wants two processes running at different tempos. Kahneman gave the popular vocabulary (2011) — System 1, fast, automatic, and intuitive; System 2, slow, effortful, and deliberative — and the layered architectures are the two systems cast in software, their arbitration problem exactly the question of when to spend the expensive slow one. The present moment is busily rediscovering them: the “reasoning” models that decide how long to think, the routers that split trivial requests from thorny ones, the coder agent that applies a one-line fix on reflex but stops to plan a multi-file refactor — all are layered architectures, re-deriving horizontal and vertical control as routing and escalation. The model families ship the layers ready-made — beside each flagship, smaller siblings often distilled from it (Hinton et al., 2015), as Google reports of Gemini’s Flash (2024): Kahneman’s two systems from two models, each billed accordingly; robotics runs the same split at steeper odds (Section 4.6). The substrate makes both layers dramatically better; it does not make the arbitration go away. It merely moves it into prompt logic and control flow — spend the slow system too freely and the agent is sluggish and expensive, too sparingly and it blunders, confidently, through problems that wanted a moment’s thought. That decision closes this chapter (Section 3.13); the hybrids’ lesson meanwhile is that the best architectures were a managed conversation between reflex and deliberation, and the managing was always the hard part.

3.5 The Belief–Desire–Intention Model

The richest of the classical architectures began in philosophy: Bratman’s account of how a creature with finite time and attention manages to act sensibly at all (1987). We cannot reconsider every option before every action, so intentions do the rationing: once settled, an intention persists, filtering the options you bother to consider thereafter, so that you are not obliged to re-derive your whole life each morning before breakfast. An intention is not a desire with the volume turned up; it is a distinct mental state whose job is commitment.

The architecture that bears its name gives an agent three attitudes — beliefs about how the world is, desires it would like to bring about, and intentions, the desires it has committed to pursuing — and runs a deliberation cycle over them: revise beliefs against new perception, promote options to intentions, plan from means to ends, act, repeat. This is the machinery behind the agent-oriented programming of Section 1.7.

The load-bearing idea, distinguishing BDI from a mere plan-then-execute loop, is the persistence of intentions. Cohen and Levesque’s phrase doubles as a definition: intention is choice with commitment (1990). Reconsider at every step and the agent spends everything deliberating; never reconsider and it charges ahead with a plan the world has invalidated — the over-cautious clerk and the fanatic, and neither is who you want flying the aeroplane. What separates them is a commitment strategy: hold an intention blindly, until the goal is believed achieved; single-mindedly, until achieved or impossible; or open-mindedly, while still believed worth pursuing. The right setting of that dial depends entirely on how fast the agent’s world moves.

None of this stayed on the page: Georgeff and Lansky’s Procedural Reasoning System put BDI to work on real-time spacecraft fault monitoring (1987), and, formalised by Rao and Georgeff (1991, 1995) and recast as AgentSpeak (1996), it survives in the Jason interpreter (Bordini et al., 2007) — a complete road from a theory of mind to a language you can run.

Now the payoff: BDI is the architecture a foundation-model agent most nearly is. The model is a superb engine for belief revision, option generation, and the means–ends reasoning that connects them (Section 3.8’s business); what it cannot supply is intention — stateless and goalless, it holds no commitment that outlives a single call. The intention lives in the harness: the task held in state, the goal carried across calls, the loop that returns to the model until the goal is met or abandoned. Read through the substrate, BDI delivers its sharpest one-line lesson: the model is the faculty of deliberation, and the harness is the will. Their question thereby returns as every agent builder’s decision — how doggedly to hold the plan before reconsidering it against the latest tool result (Section 3.13 takes it up) — and in Chapter 7 intention returns in the plural, as the joint commitments that let several agents hold to a shared plan without anyone in charge, the classical era’s most valuable bequest to the multi-agent half of the field.

Table 3.1: The four classical families as four settings of a single dial — how much the agent thinks between perceiving and acting — beside what each setting buys and costs. The classical embodiments and their modern echoes follow in Table 3.2.
Family Thinking between perceiving and acting What it buys What it costs
Reactive As little as possible — perception wired straight to action Speed, robustness, graceful degradation, situatedness No foresight or multi-step goals; cannot explain itself; hard to engineer at scale
Deliberative A great deal — model the world, plan, then act Foresight; goal-directed, inspectable plans Frame problem; slow combinatorial search; a hand-authored model as knowledge bottleneck
Hybrid (layered) Both tempos — reflex for the routine, deliberation for the considered Speed and foresight in one agent Arbitration between the layers; fiddly, hand-tuned, conceptually untidy
BDI A deliberation cycle rationed by a commitment strategy A theory of mind for bounded rationality, with a working toolchain Tuning the commitment dial between the over-cautious clerk and the fanatic
Table 3.2: The concordance: each family’s classical embodiment beside its substrate-era echo. Read the final column down the page and the chapter’s argument appears in miniature — every family the field settled on decades ago has returned, better filled.
Family Classical embodiment Modern echo
Reactive Brooks’s subsumption architecture and its walking robots The single tool-calling turn; routers, guardrails, one-shot function calls
Deliberative Shakey; STRIPS; SOAR Plan-then-execute agents, with chain-of-thought as the planning step
Hybrid (layered) TouringMachines (horizontal); InteRRaP (vertical); three-layer robotics architectures Routers and escalation; reasoning models; model tiers as System 1 / System 2
BDI Procedural Reasoning System; AgentSpeak; the Jason interpreter The model supplies deliberation; the harness supplies the intention — the will

3.6 The Modern Agent Loop as a Special Case

Strip a contemporary agent down to its skeleton and what remains is a loop. The model observes — reading the context it has been handed: the task, the history, the result of whatever it last did. It reasons — a line of thought ending in a choice of what to do next. It acts — or rather the machinery around it acts on its behalf — and the outcome folds back into the context for the next turn. This interleaving has a name, ReAct (Yao, Zhao, et al., 2023), and the loop it describes is the beating heart of every agent a framework will build for you; Lilian Weng’s much-shared anatomy draws the same skeleton from another angle (2023). Held against the catalogue we have assembled (Table 3.1), its novelty quietly drains away. In kind it is deliberative — the model reasons before each action — but not the open-loop deliberation of Section 3.3: ReAct takes one action and re-perceives before it reasons again. The modern loop is best read as sense–plan–act run closed-loop — deliberation chastened by reactivity, the reactive tradition’s hard-won lesson absorbed — and what is genuinely novel is none of the shape but the filling: for the first time the deliberating core is a general, competent reasoner, the deliberative architecture supplied at last with something to deliberate with.

3.6.1 The Harness Is the Architecture

Look at what does the acting: the chapter’s central identity snaps into focus. Everything in that loop other than the model — the code that calls it, parses the chosen action, dispatches the tool, catches the error, appends the result, decides whether to iterate or stop — is the harness, and the harness is nothing but this chapter’s “architecture” rendered in Python. When practitioners speak of loop engineering and harness engineering, they are doing agent architecture under another name — the very decisions Brooks, Ferguson, and Rao made before them, now taken in code, usually without the knowledge that they are old. One mid-2026 summer gave the two crafts canonical statements six weeks apart — loop engineering for designing the system that prompts an agent (Osmani, 2026), graph engineering for drawing the control flow an agent system must follow (Runkle & Chase, 2026) — the second delivering the verdict on its own title that this chapter would otherwise have had to: not a new idea, merely “the latest name for a well established approach”.

3.6.2 Two Knobs, Not One

The design choice has two independent knobs: how much the agent deliberates before it acts, and how far ahead it commits before stopping to perceive again. The classical families set both together — pure reaction turns both down, open-loop sense–plan–act both up — but nothing couples them, and the classical field knew it. The high-deliberation, short-commitment corner is exactly where Georgeff and Lansky’s Procedural Reasoning System sat, its founding paper pointedly titled Reactive Reasoning and Planning (1987); any BDI agent whose commitment strategy (Section 3.5) is the second knob sits there too; so does ReAct, beside model predictive control, which optimises over a finite prediction horizon and applies only the first move before re-measuring (Rawlings et al., 2017) — ReAct is, in this light, model predictive control with a foundation model standing in for both model and optimiser. What ReAct adds is not the closed loop; it is a deliberator at last worth running inside one.

%%{init: {"theme": "base", "themeVariables": {"quadrant1Fill": "#FAF7F0", "quadrant2Fill": "#F3EFE5", "quadrant3Fill": "#FAF7F0", "quadrant4Fill": "#F3EFE5", "quadrantPointFill": "#1E2EB8", "quadrantPointTextFill": "#16171B", "quadrantXAxisTextFill": "#16171B", "quadrantYAxisTextFill": "#16171B", "quadrantTitleFill": "#16171B", "quadrantInternalBorderStrokeFill": "#766F65", "quadrantExternalBorderStrokeFill": "#766F65"}}}%%
quadrantChart
    title The two knobs of the agent loop
    x-axis Little deliberation --> Much deliberation
    y-axis Short commitment --> Long commitment
    quadrant-1 Open-loop sense-plan-act
    quadrant-2 Commit without deliberating
    quadrant-3 Pure reaction
    quadrant-4 Closed-loop deliberation
    Subsumption: [0.16, 0.18]
    Shakey: [0.84, 0.86]
    ReAct: [0.92, 0.10]
    MPC: [0.80, 0.20]
    BDI and PRS: [0.66, 0.30]
Figure 3.1: The chapter’s single dial resolved into two independent knobs — how hard an agent thinks before it acts, and how far it commits before pausing to perceive again. The classical families sit on the diagonal, while the high-deliberation, short-commitment corner is the one that reactive-reasoning planners, BDI agents, model predictive control, and ReAct all reached independently — and the commit-without-deliberating corner no classical agent architecture occupied: fixed automation lives there, as Exercise 1 finds. Positions are schematic.

Figure 3.1 draws the map. There is no universally correct setting of these dials, only one appropriate to a task’s tempo, its reversibility, and its appetite for foresight; choosing it deliberately, rather than inheriting a framework’s default, is the architectural judgement this chapter exists to sharpen.

3.6.3 The Coder Agent, Anatomised

Our coder agent is a ReAct loop: its beliefs the context assembled each turn — the issue, the open files, the last command’s output; its deliberation the model’s reasoning about what to change and why; its action a single tool call; its intention the task held in state with the standing instruction to keep going until the tests pass. Its commitment strategy (Section 3.5) shows up in one concrete choice — when a test fails, does it reconsider its approach or retry the same edit? — and setting that dial badly reproduces BDI’s two failure modes: abandoning a sound plan at the first red test, or retrying a broken edit until the budget is exhausted.

3.6.4 Knowing When to Stop

The loop that confers agency is also the part most apt to misbehave. A foundation-model loop spends money and time on every pass, and a loop that does not know when to halt is not persistent but pathological. Termination is not a tidying-up detail but load-bearing architecture: a goal test, a step budget, a ceiling on cost — built from scratch in Chapter 17, made dependable in Chapter 20. The modern agent, then, is the architectural tradition’s most fortunate instance; the rest of the chapter prises open the faculty the loop exercises every turn.

“We’ll come to item one just as soon as we’ve agreed how long it deserves.”

3.7 The Reasoning Step

At the heart of the loop assembled in Section 3.6 sits the step left unopened: the model reasons. The second movement prises it open: what it means for an agent to reason and plan, what the model actually does in their place, and how an agent might know when its thinking is needed, enough, and trustworthy. Three words mark the three layers. Reasoning is the drawing of conclusions — here, conclusions about what to do. Planning is reasoning made systematic, a course of action constructed before it is taken. Metacognition is reasoning turned upon itself.

A caution at the outset: Chapter 2 allowed that the substrate can reason “after a fashion”, and postponed how much is reasoning and how much fluent performance. The postponement ends here, in an unromantic verdict: a real and powerful capacity, and an unreliable one — a tool, not a window onto a mind.

3.8 Practical Reasoning: Thinking Directed at Action

Begin with a distinction old enough to have been made in Greek. Theoretical reasoning is directed at what is the case: from things believed to some further thing to believe. Practical reasoning is directed at what to do: from what an agent wants and believes to an action (Wooldridge, 2009). Very nearly everything a foundation-model agent does when it “thinks” is practical reasoning, and to forget it is to grade an agent’s reasoning as though it were sitting an examination, when the only question that matters is whether the steps led to a sensible act.

The idea is Aristotle’s: the conclusion of a practical syllogism is not a proposition but a deed.1 A desired end furnishes the major premise — “the service must stay available”; a belief the minor — “restarting the crashed process will restore it”; and the conclusion is the restart itself. In Wooldridge’s standard account, practical reasoning is two activities (2009): deliberation, which settles which end to pursue, and means–ends reasoning, which settles how to bring it about — the pair of Section 3.5, deliberation yielding intentions and means–ends reasoning the plans that serve them.

Practical reasoning also runs against the clock. It is bounded rationality seen from the reasoning side — Bratman’s point that intentions exist precisely to spare an agent the expense of reasoning everything out afresh (1987) — so it is not the search for the optimal action but the production of a good-enough one within a budget, and the size of that budget is itself a decision, the one towards which this whole chapter is bent (Section 3.13).

With the frame in place, the loop’s reason step stops looking unprecedented. “The tests fail because the parser rejects empty input, so I will add a guard for the empty case”: a goal, a belief, a concluded action — a practical syllogism conducted in natural language, both halves folded into a single breath. What the substrate changed is not the structure but the reach: the classical practical reasoner could draw only the conclusions its designer had supplied the premises to draw, while the foundation model arrives with its own vast stock of premises and will produce a plausible practical syllogism about very nearly anything. That is the gift — and the catch, since a plausible practical syllogism is not always a sound one. Means–ends reasoning pursued systematically, with guarantees that it will not mislead, was the great achievement of classical AI under another name — planning — and it is where we turn next.

3.9 Classical Planning: Reasoning as Search

Means–ends reasoning, the second half of Section 3.8, is the part classical AI managed to make a science of. Done exhaustively, and with guarantees, it becomes planning: the construction, in advance and on paper, of a sequence of actions carrying the world from an initial state to a goal. So posed, deciding what to do becomes a problem of search through the space of states the world might occupy; the standard reference is Ghallab, Nau, and Traverso (2004).

Everything turns on the representation. The classical move, due to Fikes and Nilsson’s STRIPS (1971), is to describe the world as a set of facts and each action as a small bundle of logic: the preconditions that must hold for it to apply, and the facts it adds and deletes when it does. Restart-service requires the service to be crashed and makes it running; nothing else is disturbed. From this austere description a planner can work out, without acting, what any sequence of actions would do. The representation was standardised in the 1990s as PDDL, the Planning Domain Definition Language (McDermott et al., 1998) — the form in which a language model can today be asked to write its plans (Section 3.11).

The state space grows exponentially in the number of facts, so the whole art lies in not searching most of it. A planner can decline to fix the order of two actions until something forces the choice — partial-order planning — or expand an abstract task into subtasks and primitives in the manner of a hierarchical task network: exactly the task decomposition a modern agent performs for its subagents (Section 3.11). What turned planning into something that disposes of problems with millions of states was a better compass: the domain-independent heuristic, an estimate of how far a state lies from the goal, computed automatically by solving a relaxed version of the problem that politely ignores the inconvenient parts, such as the actions’ delete effects. Planning as heuristic search (Bonet & Geffner, 2001) is the banner of the modern field.

All this formality buys something a reflex and a fluent guess can never offer: a guarantee. A planner worth the name is sound — the plan it returns really does achieve the goal, in the world as modelled — and complete — if a plan exists, it will find one; and many are optimal into the bargain. Where a language model offers a plan that looks right, a classical planner offers one that is right.

The certainty, though, arrives with an asterisk the size of Section 3.3. Every guarantee is conditional on the model, and the planner knows precisely nothing the modeller did not encode: omit an effect, or meet a situation the model never anticipated, and the proof now guarantees a plan for a world that is not the real one. Authoring that model for an open and shifting domain is the labour that kept classical planning largely indoors — the knowledge bottleneck in its purest form, and the precise opening the foundation model strides into, offering the world knowledge the planner always had to be handed, and charging for it in exactly the coin of guarantees. A model’s planning is only its reasoning put to a use; before weighing the one we must reckon with the other.

3.10 Reasoning in Language Models

The founding observation is almost embarrassingly simple. Ask a capable model for an answer and it will give you one; ask it instead to work the problem through step by step, and it will often give you a better one. The intermediate steps — the chain of thought (Wei et al., 2022) we first met in Section 2.2 — lift accuracy markedly on arithmetic, logic, and multi-step word problems, with no change to the model’s weights.

A family of techniques followed, united by one idea — spend more computation before committing to an answer — and distinguished by what they do with the spend (Table 3.3). Self-consistency samples many chains and returns the answer most of them agree on (Wang et al., 2022) — ensembling by another name, inheriting both its gains and, as Chapter 10 will show, its hazards. Tree of thoughts searches a branching space of partial thoughts (Yao, Yu, et al., 2023) — planning-as-search come back in language. And the reasoning models use reinforcement learning to train the long deliberative trace in as a habit rather than a prompted trick (DeepSeek-AI, 2025).

Table 3.3: Four ways to spend more computation before answering — from a prompted phrase, through ensembling and search, to a habit trained into the weights. Each is an older idea returned: self-consistency is ensembling, tree of thoughts is planning-as-search.
Technique What the extra compute buys How it is obtained Reference
Chain of thought A single step-by-step trace before the answer Prompted — the instruction to think aloud Wei et al. (2022)
Self-consistency Many sampled chains; the answer most of them agree on Prompting plus sampling Wang et al. (2022)
Tree of thoughts A search over a branching space of partial thoughts — generate, score, backtrack Prompting plus search Yao, Yu, et al. (2023)
Reasoning models A long, extended deliberative trace before the answer Reinforcement learning — baked in, not prompted DeepSeek-AI (2025)

All of which sharpens the question Chapter 2 deferred: is any of this reasoning, or only a convincing impression of it — a model trained to continue text, continuing a derivation in the form it was shown? Intuition will not adjudicate; only evidence will, and the evidence is sobering on two fronts.

The first concerns faithfulness — whether the chain a model produces is the actual cause of its answer or a story told about it afterwards. Turpin and colleagues demonstrated it uncomfortably (2023): plant a subtle bias in the prompt — say, the correct-looking answer is always option A — and the model’s answers drift to follow the bias, while its chain of thought confidently justifies them without once mentioning the thing that moved it: a rationalisation, not a derivation. Perturb or truncate the chain and the answer often stands unmoved, and the measured faithfulness tends to fall as models grow more capable (Lanham et al., 2023). This ought to trouble anyone tempted to read an agent’s chain of thought as an audit trail of why it did what it did — a temptation Chapter 22 returns to — because a chain that did not cause the answer cannot explain it either.

The second concerns robustness. A genuine reasoner is indifferent to the names of the variables. On GSM-Symbolic (Mirzadeh et al., 2024), a grade-school maths benchmark whose numbers and surface details can be regenerated at will, accuracy slips when only the numbers are changed and slips further when a plausible but irrelevant clause is added — across models, by margins too large to wave away. The competence is real but fragile in a way reasoning from structure rather than surface would not be: the brittleness of Section 2.3, surfacing in the very faculty this chapter is about.

The useful verdict declines both temptations. To call language-model reasoning a mere trick is to ignore real and often large gains; to call it reasoning in the full sense is to ignore that the chain may not be the cause and the capacity may not survive a renamed variable. It is a powerful and unreliable instrument, best used as one: elicited deliberately, and — the recurring moral of this book — checked against something outside the model rather than believed because it reads well. Asking the model to plan — to commit in advance to a course it cannot yet test — inherits every doubt raised here, and adds a few of its own.

3.11 Planning with a Language Model

Take the means–ends reasoning of Section 3.9, hand it to a foundation model, and something remarkable and something alarming happen at once. The remarkable thing is that the model will plan for almost anything — deploy the service, reproduce the bug — fluently, from its vast store of how-things-are-usually-done. The knowledge bottleneck appears, at a stroke, to dissolve: nobody writes the domain model, because the model already knows, more or less, how the world works. The alarming thing Section 3.10 prepared us for: a plan that reads well is not a plan that works, and the model offers nothing, on its own, to tell the two apart.

The single most useful thing a model does as a planner is to decompose a goal too large to take in at once into an ordered set of smaller ones. Least-to-most prompting reduces a hard problem to a sequence of easier subproblems and works upward (Zhou et al., 2022); the plan-and-execute pattern that recurs through the agent frameworks does the same at the level of actions. This is the hierarchical task network of Section 3.9 with the hand-built methods struck out — the model supplies, from training, the decompositions a classical system had to be handed by an expert. Our orchestrator earns its keep exactly here, carving “implement the new export format” into write-the-serialiser, add-the-tests, and update-the-docs, and parcelling them out to its coder agents. Decomposition is the part of language-model planning that most deserves the name.

It is the next part that does not. Set to plan on the sort of well-defined benchmark a classical planner eats for breakfast, the models of the first chain-of-thought era fell embarrassingly short — the best solved about a third of the instances, and almost none once the domain’s familiar names were swapped for nonsense words (Valmeekam et al., 2023). Reasoning models have since lifted the raw scores considerably (Valmeekam et al., 2024); what has not moved is the residue — performance that sags under re-naming, and a correct plan guaranteed at no price. Worse, the model cannot reliably notice: self-verification is heir to every unreliability of Section 3.10. The model plans as it reasons — plausibly, fluently, and without guarantee.

The productive response is to stop asking it to plan alone; Figure 3.2 draws the two wirings that restore the guarantees. In one, the model is a translator, rendering the messy natural-language problem in PDDL for a sound classical planner to solve, guarantees intact (Liu et al., 2023). In the other — Kambhampati and colleagues’ LLM-Modulo framework (2024) — the model generates candidate plans and a sound external verifier accepts or rejects them in a generate-and-test loop. Their slogan is the honest summary: language models cannot plan on their own, but they can be an enormous help to planning. And the cheapest verifier of all is frequently the world itself — run the plan and see what breaks, the closed loop of Section 3.6 put to work.

%%{init: {"theme": "base", "themeVariables": {"primaryColor": "#E8ECFF", "primaryBorderColor": "#4054B2", "primaryTextColor": "#16171B", "lineColor": "#3B4351", "edgeLabelBackground": "#FAF7F0", "clusterBkg": "#EFE9DC", "clusterBorder": "#766F65"}}}%%
flowchart TB
    %% mermaid lays disconnected subgraphs out right-to-left: B is declared first so that A renders as the left panel
    subgraph B["LLM-Modulo loop"]
        direction TB
        PB["Natural-language<br/>problem"]
        PB --> B1["Model proposes a<br/>candidate plan"]
        B1 --> B2(["Sound external<br/>verifier"])
        B2 -.->|"reject, revise"| B1
        B2 -->|"accept"| B3["Verified plan"]
    end
    subgraph A["Model as translator"]
        direction TB
        PA["Natural-language<br/>problem"]
        PA --> A1["Model renders the<br/>problem in PDDL"]
        A1 --> A2["Sound classical<br/>planner solves it"]
        A2 --> A3["Plan, guarantees<br/>intact"]
    end
    classDef good fill:#DCEFE2,stroke:#1B6B5A,color:#16171B
    classDef world fill:#EFE9DC,stroke:#766F65,color:#16171B
    class A2,B2 good
    class PA,PB world
Figure 3.2: Two wirings that restore to a language model the guarantees it cannot supply alone. In series the model renders the problem in PDDL and a sound classical planner discharges it, proofs intact; in an LLM-Modulo loop the model proposes plan after plan while an external verifier turns each away until one survives. The sound component — shaded green — is what both topologies share: world knowledge from the model, certainty from elsewhere.

The coder agent shows the spectrum in miniature. When it resolves to “fix the parser, then mend the failing tests”, that plan is a fluent guess, made safe to act on not by the model’s confidence but by the test suite: the plan is checked by being run, and a red test is the verifier’s veto. The orchestrator’s decomposition is a bet; the tester settles it; and the agent systems that actually hold together wrap fluent model planning in cheap, external, automatic checks — tests, type-checkers, schema validators — rather than trust a plan because it was confidently produced.

Planning with a language model, then, inverts the classical bargain rather than escaping it: guarantees at the price of a hand-built model become a free model with the guarantees withheld, and the competent agent draws each half from wherever it is cheap. But verification only tells you that a plan has failed. To do better, the agent must take the failure and reason about what to change: reflection, and whether it genuinely helps, or merely dresses a mistake in fresh confidence, is the question we turn to next.

3.12 Reflection and Self-Critique

An agent holds a verdict — the plan failed, the test went red — and has done nothing about it yet. Reflection is the obvious next move: have the agent examine its own work and revise — a loop of maker and critic, run until the critic runs out of complaints. Two methods mark out the territory. Self-Refine keeps the whole apparatus inside a single model — generate, self-critique, revise, iterate — reporting polished results preferred to first drafts across a spread of tasks (Madaan et al., 2023). Reflexion adds a memory and a failure signal: an agent that fails a task writes itself a verbal post-mortem, stores it, and consults it on the next attempt, improving across trials by what its authors aptly call verbal reinforcement learning (Shinn et al., 2023).

Whether this is genuine self-improvement turns on what the reflection is reflecting on. If the feedback is grounded, tied to something outside the model’s own judgement (a failed unit test, a compiler error, a raised exception), reflection has real purchase. If the feedback is merely the model’s own unaided verdict on its own work, it stands on far shakier ground: passing judgement is itself reasoning, and Section 3.10 gave us no reason to expect a model to grade its answer any better than it produced it.

The evidence bears the distinction out, and sharply. Huang and colleagues examined intrinsic self-correction — a model revising its reasoning with no external feedback at all — and found that it does not reliably help and frequently hurts, the model talking itself out of correct answers about as often as into them (2023). Several reported triumphs turn out, on inspection, to have quietly handed the model an oracle that told it when to stop; withdraw it and the gains largely evaporate. Reflection is amplification, not generation: it can make excellent use of a true error signal, and it cannot conjure one out of the model’s own confidence — ungrounded, it merely launders a mistake into a more confident version of itself.

The coder agent shows the difference. When the tester returns a failing case and the coder reflects — “the loop overruns the array by one; I will tighten the bound” — the reflection is anchored to a real, externally supplied failure, and it earns its keep. When the same coder, having run nothing, merely reassures itself that its code is correct, the reflection is worth approximately nothing. This is why the running example carries a tester and a reviewer: they supply the signal that makes the coder’s reflection more than a comforting noise. A reviewer agent is precisely an externalised critic — better than self-critique to exactly the degree that it is genuinely independent of the agent it reviews, a property easy to lose and the subject of Chapter 10 and Chapter 11.

But notice what even grounded reflection asks of an agent: to decide whether its work is good enough, whether to revise or stop, whether to trust the signal. That — an agent reasoning about the quality and sufficiency of its own reasoning — is metacognition, the faculty this whole chapter has been circling. It is where we end.

3.13 Metacognition: Knowing When to Think

Is my work good enough? Should I think more, or stop? Can I trust this result? Such judgements — an agent reasoning about its own reasoning — are the province of metacognition, the word John Flavell gave the study of thinking about thinking (1979). It decides when each faculty is needed and whether to believe what it returns; every earlier section ended by deferring a decision to it.

3.13.1 Knowing What One Knows

Metacognition has two halves, and the first is self-assessment: knowing what one knows and, far more usefully, what one does not. This is the honest twin of hallucination. Kadavath and colleagues found that large models carry a surprising amount of self-knowledge: their stated confidence tracks their accuracy, and they can often predict whether they will answer a question correctly (2022). But the self-knowledge is fragile — it depends on how the question is put, and instruction-tuning seems to make a model more confident than it has any right to be. The agents we have mostly bluff; calibration is the precondition for everything else metacognitive, for thought cannot be allocated wisely by something that cannot tell a hard problem from an easy one.

What an agent needs has a name outside the language-model world: uncertainty quantification, the discipline of attaching honest error bars to a model’s predictions — at its centre the distinction calibration chases, between aleatoric uncertainty, the world’s irreducible noise, and epistemic uncertainty, the reducible part that is the model’s own ignorance. The classical toolbox is substantial — Bayesian deep learning (Gal & Ghahramani, 2016), evidential deep learning (Sensoy et al., 2018) — and almost none of it transfers cleanly to a frontier model, because the uncertainty that matters to an agent is not the wobble of a number but doubt about a meaning. The most promising methods therefore work at the level of meaning: semantic entropy samples several answers, clusters them by what they actually assert, and reads wide disagreement among the clusters as the model’s own signal that it is guessing (Farquhar et al., 2024). It is genuine progress, and a reminder (Table 3.4): the principled apparatus for knowing what one does not know was built for a different kind of model, and is having to be reinvented, still incompletely, for this one.

Table 3.4: Three ways to attach honest error bars to a model’s predictions. The classical apparatus needs access to weights that a closed frontier API withholds, so uncertainty is now estimated at the level of meaning, from sampled answers alone — the ✗ ✗ ✓ of the third column.
Method How it estimates uncertainty Usable through a closed frontier API? Reference
Bayesian deep learning Carries a distribution over the network’s weights; reads uncertainty off the spread of the predictions they produce ✗ Gal & Ghahramani (2016)
Evidential deep learning Trains a single network to output the parameters of an evidential distribution in one pass ✗ Sensoy et al. (2018)
Semantic entropy Samples several answers, clusters them by what they actually assert, treats wide disagreement among the clusters as a signal of guessing ✓ Farquhar et al. (2024)

3.13.2 Knowing How Much to Think

The second half is allocation: how much to think before acting — the decision the first movement kept deferring, and lately a real engineering lever: a reasoning model can be told how much compute to spend; an agent can route easy requests to a fast model and hard ones to a slow one. The spending pays: on easier and middling problems, allocated well, extra test-time compute can do more for accuracy than an equivalently enlarged model (Snell et al., 2024) — though on the hardest problems sheer scale still tells. But allocation is the operative word. Spent on the wrong problems it yields an agent merely slow and dear — and prone to over-thinking: reasoning itself out of a correct first instinct, or pouring a paragraph of deliberation onto a problem a reflex would have dispatched. Under-thinking and over-thinking are the same faculty failing in opposite directions — and the commitment strategy of Section 3.5, the System 1 and System 2 arbitration of Section 3.4, and the decision of when to reflect again (Section 3.12) are all this one allocation decision in different dress. A well-built agent modulates, and escalates to a human or a stronger model when it judges itself out of its depth.

3.13.3 The Faculty the Field Lacks

Of all the faculties in this chapter, metacognition is the most consequential and the least solved. An agent that reliably knew the boundary of its own competence — that set its own reasoning budget, abstained when uncertain, and knew when to stop — does not yet exist. The failures that do real damage in production are seldom the model failing to reason, but the model failing to register that it has failed, and proceeding with conviction. An agent’s most valuable single piece of self-knowledge is the knowledge of what it does not know, and it is the piece we are least able to install.

The substrate, then, gave the classical architectures the one thing they always lacked, a general reasoner to put inside them — but a reasoner that is unfaithful, brittle, overconfident, and uncalibrated, to be elicited with care, checked from outside, and supervised by a metacognition no one yet knows how to build. And throughout, every remedy — checking the plan, grounding the reflection, verifying the answer — has leaned on a capability we have used freely and not yet examined: the agent’s power to act. How an agent reaches out of its closed world of text, and what it means to grant it that power, is the subject of Chapter 4.

3.14 Summary

  • An agent architecture is the structure that turns competence into conduct — Russell and Norvig’s architecture plus program is Agent = Model + Harness in different dress: the model the swappable program, the harness the durable architecture that outlives it.
  • The classical families are four settings of one dial — how much to think between perceiving and acting: reactive, deliberative, hybrid (arbitration the hard part), and BDI, whose intention-as-persistent-commitment is precisely the standing goal a stateless substrate cannot hold. The model is the faculty of deliberation; the harness is the will.
  • The modern agent loop is a special case, not a novelty. ReAct’s observe–reason–act cycle is sense–plan–act run closed-loop, its reasoning core at last any good — and a loop that cannot stop is not persistent but pathological: termination is load-bearing architecture.
  • The loop’s “reason” step is practical reasoning, the ancient pairing made mechanical. Deliberation settles what to pursue; means–ends reasoning settles how.
  • Language-model reasoning and planning are real, powerful, and unguaranteed. The stated chain is not always the true cause — a tool, not a confession — and a drafted plan’s correctness is the hard part: let a verifier, a classical planner, or the world check it. Reflection improves an agent only when grounded in external signal; ungrounded, it launders the mistake into a more confident one.
  • Metacognition — when to think, how much, whether to trust the result — is the chapter’s most consequential and least solved problem. An agent that reliably knew when it was out of its depth would change the field.

3.15 Exercises

Exercise 1. Place each of the following designs on the two knobs of Section 3.6 — how much it deliberates before each action, and how far it commits before perceiving again — then name the classical family, or the corner of Figure 3.1, that it instantiates, and one failure mode it inherits from that position. (a) A guardrail that reads each outbound message an agent drafts and, in a single model call carrying no memory of earlier calls, passes or blocks it. (b) A release agent that reasons at length to draft a forty-step migration script, then executes all forty steps without a further model call. (c) A support agent that reasons at length before composing each reply, sends it, and does nothing more until the customer answers. (d) A nightly maintenance job that runs a fixed shell script and consults no model at all. (e) One of the four sits in the corner of Figure 3.1 that the classical field left unoccupied: say which, and explain why a design may live there only while its world stands still.

Exercise 2. An orchestrator dispatches each incoming issue down one of two paths. The reflex path is a single call to a small model, costing 1 unit; it resolves an easy issue with probability 0.9 and a hard one with probability 0.15. The deliberative path is a full plan-then-execute pass, costing 8 units and resolving either kind with probability 0.95. A fraction 0.25 of issues are hard. The test suite catches every reflex failure at no cost, and a caught failure is escalated to the deliberative path; deliberative failures are handed to a human under every policy alike, and may be ignored in the comparison. (a) Compute the expected cost per issue of three policies: deliberate-always; reflex-first, in which every issue tries the reflex path and failures escalate; and route, in which a classifier call costing 0.2 labels the issue — with accuracy 0.8 on either kind — and labelled-easy issues go reflex-first while labelled-hard issues go straight to deliberation. (b) Show that at these numbers not even a perfect router — accuracy 1, fee waived — beats reflex-first, and isolate the one-line inequality responsible. (c) Now let a quarter of reflex failures slip past the tests, each costing 40 units to repair downstream; recompute reflex-first, route, and the perfect router. Which policy wins? (d) In at most two sentences: when does a router earn its keep, and how does your answer connect to the warning of Section 3.2 about reflexes wired to actions that cannot be taken back?

Exercise 3. A coder agent executes its plan one step at a time, at a cost of one unit per step. Immediately before each step, independently with probability q, the world shifts — a teammate merges a conflicting change, a dependency releases — and the plan, if still valid, becomes invalid; steps executed while the plan is invalid accomplish nothing. The agent reconsiders after every k-th step: reconsideration costs r units, detects any invalidation, and replans, so each block of k steps begins with a valid plan. (a) Show that the expected number of useful steps in a block is \sum_{i=1}^{k} (1 - q)^{i}, and find the sum’s closed form. (b) Justify that the long-run expected cost per useful step is

c(k) \;=\; \frac{k + r}{\sum_{i=1}^{k} (1 - q)^{i}}.

  1. For r = 2 and q = 0.25, evaluate c(k) for k \in \{1, 2, 3, 6, 12\} and pick the best interval on the menu. (d) Repeat for q = 0.02. (e) Show that c(k) \to \infty as k \to \infty whenever q > 0, and read your results as settings of the commitment dial of Section 3.5 — the interior optimum k^{*} interpolating between the cautious (k = 1) and bold (k \to \infty) agents that Chapter 1’s Exercise 6 could price only at the poles: name the classical failure mode waiting at each end of the menu.

Exercise 4. For planning purposes the team’s world has five facts — patch_written, tests_pass, review_approved, merged, deployed — and five actions, three of them already modelled STRIPS-fashion: run_tests (precondition patch_written; adds tests_pass; deletes nothing), merge (precondition review_approved and tests_pass; adds merged; deletes nothing), and deploy (precondition merged and tests_pass; adds deployed; deletes nothing). The initial state is {tests_pass} — the main branch is green and nothing else has happened — and the goal is {deployed, tests_pass}. (a) Formalise the two remaining actions from their informal descriptions: writing the patch produces a written patch and invalidates both the suite’s verdict and any existing approval, since the diff has changed; requesting review requires a written patch and a green suite, and yields approval. Give each action’s precondition, add, and delete lists. (b) The orchestrator proposes the plan write_patch, request_review, run_tests, merge, deploy, in that order. Trace the state through it by hand and identify the first step at which it fails, naming the missing fact. (c) Give a valid plan and prove that no shorter one exists, arguing from which actions are the sole producers of which facts. (d) Compute the delete-relaxation estimate of the initial state’s distance from the goal — the length of the shortest plan when every delete list is ignored — and say in one sentence what real work the relaxation’s optimism has hidden. (e) The proof in (c) is conditional on the model. Name one contingency of real continuous-integration practice the model omits, and state precisely what becomes of the planner’s guarantee when it arises.

Exercise 5. Before touching the code, a coder agent asks its model two questions, sampling eight answers to each at nonzero temperature. To “What does the --frozen flag of our build tool do?” the eight samples are lexically all different — no two word-for-word alike — but every one asserts the same thing: the flag stops the lockfile being updated. To “Which module validates the configuration schema?” the samples are “config.py” (three times), “the config module” (once), “schema.py” (three times), and “validation lives in schema.py” (once). (a) For each question compute the naive entropy over distinct answer strings, H = -\sum_{c} \hat{p}_c \ln \hat{p}_c, where \hat{p}_c is the fraction of samples showing string c. (b) Cluster each question’s samples by what they assert, and compute the semantic entropy — the same formula taken over the clusters. (c) The agent has the budget to verify exactly one of the two answers against the repository. Which question does each measure tell it to spend the check on, and which measure is right? (d) In two sentences: what property must the clustering relation have for (b) to mean anything, and what does semantic entropy degenerate into if answers are clustered by exact string match?

Exercise 6. A coder agent triages each incoming subtask to one of two routes: fast — a single pass, no extended reasoning — costing 1 unit of compute, or slow — extended reasoning plus a grounded reflection round — costing 6 units. A subtask is easy with probability 0.7 and hard with probability 0.3; the fast route succeeds with probability 0.95 on easy tasks and 0.40 on hard ones, the slow route with probability 0.97 and 0.85 respectively; and every failure costs L = 30 units of human rework. Take the expected total cost per subtask — compute plus expected rework — as the yardstick. (a) Compute the expected cost of always-fast and of always-slow, and name the metacognitive failure the loser exemplifies. (b) Rather than consult an external difficulty classifier, the agent reads its own uncertainty: it samples a few cheap answers and takes the semantic entropy of Exercise 5 — the agent disagreeing with itself — as a self-issued signal that the task is hard, routing every flagged subtask slow. On genuinely hard tasks this self-assessment fires with probability 0.75; on easy ones the agent frets without cause with probability 0.15; and the signal rides free on the fast pass it reads. Compute the expected cost under this self-assessment router. (c) Compute the expected cost under perfect self-assessment — the agent that routes every hard task slow and every easy one fast — and hence the fraction of the value of perfect self-assessment that the agent’s own noisy confidence signal captures. (d) Find the rework cost L at which always-slow begins to beat always-fast, and answer in one sentence: is “how much should this agent think?” a property of the task alone, or also of how well an agent knows its own mind?

Further exercises for this chapter continue in the web edition’s exercise bank.

“Reopening the flight plan now would set a dangerous precedent.”

  1. The practical syllogism is set out in Aristotle’s De Motu Animalium, with related discussion in the Nicomachean Ethics.↩︎