%%{init: {"theme": "base", "themeVariables": {"primaryColor": "#E8ECFF", "primaryBorderColor": "#4054B2", "primaryTextColor": "#16171B", "lineColor": "#3B4351", "edgeLabelBackground": "#FAF7F0", "clusterBkg": "#EFE9DC", "clusterBorder": "#766F65"}}}%%
flowchart TB
%% mermaid lays disconnected subgraphs out right-to-left: B is declared first so that A renders as the left panel
subgraph B["LLM-Modulo loop"]
direction TB
PB["Natural-language<br/>problem"]
PB --> B1["Model proposes a<br/>candidate plan"]
B1 --> B2(["Sound external<br/>verifier"])
B2 -.->|"reject, revise"| B1
B2 -->|"accept"| B3["Verified plan"]
end
subgraph A["Model as translator"]
direction TB
PA["Natural-language<br/>problem"]
PA --> A1["Model renders the<br/>problem in PDDL"]
A1 --> A2["Sound classical<br/>planner solves it"]
A2 --> A3["Plan, guarantees<br/>intact"]
end
classDef good fill:#DCEFE2,stroke:#1B6B5A,color:#16171B
classDef world fill:#EFE9DC,stroke:#766F65,color:#16171B
class A2,B2 good
class PA,PB world
5 Reasoning, Planning, and Metacognition
At the heart of the loop assembled in Section 4.6 sits a single step that was left unopened: the model reasons. The modern agent, we established, is deliberation run closed-loop; and whether to spend that deliberation — when an agent should think and when it should simply act — is the decision on which a loop is won or lost. This chapter prises open the reasoning step: what it means for an agent to reason and to plan, what the foundation model actually does when it appears to think, and — the hardest and least settled question in these pages — how an agent might know when its thinking is needed, when it is enough, and when it can be trusted.
Three words in the title mark three layers of one faculty. Reasoning is the drawing of conclusions — here, conclusions about what to do, which is a narrower and older subject than it first looks. Planning is reasoning made systematic: the construction of a course of action before any of it is taken. Metacognition is reasoning turned upon itself: an agent’s judgement about its own thinking — how confident it should be, how hard a problem is, whether to deliberate further or act now. As with architecture in the previous chapter, none of these is new. Practical reasoning has a literature running back to Aristotle; planning was among the first things classical AI tried to automate, and largely succeeded at; metacognition is the youngest of the three, and the one whose question the substrate has changed most. The pattern of the book holds: the classical theory supplies the frame, and the foundation model supplies, at last, something general to put inside it.
A caution belongs at the outset, because this is the chapter where an old debt falls due. In Chapter 3 we allowed that the substrate can reason “after a fashion”, and postponed the awkward question of how much of that is reasoning and how much a fluent performance of reasoning. The postponement ends here. We will look at what the model produces when it is invited to think step by step, at the evidence that those steps genuinely drive its answers and the evidence that they sometimes do not, and we will arrive at a verdict that is useful precisely because it is unromantic: the reasoning of a language model is a real and powerful capacity, and an unreliable one, to be used as a tool and not mistaken for a window onto a mind.
The chapter proceeds from the old to the new and then to the unsolved. The first two sections recover the classical account — practical reasoning as the link between an agent’s ends and its acts, and planning as the systematic search for a course between them. The next two examine what the foundation model does in their place: reasoning in language, with all its brilliance and all its unreliability, and planning by a model that supplies the world knowledge classical planners never had but withholds the guarantees they prized. We then turn to reflection — an agent improving its own work by examining it — and to the evidence about when that helps and when it merely launders a mistake into a more confident version of itself. We close on metacognition: the question Chapter 4 left on the table, of when an agent should think and how much, which is at once the most practical decision an agent designer makes and the one the field is furthest from answering well.
5.1 Practical Reasoning: Thinking Directed at Action
Begin with a distinction old enough to have been made in Greek and useful enough to have survived the journey. Not all reasoning is of a kind. Theoretical reasoning is directed at what is the case: it begins from things believed and arrives at some further thing to believe, so that the conclusion of a theoretical inference is a proposition. Practical reasoning is directed at what to do: it begins from what an agent wants and what it believes, and arrives at an action (Wooldridge, 2009). The distinction is not pedantry. An agent reasons in order to act, and very nearly everything a foundation-model agent does when it “thinks” is practical reasoning in this strict sense — working out, from a goal and a reading of its situation, what to do next. To forget it is to grade an agent’s reasoning as though it were sitting an examination, marking each step for truth, when the only question that finally matters is whether the steps led to a sensible act.
The idea is Aristotle’s. He noticed that a chain of practical reasoning differs from a theoretical one in where it comes to rest: the conclusion of a practical syllogism is not a proposition but a deed.1 A desired end furnishes the major premise — “the service must stay available”; a belief about the world furnishes the minor — “restarting the crashed process will restore it”; and the conclusion is the restart itself. Modern treatments keep the shape and add a division of labour. Practical reasoning, in Wooldridge’s standard account, is two activities and not one (2009): deliberation, which settles which end to pursue, and means–ends reasoning, which settles how to bring it about. Deliberation decides where to go; means–ends reasoning works out the route. The reader of Section 4.5 will recognise the pair at once, for it is the same one: deliberation is what yields intentions, and means–ends reasoning is what yields the plans that serve them.
What makes practical reasoning hard — and what most sharply separates it from the theoretical kind — is that it runs against the clock. The world will not wait while an agent weighs every end and every means, and an agent that insisted on doing so would never act at all. This is bounded rationality again, seen now from the reasoning side: Bratman’s point that intentions exist precisely to spare an agent the expense of reasoning everything out afresh, settling some questions in advance and taking them off the table (1987). Practical reasoning is therefore not the search for the optimal action but the production of a good-enough one within a budget — and the size of that budget, how much to deliberate before committing, is itself a decision, and the one towards which this whole chapter is bent (Section 5.6).
With the frame in place, the modern agent loop stops looking unprecedented and starts looking familiar. The reason step of the agent loop — the model’s “thought” before it acts — is a practical syllogism conducted in natural language. “The tests fail because the parser rejects empty input, so I will add a guard for the empty case”: a goal, a belief about the world, a concluded action, exactly as Aristotle drew it, only generated token by token rather than hand-coded. Both halves are usually present, folded into a single breath — the model deliberates (of the several things wrong, the parser first) and reasons from means to end (the fix is a guard clause) in one stretch of text. This is why the classical vocabulary keeps earning its place. When a contemporary agent “thinks about what to do next”, it is not performing some operation the field has no name for; it is doing practical reasoning, the very thing agents have been described as doing since there were agents to describe.
What the substrate changed is not the structure of practical reasoning but its reach. The classical practical reasoner could draw only the conclusions its designer had supplied the premises and the rules to draw; the foundation model arrives with its own vast stock of premises about how the world works, and will produce a plausible practical syllogism about very nearly anything. That is the gift — and, as the rest of the chapter will insist, the catch, since a plausible practical syllogism is not always a sound one. Means–ends reasoning pursued systematically, and with guarantees that it will not mislead, was the great achievement of classical AI under another name — planning — and it is where we turn next, before asking how much of those guarantees survives when the planner is a language model.
5.2 Classical Planning: Reasoning as Search
Means–ends reasoning, the second half of Section 5.1, is the part classical AI managed to make a science of. Done casually, it is the everyday business of working out how to get from where you are to where you would like to be. Done systematically — exhaustively, and with guarantees — it becomes planning: the construction, in advance and on paper, of a sequence of actions that will carry the world from its present state to a desired one. Stated formally, a planning problem is four things — an initial state, a goal, a set of actions each with the conditions under which it may be taken and the effects it brings about, and the demand for a sequence of those actions leading from the one to the other. So posed, deciding what to do becomes a problem of search through the space of states the world might occupy, and planning becomes the most thoroughly studied form of means–ends reasoning there is, the standard reference for which is Ghallab, Nau, and Traverso (2004).
Everything turns on the representation. The classical move, due to Fikes and Nilsson’s STRIPS (1971) and first met in Section 4.3, is to describe the world as a set of facts and each action as a small bundle of logic: the preconditions that must hold for it to apply, and the facts it adds and deletes when it does. Restart-service requires the service to be crashed and makes it running; nothing else about the world is disturbed. From this austere description a planner can work out, without taking a single action, what any sequence of them would do. The representation was standardised in the 1990s as PDDL, the Planning Domain Definition Language (McDermott et al., 1998), so that rival planners could be set the same problems at competition — and, as Section 5.4 will note, so that a language model can today be asked to write its plans in a form a classical planner will agree to check.
The space to be searched is, for any interesting problem, astronomically large; the number of states grows exponentially in the number of facts, so the whole art lies in not searching most of it. A planner can work forward from the initial state or backward from the goal; it can decline to fix the order of two actions until something forces the choice, the strategy of least-commitment or partial-order planning; or it can plan hierarchically, expanding an abstract task — “deploy the release” — into subtasks and those into primitive actions, in the manner of a hierarchical task network. That last idea is worth marking in passing, because it is exactly the task decomposition a modern agent performs when it breaks a job into pieces for itself or its subagents (Section 5.4); the technique is a good deal older than it looks.
What turned planning from a laboratory curiosity into something that disposes of problems with millions of states was not a faster computer but a better compass. The decisive advance was the domain-independent heuristic: a way of estimating, automatically and from the structure of the problem itself, how far any given state lies from the goal — typically by solving a relaxed version of the problem that politely ignores the inconvenient parts, such as the actions’ delete effects — and using that estimate to steer the search towards the goal rather than blunder about the state space at random. Planning as heuristic search (Bonet & Geffner, 2001) is the banner of the modern field, and it is why a present-day classical planner, handed a well-specified problem, is genuinely formidable.
All this formality buys something a reflex and a fluent guess can never offer: a guarantee. A planner worth the name is sound — the plan it returns really does achieve the goal, in the world as modelled — and complete — if a plan exists, it will find one; and many are optimal into the bargain, returning the shortest or cheapest plan there is. This is the deep attraction of planning, and the reason it never went away: where a language model offers a plan that looks right, a classical planner offers one that is right, with the proof residing in the structure of the search. It does not hallucinate a step, and it does not stop early because its patience has run out. It trades flexibility for certainty, and for the right sort of problem that is an excellent trade.
The certainty, though, arrives with an asterisk the size of Section 4.3. Every guarantee is conditional on the model — the hand-authored description of states, actions, preconditions, and effects — and the planner knows precisely nothing the modeller did not encode. Specify the domain correctly and completely and the planner is unbeatable upon it; omit an effect, or meet a situation the model never anticipated, and the proof now guarantees a plan for a world that is not the real one. Authoring that model, exhaustively, for an open and shifting domain is the labour that kept classical planning largely indoors — the knowledge bottleneck once more, in its purest form. Sound reasoning over a model you cannot build is reasoning you cannot use.
This is the precise opening the foundation model strides into, offering to supply the world knowledge the planner always had to be handed, and to charge for it in exactly the coin of guarantees. That offer, and its price, is the business of Section 5.4. But a model’s planning is only its reasoning put to a particular use, and before we can weigh the one we must reckon with the other — with what a language model is actually doing when it appears to think. To that question we now turn.
5.3 Reasoning in Language Models
The observation that founded the study of language-model reasoning is almost embarrassingly simple. Ask a capable model for an answer and it will give you one; ask it instead to work the problem through step by step, and it will often give you a better one. The intermediate steps — the chain of thought (Wei et al., 2022) we first met in Section 3.2 — are not decoration: on arithmetic, logic, and multi-step word problems they lift accuracy markedly, and they do so with no change to the model’s weights, summoned by little more than the instruction to think aloud. Something in the substrate supports the serial working-through we recognise as reasoning, and a phrase is enough to switch it on. This is the capacity that earned the model its cognitive adjective, and the one we undertook, in Chapter 3, to examine without flattery.
The phrase was only the beginning. What followed was a family of techniques united by one idea — that a model reasons better when it is allowed to spend more computation before committing to an answer — and distinguished by what they do with the extra spend. Self-consistency samples not one chain but many and returns the answer most of them agree on (Wang et al., 2022), which is ensembling by another name and inherits both its gains and, as Chapter 13 will show, its hazards. Tree of thoughts abandons the single chain altogether and searches a branching space of partial thoughts, generating candidate steps, scoring them, and backtracking from dead ends (Yao et al., 2023) — which is, unmistakably, planning-as-search come back in language, with the model serving as both the generator of moves and the heuristic that judges them. And the reasoning models that now headline every release go further still, using reinforcement learning to train the long deliberative trace in as a learned habit rather than a prompted trick (DeepSeek-AI, 2025). The trajectory is plain: from coaxing reasoning with a phrase, to ensembling it, to searching with it, to baking it in.
| Technique | What the extra compute buys | How it is obtained | Reference |
|---|---|---|---|
| Chain of thought | A single step-by-step trace before the answer | Prompted — the instruction to think aloud | Wei et al. (2022) |
| Self-consistency | Many sampled chains; the answer most of them agree on | Prompting plus sampling | Wang et al. (2022) |
| Tree of thoughts | A search over a branching space of partial thoughts — generate, score, backtrack | Prompting plus search | Yao et al. (2023) |
| Reasoning models | A long, extended deliberative trace before the answer | Reinforcement learning — baked in, not prompted | DeepSeek-AI (2025) |
All of which sharpens, rather than settles, the question Chapter 3 deferred: is any of this reasoning, or only a convincing impression of it? The generous reading is hard to resist — the model sets out premises, works through steps, and arrives at a conclusion that follows, exactly as a person reasoning carefully would. The deflationary reading is just as available: a model trained to continue text will, shown the form of a derivation, continue it in that form, and the product can wear every outward sign of reasoning while being made by nothing of the kind. Intuition will not adjudicate between these; only evidence will, and the evidence is sobering on two fronts.
The first concerns faithfulness — whether the chain a model produces is the actual cause of its answer or a story told about it afterwards. Turpin and colleagues supplied the uncomfortable demonstration (2023): plant a subtle bias in the prompt — arrange, say, for the correct-looking answer always to be option A — and the model’s answers drift to follow the bias, while its chain of thought goes on confidently justifying them with reasoning that never once mentions the thing that actually moved it. The steps are fluent, plausible, and post-hoc: a rationalisation, not a derivation. This ought to trouble anyone tempted to read an agent’s chain of thought as an audit trail of why it did what it did — a temptation Chapter 26 returns to — because a chain that did not cause the answer cannot explain it either.
The second concerns robustness. A genuine reasoner is indifferent to the names of the variables: rename the quantities in a problem, or pad it with a true but irrelevant sentence, and a sound method returns the same answer. Language-model reasoning frequently does not. On GSM-Symbolic (Mirzadeh et al., 2024), a version of a grade-school maths benchmark whose numbers and surface details can be regenerated at will, accuracy slips when only the numbers are changed and slips further when a plausible but irrelevant clause is added — across models, and by margins too large to wave away. The competence is real, but it is fragile in a way a method reasoning from the structure of a problem rather than its surface would not be; it is the brittleness of Section 3.3, surfacing now in the very faculty this chapter is about.
What, then, is the verdict? The useful one declines both temptations. To call language-model reasoning a mere trick is to ignore real, repeatable, and often large gains, and the plain fact that reasoning models now dispatch problems their predecessors could not. To call it reasoning in the full sense — faithful, robust, answerable for its every step — is to ignore that the chain may not be the cause and the capacity may not survive a renamed variable. It is a powerful and unreliable instrument, and it is best used as one: elicited deliberately, sampled and searched where the stakes justify the spend, and — the recurring moral of this book — checked against something outside the model rather than believed because it reads well. The chain of thought is not, after all, a window onto a mind; it is more generated text, produced by the same machinery and heir to the same failures.
That the platforms have lately taken to retaining this reasoning from one turn to the next while declining to show it only sharpens the warning:2 a trace you are invited to depend on but forbidden to inspect is, by this chapter’s lights, the worst of both worlds — trusted by the machine, unauditable by you. Which is exactly why asking the model to plan — to commit in advance to a course of action it cannot yet test — inherits every doubt raised here, and adds a few of its own. That is the next section’s business.
5.4 Planning with a Language Model
Take the means–ends reasoning of Section 5.2, hand it to a foundation model, and something remarkable and something alarming happen at once. The remarkable thing is that the model will plan for almost anything. Ask it for a sequence of steps to reach a goal — deploy the service, reproduce the bug, carry out the migration — and it produces one, fluently, drawing on the vast store of how-things-are-usually-done it absorbed in training. The knowledge bottleneck that kept classical planning indoors (Section 4.3) appears, at a stroke, to dissolve: nobody writes the domain model, nobody enumerates the actions and their preconditions, because the model already knows, more or less, how the world works. The alarming thing is the one Section 5.3 prepared us for. A plan that reads well is not a plan that works, and the model offers nothing, on its own, to tell the two apart.
The single most useful thing a model does as a planner is to decompose: to break a goal too large to take in at once into an ordered set of smaller ones. Least-to-most prompting makes the move explicit, cracking a hard problem by first reducing it to a sequence of easier subproblems and working upward (Zhou et al., 2022); the plan-and-execute pattern that recurs through the agent frameworks does the same at the level of actions, drafting the whole plan first and then carrying out its steps in turn. This is the hierarchical task network of Section 5.2 with the hand-built methods struck out — the model supplies, from training, the decompositions a classical system had to be handed by an expert. Our orchestrator earns its keep exactly here: given “implement the new export format”, it carves the task into write-the-serialiser, add-the-tests, and update-the-docs, and parcels them out to its coder agents. The decomposition is genuinely valuable, and it is the part of language-model planning that most deserves the name.
It is the next part that does not. Generating a plausible plan is easy; knowing whether it is correct is hard, and the model is conspicuously worse at the second than the first. Set to plan autonomously on the sort of well-defined benchmark a classical planner eats for breakfast, even strong models succeed only occasionally — in one careful study the best of them solved around an eighth of the instances, the rest undone by missing preconditions, impossible steps, and goals quietly left unmet (Valmeekam et al., 2023). Worse, the model cannot reliably notice: asking it to check its own plan is asking it to reason about the plan, and self-verification is heir to every unreliability of Section 5.3, so a confident review is no warrant of a correct one. The model plans as it reasons — plausibly, fluently, and without guarantee.
The productive response is not to make the model plan better in isolation but to stop asking it to plan alone. Pair it with an external check, and the division of labour comes clear. In one arrangement the model is a translator: it turns the messy natural-language problem into the formal language of classical planning — PDDL — and hands it to a sound classical planner, which solves it with the guarantees intact and hands the answer back (Liu et al., 2023). The model does what it is uniquely good at, turning the world into a model — the very thing whose absence was the classical bottleneck — and the planner does what it is uniquely good at, sound search. In another, more general arrangement, which Kambhampati and colleagues call an LLM-Modulo framework (2024), the model generates candidate plans and an external, sound verifier accepts or rejects them in a generate-and-test loop — the model a fertile source of guesses, the verifier the arbiter of correctness. Their slogan is the honest summary of the whole section: language models cannot plan on their own, but they can be an enormous help to planning. And the cheapest verifier of all is frequently the world itself — run the plan and see what breaks, the closed loop of Section 4.6 put to work.
The coder agent shows the spectrum in miniature. When it resolves to “fix the parser, then mend the failing tests”, that plan is, at the moment of writing, wholly unverified — a fluent guess that might be wrong in any of a dozen ways. What makes it safe to act on is not the model’s confidence but the test suite: the plan is checked by being run, and a red test is the verifier’s veto. The orchestrator’s decomposition is a bet; the tester is what settles it. This is why the agent systems that actually hold together tend to wrap fluent model planning in cheap, external, automatic checks — tests, type-checkers, schema validators, the environment’s own complaints — rather than trust a plan because it was confidently produced. The moral generalises the classical lesson precisely: the value of planning was always in its guarantees, and when the planner cannot supply them, you must find them somewhere else.
Planning with a language model, then, inverts the classical bargain rather than escaping it. Classical planning gave you guarantees at the price of a model you had to build by hand; model-based planning gives you the model for nothing and withholds the guarantees. The competent agent draws each half from wherever it is cheap — the model for world knowledge and decomposition, an external verifier for certainty — and declines to pretend the second can be skipped. But verification only tells you that a plan has failed; it does not, by itself, produce a better one. For that, the agent must take the failure — the red test, the rejected plan, the tool’s complaint — and reason about what to change. That act of examining one’s own work and revising it is reflection, and whether it genuinely helps, or merely dresses a mistake in fresh confidence, is the question we turn to next.
5.5 Reflection and Self-Critique
An agent holds a verdict — the plan failed, the test went red — and has done nothing about it yet. Reflection is the obvious next move: have the agent examine its own work, or the trail of its own attempts, and revise. The appeal is considerable, because a model that could reliably criticise and improve its own output would lift itself by its own bootstraps, reaching better answers with no help from outside. The basic shape is a loop of maker and critic: produce a draft, generate a critique of it, revise in the light of the critique, and go round again until the critic runs out of complaints.
Two methods mark out the territory. Self-Refine keeps the whole apparatus inside a single model: it generates an output, prompts the same model to feed back on that output, then to revise it, and iterates until satisfied, reporting that the polished results are preferred to the first drafts across a spread of tasks (Madaan et al., 2023). Reflexion adds a memory and a failure signal: an agent that fails a task writes itself a few sentences of verbal post-mortem — “I assumed the list was sorted; it was not” — stores that reflection, and consults it on the next attempt, improving across trials by what its authors aptly call verbal reinforcement learning (Shinn et al., 2023). Both look like genuine self-improvement: a system getting better by thinking about where it went wrong.
Whether it is genuine, though, turns on a single distinction that the surface similarity of these methods hides — what the reflection is reflecting on. If the feedback is grounded, tied to something outside the model’s own judgement (a failed unit test, a compiler error, a raised exception, a search that came back empty), then reflection has real purchase, because the error signal is real: the agent knows that it was wrong, and usually has a solid clue as to why. If the feedback is merely the model’s own unaided verdict on its own work — “does this look right? yes, it looks right” — then reflection stands on far shakier ground, because passing judgement is itself reasoning, and Section 5.3 gave us no reason to expect a model to be any better at grading its answer than at producing it.
The evidence bears the distinction out, and sharply. Huang and colleagues examined intrinsic self-correction — a model revising its reasoning with no external feedback at all — and found that it does not reliably help and frequently hurts, the model talking itself out of correct answers about as often as into them (2023). Several reported triumphs of self-correction turn out, on inspection, to have quietly handed the model an oracle that told it when to stop, which is external feedback wearing a disguise; withdraw the oracle and the gains largely evaporate. The lesson is not that reflection is worthless but that it is amplification, not generation: it can make excellent use of a true error signal, and it cannot conjure one out of the model’s own confidence. Reflexion works because it reflects on real failures the environment handed it; Self-Refine works best on the tasks where its self-feedback happens to track something checkable.
The coder agent shows the difference in a single contrast. When the tester returns a failing case and the coder reflects — “the loop overruns the array by one; I will tighten the bound” — the reflection is anchored to a real, externally supplied failure, and it earns its keep. When the same coder, having run nothing, merely asks itself whether its code is correct and reassures itself that it is, the reflection is ungrounded and worth approximately nothing. This is the entire reason the running example carries a tester and a reviewer: they exist to supply the external signal that makes the coder’s reflection more than a comforting noise. A reviewer agent, at its best, is precisely an externalised critic — better than self-critique to exactly the degree that it is genuinely independent of the agent it reviews, a property easy to lose and the subject of the collective-judgement and debate chapters to come (Chapter 13 and Chapter 14).
Reflection, then, is real but conditional: a powerful amplifier when there is a true signal to amplify, an echo chamber when there is not. The design rule follows without argument — build the external check first, the test or the verifier or the independent critic, and let the agent reflect against that; never appoint the model the sole honest judge of its own work. But notice what even grounded reflection asks of an agent: to look at its own output and decide whether it is good enough, whether to revise once more or to stop, whether the signal it is reacting to is one it ought to trust. That — an agent reasoning about the quality and sufficiency of its own reasoning — is metacognition, the same faculty this whole chapter has been circling: knowing when to think harder, when to act, and when to believe the result. It is where we end.
5.6 Metacognition: Knowing When to Think
Is my work good enough? Should I think more, or stop? Can I trust this result? Judgements of that kind — an agent reasoning about its own reasoning — are the province of metacognition, the word the psychologist John Flavell gave the study of thinking about thinking (1979). It is the supervisor of the faculties this chapter has assembled: it does not itself reason, plan, or reflect, but decides when each is needed and whether to believe what it returns. And it is the chapter’s destination because every earlier section ended by deferring a decision to it — when to deliberate, whether to trust the chain, whether the plan is sound, whether to reflect once more. All of those are one faculty’s work, and we have put off naming it until now.
5.6.1 Knowing What One Knows
Metacognition has two halves, and the first is self-assessment: knowing what one knows and, far more usefully, what one does not. This is the honest twin of hallucination — a model that fabricates with confidence is exactly a model that cannot tell its knowledge from its guesses. The evidence here is genuinely two-sided. Kadavath and colleagues found that large models, asked in the right way, carry a surprising amount of self-knowledge: their stated confidence tracks their accuracy, and they can often predict whether they will answer a question correctly, the ability improving as models grow (2022). But the self-knowledge is fragile — it depends on how the question is put, and the instruction-tuning that makes a model helpful seems also to make it more confident than it has any right to be. An agent that reliably knew when to say “I don’t know”, and meant it, would be transformative; the agents we have mostly bluff. Calibration is the precondition for everything else metacognitive, for thought cannot be allocated wisely by something that cannot tell a hard problem from an easy one.
What an agent needs here has a name and a long history outside the language-model world: uncertainty quantification, the discipline of attaching honest error bars to a model’s predictions. Its central distinction is the one calibration is really chasing — between aleatoric uncertainty, the irreducible noise in the world, and epistemic uncertainty, the reducible part that is the model’s own ignorance; to know what one does not know is, precisely, to estimate one’s epistemic uncertainty.
The classical toolbox is substantial — Bayesian deep learning, which carries a distribution over a network’s weights and reads uncertainty off the spread of the predictions they produce (Gal & Ghahramani, 2016), and evidential deep learning, which trains a single network to output the parameters of an evidential distribution in one pass (Sensoy et al., 2018). The awkward fact is that almost none of it transfers cleanly to a frontier model: one cannot cheaply keep a posterior over hundreds of billions of weights, an ensemble of such models is ruinous, and the uncertainty that matters to an agent is not the wobble of a number but doubt about a meaning — whether a claim is true, not how far a logit trembles. The most promising language-model methods therefore work at the level of meaning rather than tokens: semantic entropy, for one, samples several answers, clusters them by what they actually assert, and reads wide disagreement among the clusters as the model’s own signal that it is guessing (Farquhar et al., 2024). It is genuine progress, and also a reminder in miniature of the pattern: the principled apparatus for knowing what one does not know was built for a different kind of model, and is having to be reinvented, still incompletely, for this one.
| Method | How it estimates uncertainty | Usable through a closed frontier API? | Reference |
|---|---|---|---|
| Bayesian deep learning | Carries a distribution over the network’s weights; reads uncertainty off the spread of the predictions they produce | ✗ | Gal & Ghahramani (2016) |
| Evidential deep learning | Trains a single network to output the parameters of an evidential distribution in one pass | ✗ | Sensoy et al. (2018) |
| Semantic entropy | Samples several answers, clusters them by what they actually assert, treats wide disagreement among the clusters as a signal of guessing | ✓ | Farquhar et al. (2024) |
5.6.2 Knowing How Much to Think
The second half is allocation: how much to think before acting — the decision Section 4.4 and Section 4.6 deferred to this chapter, and which has lately become a real engineering lever rather than a figure of speech. A reasoning model can be made to spend more or less compute before it answers; an agent can route an easy request to a fast model and a hard one to a slow one, in the manner of the hybrids, or send a routine step straight to action and a fraught one around an extended loop. And the spending pays: on easier and middling problems, allocated well, extra test-time compute can do more for accuracy than an equivalent enlargement of the model itself (Snell et al., 2024) — though on the hardest problems sheer scale still tells. The operative phrase is allocated well. Spent on the wrong problems it yields an agent that is merely slow and dear — and, in the failure the enthusiasm forgets, prone to over-thinking: reasoning itself out of a correct first instinct, or pouring a paragraph of deliberation onto a problem a reflex would have dispatched. Under-thinking and over-thinking are the same faculty failing in opposite directions.
Seen this way, several threads of the last two chapters turn out to be one. The commitment strategy of Section 4.5 — how doggedly to hold a plan before reconsidering — the System 1 and System 2 arbitration of Section 4.4, and the decision of when to reflect again from Section 5.5 are each a metacognitive control, settling how much of which faculty to spend. A well-built agent does not think a fixed amount; it modulates, attempting an easy task briefly, deliberating at length over a hard one, and escalating to a human or a stronger model when it judges itself out of its depth. The frontier of the field sits exactly here — agents that set their own reasoning budget, abstain when uncertain, and know when to stop — which is a courteous way of saying the field has for the most part not arrived.
5.6.3 The Faculty the Field Lacks
For of all the faculties in this chapter, metacognition is at once the most consequential and the least solved. We can elicit reasoning, scale it, plan with it, and reflect with it; an agent that reliably knew the boundary of its own competence — that thought hard exactly when thought would help and not otherwise, and admitted uncertainty instead of manufacturing an answer — does not yet exist. The gap is precisely the one between an arresting demonstration and a dependable system. The failures that do real damage in production are seldom the model failing to reason; they are the model failing to register that it has failed, and proceeding with conviction. An agent’s most valuable single piece of self-knowledge is the knowledge of what it does not know, and it is the piece we are least able to install.
So the chapter closes where Section 5.1 opened it: with reasoning directed at action, now wrapped in the judgement that ought to govern it. The substrate gave the classical architectures the one thing they always lacked, a general reasoner to put inside them — but a reasoner that is unfaithful, brittle, overconfident, and uncalibrated, to be elicited with care, checked from outside, and supervised by a metacognition no one yet knows how to build. That is the honest state of the art. And throughout, every remedy has leaned on a capability we have used freely and not yet examined: the agent’s power to act — to run the test, call the planner, query the world — which supplied the external grounding each section turned on. How an agent reaches out of its closed world of text and acts upon the world, and what it means to grant it that power, is the subject of Chapter 6.
5.7 Summary
- Reasoning, planning, and metacognition are three layers of one faculty — drawing conclusions about what to do, laying out a course of action in advance, and judging one’s own thinking. None is new; the classical theory frames each, and the substrate is what finally fills it.
- Practical reasoning links an agent’s ends to its acts. Deliberation settles what to pursue, means–ends reasoning settles how, and the ReAct loop’s “reason” step is that ancient pairing made mechanical — which is why the classical account still describes it.
- Classical planning is means–ends reasoning made systematic — search from a state to a goal, with real guarantees of soundness and, sometimes, optimality, bought at the price of a complete and hand-authored model: the knowledge bottleneck again.
- Language-model reasoning is real, powerful, and unreliable. Chain of thought and its kin genuinely improve performance, but the stated steps are not always the true cause, and fluency is not faithfulness. Treat the chain as a tool, not a confession.
- A language model plans with the world knowledge classical planners lacked, and without their guarantees. It decomposes tasks and drafts plans freely; knowing the plan is correct is the hard part, and the surest remedy is to let something else — a verifier, a classical planner, the world itself — check it.
- Reflection helps when it is grounded and flatters when it is not. An agent revising against external feedback improves; an agent grading its own homework often merely launders the mistake into a more confident one. External signal is what makes self-critique work.
- Metacognition — knowing when to think, how much, and whether to trust the result — is the most consequential and least solved problem in the chapter. It is the decision Chapter 4 deferred: spend deliberation where it earns its cost and nowhere else. An agent that reliably knew when it was out of its depth would change the field; today’s, for the most part, do not.
5.8 Exercises
Exercise 1. In one afternoon of the software-engineering team’s life, three agents each record a thought and act on it. The first coder: “The nightly build is red because commit 4f2a renamed parse_row without updating its callers; reverting that commit will make the build green. I will revert it.” (In fact the build is red because a CI credential has expired; the rename is harmless.) The second coder: “The linter reports four style warnings in utils.py; clean code is easier to review; I will spend my remaining budget on the warnings.” (The release-blocking test failure it was assigned sits untouched.) The third coder: “Users cannot log in; the logs show the auth worker has crashed; restarting the web front-end will restore login. I will restart the front-end.” (The logs are right, and the front-end is healthy.) (a) Cast each thought as a practical syllogism, identifying the major premise, the minor premise, and the concluded act. (b) One agent holds a false belief, one fails at deliberation, and one fails at means–ends reasoning; say which is which, justifying each in a sentence. (c) For each agent, name the cheapest run-time check that would expose the failure. (d) An auditor examines only whether each stated belief is true. Which of the three failures does the audit miss, and what does that say about grading practical reasoning as though it were theoretical?
Exercise 2. For planning purposes the team’s world has five facts — patch_written, tests_pass, review_approved, merged, deployed — and five actions, three of them already modelled STRIPS-fashion: run_tests (precondition patch_written; adds tests_pass; deletes nothing), merge (precondition review_approved and tests_pass; adds merged; deletes nothing), and deploy (precondition merged and tests_pass; adds deployed; deletes nothing). The initial state is {tests_pass} — the main branch is green and nothing else has happened — and the goal is {deployed, tests_pass}. (a) Formalise the two remaining actions from their informal descriptions: writing the patch produces a written patch and invalidates both the suite’s verdict and any existing approval, since the diff has changed; requesting review requires a written patch and a green suite, and yields approval. Give each action’s precondition, add, and delete lists. (b) The orchestrator proposes the plan write_patch, request_review, run_tests, merge, deploy, in that order. Trace the state through it by hand and identify the first step at which it fails, naming the missing fact. (c) Give a valid plan and prove that no shorter one exists, arguing from which actions are the sole producers of which facts. (d) Compute the delete-relaxation estimate of the initial state’s distance from the goal — the length of the shortest plan when every delete list is ignored — and say in one sentence what real work the relaxation’s optimism has hidden. (e) The proof in (c) is conditional on the model. Name one contingency of real continuous-integration practice the model omits, and state precisely what becomes of the planner’s guarantee when it arises.
Exercise 3. An agent attacks a hard question by sampling m chains of thought independently at nonzero temperature and returning the majority answer, with m odd. Each chain yields the correct answer with probability p; assume, pessimistically, that every incorrect chain lands on the same wrong answer, so that the vote is between two candidates and the majority is correct with probability
P_m \;=\; \sum_{k=(m+1)/2}^{m} \binom{m}{k}\, p^{k} (1-p)^{m-k}.
- For p = 0.7, compute P_5. (b) Find the smallest odd m for which P_m \ge 0.9. (c) Compute P_5 for p = 0.4, and state — justifying it from the behaviour of P_m as m grows — what majority voting does for a model systematically inclined towards one wrong answer. (d) An engineer proposes an adaptive spend: sample three chains; if they agree unanimously, return their answer; otherwise sample six more and take the majority of all nine. For p = 0.7, compute the expected number of chains consumed per question and the resulting accuracy, and compare both figures with the fixed m = 9 scheme. What signal is the adaptive rule reading, and which faculty of this chapter is it a first, crude implementation of? (e) Name the assumption every part above leans on, and explain why chains sampled from a single model are exactly where it fails — a hazard Chapter 13 returns to.
Exercise 4. Before touching the code, a coder agent asks its model two questions, sampling eight answers to each at nonzero temperature. To “What does the --frozen flag of our build tool do?” the eight samples are lexically all different — no two word-for-word alike — but every one asserts the same thing: the flag stops the lockfile being updated. To “Which module validates the configuration schema?” the samples are “config.py” (three times), “the config module” (once), “schema.py” (three times), and “validation lives in schema.py” (once). (a) For each question compute the naive entropy over distinct answer strings, H = -\sum_{c} \hat{p}_c \ln \hat{p}_c, where \hat{p}_c is the fraction of samples showing string c. (b) Cluster each question’s samples by what they assert, and compute the semantic entropy — the same formula taken over the clusters. (c) The agent has the budget to verify exactly one of the two answers against the repository. Which question does each measure tell it to spend the check on, and which measure is right? (d) In two sentences: what property must the clustering relation have for (b) to mean anything, and what does semantic entropy degenerate into if answers are clustered by exact string match?
Exercise 5. A coder agent’s first draft of a patch is correct with probability q. A critic reviews the draft: a correct draft it nevertheless declares wrong with probability a; an incorrect draft it catches with probability b. Whenever the critic declares the draft wrong, the agent revises: a revised correct draft remains correct with probability r_c, and a revised incorrect draft becomes correct with probability r_w. Drafts the critic passes are left alone. (a) Derive the probability q' that the patch is correct after one critique-and-revise round, and the exact condition under which the round helps (q' > q). (b) The team’s tester is a deterministic suite with total coverage — a = 0, b = 1. Compute q' for q = 0.5 and r_w = 0.4. (c) Now remove the tester and let the model grade its own work: a = 0.25, b = 0.35, r_c = 0.6, r_w = 0.4. Compute q' for q = 0.5 and for q = 0.8; derive the threshold value of q above which this critic is a net harm; and say in one sentence what this implies for bolting intrinsic self-correction onto ever more capable models. (d) With the deterministic suite of (b), the loop now runs up to k rounds, revising after every red run. Derive the probability that the patch is correct after k rounds, and find the smallest k for which it exceeds 0.95.
Exercise 6. A coder agent triages each incoming subtask to one of two routes: fast — a single pass, no extended reasoning — costing 1 unit of compute, or slow — extended reasoning plus a grounded reflection round — costing 6 units. A subtask is easy with probability 0.7 and hard with probability 0.3; the fast route succeeds with probability 0.95 on easy tasks and 0.40 on hard ones, the slow route with probability 0.97 and 0.85 respectively; and every failure costs L = 30 units of human rework. Take the expected total cost per subtask — compute plus expected rework — as the yardstick. (a) Compute the expected cost of always-fast and of always-slow, and name the metacognitive failure the loser exemplifies. (b) Rather than consult an external difficulty classifier, the agent reads its own uncertainty: it samples a few cheap answers and takes the semantic entropy of Exercise 4 — the agent disagreeing with itself — as a self-issued signal that the task is hard, routing every flagged subtask slow. On genuinely hard tasks this self-assessment fires with probability 0.75; on easy ones the agent frets without cause with probability 0.15; and the signal rides free on the fast pass it reads. Compute the expected cost under this self-assessment router. (c) Compute the expected cost under perfect self-assessment — the agent that routes every hard task slow and every easy one fast — and hence the fraction of the value of perfect self-assessment that the agent’s own noisy confidence signal captures. (d) Find the rework cost L at which always-slow begins to beat always-fast, and answer in one sentence: is “how much should this agent think?” a property of the task alone, or also of how well an agent knows its own mind?
Exercise 7. The team’s reviewer agent is shown two candidate patches for the same ticket, each preceded by an author line, and emits a chain of reasoning followed by a verdict on which to merge. An engineer suspects the verdict is driven by the author line — “authored by: staff engineer” against “authored by: intern” — rather than by the code. (a) Design the controlled experiment: what is varied, what is held fixed, how the author labels are counterbalanced across pairs, what ground truth is needed for each pair, and the two measurements to record. (b) The experiment is run on 100 pairs in which a hidden test suite judges the intern-labelled patch strictly better. The reviewer selects the staff-labelled patch 68 times; across the 100 chains the author line is mentioned three times; the chains otherwise discuss correctness, style, and coverage, fluently and in detail. State exactly what these numbers establish, and give the correct name for the relation the chains bear to the verdicts. (c) A colleague proposes asking the reviewer, in a follow-up turn, “Did the author line influence your decision?” Explain why the answer settles nothing, whichever way it comes out. (d) Propose the remedy, and say whether it works by improving the reviewer’s reasoning or by removing something from its world.
Exercise 8. Implement the verifier half of a generate-and-test loop for the domain of Exercise 2. Represent an action as a frozen dataclass carrying its name and its precondition, add, and delete sets, and write a function validate(state, plan, goal) returning one of three verdicts: the plan is valid; some step is inapplicable — report the first such step and the facts its precondition lacks; or the plan runs to completion but leaves the goal unmet — report the missing facts. (a) Implement it, standard library only. (b) Run it on three plans the language model has proposed: P_1, the five-step plan of Exercise 2(c); P_2, which skips review — write_patch, run_tests, merge, deploy; and P_3, which is P_1 followed by a second write_patch, because the model helpfully started on the next ticket. Report each verdict. (c) In two sentences: which classical guarantee does this verifier restore to the loop, which can it not restore, and on what does the restored one remain conditional?
Exercise 9. Build a perturbation harness in the manner of GSM-Symbolic (Mirzadeh et al., 2024) for one template problem from the team’s world: “{name} maintains a CI pipeline that runs {n} test suites, each containing {m} tests. Tonight {k} tests were flaky, and each flaky test was executed one extra time. How many test executions did tonight’s run perform?” — ground truth n \cdot m + k. (a) Write a seeded generator, standard library only, that emits matched instances in three conditions: (i) the base instance with constants (n, m, k) = (4, 25, 6); (ii) the same structure with fresh name and numbers; (iii) as (ii), plus one numerically flavoured but irrelevant sentence, such as “Each retry also used 3 extra minutes of runner time.” The answer must be computed programmatically, and the same seed must reproduce the same triple. (b) State the evaluation protocol for a model under test, and the two contrasting accuracy signatures — the pattern that would indict matching on the problem’s surface, and the pattern a method reasoning from its structure would show. (c) Explain why the ground truth must come from the formula rather than from the strongest available model’s answers, naming the finding of this chapter that the shortcut would collide with.
Exercise 10. A vendor pitches DeepPlan, an autonomy upgrade for the orchestrator, as five stages: (i) the model produces a step-by-step plan for the ticket; (ii) the model re-reads and revises its own plan twice, with no external input (“double reflection”); (iii) five whole plans are sampled and the majority plan is adopted (“plan-level self-consistency”); (iv) the adopted plan’s steps are executed end to end, with nothing inspected until all steps have run (“uninterrupted execution”); (v) the model appends a paragraph certifying the run’s success (“metacognitive audit”). No test, validator, or other external check appears anywhere. (a) For each stage, say whether the chapter’s evidence supports it as specified, naming the specific finding or argument that bears on it. (b) Two of the stages are not merely unsupported but defective in their very mechanics; identify them and say precisely why. (c) Redesign the pipeline under the same token budget: give your stages and, for each, the external signal that grounds it.
Exercise 11 (lab). Couple a live model to the verifier of Exercise 8, LLM-Modulo fashion. Describe the domain of Exercise 2 to the model in natural language — facts, actions with preconditions and effects, initial state, and goal — and ask it for a plan; validate the reply; if invalid, allow up to five revision rounds under one of two feedback regimes: bare — “the plan is invalid; try again” — or grounded — the validator’s verdict verbatim, for example “step 2, request_review: precondition tests_pass not satisfied”. A strong model may well solve this five-action domain at the first attempt; if it does so in more than about half your trials, enlarge the domain — add staging, migration, and rollback facts and actions — until first-try validity falls below one half, then run at least twenty attempts per regime. Record, per regime, the distribution of rounds to a valid plan, and, throughout, every case in which an invalid plan arrives wrapped in an assertion of confidence. (a) Report the comparison between regimes and relate it to the chapter’s account of what makes reflection work. (b) Relate the confident-but-invalid cases to the chapter’s account of self-assessment. (c) Quote the transcript moment, if it occurs, where the model repairs precisely the fault the validator named, and say what this shows the model can do with a true error signal that it could not do for itself. Record the model identifier and the date beside your results.