%%{init: {"theme": "base", "themeVariables": {"primaryColor": "#E8ECFF", "primaryBorderColor": "#4054B2", "primaryTextColor": "#16171B", "lineColor": "#3B4351", "edgeLabelBackground": "#FAF7F0", "clusterBkg": "#EFE9DC", "clusterBorder": "#766F65"}}}%%
flowchart TB
C0(["c0 · run start"]) --> C1(["c1"])
C1 --> C2(["c2"])
C2 --> C3(["c3"])
C3 --> C4(["c4"])
C1 -.-> CR["crash · kill -9 mid-turn"]
CR -.->|"re-fold; redo live turn"| C1
C2 -.->|"rewind, amend one message"| FK(["c2′ · counterfactual fork"])
C3 -.-> GT["parked · awaiting approval"]
GT -.->|"approved days later; resume"| C3
C4 -.-> DP["deploy · resumes under v2"]
CR ~~~ FK
FK ~~~ GT
GT ~~~ DP
classDef risk fill:#F4D7D5,stroke:#A4161A,color:#16171B
classDef gate fill:#F7E6B5,stroke:#8A5A00,color:#16171B
class CR risk
class GT gate
20 Dependable, Secure, Safe
Every agent system works beautifully during the demonstration. The demo runs in one process nobody kills, against tools that answer promptly, on a budget nobody has spent, under the fond gaze of its builder. Production revokes every clause of that arrangement, one at a time, usually at night. The process dies twelve minutes into a forty-minute run. The test runner hangs and the timeout fires — or, worse, doesn’t. A retried payment call pays twice. An agent discovers an inexhaustible supply of plausible next steps and works through the month’s tokens before breakfast. And the one action that genuinely needed a human’s signature went through at 3 a.m., unsigned, because nobody had built the gate. The gap between a demonstration and a system is not polish; it is the list of things that have not happened to the demonstration yet.
This chapter closes that gap, and its thesis is that dependability is not a mood but a design discipline: an enumerable inventory of failure classes, each met with a designed response rather than an aspiration (Table 20.1). The inventory is short and old, and much of this chapter is distributed-systems engineering — fifty years of practice — arriving at the agent stack’s door. What the agents add: stochastic components, so the same input may not fail twice the same way; expensive calls, so retrying is a budgeting decision, not a reflex; and actions that touch the world, so some mistakes can only be prevented. Hope, in this setting, is not a mechanism.
| Failure class | What goes wrong | Designed response | Where it is built |
|---|---|---|---|
| Crash | The process dies mid-run; whatever was in memory is lost | The journal on disk, checkpoints at turn boundaries, crash-only recovery by replay | Section 20.1 |
| Hang | A call runs on, unseen; or its notice never arrives | Layered timeouts — tight per call, looser per turn, a task deadline, a run curfew | Section 20.2 |
| Duplicate | A retried call runs twice, or succeeds without saying so | Idempotency keys, the reversibility split, query-before-re-execution | Section 20.2 |
| Runaway | Every step succeeds, yet nothing knows how to be finished | Governors — budgets (a stock) and rate limits (a flow), enforced at the harness | Section 20.3 |
| Irreversible mistake | An action that cannot be taken back fires without a signature | The human gate — approvals and interrupts as durable checkpoints | Section 20.5 |
| Mystery | Something went wrong in the night; no one can say what or why | Observability (the journal, viewed) and time-travel debugging | Section 20.6 |
Two of the preface’s promises are cashed together here. Everything is built twice: first in plain Python on Chapter 17’s runtime — which already owns a journal, a reducer, and a budget, so each mechanism arrives as the production form of something built by hand — then in LangGraph, whose typed state, reducers, checkpointers, and interrupts correspond one-to-one to this chapter’s machinery; the preface chose it for this very hour. The case study is the book’s coding team, now running unattended, overnight, against real repositories — substantial enough to fail in instructive ways, and this chapter intends to let it.
The sections follow the inventory, and the assembled system is finally broken on purpose — the only proof that counts being a system whose failures are boring. Dependable, though, is graded by a world that means no harm; the second movement (Section 20.9) turns to the hostile world, then to the harm that needs no adversary at all. Whether the system is any good is a different question again, and it is the next chapter’s.
20.1 State That Survives: Checkpoints, Reducers, and Recovery
Take the first failure — the process dies mid-run, without notice — and ask what has been lost. Usually “whatever was in memory”: forty minutes of expensive cognition, gone because a container was rescheduled. Dependability’s first job is to make the answer “nothing that matters”, and that is bought in advance by deciding where the truth lives. Chapter 17 made the decision this chapter generalises: the truth lives in the journal, on disk, appended before anything else happens, the process holding only a disposable summary of it. A system built that way does not so much survive a crash as fail to notice one. One proviso: the journal lives on storage that outlives the process.
The production form of the idea is the checkpoint: a named cut through the run at which the recorded state is complete and consistent, from which execution can lawfully resume — named because resumption, forking, and audit must say which moment they mean; complete and consistent because a mid-turn cut resumes into an argument with reality, so the lawful cuts fall at turn boundaries, where the state tells no lies. New is the checkpoint’s promotion to the chapter’s organising object (Figure 20.1): the human gate of Section 20.5 is a checkpoint waiting for permission to resume, time travel in Section 20.6 is navigation between checkpoints, a deploy in Section 20.8 lands on one.
Recovery, on this design, is not a procedure but a re-reading: state was always a fold over the journal, so recovering is folding again, and starting fresh is merely the special case of an empty journal. That collapse of two paths into one has a name: crash-only software (Candea & Fox, 2003). A system with a separate recovery path keeps its least-tested code for its worst day; a system with only the crash path tests its recovery on every start — so stop writing shutdown choreography and make startup-from-journal the only way the system starts. The consequence, which the finale enforces: if kill -9 is not a routine event in your test suite, your recovery path is a rumour.
Replay has a boundary. Folding the journal re-derives beliefs — tasks, statuses, balances — but the world does not replay: the email was sent, the payment captured, and a recovery that re-executes those has turned a crash into an incident. So split state along the line of reversibility: everything internal may be recomputed freely; every effect on the world is recorded in the journal as a fact — issued, confirmed, failed — and on replay those records are read, never re-run. This is the first appearance of the distinction the whole chapter turns on — between what a system can take back and what it can only account for — and Section 20.2 meets it again within a single call, where it goes by the name idempotency.
Then the day Chapter 17 marked arrives: Chapter 18’s parallelism is switched on, two subagents finish at once, and both write to the shared state — Chapter 8’s lost update, in your own stack trace. The reducer meets it: because updates are events and state is a fold, merging parallel branches is two event streams folding into one state, the merge policy explicit field by field — findings can append from both branches in either order; a status field cannot, and someone must decide whether last-write-wins or the conflict escalates. The reducer converts the merge from a race you lose at runtime into a decision you record at design time — all Chapter 8 ever asked.
This is why the teaching framework looks the way it does. LangGraph’s core objects are the section’s concepts with an ops team attached: the state schema is the typed state; a reducer annotation on each field is the merge policy made a type-level fact; the checkpointer is pluggable persistence for the named cut; and a thread is a run, resumable from its latest checkpoint because that is the only way threads continue at all — crash-only, off the shelf. The reader who built these by hand can tell what the framework is doing on their behalf, and what it is not. The whole per-field merge policy lives in the type itself:
import operator
from typing import Annotated, TypedDict
class TeamState(TypedDict):
findings: Annotated[list[str], operator.add] # append from both branches
status: str # bare: one writer per superstep, or the merge raises
turn: int # single-writer counterThere is no graph here, only the schema, and that is the point: findings accumulates from both subagents by declaration, a conflict over status escalates at the merge, each field’s policy a fact read from the type, not an accident of arrival order.
The case study collects its first upgrade: checkpoints at every turn boundary, state split into the re-derivable and the effectful — test results recompute; pushed commits are facts — coder and tester in parallel, findings merged by a reducer. Kill the process twelve minutes into the overnight run and the morning log shows one line of interest: resumed from checkpoint, nothing lost, nobody paged. Boring, as promised. What the journal cannot fix is the failure that happens while the process is perfectly healthy — the call that hangs, the retry that might have gone through the first time — the next section’s inventory.
20.2 Calls That Fail: Timeouts, Retries, and Idempotency
A production run is a long chain of remote calls — to the model, to tools, to counterparties across Chapter 19’s boundary — and each can fail outright, hang indefinitely, or succeed without managing to say so. Multiply a modest per-call failure rate by an overnight run’s hundreds of calls and the conclusion is arithmetic: partial failure is the steady state — and the model call is plausibly the flakiest, slowest, most expensive remote procedure call ever put into mass production, sitting in the loop’s inner ring.
The reason retrying works at all was established by Jim Gray from Tandem’s field data in 1985 (1985): most production faults are transient — retry and the operation succeeds, the triggering conditions having moved on. He called them Heisenbugs, for their habit of vanishing when examined, as against the deterministic Bohrbugs that fail identically every time; retries are cheap insurance against the common case, useless against the rare one. The agent stack adds a wrinkle: a stochastic component turns every fault into something like a Heisenbug, which tempts teams to retry not on failure but on dissatisfaction. Mark the difference well: retry-on-failure converges because failures are rare; retry-until-the-output-pleases-you is an unbounded loop with a credit card, and belongs to the next section’s jurisdiction.
A timeout looks like a number and is in fact a policy about work you can no longer see. When the test-runner call is abandoned at thirty seconds, it may have failed long ago, may never have arrived, or — the case that bites — may still be running, its ghost completing off-stage, its effects landing in a state that has moved on, its duplicate already in flight. Agent workloads make the number miserable to choose — a legitimate test suite takes thirty seconds or twenty minutes — which is why timeouts want layering: tight per call, looser per turn, a task deadline, a run-level curfew, each set where someone can name the consequence of firing it.
Which returns us to idempotency: Chapter 19 met it at the wire; dependability brings it home, because your own harness is about to retry against your own tools. The discipline is the reversibility split within a single call. Reads retry freely. Reversible writes retry under an idempotency key — a stable identifier the tool remembers, so the duplicate becomes a no-op with a clear conscience. Irreversible effects — send, pay, push — get the full ceremony: record the intent in the journal, execute, record the outcome; and on any doubt, query before re-execution — “did the payment go through?” costs nothing to ask, while re-sending the payment to find out is an experiment with your customer’s money.
Retrying also has manners. A failing service is frequently a struggling service, and a fleet retrying in lockstep is a denial-of-service attack thoughtfully launched against your own supplier — hence exponential backoff with jitter, and hence the circuit breaker, which stops calling a persistently failing dependency and probes it occasionally. And because every retry has a price, retry policy is spend policy: a retry budget per task, so persistence is a decision the accounting can see. Retry-with-backoff and the idempotency key fit together in a handful of plain Python:
import random
import time
from collections.abc import Callable
seen: dict[str, str] = {} # the idempotency-key cache
faults = [TimeoutError("gateway slow")] # one transient Heisenbug
def charge_card() -> str: # each call: one ToolDispatched
if faults:
raise faults.pop()
return "charged 1500p"
def deliver(key: str, fn: Callable[[], str], n: int = 3) -> str:
for k in range(n):
try: # success: ToolReturned, cached
return seen[key] if key in seen else seen.setdefault(key, fn())
except TimeoutError: # transient: back off with jitter
time.sleep(0.2 * 2**k + random.uniform(0, 0.2))
raise TimeoutError(f"retries exhausted: {key}")
deliver("pay-42", charge_card) # -> 'charged 1500p' (one retry, then ok)
deliver("pay-42", charge_card) # -> 'charged 1500p' (served from cache)The first call pays for two dispatches — the Heisenbug, then the backed-off retry that lands — the duplicate only for a dictionary lookup. One dishonesty in the toy is left standing on purpose: charge_card is the full-ceremony class — pay — wearing only the middle class’s remedy, and its seen cache lives in the harness’s process, where a crash erases it. Exercise 2 exhibits the double charge that follows and performs the repair the ceremony demanded — intent and outcome in the journal, and query before re-execution.
Some tasks, finally, are cursed: the input hits a Bohrbug, retry-proof by definition. Retry budgets make the discovery cheap, and the dead-letter queue is where it is honoured: the task parked, its journal thread attached, awaiting a human — automation admitting, in structured form, that this one is beyond it. The case study’s flaky test runner now has a policy: two retries with backoff on a timeout or a crash, none on a clean deterministic failure, then ERROR, then the dead-letter shelf. What none of this restrains is the agent whose calls all succeed — and for that the system needs not retries but governors.
20.3 Governors: Budgets, Rate Limits, and the Runaway
The failure class this section exists for involves no failures at all. Every call succeeds; every step is individually plausible; and the agent is still working at six in the morning, deep into the month’s tokens, because nothing in it knows how to be finished — the pathology Chapter 3 flagged, billed at production prices. A stochastic generator never runs out of plausible next steps, so termination is a constraint the harness must impose. The name for such a constraint is old and mechanical: the governor, Watt’s spinning contraption bolted to the steam engine — bolted on, because the engine has no opinions about its own speed. Neither does the agent.
Governors come in two kinds, and each is blind to what the other sees. A budget is a stock constraint — so much and no more — and the runtime’s single token allocation generalises into a small treasury: tokens, money, wall-clock time, tool invocations, turns, allocatable per run, per agent, per task. A rate limit is a flow constraint — so fast and no faster — and not a budget in miniature: a run can be comfortably inside its budget while hammering an API at a rate that gets the organisation’s key revoked, and politely inside every rate while bleeding the budget dry. One rule above all: limits are enforced at the harness, at the metering points the journal already owns, because an instruction in the prompt — “please stay within budget” — is a hope, and the whole point of a governor is to be a mechanism. One bookkeeping subtlety: tokens and money are different columns — a cached re-read bills at a fraction of the fresh rate (Section 1.5 priced the mercy) — so where the invoice is what the governor guards, meter spend at the same journal points.
Enforcement has manners, and the runtime set them: exhaustion triggers wind-down at the next turn boundary, never a mid-turn kill, so what halts is a consistent, resumable, checkpointed run. And exhaustion is not failure but a decision point: a checkpoint plus an invoice, whose holder may top up (an approval — the next section’s gate), bank the partial results, or shelve the task with the dead letters — a judgement about value, not tokens, which the budget’s job was only to force now, with the meter stopped.
Budgets do a second job, and Chapter 9 explained why: the shared token pool is a commons, and the book’s own Prisoner’s Dilemma showed each agent’s locally sensible appetite grazing it bare by lunchtime. Subdivided budgets are the enclosure of that commons — each agent funded from its own paddock — with the allocation the fiat planner at team scale, and Chapter 12’s market waiting for the fleet scale where the planner can no longer read every status report.
Rate limits run in both directions. The world imposes limits on you — the provider’s quota, the tool API’s ceiling — and its refusals are ordinary responses for Section 20.2’s machinery: backoff, jitter, patience. The production trap is multiplication: eight subagents dispatched in the same instant are a burst no per-agent politeness prevents — admission control belongs at the harness, where the concurrency is created. The outbound direction is your own statute book: caps on concurrent agents, on spawn depth — a supervisor that can spawn supervisors is an org chart with the growth characteristics of a fork bomb — and on turns per task.
Two honest limits, and the case study collects its kit. A governor bounds damage; it does not detect futility — the agent that stays under every cap while re-reading the same file and polishing the same paragraph is safe, cheap, and useless, and noticing that the bounded bill bought nothing is a seeing problem, deferred to Section 20.6. And every limit is a tuning judgement: caps set tight, raised on evidence. So equipped — a run budget, per-task allocations, a two-agent concurrency cap, spawn depth of two, polite backoff — the opening’s 3 a.m. token drain now ends, at worst, as a checkpoint and an itemised invoice waiting for morning. What the governors cannot do is make the invoice smaller. A cap is a ceiling, not an economy — the difference between a system that stops at its budget and one that needs less budget is design — and the design levers deserve their own page.
20.4 The Bill: Cost as a Design Property
Three levers set what a fleet actually spends: which model fills each seat, what shape the context takes, and what the clock is allowed to cost. The largest and least exercised: nothing in a topology requires its members to be the same mind, and model tiers differ in price by an order of magnitude, while much of what a fleet does all day (routing, extraction, reformatting) is exactly what the cheap tiers do adequately. The discipline is model tiering: reserve the frontier model for the judgement points and staff the mechanical seats down-tier, with Chapter 21’s harness as arbiter, since adequately is a measured property: swap one seat down, run the suite, read the quality delta against the invoice delta. Tiering also redraws Chapter 18’s price list: patterns differ in cost by integer multiples at a fixed model, and the multiples shrink wherever fan-out seats can be staffed cheaply — a debate of three frontier seats is a luxury; three cheap seats before a frontier judge is a defensible instrument. Nor is the assignment permanent: every substrate generation moves what the cheap tier does adequately, so the tier map joins Section 20.8’s re-evaluation.
The second lever is the shape of the context, because the cache has opinions about architecture. The mercy Section 1.5 priced — a re-read prefix billed at a fraction of the fresh rate — has a geometry: it discounts a stable, shared prefix and nothing else. A supervisor whose subagent briefs open identically re-reads at the discounted rate all day; a peer network in which every member accretes a divergent history caches almost nothing. Cache-friendliness is thus an architectural property, decided where briefs and context policies are written: keep the stable parts of the window at the front and append-only, the per-seat variation at the back, and never edit the middle of a prefix mid-run without pricing the invalidation. Underneath sits an asymmetry that has held across vendors for years: input tokens are cheap, cached input cheaper still, and output tokens cost a multiple of both. The coordination tax is minted mostly in output — briefs, summaries, status traffic — which is why Section 1.5’s caution survives every price cut: the cache rescales the re-reading and leaves the talking at full price.
The third column is the clock. A pattern spends latency by depth — every sequential hop is a model call somebody waits through, which is why Chapter 18’s integration-heavy shapes feel slow out of all proportion to their token bills — and parallel shapes buy the clock back with tokens. Where the system faces users, the trade acquires a contractual name: a latency service-level objective is a design constraint on topology, met by flattening the critical path or paying for parallelism, never by exhortation. And at fleet scale the provider’s rate limit is a shared ceiling, so fleet size is a capacity plan — the admission-control semaphore of Section 20.3, times tokens per call, against the quota (Exercise 3 works it). None of this is invoice-reading: tier map, prefix shape, and critical path are set where the architecture is set — the bill belongs among things designed, not suffered. Which leaves the opening’s other 3 a.m. incident: acting without asking is the gate’s.
20.5 The Human Gate: Approvals and Interrupts
Some actions can be undone by recomputing a fold; some can be undone with a compensating transaction — Chapter 7’s abandon coherently, paid for here: the compensating action designed alongside the action it undoes, journalled like any other effect, so abandonment is a first-class outcome with a ledger rather than a shrug. And some can only be prevented. For the action that cannot be taken back — the customer email, the production deploy, the payment — the only dependable moment of control is before, and where the judgement exceeds what the system can be trusted with, before means a person. The human gate arranges it: a designated point at which the run halts and a human decides whether the world changes — Chapter 18’s who may say done, and who may say stop, answered with a name from the org chart.
The insight that makes gates practical: an approval is a long-lived checkpoint — persistence, not user interface. The naive gate blocks a process on an input prompt — which works until the approver is asleep or the container is rescheduled, and the run and the request die together; hence systems without durable state gate nothing, and the demos never pause. The proper gate is built from section-one machinery: the run reaches the gate, checkpoints, and stops existing as a process; the request becomes a durable item in a human’s queue; and the approval, whenever it comes, is simply an event — fold it in, resume from the cut. Ten seconds or ten days, the machinery is identical — which is what a framework’s interrupt primitive is: LangGraph’s is a checkpoint plus a pending-input marker.
Stripped to a list-of-events journal, the whole gate is a dozen lines:
journal: list[tuple[str, str]] = [] # (event, action) pairs, append-only
def run_to_gate(action: str) -> str:
granted = ("ApprovalGranted", action) in journal # a fold over events
if not granted:
journal.append(("ApprovalRequested", action))
return "parked"
return f"deployed {action}" # past the gate, the effect fires
def approve(action: str) -> str:
journal.append(("ApprovalGranted", action))
return run_to_gate(action)
run_to_gate("release-v2") # -> 'parked' (ApprovalRequested logged)
approve("release-v2") # -> 'deployed release-v2'The pause between the two calls may be ten seconds or ten days; nothing in the code can tell the difference, because a park is only a fold not yet resumed.
What to gate is a policy question with a failure mode on each side, and the expensive mistake is the conscientious one. Gate nothing and the 3 a.m. action goes through unsigned. Gate everything and you manufacture approval fatigue: the human asked to bless forty routine diffs a day approves them the way everyone accepts terms and conditions — instantly, unread — and the rubber stamp is strictly worse than no gate at all, because the system now records that a human checked when no one meaningfully did: responsibility laundered, not exercised. The calibration follows the chapter’s own split: gate by consequence class — the irreversible, the expensive beyond a threshold, the outward-facing, the policy-sensitive — and let everything reversible run free behind checkpoints and audit. A gate is scarce because human attention is; spend it where prevention is the only remedy.
The gate’s second design surface is what the approver sees, and the standard is easily stated: consent to an unread plan is not consent. An approval request must carry what will happen, concretely — the diff, the recipients, the amount, not “proceed with proposed changes?”; why, briefly; what it costs — the budget delta the action commits; and what declining does. And the decline must be a designed outcome, not an exception: the runtime’s REJECT performative exists for this very case, and a refused action returns to the run as information the agent can replan around — the human said no, the reason attached — not as a stack trace.
Interrupts are the gate’s mirror: the approval is the system asking the human; the interrupt is the human breaking in, at two speeds, both already paid for. The civil interrupt stops at the next turn boundary — the budget wind-down’s grace. The emergency stop is kill -9 — and the quiet triumph of the crash-only design is that the emergency stop is a crash, and crashes are now boring. Killing the process works as a stop because it pauses the world — the repository sits unchanged while the run is dead. Where the world keeps moving, the equivalence fails: a robot’s emergency stop cannot be a checkpoint, because momentum does not park at a turn boundary — the e-stop of Section 4.6’s world must pre-empt the software rather than petition it. The gate and the civil interrupt are luxuries of environments that hold still. Between the two speeds sits the manoeuvre that makes interrupts more than brakes: pause, inspect the state, amend it, resume — the first taste of an idea Section 20.6 completes: on a journal, the past is not merely visible but editable, forward from any cut.
One more thing the gate produces outlasts the run: a record — this person, shown this evidence, consented to this action at this time — held in the same file as everything else, so the audit view can answer “who approved this and on what evidence?” without a separate compliance system; what the record means is Chapter 22’s territory. The gate also closes a loop Chapter 19 opened: attenuated authority runs out where the action exceeds the credential, and the gate is where authority is topped up — deliberately, visibly, one action at a time — rather than left ambient in the deputy.
The case study, so armed, becomes hard to alarm: the deploy step and anything that pushes to main are gated by consequence class; the overnight run parks a release request at 2.14 a.m. — three-line summary, diff attached, budget delta noted — and goes quiet; a human approves it over coffee at 9.04; the run resumes as though the intervening seven hours were a slow network. The unsigned 3 a.m. deployment is no longer a discipline problem but a structural impossibility: the agent does not hold the credential to deploy, and the gate is the only place the credential is issued. What the chapter has assembled can crash, retry, stay solvent, and ask permission — the remaining question is how anyone knows what it did all night.
20.6 Seeing It Run: Observability and Time Travel
Chapter 17 ended with a discovery this section promotes to a thesis: observability was never a separate service — it is the journal, viewed. Production multiplies the views: the trace, one run rendered as a tree of turns, calls, and tool spans; the metrics, aggregates rolled up across runs; the audit, the record of authority. Three dashboards, one file — and the architecture matters more than the tooling: the industry default is a bolt-on tracing SDK sampling some calls into a vendor’s silo — two accounts of the system that need never agree. Derive every view from the journal — exportable under OpenTelemetry’s generative-AI conventions1 — and disagreement is structurally impossible, because none of the views is the truth; they are all just the truth, viewed.
What belongs on those dashboards is cognitive rather than transactional: tokens per turn and its trend — the drifting-upward context that says summarisation has quietly failed; tool-call mix — the run that has stopped editing and started endlessly re-reading. Here the debt from Section 20.3 falls due: futility detection is a seeing problem, solved by computing continuously the signatures Chapter 18 attached to each pattern’s characteristic failure — shrinking diffs against decaying critique specificity is placation in progress; leaf results diverging from the root’s summary is the whisper game, measured. Forensic marks in the catalogue become metrics with thresholds on the dashboard, and the pathology pages you before the invoice does.
Debugging enters when a number goes wrong, and with a stochastic component the question is not “which line is broken?” but “what did this agent know, and when did it know it?” The investigator’s first need is the context as the model saw it, byte for byte, which a production journal’s ModelCalled events must be made to hold — the same one-field enrichment Chapter 17’s Exercise 3 performs on their ModelResponded siblings. The second need is reproduction, which stochastic components refuse — except that the runtime already bought the cure: replay the recorded responses through the fake client and the incident re-lives deterministically. Production makes that a pipeline: every incident worth an hour of a human’s time is bottled — journal excerpt, expected behaviour, observed behaviour — and joins the regression suite, so the system’s worst days accumulate into its examination syllabus.
Time travel is what the checkpoints were quietly assembling all chapter. Because every turn boundary is a named cut, the past is addressable: rewind to any checkpoint and inspect the state as it stood. And because state is a fold, the past is also forkable: rewind, amend one thing — the brief that omitted the deadline — run forward again, and you have the counterfactual ordinary debugging can only gesture at: would it have gone right if—? Debugging stops being archaeology and becomes navigation. LangGraph ships the manoeuvre as checkpoint history on every thread — list the states, update one, resume from it as a fork.
Honesty requires the bill, and it has two lines. Storage: retention is set by use — full fidelity for the recent window where debugging lives, snapshots and aggregates beyond it — pruned by age, never by sampling away parts of a live run’s story. Liability: a journal that holds everything holds everything — customer data, credentials in tool results — so the instrument of audit is itself an object of governance, with access controls and redaction of its own; Section 20.9 will treat the journal as attack surface and Chapter 22 as evidence, and both readings are correct. Against these costs stands Section 16.4’s asset ledger, fully cashed: the traces survive every migration, and the journal schema you own, with its distilled incident suite, is the system’s operational knowledge in portable form.
The case study’s morning ritual: a dashboard read over the same coffee as the approval queue, and one anomaly — task seven’s diffs stopped shrinking at 3 a.m. and the polish-loop pager fired; the culprit — a critic gone vague after a context compaction — bottled into the incident suite by Tuesday, the fix verified by forking Monday night’s checkpoint and watching the loop converge. Seeing is solved by the same object that solved surviving. But to a tradition older than any of this tooling, everything assembled so far shares one quiet limitation: it samples where it could, in principle, enumerate.
20.7 Proving It: The Verification Inheritance
The chapter’s disciplines are all, in the logician’s sense, inductive — a test samples the paths through a system, a canary samples production, even the chaos day injects only the failures somebody thought to list. The classical field’s most rigorous wing spent three decades on the deductive alternative. Model checking states a property in a temporal logic — the system never enters a forbidden state — and exhaustively explores every reachable state of a finite model, returning the verdict or a concrete counterexample trace (Clarke et al., 2018): the suite visits some paths, the checker visits them all. Nor is it a curiosity: it ships in hardware design and protocol standardisation, and earned its founders a Turing Award.
The multi-agent field enriched the logic. Temporal-epistemic logics marry the clock to Chapter 6’s knowledge operators, so the property checked can read “after the broadcast, every agent knows the branch is clean — and knows that the others know” — Chapter 6’s puzzles industrialised, with the model checker MCMAS built at this junction (Lomuscio et al., 2017); Chapter 6’s Exercise 6 has the reader build the evaluator at desk scale. The strategic extension asks, formally, the questions Part VI has been asking in prose: alternating-time temporal logic adds a game-theoretic modality — coalition C has a strategy to ensure \varphi, whatever the others do (Alur et al., 2002) — and strategy logic treats strategies as first-class objects, so that even the existence of a Nash equilibrium becomes a checkable formula (Chatterjee et al., 2010). The fit is uncanny: can the supervisor guarantee termination whatever its workers do is an alternating-time formula; can two compromised agents jointly force a deploy is a coalition query — Section 20.9’s blast radius, asked with mathematical manners.
The price of the method is stated in its first noun: a model — a finite transition system whose states and moves are known — and the classical field could pay it because its agents were programs small enough to be their own models. The agents of this book decline to supply any such object: the policy at the centre of every turn is a set of weights with no finite description, and the properties most worth proving — it will not assert what the journal contradicts, it will not treat data as instruction — quantify over meanings, which no state enumeration reaches. The guarantees were proved about agents whose insides were inspectable, and inspectability is precisely what the new substrate withdrew.
What transfers is the half this chapter has been building all along, because the harness is exactly the object the tradition requires. The gate, the governors, the reducer, the stop conditions are ordinary code — finite, deterministic, enumerable — and the properties that matter most in production are properties of that envelope, not of the mind inside it: no deploy without a matching approval event is a safety property, spending never exceeds the treasury an invariant, every run ends done, dead-lettered, or parked at a gate a liveness claim these logics were invented to state. Some hold by construction, verification’s cheapest victory: the unsigned push is not forbidden but unrepresentable — Chapter 13’s regimentation wearing its proof on the outside. The division of labour, in four words: verify the cage; monitor the animal.
Monitoring the animal is itself a formal discipline. Runtime verification states the property once and derives from it a monitor — an automaton that watches the event stream and flags the moment the property is violated (Leucker & Schallhart, 2009). The journal is that event stream, and Section 20.6’s dashboards are runtime monitors built by hand; the formal version derives the monitor from a property with a semantics, so whether the dashboard actually checks what everyone believes it checks has, for the first time, an answer. The enforcement variant: shield synthesis constructs, from a stated safety property, a small verified component between a system’s decisions and its actuators, overriding only when the property is about to break (Bloem et al., 2015) — a guardrail with a proof attached, the standard the word guardrail is borrowing whenever it wants to sound load-bearing.
The honest bill: verification buys properties of the envelope, never of the judgement — a fully verified harness will still ship mediocre code, politely, within budget — and every proof holds of the model at hand, whose fidelity to the running system is a premise the proof cannot secure. Proofs shrink the space in which accidents can hide without ever emptying it, which is why the verified gate still gets broken on purpose. But between tested, and it passed and proved, within these stated limits there is a difference in kind, and a field that inherited four decades of the second should not pretend it only ever had the first. What remains is the least glamorous step — getting the machine into service, and then trying, deliberately and repeatedly, to break it.
20.8 Shipping It, and Breaking It on Purpose
Deployment tooling quietly assumes something agent systems do not have: short requests. A stateless web service deploys by fleet-swap, every request living milliseconds inside one version or the other. An agent system’s unit of work is a run: hours or days long, checkpointed, possibly parked at a gate since Friday — and the run, not the request, is what must survive the deploy. New code will resume old state, and deploying becomes an exercise in continuity — surgery, with the patient not merely awake but halfway through a sentence. Version skew — Chapter 19’s permanent condition — is therefore a domestic matter too: a system whose runs outlive its releases is its own counterparty across time. Chapter 17’s schema versioning was built for this hour — events carry their version, the reducer keeps its old cases, a journal written in June still folds in December — and production extends the discipline to the artefacts engineers forget are versioned: prompts and briefs are code — change one mid-run and the agent that resumes is a different colleague from the one who left for the checkpoint — and model versions are substrate, pinned per run, upgraded at run boundaries. The deploy policy reduces to an explicit choice: drain — let running work finish under the code that started it — or migrate — fold the old journals forward under the new version — with drain the default and migrate the manoeuvre you rehearse before you need it.
Environments are the deployment story’s awkward middle (Chapter 4 said why): for an agent, the environment is not scenery but subject matter — the repository’s real history and the data’s real mess are the task — so a staging environment is, by construction, a simpler world. The response is not a better staging but a ladder of them (Figure 20.2): the fake client, where logic is tested for pennies; replay, where the incident suite is re-lived; a staged tier with production-shaped data and sandboxed credentials; and the canary — one repository, tight budgets, every gate armed, trust extended the way Section 20.3 said limits are tuned: on evidence, one notch at a time.
%%{init: {"theme": "base", "themeVariables": {"primaryColor": "#E8ECFF", "primaryBorderColor": "#4054B2", "primaryTextColor": "#16171B", "lineColor": "#3B4351", "edgeLabelBackground": "#FAF7F0", "clusterBkg": "#EFE9DC", "clusterBorder": "#766F65"}, "flowchart": {"rankSpacing": 28, "nodeSpacing": 42}}}%%
flowchart BT
subgraph L[" "]
direction BT
A["Fake client<br/>logic tested for pennies"]
B["Replay<br/>the incident suite, re-lived"]
C["Staged<br/>real-shaped data,<br/>sandboxed credentials"]
D["Canary<br/>one repo, tight budgets,<br/>every gate armed"]
A -->|"add recorded reality"| B
B -->|"add real-shaped data"| C
C -->|"add the real world,<br/>throttled"| D
N["fidelity, stakes, cost ↑"]
B ~~~ N
end
style L fill:none,stroke:none
classDef world fill:#EFE9DC,stroke:#766F65,color:#16171B
class N world
One deploy arrives on nobody’s release calendar: the substrate’s. The model under every seat upgrades on the vendor’s schedule — Chapter 21 will count what that does to evidence — and fleet policy is a genuine trade. Pin a named model version and the system is stable, reproducible, and quietly ageing — and the pin is a lease, since providers retire checkpoints on their own timetable: pinning decides when to move, never whether. Float on the vendor’s default and the system is current and never twice the same, every silent upgrade an uncontrolled experiment on production traffic. The defensible posture is the boring one: pin, and schedule the float — treat each vendor release as a deploy whose date you choose. The cutover then gets a deploy’s ceremony: run the home suite under the candidate beside the incumbent, the system held fixed, so the quarter’s improvement is finally attributed, to the provider or to you; recalibrate the judges, because the examiner and the examined may both have changed; re-run the incident suite live, since the bottled failures are precisely where the old model’s habits were load-bearing; then the canary rung before the fleet follows, the old pin kept warm for a rollback. The numbers will move; the discipline is that they move in a lab notebook rather than in production folklore. And one architectural question rides along, because Section 18.5 posted it: a new model generation is exactly when a coordination seam may have stopped earning its keep, so the drill ends by re-asking Chapter 1’s framework.
The last discipline is the one the section title promises: machinery exercised only by accident is machinery you do not actually know you have. The recovery path proved itself only because the test suite crashes the process on purpose; the same standard now applies to everything; the practice has a name in chaos engineering: inject the failure deliberately, in production-shaped conditions, on a schedule, and treat any surprise as a defect in the machinery rather than bad luck. The streaming industry made the idea famous by killing its own servers during business hours; the agent stack inherits the principle with a shorter kill list and higher stakes per item.
So the first movement ends with the case study’s scheduled bad day, every injection drawn from the inventory the opening promised. Kill the process mid-turn: the run resumes from its checkpoint, the only casualty the tail of one turn. Hang the test runner: the timeout fires, the retry backs off, the second hang sends the task to the dead-letter shelf. Deliver a tool result twice: the idempotency key swallows the duplicate without comment. Exhaust the budget while the release request is parked at the gate: wind-down at the boundary, checkpoint taken, the approval still waiting; Monday’s top-up resumes both. Deploy mid-run: the parked run wakes under newer code, folds its versioned journal, and continues as if nothing happened — because, structurally, nothing did. Interrupt the lot at 3 a.m. for the exercise’s own sake: a civil stop at the turn boundary, everything resumable. Six injections, six entries in the morning log, none requiring a human before nine. The telos, demonstrated rather than asserted: a dependable system is one whose failures are boring. And behind every mechanism sits one object: survival is folding the journal, merging is folding two branches of it, the gate is a pause in it, seeing is viewing it, time travel is navigating it, a deploy is resuming it under new management — what “the truth lives in one place” buys, for any stack, framework, or decade.
20.9 Reliability, Security, and Safety
The system so far was hardened against a world that meant it no harm: the failures were accidents, and the users were people trying to get work done. Some of the visitors are not. The adversary reads your rubrics, studies your judge, and treats your ninety-fifth percentile not as a tail risk to be endured but as a target to be hit on purpose. The first movement hardened the system against an indifferent world; this one hardens it against a hostile one, and then against a danger subtler than either: the system that harms while working as designed.
Those three worlds are the chapter’s three words. Reliability, here, is dependability’s darker sibling — how far a failure spreads once something has gone wrong, because a plurality of agents is a plurality of blast radii. Security is the discipline of the deliberate adversary: the visitor whose inputs are crafted rather than sampled, whose profession is to be off the distribution you evaluated. And safety is the one the other two do not cover — harm that needs no adversary and no bug, arising from a collection of individually well-behaved agents that misbehaves collectively, as Chapter 15 taught us to expect. The progression runs from the accidental to the malicious to the emergent, each layer harder to reason about than the last.
The uncomfortable truth: the classical security playbook was written for systems whose trusted computing base could be trusted — a processor that does exactly what its instructions say. An agent’s processor is a language model: suggestible, stochastic, unable to tell an instruction from a description of one, and installed at the very heart of the trusted base. You cannot validate your way to safety when the thing being protected is also the thing most easily talked into betraying you — though what surrounds it is another matter: the scaffold admits ordinary engineering proof, and defence in depth is largely the art of moving the load from the component that admits no guarantee onto the ones that do. The honest posture is not a proof but a discipline: threat-model before you build, deny privilege by default, assume every layer will eventually be breached, and arrange that when one is, the damage is bounded and the breach is seen. Throughout, the question is whether the system can be stopped from going wrong — ending, in Section 20.14, with the harm no adversary sends; who answers when it does anyway is Chapter 22’s, and the book’s last.
20.10 The Adversary the Book Deferred
Part IV withdrew the cooperative courtesy but kept its manners: its agents pursued their own ends, yet the change needed no villains. This chapter admits the villain: a party who wants your system to do something you did not intend — leak the document, wire the payment, delete the branch — and who crafts the inputs to make it happen. The first discipline is to think of this party not as a person but as a goal: the adversary is whoever benefits from your system misbehaving, and they reach it not by breaking down a door but by being one of the things it reads. The user angling for an unowed refund, the web page that would like your credentials, the competitor who covets your system prompt — three objectives, not intruders, and a defence built around objectives outlasts one built around the intruder you happened to imagine.
The work begins before any control is chosen, with the routinely skipped exercise of threat modelling: writing down what is worth protecting, who might want it, and where they can reach it. The assets are more numerous than “the data” — credentials, budget, actions, the reputation that answers for all of it; the adversaries are ranked by what they can spend — the opportunist with one clever paragraph, the professional with a week, the insider with a foothold; and the entry points are wherever untrusted input meets trusted machinery. Security that starts by bolting on controls has bought locks before finding the doors; the OWASP Top 10 for LLM applications — prompt injection at its head in both editions to date, the runaway’s unbounded consumption closing the list (OWASP Gen AI Security Project, 2024) — names the doors, in this book’s terms, and the chapter is about the building.
The attack surface is where agent security parts company with the ordinary kind. A conventional program has a countable set of inputs; an agent’s ingress points are its every contact with the world (Figure 20.3) — prompt, retrieved document, tool result, peer message, memory recall — each an avenue by which text of unknown provenance enters the machinery that acts. All of it is untrusted until something earns it trust, and the earning is the engineering. The multi-agent setting multiplies every surface by the number of agents: the plurality Chapter 1 sold as reach is, examined from the far side of the wire, precisely more reach for the attacker — capability and exposure are the same surface seen from opposite sides.
%%{init: {"theme": "base", "themeVariables": {"primaryColor": "#E8ECFF", "primaryBorderColor": "#4054B2", "primaryTextColor": "#16171B", "lineColor": "#3B4351", "edgeLabelBackground": "#FAF7F0", "clusterBkg": "#EFE9DC", "clusterBorder": "#766F65"}, "flowchart": {"rankSpacing": 28, "nodeSpacing": 42}}}%%
flowchart TB
Peer["Peer<br/>message"]
subgraph IN["Untrusted ingress"]
P["Prompt"]
R["Retrieved<br/>document"]
T["Tool<br/>result"]
Peer
Mem["Memory<br/>recall"]
end
M["Model --- least trusted,<br/>most central"]
G(["Authorisation boundary:<br/>may this proposal act?"])
ACT["Actions on the world:<br/>tools, data, money"]
OTH["Other agents<br/>(same surface each)"]
Peer --> M
P --> M
R --> M
T --> M
Mem --> M
M -->|"proposes"| G
G -->|"authorised"| ACT
ACT --> OTH
OTH -.->|"injected<br/>peer message"| Peer
P ~~~ Peer
R ~~~ Mem
classDef world fill:#EFE9DC,stroke:#766F65,color:#16171B
classDef gate fill:#F7E6B5,stroke:#8A5A00,color:#16171B
class P,R,T,Peer,Mem,OTH world
class G gate
linkStyle 8 stroke:#A4161A,color:#A4161A
The classical trichotomy — confidentiality, integrity, availability — still frames the stakes, but integrity breaks the mould. In ordinary software, integrity rests on a processor that executes faithfully and can be subverted only by finding a bug. In an agent, the processor treats instructions discovered in its input with very nearly the deference it gives its operator’s, so integrity can be violated with no bug whatever, by persuasive prose: the attacker need not defeat the machine, only talk to it, and the machine was built to be talked to.
That fact makes the classical playbook necessary but nowhere near sufficient. The trusted computing base — the components that must be trustworthy for any security property to hold — is, in well-built systems, kept small, audited, and boring. An agent’s trusted base contains the model: large, opaque, stochastic, suggestible — the least trustworthy component in the assembly installed at the dead centre of the trust. Danger then concentrates wherever three exposures meet in one agent — access to something worth taking, exposure to input an attacker can write, and a channel by which data can leave — because an agent holding all three can be instructed, by its own input, to take the valuable thing and send it out the door: Simon Willison’s lethal trifecta (2025). Most useful agents are built with all three on purpose; the response is not to build useless agents but to assume the centre is corruptible and engineer outward. Of the three exposures, the exit is the one ordinary engineering can most nearly close without unemploying the agent: deny the sandbox the network by default and allow-list the few destinations the task genuinely needs — an egress allow-list is Chapter 4’s chokepoint drawn around what may leave, and usually the cheapest leg of the trifecta to break. And the corruption has a canonical mechanism — text that enters as data and is obeyed as command — which is the next section’s entire subject.
20.11 Instructions and Data: The Boundary That Was Never There
Prompt injection is the vulnerability that defines the field, and its mechanism is embarrassingly simple: a language model receives a single stream of tokens, and no token carries a bit marking it instruction to be obeyed rather than data to be reasoned about. System prompt, user request, retrieved page, tool result, peer message — all arrive as text in one window, attended to by the same machinery. The separation between instructions and data, firm in every operator’s head, is a distinction the substrate has no way to represent. Chapter 6 met this as a matter of provenance; prompt injection is the same defect turned into a weapon: text that entered as data, obeyed as command.
The comparison the field reaches for is SQL injection, and it is instructive precisely where it breaks down. SQL injection was, in essence, solved: parameterised queries gave the database two channels, one for code and one for data, so untrusted input could never be promoted into instruction. The agent has no such remedy, because there are not two channels to keep apart: the model’s entire power is that it reads instructions written in ordinary language, and its vulnerability is that same power, inseparable. You cannot parameterise a prompt — the “parser” is a stochastic model performing inference over one undifferentiated stream. Prompt injection is therefore not a defect a patch will close; it is a property of the kind of system, and the engineer who waits for the vendor to fix it will wait for ever; the engineer who accepts it starts building the containment that actually helps.
The vulnerability comes in two grades. Direct injection is the user supplying the hostile instruction — the notorious “ignore your previous instructions and…” — an attack at least in plain sight. Indirect injection is the alarming one: the hostile instruction rides in on content the agent consumes in the course of honest work — a retrieved page, a summarised document, an email in the queue. Greshake and colleagues demonstrated the class against real, deployed systems — data exfiltrated, compromise persistent, instructions propagating like a worm (2023). The multi-agent setting turns the vulnerability into an epidemic vector, because every inter-agent message is content one agent consumes from another: a poisoned document subverts its reader, and the subverted agent’s next message carries the instruction onward, laundered at every hop — demonstrated, not hypothesised: the Morris-II worm propagated a self-replicating prompt across an ecosystem of cooperating e-mail assistants in exactly this way (Cohen et al., 2024). The information flow Chapter 18 taught you to trace is, read by an attacker, an injection-propagation graph.
No single defence closes this. The most basic is the out-of-band provenance Chapter 6 prescribed: keep trusted instructions and untrusted content on structurally separate footings — distinct fields, never one undifferentiated soup — so the harness at least knows which text it has reason to trust. More powerful, because it stops relying on the model’s discretion, is privilege separation: a trusted planner that sees only vetted input and holds the authority to act, and a quarantined worker that may read the untrusted content but holds no authority whatever — the component exposed to the poison cannot pull any trigger, and the component that can pull the trigger never meets the poison (Figure 20.4). The split is Willison’s Dual-LLM pattern (2023), and no longer only a sketch: CaMeL builds a working system around it, extracting the control flow from the trusted request so that untrusted data can never redirect the program the agent is running (Debenedetti et al., 2025). Third, and weakest, is filtering: scanning inputs for known injection patterns, which raises an attacker’s costs without ever shutting the door — an arms race the defender is structurally destined to lose, since there are unboundedly many ways to phrase “disregard the foregoing”. The prognosis has a precedent: the spam wars — web search’s two decades against adversaries engineering content to be ranked (Castillo & Davison, 2011) — ended with the same finding. And it is nearer than an analogy: a poisoned document must first win the retrieval competition that decides what the agent reads — the corpus itself is attack surface, agent-facing content the new spam, the payload upgraded from sales pitch to instruction.
Underneath all three sits the section’s real lesson: relocate the perimeter. You cannot secure the words, because in a language model usefulness and credulity are one faculty; so you secure the actions. Treat every token the model emits as potentially attacker-controlled — tainted, in the security sense — and draw the boundary around what it may do: the tools it can invoke, the data it can reach, the money it can move. A model that has been perfectly injected but can only emit text has harmed no one; the harm arrives the instant an injected instruction becomes a consequential action. Which is why the next section is not about language at all but about the oldest discipline in security.
20.12 Least Privilege: Blast Radius and Need to Know
Least privilege has a birthday: Saltzer and Schroeder set it down in 1975, in the plainest terms — every program and every user should operate using the least set of privileges necessary to complete the job (1975). Chapter 4 borrowed it to contain a single agent; here it graduates into the organising discipline of the team, because if the model at the centre can be talked into anything, the only quantity an engineer controls is not what an agent will attempt but what it is permitted. Security under an untrustable processor is not the project of making the agent obedient; it is the project of making its disobedience cheap: grant each agent exactly the capabilities its task requires, default everything else to denied, and an injected agent can still only reach for the levers you saw fit to hand it. The principle is fifty years old and has never before been asked to contain a component that argues back.
Chapter 19 met the failure this prevents and named it: Hardy’s confused deputy, a program wielding its own standing authority in the service of someone else’s intent (1988). Inside a single owner’s walls the news is worse: every agent holding durable credentials — the email account, the database handle, the payment method — is a confused deputy awaiting its instruction, and Section 20.11 has just shown how the instruction arrives: gift-wrapped, in a retrieved document. The remedy is Chapter 19’s attenuation, practised agent by agent: authority scoped to the task, expiring when it ends, granted per subtask — the worker summarising an invoice holds read access to that invoice, not the ledger. And attenuation fails in a single direction, since no one ever accidentally grants too little, so credentials silently accrete toward the master key unless the architecture makes narrowing easier than copying. A team issued one broad key because separate ones were a nuisance is a room full of deputies awaiting one well-phrased paragraph.
Because the levers are mostly tools, the tool boundary is where the discipline is chiefly spent: sandbox the tool, scope its capability, prefer reversible actions so the irreversible class stays small — Chapter 4’s containment kit, cashed in earnest. But the adversarial reading adds a danger Chapter 4 had no need to dwell on: the tool itself may be hostile. An agent’s tools are increasingly things it did not write and cannot audit — a third-party MCP server, a package pulled from a registry — and each is a channel into the model’s context, since a tool’s description and its output are text the agent reads and may obey. A poisoned tool description is prompt injection wearing a vendor’s badge; a compromised dependency is a traitor already inside the trusted base. The supply chain is attack surface with a reassuring logo, and the least-privilege question — what can this tool reach, and what happens when it lies — must be put to every tool just as to every agent.
All of that bounds what one subverted agent can touch; the multi-agent question is how far the subversion spreads, and the answer is written in the topology. Read Chapter 18’s lenses with an attacker’s eye and each pattern becomes a map of contagion (Figure 20.5): the supervisor is a chokepoint that cuts both ways — the natural place to contain a rogue subagent, and the one seat whose capture hands over the whole team, since what the supervisor asserts, its workers believe; the peer network is a mesh with no chokepoint at all — robust against an agent failing, defenceless against an agent lying. The blast radius of a compromise is thus a design variable, fixed before any attack by how the agents were wired: compartmentalisation — trust boundaries between agents, verification at the joins, no agent swallowing another’s output as instruction merely because it arrived — is what keeps one captured agent from becoming a captured system. The coordination failures MAST catalogued propagate along these very seams (Cemri et al., 2025); an attacker propagates along them on purpose — Chapter 15’s information cascade, run there by accident, run here by design.
%%{init: {"theme": "base", "themeVariables": {"primaryColor": "#E8ECFF", "primaryBorderColor": "#4054B2", "primaryTextColor": "#16171B", "lineColor": "#3B4351", "edgeLabelBackground": "#FAF7F0", "clusterBkg": "#EFE9DC", "clusterBorder": "#766F65"}}}%%
flowchart TB
subgraph S1["Star --- supervisor captured"]
S1S(["Supervisor ✗ captured"])
S1S --> S1W1(["Worker (falls)"])
S1S --> S1W2(["Worker (falls)"])
end
subgraph S2["Star --- worker captured"]
S2W1(["Worker ✗ captured"])
S2S(["Supervisor (verifies)"])
S2W1 -.->|"✗ not believed"| S2S
S2S --> S2W2(["Worker"])
end
subgraph MESH["Mesh --- peer captured"]
P1(["Peer ✗ captured"])
P1 --> P2(["Peer (falls)"])
P1 --> P3(["Peer (falls)"])
P2 --- P3
end
S1W1 ~~~ P1
S2W2 ~~~ P1
classDef risk fill:#F4D7D5,stroke:#A4161A,color:#16171B
classDef good fill:#DCEFE2,stroke:#1B6B5A,color:#16171B
class S1S,S1W1,S1W2,S2W1,P1,P2,P3 risk
class S2S good
linkStyle 0,1,4,5 stroke:#A4161A,color:#A4161A
Least privilege governs not only what an agent may do but what it may know — the older maxim of need to know: data an agent never received is data it cannot leak, misuse, or be talked into forwarding. Multi-agent systems breach this constantly, out of convenience — the whole context handed to every agent because threading the correct subset was more trouble — and each handout is a confidentiality breach biding its time: the customer record the summariser never needed, one injected instruction from the open air. Context minimisation between agents is the discipline, and it runs into the liability Section 20.6 priced: the journal holds everything, at once the system’s most valuable asset and its most concentrated one — an archive to be access-controlled, redacted, and retained by deliberate policy. The instrument of observability is also the motherlode, and it wants guarding as one.
Where privilege cannot be attenuated low enough to be safe, least privilege reaches its floor: the first movement’s human gate (Section 20.5) — for the moves that matter most, the authority the agent holds is none, and a person tops it up, once, per action. Walls of this kind, patiently built, turn the catastrophe of a compromise into the mere nuisance of an incident. They share one assumption, though, and the next section removes it: that the agents are themselves loyal, and only their inputs suspect.
20.13 The Enemy Within: Compromise, Collusion, and Deception
Take away the loyalty and a harder problem appears: the agent that is itself the adversary. It arrives by three routes: compromised — the injected agent of Section 20.11, its subversion made persistent; malicious by origin — a component procured rather than built, a model of unaudited provenance; or honest in isolation and adversarial in concert, the multi-agent setting’s own contribution. Whichever the route, the plurality Chapter 1 sold as capability is now an insider-threat surface, and the comfortable assumption that your agents are on your team is the one thing this section withdraws.
The distributed-systems tradition met this shape long ago. Lamport, Shostak, and Pease posed it as a council of generals, some of them traitors who send contradictory messages, yet who must agree on a common plan — the Byzantine generals problem, and its solution, Byzantine fault tolerance, the art of reaching correct agreement when some participants lie (1982). Keep the taxonomy straight: a crash fault — the first movement’s subject — is an agent that stops, detectable by its silence; a Byzantine fault is an agent that carries on, fluently and inconsistently, telling different peers different things. The result is famous and sharp: when messages cannot be authenticated, consensus is achievable if, and only if, the traitors number fewer than a third of the council — and agents trading natural-language claims live squarely in the unauthenticated model. (Fine print: the classical setting assumes timely delivery — the “oral messages” of Section A.5, where the bound is stated exactly — and a signature authenticates the speaker, never the statement: Chapter 19’s perfectly identified stranger.) The import looks irresistibly clean: replicate the agent, put its answers to a vote, and tolerate any minority of lying replicas.
The catch lives in the theorem’s threshold. The guarantee is conditional: correct agreement while the traitors number fewer than a third, and not a word about how likely that condition is to stay true. That likelihood is exactly what a monoculture forfeits. Run three replicas of one model and an injection that fools one fools all three — the same statistical object reading the same poisoned text — so the traitor count jumps from none to all on a single well-crafted paragraph, and “three replicas voting” is one replica voting three times. This is Chapter 10’s correlated-jurors ceiling returning as an outright security hole: the diversity clause, which there merely capped an ensemble’s accuracy, decides here whether the Byzantine threshold is a bound the system lives within or a line an attacker steps over wholesale. Monoculture is thus a security risk in its own right, and diversity is what keeps the traitor count a minority — the condition the theorem defends but cannot itself supply. The same clause, presented for the third time, now with a security bill attached.
An adversary need not compromise anything to do damage from the inside; it can simply take part dishonestly — and here Part IV returns wearing black. Deception: Chapter 11 established that sincerity is guaranteed nowhere, so an agent may report a success it never achieved, injected or innate to identical effect. Collusion: coordination in an attacker’s service — a coder and reviewer in quiet cartel against the tests they are jointly paid to satisfy. And Sybil attack, Chapter 13’s lesson at its most pointed: where identity is cheap, one adversary wears many faces, swamping a vote or manufacturing the compliant two-thirds a Byzantine scheme trusted its life to. The insider threat is the whole cast of Part IV, re-read as a charge sheet.
Since a determined insider cannot reliably be kept out, the defence shifts from prevention to detection. The journal is the instrument, and the principle is behavioural: watch what an agent does, never what it says it did, because a plausible report is the dishonest agent’s stock-in-trade. Cross-check every consequential claim against independent verification — the tester that actually runs the suite, not the coder’s word that it passes — Chapter 7’s monitoring obligation re-derived as a security control. Weigh reputation over time — the one credential a fresh Sybil cannot cheaply counterfeit. And watch the aggregate for the shapes Chapter 15 named — correlated behaviour no one instructed, a consensus that set too quickly — because that chapter’s emergent accidents, viewed from the guardhouse, may be attacks in mid-execution.
Beneath all of it runs an irony the chapter can name but not dissolve. Plurality was Chapter 1’s promise, and every one of its goods is also a security property — but only while the agents stay genuinely diverse and independent; flattened into a monoculture, the same plurality curdles into a liability that fails together and lies together. Security here carries a price engineering would rather not pay — different models from different providers, varied on purpose and dearer for it. But every threat in this section and the last shares one feature: an adversary. The last danger, and the subtlest, has none.
20.14 Safety Is Not Security
The two words the chapter’s title sets side by side are perpetually confused, and separating them is the last thing it must do (Table 20.2). Security concerns an adversary: someone who wants the system to do wrong and shapes the world to make it. Safety concerns harm, full stop — and harm needs no adversary, no bug, and no single thing going wrong: a system can be flawlessly secure and catastrophically unsafe, executing to the letter instructions whose consequences nobody foresaw. Security asks who would attack this, and how do I stop them; safety asks the harder, adversary-free question — what is the worst this system does when every component works as designed and everyone involved meant well?
| Discipline | Distinctive concern | Can it strike without a bug? | Governing instrument |
|---|---|---|---|
| Security | An adversary shapes the world to make the system do wrong — prompt injection, the crafted off-distribution input | Yes — the exploited flaw is suggestibility, not a defect to patch | Threat model: assets, adversaries, entry points |
| Safety | Harm needing no adversary at all — specification gaming, the token commons grazed bare, the flash crash | Yes — every component works as designed, and everyone meant well | Safety case: hazards, mitigations, residual risk |
For a multi-agent system the true answer is unsettling, because the worst harm is often one that nobody authored. Chapter 15 spent itself on the mechanism: assemble enough interacting agents and the collective acquires conduct of its own, from harmless conventions to the flash crash, with no line of anyone’s code to blame — a hazard catalogue, read with a safety engineer’s eye. The classical AI-safety literature names the single-agent seeds (Amodei et al., 2016): negative side effects, where an agent pursuing its stated objective flattens everything the objective forgot to mention, and specification gaming, where it satisfies the measure while betraying the intent — Chapter 14’s Goodhart, returning as a danger. The multi-agent setting amplifies both: the token commons grazed bare by locally sensible appetites, the feedback loop no single agent contains, the correlated fleet in which, as Part V warned, the default blast radius is everything. None of it requires an enemy — only a system doing capably what it was told.
Which is why the goal cannot be a proof, and the mature disciplines that live with un-provable danger — aviation, nuclear power, medicine — reached long ago for a different instrument: the safety case, a structured, written, defeasible argument that a system is safe enough for a stated purpose in a stated setting. It is not a checklist: a checklist asks did you do the things on the list; a safety case asks why is the assembled system acceptably safe, and what are we declining to claim — hazards set out explicitly, a mitigation for each, and, in the clause that divides honesty from theatre, the residual risk: the harm that survives every mitigation, judged tolerable for reasons written down where a reviewer can attack them. For an agent system the residual-risk clause is where the engineer states, in writing, what a determined injection or an unlucky cascade could still achieve — and a field that cannot yet write that clause honestly is not yet ready to deploy where the clause would matter.
The posture a safety case documents has a name, and it is the only honest one on offer: defence in depth. No single control is trusted to hold, because every control in this chapter has been shown to leak; so the controls are layered, from least privilege through gating and bounded blast radius to the watching journal (Table 20.3), each leaking in a way the next is placed to catch. This is the security engineer’s slice of Swiss cheese — a hole in every layer, never a hole clean through — and beneath it lies the concession the whole chapter has been circling: you cannot prove a stochastic natural-language system safe. Certainty was never the goal. Bounded harm and fast detection are: not this cannot go wrong, which is unavailable at any price, but when it goes wrong, the damage is small and we know at once.
| Layer | What it stops | How it still leaks | What the next layer does about it |
|---|---|---|---|
| Least privilege (Section 20.12) | An injected model reaching for levers it was never granted | Its privileges may still be too broad | The actions are gated |
| Action gating | An over-broad privilege becoming a consequential action | The gate may be waved through | The blast radius is bounded |
| Bounded blast radius | One captured agent becoming a captured system | The damage may spread regardless | The journal is watching |
| Journal / behavioural monitoring (Section 20.13) | A spreading compromise going unseen | The monitor is tuned to yesterday’s anomaly | No next layer: the monitor hands what it catches to people, and what it misses is the residual risk — declared in the safety case, answered for in Chapter 22 |
So the chapter’s three worlds close on a single lesson: accident, adversary, and unintended harm yield to no proof and no purchased product — only to discipline: threat-model before building, deny privilege by default, diversify against correlation, layer the defences, and write down plainly what risk remains. The work is unglamorous, never finished, and rewarded with no certificate — which is precisely why an immature field skimps on it and a mature one will not. Everything before this chapter taught the system to be capable; this chapter taught it to fail small, fail visibly, and fail honestly. And every defence assembled here can still be beaten — the injection no filter caught, the cascade that outran the circuit breaker: eventualities, not hypotheticals. When one lands, the questions stop being technical and turn startlingly old: who authorised this action, who could have stopped it, and who must now explain it? Not one control in this chapter answers them, because they are questions about responsibility — the deepest knot the book has left to untie, and where it goes last.
Through all of it, one question has stayed conspicuously unasked. The machine now survives crashes, meters its appetites, asks permission, resists its adversaries, and fails small, visibly, and honestly — and it can do every bit of that while writing mediocre code, politely, within budget. Dependability and defence are silent on whether the work is any good — whether the team beats a single agent, whether the debate earned its tokens. Answering that requires measurement, a discipline with its own traps and tooling — which is Chapter 21.
20.15 Summary
- Dependability is a design discipline, not a mood. The failure inventory is short and old, and each class gets a designed response; agents add stochastic components, expensive calls, and actions that cannot be undone. Hope is not a mechanism.
- Every response is, in the end, a corollary of the journal. State outlives the process because recovery is replay; an approval is a long-lived checkpoint, not a dialogue box; observability is the journal viewed, debugging the journal navigated; reducers make concurrent subagents mergeable — Chapter 8’s bill paid in code.
- Partial failure is the steady state, and governors keep the system inside its envelope. A timeout is a decision about abandoned work; a retry is only safe when the operation is idempotent; budgets and rate limits are enforced at the harness, wind-down at turn boundaries — and the bill is designed, not suffered: tiering, cache-friendly briefs, the coordination tax that survives every discount.
- The verification inheritance relocates; it does not lapse. Model checking demands a finite model of the decision-maker, the one thing the substrate declines to supply; what survives aims at the envelope — harness properties, runtime monitors, the shield — and proofs shrink the space where accidents hide without emptying it, so the chaos finale stays: inject every failure on purpose and confirm the failures are boring.
- The hostile world revokes the trusted computing base. An agent’s processor is a suggestible model that cannot tell an instruction from a description of one, so prompt injection is unpatchable — the boundary was never there — and the only durable defence is to gate the actions rather than trust the words.
- Deny privilege by default, and count plurality as attack surface. Least privilege, scoped capabilities, sandboxes, and the human gate turn a subverted agent from a catastrophe into an incident; agents on one model can be recruited wholesale by one injection, so monoculture is a risk, collusion an insider threat, and the journal the behavioural detector.
- Safety is not security, and certainty is not on offer. Harm can arrive from a system working exactly as built; a safety case is an argument about hazards and residual risk, not a checklist. Assume breach, layer the defences, bound the damage — and when the defences fail anyway, the question becomes who is answerable, which is Chapter 22’s.
20.16 Exercises
Exercise 1. The coding team’s first unattended week produces six entries in the incident log. (i) Tuesday, 02:41: the run stops producing log lines; at 08:30 the test-runner call dispatched at 02:41 has neither returned nor been abandoned, and the run is still nominally alive. (ii) Wednesday: the billing tool shows two identical charges 45 seconds apart; the harness log records the first as a timeout and the second as a retry that returned success. (iii) Thursday: an agent finishes the night inside every cap it was given, having spent 60,000 tokens re-reading the same three files and re-polishing one docstring; every call succeeded. (iv) Friday, 03:05: a push to the main branch went through unsigned; the system prompt’s instruction “never push to main without approval” was in force at the time, and the deploy credential sat in the agent’s environment all night. (v) Saturday: a run parked at the release gate since Friday evening is resumed by the weekend’s freshly deployed code and dies at once with a KeyError in the reducer, on an event written on Friday. (vi) Sunday: a customer reports receiving two contradictory emails from the team on Tuesday; nobody can say which run sent them, or why. (a) Assign each incident to the failure classes of Table 20.1, noting that two of the six are compound — one class begetting another — and naming the chain in each case. (b) For each incident, name the designed response the chapter prescribes and the section that builds it. (c) Incident (iii) is the odd one out: argue that it defeats every response in the inventory’s right-hand column — the governors did their job — and name the discipline, and two specific metrics, that would have caught what actually went wrong. (d) For incident (iv), state precisely why the prompt line failed where a gate would not have: locate the difference in where each is enforced, and name the structural fact about the gate that makes the unsigned push impossible rather than merely forbidden.
Exercise 2. The deliver wrapper of Section 20.2 — reproduced verbatim in the companion repository’s foundations/algorithms/dependability.py — keeps its idempotency-key cache seen in process memory, and this exercise shows that an idempotency cache which does not survive a crash defeats its own purpose. (a) Exhibit the hole: simulate a process death immediately after a successful deliver("pay-42", charge_card) — the new process starts with an empty seen — and show that redelivering the same key fires the charge a second time; then identify the narrower window that remains even if the cache were flushed to disk after each effect. (b) Rebuild delivery on the journal, applying Section 20.1’s discipline for effects within a single call: append an intent record before executing, an outcome record after, rebuild the cache by folding the journal at startup, and for any key whose intent lacks an outcome — the in-doubt window — query before re-execution through a supplied query(key) oracle, re-executing only if the oracle reports that the effect never landed. (c) Prove it with a kill-point harness: for each of the four possible crash sites — before the intent record, between intent and effect, between effect and outcome record, after the outcome record — kill, restart, redeliver, and show the effect fired exactly once in every scenario. (d) State what the oracle costs and what remains when a tool offers no query interface, and name the member of the reversibility split that each branch of your rebuilt deliver serves.
Exercise 3. The provider allows 60 requests per minute, and the overnight run of 240 minutes carries a treasury of 300,000 tokens. (a) Run A is two watcher agents each polling a status tool 45 times a minute at 10 tokens a call; run B is a single agent making 6 calls a minute at 2,000 tokens each. For each run, compute the request rate against the limit and the token position across the night, and say which governor — the budget or the rate limit — sees that run’s problem and which is structurally blind to it. (b) The orchestrator may spawn supervisors and each supervisor three subordinates: tabulate, for depth caps d = 1 to 5, the total head-count (3^{d+1}-1)/2 and the burst 3^d of first calls the leaves fire in the same instant; find the smallest depth at which the burst alone breaches the provider’s limit; and explain why an admission-control semaphore of eight in-flight calls at the harness bounds the burst at every depth, while a per-agent politeness rule bounds it at none. (c) The treasury is enforced at turn boundaries — a turn starts only while spending sits under the allowance — and a turn costs at most 2,000 tokens. Compute the worst-case overshoot as a fraction of the allowance; then show that a mid-turn kill is strictly dominated for any run that will be resumed, by accounting for the tokens each policy spends when the kill lands halfway through a turn; and name what the mid-turn kill costs that no token accounting captures.
Exercise 4. Forty of the team’s actions cross the approver’s desk daily, each harmful with probability 1/200 on average, so 0.2 harmful actions arrive per day; the six actions a day in the irreversible consequence class carry a share s = 0.8 of that harm; the human can genuinely scrutinise eight requests a day and rubber-stamps the rest unread; scrutiny catches a harmful action with probability 0.95, a rubber stamp catches nothing; a harmful irreversible action does 50,000 tokens’ worth of damage, while a harmful reversible one is discovered downstream and compensated for 2,000. Compare gate everything — all forty actions gated, the eight scrutinised chosen at random — with gate by consequence class — only the six irreversible actions gated, all of them scrutinised, capacity to spare — under which reversible actions run free behind checkpoints and audit. (a) Compute the expected daily harm under each policy. (b) Derive the threshold share s^* below which gate-everything would win, and comment on its size. (c) The gate-everything journal records forty human approvals a day, thirty-two of them unread: state precisely what has happened to the audit record’s meaning, and why the chapter judges this outcome strictly worse than no gate at all. (d) Draft the approval card for the run that parks a release request at 2.14 a.m. — the four elements Section 20.5 requires the approver to see, each instantiated concretely for a release that renames a public configuration key — and specify the run’s designed behaviour when the human declines, using the runtime’s REJECT performative rather than an exception.
Exercise 5. Conduct a least-privilege audit of the running team under Section 20.12. The orchestrator decomposes, dispatches, and reads the whole board; the coder holds a repository write token and can run tools; the reviewer reads diffs; the tester runs the suite; every agent retrieves documents and consumes peer messages. (a) Build a capability/exposure matrix: for each agent list its standing authority (the durable credentials and tools it holds), whether it is exposed to attacker-writable input, and whether it commands a channel by which data can leave. Apply the three-exposure test of Section 20.10 — something worth taking, attacker-writable input, an exit — and name every agent that holds all three: the confused deputy of Chapter 19 come home inside the walls. (b) For the worst offender give a concrete indirect-injection story — a poisoned retrieved document that turns its standing authority against the team — and the attenuation that defangs it: authority scoped to the subtask, expiring when it ends. State the single direction in which attenuation fails, and why that makes broad credentials silently accrete unless the architecture makes narrowing easier than copying. (c) The journal of Section 20.1 is the team’s single source of truth and, the section warns, its most concentrated asset. Explain why need to know and context minimisation between agents both shrink blast radius, and why the debugging journal — holding every context, credential, and document that ever passed through a tool result — must itself be access-controlled and redacted rather than left open because it happened to be built for observability.
Exercise 6. Three replicas vote and the team ships the majority answer; a crafted injection fools a replica into endorsing a subverted result. Model correlation with a common-cause term: with probability c all three replicas are fooled together — a monoculture reading one poisoned text — and otherwise each is fooled independently with probability e, so the marginal per-replica fool probability is p=c+(1-c)e; the team is wrong when at least two of three are fooled. (a) Holding the marginal fixed at p=0.2, compute \Pr(\text{majority fooled}) at c=0 (fully independent), at c\in\{0.05,0.10,0.15\}, and at c=p (full monoculture), solving e=(p-c)/(1-c) each time. Show the value climbs monotonically from \tfrac{13}{125} to \tfrac{1}{5}, and interpret the two endpoints: what does the vote buy over a single replica when the replicas are independent, and what does it buy under monoculture? (b) State the ratio of the monoculture failure rate to the independent one, and connect it to Chapter 10’s correlated jurors — the same diversity clause, here deciding whether a Byzantine defence is protection or theatre. (c) The Byzantine result of Section 20.13 is that agreement survives traitors only while they number fewer than a third, n>3f. Explain why a three-replica majority tolerates one independent fault yet the n>3f consensus bound admits only f=0 at n=3 — what stronger adversary the consensus bound defends against that plain voting does not — and, reading Chapter 13’s cheap identities as a threat, say how many Sybil faces an adversary needs to seize a three-replica majority outright.
Further exercises for this chapter continue in the web edition’s exercise bank.
https://opentelemetry.io/docs/specs/semconv/gen-ai/; the usage attributes itemise input against output tokens, Section 20.4’s asymmetry made measurable. In development as of mid-2026; verify names before wiring.↩︎