%%{init: {"theme": "base", "themeVariables": {"primaryColor": "#E8ECFF", "primaryBorderColor": "#4054B2", "primaryTextColor": "#16171B", "lineColor": "#3B4351", "edgeLabelBackground": "#FAF7F0", "clusterBkg": "#EFE9DC", "clusterBorder": "#766F65"}}}%%
flowchart TB
C0(["c0 · run start"]) --> C1(["c1"])
C1 --> C2(["c2"])
C2 --> C3(["c3"])
C3 --> C4(["c4"])
C1 -.-> CR["crash · kill -9 mid-turn"]
CR -.->|"re-fold; redo live turn"| C1
C2 -.->|"rewind, amend one message"| FK(["c2′ · counterfactual fork"])
C3 -.-> GT["parked · awaiting approval"]
GT -.->|"approved days later; resume"| C3
C4 -.-> DP["deploy · resumes under v2"]
CR ~~~ FK
FK ~~~ GT
GT ~~~ DP
classDef risk fill:#F4D7D5,stroke:#A4161A,color:#16171B
classDef gate fill:#F7E6B5,stroke:#8A5A00,color:#16171B
class CR risk
class GT gate
23 Building Dependable Agent Systems
Every agent system works beautifully during the demonstration. The demo runs in one process that nobody kills, against tools that answer promptly, on a budget nobody has spent, under the fond gaze of the person who built it and can nudge it past any awkwardness. Production revokes every clause of that arrangement, one at a time, usually at night. The process dies twelve minutes into a forty-minute run. The test runner hangs and the timeout fires — or, worse, doesn’t. A retried payment call pays twice. An agent discovers an inexhaustible supply of plausible next steps and works through the month’s tokens before breakfast. And the one action that genuinely needed a human’s signature went through at 3 a.m., unsigned, because nobody had built the gate. The gap between a demonstration and a system is not polish; it is the list of things that have not happened to the demonstration yet.
This chapter closes that gap, and its thesis is that dependability is not a mood but a design discipline: an enumerable inventory of failure classes, each met with a designed response rather than an aspiration. The inventory, happily, is short and old — crashes, hangs, duplicates, runaways, irreversible mistakes, and mysteries — and a neighbouring discipline has spent fifty years learning to survive it; much of this chapter is distributed-systems engineering arriving at the agent stack’s door. What the agents add is character: components that are stochastic, so the same input may not fail twice the same way; calls that are expensive, so retrying is a budgeting decision and not a reflex; and actions that touch the world, so some mistakes cannot be rolled back, only prevented. Hope, in this setting, is not a mechanism.
| Failure class | What goes wrong | Designed response | Where it is built |
|---|---|---|---|
| Crash | The process dies mid-run, without notice; whatever was in memory is lost | The journal on disk, checkpoints at turn boundaries, crash-only recovery by replay | Section 23.1 |
| Hang | A call runs on, unseen; or its notice never arrives | Layered timeouts — tight per call, looser per turn, a task deadline, a run-level curfew | Section 23.2 |
| Duplicate | A retried call runs twice, or succeeds without managing to say so | Idempotency keys, the reversibility split, and query-before-re-execution | Section 23.2 |
| Runaway | Every step succeeds, yet nothing in the agent knows how to be finished | Governors — budgets (a stock) and rate limits (a flow), enforced at the harness | Section 23.3 |
| Irreversible mistake | An action that cannot be taken back fires without a signature | The human gate — approvals and interrupts, engineered as durable checkpoints | Section 23.4 |
| Mystery | Something went wrong in the night, and no one can say what it did or why | Observability (the journal, viewed) and time-travel debugging | Section 23.5 |
The method is the book’s, and this is the chapter where two of the preface’s promises are cashed together. Everything is built twice: first in plain Python on Chapter 20’s runtime — which already owns a journal, a reducer, and a budget, so each dependability mechanism arrives as the production form of something you have already built by hand — and then in LangGraph, the book’s teaching framework, chosen back in the preface for this very hour: its typed state, reducers, checkpointers, and interrupts correspond one-to-one to the machinery this chapter constructs, so the framework arrives as a convenience you understand rather than a magic you depend upon. The running case study is the book’s coding-agent team, now expected to earn its keep: running unattended, overnight, against real repositories — a system chosen precisely because it is substantial enough to fail in instructive ways, and this chapter intends to let it.
The sections follow the inventory. State that survives the process — typed state, reducers, checkpoints, and recovery, where concurrency finally collects the bill Chapter 11 wrote and Chapter 20 postponed (Section 23.1). Calls that fail — timeouts, retries, and the idempotency that makes retrying safe (Section 23.2). Governors — budgets and rate limits, and the runaway agent they exist to stop (Section 23.3). The human gate — approvals and interrupts, engineered as state rather than as ceremony (Section 23.4). Seeing it run — observability and time-travel debugging, the journal viewed and then navigated (Section 23.5). And shipping it — deployment for runs that outlive their processes, followed by the only proof that counts: breaking the assembled system on purpose and watching it be boring about it (Section 23.6). A system that survives all that is dependable. Whether it is any good is a different question, with a discipline of its own, and it is the next chapter’s.
23.1 State That Survives: Checkpoints, Reducers, and Recovery
Take the first failure from the inventory — the process dies, mid-run, without notice — and ask the only question that matters: what, precisely, has been lost? For most software the honest answer is “whatever was in memory”, which for an agent system means the conversation so far, the tasks in flight, the tool results paid for and not yet used — forty minutes of expensive cognition, gone because a container was rescheduled. Dependability’s first job is to make the honest answer “nothing that matters”, and that answer cannot be bolted on afterwards; it is bought in advance, by deciding where the truth lives. Chapter 20 made the decision this chapter now generalises: the truth lives in the journal, on disk, appended before anything else happens — and the process holds only a disposable summary of it. A system built that way does not so much survive a crash as fail to notice one.
The production form of the idea is the checkpoint: a named cut through the run at which the recorded state is complete and consistent, from which execution can lawfully resume. The definition earns each of its adjectives. Named, because resumption, forking, and audit all need to say which moment they mean; complete and consistent, because a cut taken mid-turn — after the model was called, before its response was recorded — resumes into an argument with reality; and the runtime already knew where the lawful cuts fall: at turn boundaries, the same places its budget wind-down chose, because between turns the state tells no lies. What is new here is the checkpoint’s promotion from persistence detail to the chapter’s load-bearing object: the human gate of Section 23.4 is a checkpoint waiting for permission to be resumed; time travel in Section 23.5 is navigation between checkpoints; a deploy in Section 23.6 lands on one. Get the checkpoint right and most of this chapter’s remaining machinery is a corollary.
Recovery, on this design, is not a procedure; it is a re-reading. State was always a fold over the journal, so recovering from a crash is folding again — snapshot plus tail, as the runtime arranged — and starting fresh is merely the special case where the journal is empty. That collapse of two paths into one is worth dwelling on, because it is the deepest idea in the section and it has a name: crash-only software (Candea & Fox, 2003). Candea and Fox’s observation was that a system with a separate, special recovery path keeps its least-tested code for its worst day — the graceful-shutdown path gets exercised nightly while the crash path waits, untried, for production to try it — whereas a system with only the crash path tests its recovery on every single start. The design consequence is cheerfully brutal: stop writing shutdown choreography and make startup-from-journal the only way the system ever starts. The operational consequence is the discipline this chapter’s finale will enforce: if kill -9 is not a routine event in your test suite, your recovery path is a rumour.
Replay has a boundary, though, and drawing it wrongly is this section’s characteristic mistake. Folding the journal re-derives beliefs — tasks, statuses, balances — but the world does not replay: the email was sent, the payment captured, the commit pushed, and a recovery that re-executes those on its way back to the present has turned a crash into an incident. The discipline is to split state along the line of reversibility. Everything internal is re-derivable and may be recomputed freely; every effect on the world is recorded in the journal as a fact — issued, confirmed, failed — and on replay those records are read, never re-run. The effect’s record, not the effect, is what recovers. This is the first appearance of a distinction the whole chapter turns on — between what a system can take back and what it can only account for — and Section 23.2 will meet it again within a single call, where it goes by the name idempotency.
Then the day Chapter 20 marked in advance arrives: the parallelism Chapter 21’s supervisor promised is switched on, two subagents finish at once, and both write to the shared state. Single-threading was the training wheel; production takes it off, and Chapter 11’s lost update — the classical failure the book has been promising since the warehouse — turns up in your own stack trace. The reducer is the discipline that meets it, and it is worth seeing why the journal design makes the problem tractable rather than merely familiar: because updates are events and state is a fold, merging parallel branches is not “two writers scribble on one object” but “two event streams fold into one state” — and the fold makes the merge policy explicit, field by field. A list of findings can append from both branches in either order; a status field cannot, and someone must decide whether last-write-wins, first-write-wins, or the conflict escalates. The reducer converts the merge from a race you lose at runtime into a decision you record at design time — which is all Chapter 11 ever asked — and where the contested resource is genuinely outside the state object, the repository itself, that chapter’s locks stand ready, at the location the design marked for them.
All of which is why the book’s teaching framework looks the way it does, and this is the hour the preface chose it for. LangGraph’s core objects are the section’s concepts with an ops team attached: the state schema is the typed state, declared once; each field carries a reducer annotation — append here, overwrite there — which is the per-field merge policy made a type-level fact, so that parallel branches merge by declaration rather than by accident; the checkpointer is pluggable persistence for the named-cut-per-superstep this section defined, from an in-memory toy to a production database by swapping one object; and a thread is a run, resumable from its latest checkpoint because resumption-from-checkpoint is the only way threads continue at all — crash-only, off the shelf. Nothing in the list should feel new, and that is the pedagogy working as designed: the reader who built the journal, the reducer, and the snapshot by hand in Chapter 20 is now looking at the same machinery with better packaging, and can tell — which is the point — what the framework is doing on their behalf, and what it is not.
In LangGraph that packaging is a few lines, and the whole per-field merge policy lives in the type itself — an Annotated reducer where parallel branches must combine, a bare field where successive steps overwrite and a second writer in the same superstep is rejected at the merge:
import operator
from typing import Annotated, TypedDict
class TeamState(TypedDict):
findings: Annotated[list[str], operator.add] # append from both branches
status: str # bare: one writer per superstep, or the merge raises
turn: int # single-writer counterThere is no graph here, only the schema, and that is the point: findings accumulates from both subagents by declaration, a concurrent conflict over status escalates as an error at the merge, and each field’s policy is a fact the framework reads from the type rather than an accident of arrival order.
The case study collects its first upgrade. The coding team now checkpoints at every turn boundary, splits its state into the re-derivable and the effectful — test results recompute; pushed commits are facts — and runs its coder and tester in parallel, their findings merged by a reducer that appends findings and escalates conflicting statuses. Kill the process twelve minutes into the overnight run and the morning log shows one line of interest: resumed from checkpoint, nothing lost, nobody paged. Boring, as promised. What the journal cannot fix is the failure that happens while the process is perfectly healthy — the tool call that hangs, the model call that dies halfway, the retry that might have gone through the first time — and that is the next section’s inventory.
23.2 Calls That Fail: Timeouts, Retries, and Idempotency
A production run is a long chain of remote calls — to the model, to tools, to MCP servers, to counterparties across Chapter 22’s boundary — and each one can fail outright, hang indefinitely, or succeed without managing to say so. Multiply a modest per-call failure rate by the hundreds of calls in an overnight run and the conclusion is arithmetic, not pessimism: partial failure is the steady state. Some call is always failing; the design question is never whether to handle it but which designed response each failure meets — and the agent stack sharpens the question with an unflattering superlative, because the model call is plausibly the flakiest, slowest, and most expensive remote procedure call ever put into mass production, and it sits in the loop’s inner ring.
The reason retrying works at all — rather than being superstition with a sleep statement — was established by Jim Gray from Tandem’s field data in 1985, in one of the founding documents of dependable computing (1985). Most production software faults, Gray found, are transient: retry the operation and it succeeds, because the triggering conditions — a race, a full buffer, a passing overload — have moved on. He called them Heisenbugs, for their habit of vanishing when examined, as against the deterministic Bohrbugs that fail identically every time; and the practical asymmetry is the whole strategy: retries are cheap insurance against the common case, and useless against the rare one. The agent stack adds a genuinely new wrinkle: a stochastic component turns every fault into something like a Heisenbug — the model that produced nonsense may well produce sense on the second sampling — which tempts teams to retry not on failure but on dissatisfaction. Mark the difference well: retry-on-failure converges because failures are rare; retry-until-the-output-pleases-you is an unbounded loop with a credit card, and belongs to the next section’s jurisdiction.
A timeout looks like a number and is in fact a policy, and the policy concerns work you can no longer see. When the test-runner call is abandoned at the thirty-second mark, three pasts are consistent with your present: the call failed long ago and the notice was lost; it never arrived; or — the case that bites — it is still running, and will complete at second forty-five, after you have given up, moved on, and perhaps already dispatched the retry. The abandoned call is not cancelled by your impatience; its ghost completes off-stage, its effects land in a state that has moved on, and its duplicate is now in flight. Agent workloads make the number itself miserable to choose — a legitimate test suite takes thirty seconds or twenty minutes, and no single deadline suits both — which is why timeouts want layering: a tight one per call, a looser one per turn, a task-level deadline, a run-level curfew, each set at the altitude where someone can name the consequence of firing it.
Which returns us, as Section 23.1 promised, to idempotency — Chapter 22 met it at the wire, where a counterparty’s retry must be safe to receive twice; dependability brings the same obligation home, because your own harness is about to retry against your own tools. The discipline is the reversibility split of Section 23.1 applied within a single call. Reads retry freely. Reversible writes retry under an idempotency key — the request carries a stable identifier, the tool remembers what it has already done, and the duplicate becomes a no-op with a clear conscience. Irreversible effects — send, pay, push — get the full ceremony: record the intent in the journal, execute, record the outcome; and on any doubt, query before re-execution — “did the payment go through?” is a question, and asking it costs nothing, while re-sending the payment to find out is an experiment with your customer’s money.
Retrying also has manners. A failing service is frequently a struggling service, and a fleet of agents retrying immediately and in lockstep is a distributed denial-of-service attack you have thoughtfully launched against your own supplier — hence exponential backoff with jitter, so the retries spread out and thin out; hence the circuit breaker, the pattern that stops calling a persistently failing dependency altogether and probes it occasionally, preferring a fast, honest failure to a queue of doomed hopefuls. And because every one of these retries is a model call or a tool run with a price on it, retry policy in an agent system is spend policy: a retry budget per task, counted in the same currency as everything else, so that persistence is a decision the accounting can see.
Retry-with-backoff and the idempotency key fit together in a handful of plain Python — a retry wrapper that backs off with jitter around a deliberately flaky effect, and an idempotency-key cache that turns the second delivery of the same request into a cache hit:
import random
import time
from collections.abc import Callable
seen: dict[str, str] = {} # the idempotency-key cache
faults = [TimeoutError("gateway slow")] # one transient Heisenbug
def charge_card() -> str: # each call: one ToolDispatched
if faults:
raise faults.pop()
return "charged 1500p"
def deliver(key: str, fn: Callable[[], str], n: int = 3) -> str:
for k in range(n):
try: # success: ToolReturned, cached
return seen[key] if key in seen else seen.setdefault(key, fn())
except TimeoutError: # transient: back off with jitter
time.sleep(0.2 * 2**k + random.uniform(0, 0.2))
raise TimeoutError(f"retries exhausted: {key}")
deliver("pay-42", charge_card) # -> 'charged 1500p' (one retry, then ok)
deliver("pay-42", charge_card) # -> 'charged 1500p' (served from cache)The first call pays for two dispatches — the Heisenbug, then the backed-off retry that lands — while the duplicate pays only for a dictionary lookup, because the key was already spent; and had every attempt failed, deliver would have raised rather than returned, leaving the caller a failure to route instead of a silence to discover.
Some tasks, finally, are cursed. The input hits a deterministic bug — a Bohrbug, retry-proof by definition — and the fourth attempt fails like the first, only later and at four times the cost. Retry budgets exist so this discovery is cheap, and the dead-letter queue is where the discovery is honoured: the task parked, its journal thread attached, awaiting a human — automation admitting, in a structured way, that this one is beyond it. The case study’s flaky test runner, which Chapter 20 could only absorb and shrug at, now has an actual policy: two retries with backoff on a timeout or a crash, no retries on a clean deterministic failure, then ERROR, then the dead-letter shelf with the full trace stapled on — and the overnight run keeps going on the tasks that are not cursed. What none of this machinery restrains is the agent whose calls all succeed — hundreds of them, each plausible, each billed — and for that the system needs not retries but governors.
23.3 Governors: Budgets, Rate Limits, and the Runaway
The failure class this section exists for involves no failures at all. Every call succeeds; every step is individually plausible; and the agent is still working at six in the morning, deep into the month’s tokens, because nothing in it knows how to be finished — the pathology Chapter 4 flagged the day the loop was introduced, now billed at production prices. The root cause is structural and worth stating plainly: a stochastic generator never runs out of plausible next steps. A classical planner halts when its search space is exhausted; a language model’s search space is never exhausted, so termination is not a behaviour the system will exhibit — it is a constraint the harness must impose. The name for such a constraint is old and mechanical: the governor, Watt’s spinning contraption bolted to the steam engine to throttle it before it shook itself apart — bolted on, note, because the engine has no opinions about its own speed. Neither does the agent.
Governors come in two kinds, and the pair matters because each is blind to what the other sees. A budget is a stock constraint — so much and no more — and the runtime’s single token allocation now generalises into a small treasury of them: tokens, money, wall-clock time, tool invocations, turns, each allocatable per run, per agent, and per task. A rate limit is a flow constraint — so fast and no faster — and it is not a budget in miniature: a run can be comfortably inside its budget while hammering an API at a rate that gets the organisation’s key revoked, and can be politely inside every rate while bleeding the budget dry at a sustainable pace. Dependability wants both gauges on the dashboard, and one rule above all, inherited from the opening’s dictum: limits are enforced at the harness, at the metering points the journal already owns, because an instruction in the prompt — “please stay within budget” — is a hope, and the whole point of a governor is to be a mechanism.
Enforcement has manners, and the runtime already set them: exhaustion triggers wind-down at the next turn boundary, never a mid-turn kill, so that what halts is a consistent, resumable, checkpointed run rather than wreckage. What production adds is the recognition that exhaustion is not failure — it is a decision point. A budget-exhausted run is a checkpoint plus an invoice, and it presents its holder with three options: top up — which is an approval, and belongs to the next section’s gate; wind down and bank the partial results; or conclude the task is not worth its appetite and shelve it with Section 23.2’s dead letters. Which of the three is right is a judgement about value, not about tokens — the budget’s job was only to force the judgement to happen now, with the meter stopped, rather than at month’s end with the finance report.
Budgets do a second job in a multi-agent system, and Chapter 12 explained why it is needed: the shared token pool is a commons, and the book’s own Prisoner’s Dilemma showed each agent’s locally sensible appetite grazing it bare by lunchtime. Subdivided budgets are the enclosure of that commons — each agent funded from its own paddock, spending against its own allocation, so that the tester’s thoroughness no longer competes with the coder’s context for the same blades of grass. The allocation itself is the fiat planner at team scale — the orchestrator divides, and at four agents the central planner can still read every status report — with Chapter 15’s market waiting, design and warnings supplied, for the fleet scale where it cannot.
Rate limits, meanwhile, run in both directions, and the inbound direction is not optional. The world imposes limits on you — the model provider’s quota, the tool API’s ceiling — and its refusals arrive as ordinary responses to be handled by Section 23.2’s machinery: backoff, jitter, and patience, not indignation. The production trap is multiplication: Chapter 21’s parallelism promises arrive at the provider’s door all at once, and eight subagents dispatched in the same instant are a burst no per-agent politeness prevents — admission control belongs at the harness, where the concurrency is created. The outbound direction is your own statute book: caps on concurrent agents, on spawn depth — a supervisor that can spawn supervisors is an org chart with the growth characteristics of a fork bomb — and on turns per task, the loop’s stop conditions promoted from hygiene to fleet policy.
Two honest limits on the governors themselves, and then the case study collects its kit. First, a governor bounds damage; it does not detect futility — the agent that stays under every cap while re-reading the same file and polishing the same paragraph is safe, cheap, and useless, and noticing that the bounded bill bought nothing is a seeing problem, deferred to Section 23.5. Second, every limit is a tuning judgement, and the first overnight runs exist to make it: caps set tight, raised on evidence. So equipped, the coding team gets a run budget, per-task allocations, a two-agent concurrency cap, spawn depth of two, and backoff that treats the provider’s refusals as traffic rather than insult — and the opening’s 3 a.m. token drain now ends, at worst, as a checkpoint and an itemised invoice waiting politely for morning. Which leaves the opening’s other 3 a.m. incident: the action that went through unsigned. Spending too much without asking is the governors’ problem; acting without asking is the gate’s.
23.4 The Human Gate: Approvals and Interrupts
Some actions can be undone by recomputing a fold; some can be undone with an apology and a compensating transaction; and some can only be prevented. The reversibility split that has organised this chapter reaches its sharpest consequence at that third class: for the action that cannot be taken back — the customer email, the production deploy, the payment, the tweet — the only dependable moment of control is before, and where the judgement required exceeds what the system can be trusted to exercise, before means a person. The human gate is the control structure that arranges it: a designated point at which the run halts and a human decides whether the world changes. Chapter 21’s control lens asked of every pattern who may say done, and who may say stop; the gate is that question answered, for the actions that matter most, with a name from the org chart rather than the architecture diagram.
The engineering insight that makes gates practical is the one Section 23.1 trailed: an approval is a long-lived checkpoint, and it belongs to the persistence story, not the user-interface story. The naive gate blocks a process on an input prompt — which works until the approver is asleep, the weekend intervenes, or the container is rescheduled, at which point the run and the request die together; this is why systems without durable state gate nothing, and why the demos never pause. The proper gate is built from the section-one machinery: the run reaches the gate, checkpoints, and stops existing as a process; the request becomes a durable item in a human’s queue; and the approval, whenever it comes, is simply an event — fold it in, resume from the cut. Ten seconds or ten days, the machinery is identical, because a pause that survives a weekend is just a resumption that took longer. This, incidentally, is what a framework’s interrupt primitive is — LangGraph’s version is a checkpoint plus a pending-input marker, resumed by supplying the human’s payload — and Section 23.1 has already taught what that costs and why it works.
Stripped to a list-of-events journal, the whole gate is a dozen lines: reaching it appends an ApprovalRequested event and returns — the process stops existing — while approve appends the human’s ApprovalGranted and folds the run forward from the same cut:
journal: list[tuple[str, str]] = [] # (event, action) pairs, append-only
def run_to_gate(action: str) -> str:
granted = ("ApprovalGranted", action) in journal # a fold over events
if not granted:
journal.append(("ApprovalRequested", action))
return "parked"
return f"deployed {action}" # past the gate, the effect fires
def approve(action: str) -> str:
journal.append(("ApprovalGranted", action))
return run_to_gate(action)
run_to_gate("release-v2") # -> 'parked' (ApprovalRequested logged)
approve("release-v2") # -> 'deployed release-v2'The pause between the two calls may be ten seconds or ten days, and nothing in the code can tell the difference, because a park is only a fold that has not been resumed yet.
What to gate is a policy question with a failure mode on each side, and the expensive mistake is the conscientious one. Gate nothing and the 3 a.m. action goes through unsigned. Gate everything and you manufacture approval fatigue: the human asked to bless forty routine diffs a day approves them the way everyone accepts terms and conditions — instantly, unread, with a faint sense of duty discharged — and the rubber stamp is strictly worse than no gate at all, because the system now records that a human checked, when no one meaningfully did: responsibility has been laundered, not exercised. The calibration follows the chapter’s own split: gate by consequence class — the irreversible, the expensive beyond a threshold, the outward-facing, the policy-sensitive — and let everything reversible run free behind checkpoints and audit. A gate is a scarce resource because human attention is; spend it where prevention is the only remedy.
The gate’s second design surface is what the approver sees, and the standard is easily stated: consent to an unread plan is not consent. An approval request must carry what will happen, concretely — the diff, the recipients, the amount, not “proceed with proposed changes?”; why, briefly — the task and the reasoning at paragraph length, not the transcript; what it costs — the budget delta the action commits; and what declining does. This is Chapter 21’s planner–executor offer — the complete statement of intent, reviewable before anything touches the world — delivered at action grain rather than plan grain. And the decline must be a designed outcome, not an exception: the runtime’s REJECT performative exists for this very case, and a refused action should return to the run as information the agent can replan around — the human said no, the reason is attached, the run continues — rather than as a stack trace.
Interrupts are the gate’s mirror: the approval is the system asking the human; the interrupt is the human breaking in — because a person saw the trace going somewhere expensive, embarrassing, or wrong, and dependability’s control question includes who may stop this, and how fast. On a checkpointed runtime the answer has two speeds, both already paid for. The civil interrupt stops at the next turn boundary — the same grace the budget wind-down uses, yielding a consistent, resumable run and no lost work. The emergency stop is kill -9 — and the quiet triumph of the crash-only design is that the emergency stop is a crash, and crashes are now boring: even a panic costs nothing but the tail of a turn. Between the two speeds sits the manoeuvre that makes interrupts more than brakes: pause, inspect the state, amend it — redirect the task, correct a mistaken belief, prune a doomed subtree — and resume, which is the first taste of an idea Section 23.5 will complete: on a journal, the past is not merely visible but editable, forward from any cut.
One more thing the gate produces, and it outlasts the run: a record. An approval is a fact with organisational weight — this person, shown this evidence, consented to this action at this time — and the journal holds it in the same file as everything else, which is why the audit view Chapter 20 promised can answer “who approved this and on what evidence?” without a separate compliance system. What that record means — who carries the responsibility the signature creates, and what an organisation owes the person it puts at the gate — is Chapter 26’s territory, and the present section leaves it there. The gate also closes a loop Chapter 22 opened: attenuated authority runs out precisely where the action exceeds the credential, and the gate is where authority is topped up — deliberately, visibly, one action at a time — rather than left lying ambient in the deputy.
The case study, so armed, becomes hard to alarm. The deploy step and anything that pushes to the main branch are gated by consequence class; the overnight run works through its tasks, parks a release request at 2.14 a.m. — three-line summary, diff attached, budget delta noted — checkpoints, and goes quiet; a human reads it over coffee at 9.04 and clicks approve; the run resumes as though the intervening seven hours were a slow network. The opening’s unsigned 3 a.m. deployment is no longer a discipline problem but a structural impossibility: the agent does not hold the credential to deploy, and the gate is the only place the credential is issued. What the chapter has now assembled can crash, retry, stay solvent, and ask permission — and the remaining question is how anyone knows what it did all night, which is where the journal stops being infrastructure and becomes an instrument panel.
23.5 Seeing It Run: Observability and Time Travel
Chapter 20 ended its persistence argument with a discovery this section promotes to a thesis: observability was never a separate service — it is the journal, viewed. What production adds is that the views multiply and acquire the ops tradition’s names. The trace is one run rendered as a tree — turns containing model calls containing tool spans, each with its timing and cost. The metrics are the aggregates — spend per task, latency per tool, dead-letter counts — rolled up across runs. The audit is the record of authority — who approved what, on what evidence. Three dashboards, one file, and the architecture matters more than the tooling: the industry default is a bolt-on tracing SDK that samples some calls into a vendor’s silo, at which point the system has two accounts of itself that need never agree — observability with a split brain. Derive every view from the journal and disagreement is structurally impossible, because none of the views is the truth; they are all just the truth, viewed.
What should be on those dashboards is not what the web tradition watches, because the semantics are cognitive rather than transactional. Requests per second matter less than tokens per turn and its trend — the drifting-upward context that says summarisation has quietly failed; tool-call mix — the run that has stopped editing and started endlessly re-reading; rounds per loop and messages per task, the coordination overhead made a number. And here the debt from Section 23.3 falls due: governors bound the bill but cannot tell whether it bought anything, and futility detection is a seeing problem — solved by computing, continuously, the very signatures Chapter 21 attached to each pattern’s characteristic failure. Shrinking diffs against decaying critique specificity is placation in progress; revisions cycling without convergence is the polish loop; leaf results diverging from the root’s summary is the whisper game, measured. In the catalogue these were forensic marks to look for afterwards; on the dashboard they are metrics with thresholds, and the pathology pages you before the invoice does.
Debugging enters when a number goes wrong and someone must find out why, and with a stochastic component in the loop the question has a particular shape: not “which line is broken?” but “what did this agent know, and when did it know it?” The bad output is almost never a bad function; it is a defensible decision made against a context that was missing something, or contained something it should not have — so the investigator’s first need is the context as the model saw it, byte for byte, not as anyone remembers or summarises it, and the journal’s ModelCalled events hold it. The second need is reproduction, which stochastic components refuse — except that the runtime already bought the cure: replay the recorded responses through the fake client and the incident re-lives deterministically, as many times as the investigation needs. Production’s contribution is to make that a pipeline rather than a party trick: every incident worth an hour of a human’s time is bottled — journal excerpt, expected behaviour, observed behaviour — and joins the regression suite as a deterministic test, so the system’s worst days accumulate into its examination syllabus.
Time travel is what the checkpoints were quietly assembling all chapter, and it completes the manoeuvre Section 23.4 began. Because every turn boundary is a named cut, the past is addressable: rewind the run to any checkpoint and inspect the state exactly as it stood — no reconstruction, no memory, no “it must have been about here”. And because state is a fold, the past is also forkable: rewind, amend one thing — the brief that omitted the deadline, the tool result that lied, the belief the agent should never have adopted — and run forward again, and you have the counterfactual that ordinary debugging can only gesture at: would it have gone right if—? Debugging stops being archaeology — sifting fragments for what probably happened — and becomes navigation: go there, look, change it, watch. LangGraph ships the whole manoeuvre as checkpoint history on every thread — list the states, update one, resume from it as a fork — machinery whose every line Section 23.1 has already accounted for.
Honesty requires the bill, and it has two lines. Storage first: contexts are large, journalling them in full is expensive, and the discipline is a retention policy set by use — full fidelity for the recent window where debugging lives, snapshots and aggregates beyond it — with the one-truth principle intact: prune the archive by age, never by sampling away parts of a live run’s story. Liability second: a journal that holds everything holds everything — customer data, credentials in tool results, the confidential document an agent read — so the instrument of audit is itself an object of governance, with access controls and redaction of its own; Chapter 25 will treat the journal as attack surface and Chapter 26 as evidence, and both readings are correct. Against these costs stands Section 19.4’s asset ledger, now fully cashed: the traces are what survive every migration — frameworks come and go, and the journal schema you own, with the incident suite distilled from it, is the operational knowledge of the system in portable form.
The case study’s morning ritual, then: a dashboard read over the same coffee as the approval queue — spend per task against budget, the dead-letter shelf, progress signatures all green — and one anomaly: task seven’s diffs stopped shrinking at 3 a.m. and the polish-loop pager fired, so the run was interrupted at a turn boundary, its last four checkpoints inspected, and the culprit — a critic gone vague after a context compaction — bottled into the incident suite by Tuesday, the fix verified by forking Monday night’s checkpoint and watching the loop converge this time. Seeing, in short, is solved by the same object that solved surviving. What remains is the world’s least glamorous step — getting this machine off the laptop and into service, and proving the whole chapter’s kit by trying, deliberately and repeatedly, to break it.
23.6 Shipping It, and Breaking It on Purpose
Deployment is a solved problem for the systems the tooling grew up on, and the solution quietly assumes something agent systems do not have: short requests. A stateless web service deploys by fleet-swap — new code up, old code down, and every request lives a few hundred milliseconds inside one version or the other, none the wiser. An agent system’s unit of work is a run: hours or days long, checkpointed, possibly parked at an approval gate since Friday — and the run, not the request, is what must survive the deploy. New code will resume old state; the sprint that ships on Tuesday inherits the runs that started on Monday; and deploying becomes an exercise in continuity — surgery, with the patient not merely awake but halfway through a sentence.
Which makes version skew — Chapter 22’s permanent condition between estates — a domestic matter too, because a system whose runs outlive its releases is its own counterparty across time. Chapter 20’s schema versioning was built for this hour: events carry their version, the reducer keeps its old cases, and a journal written in June still folds in December. Production extends the same discipline to the artefacts engineers forget are versioned. Prompts and briefs are code — change one mid-run and the agent that resumes is a different colleague from the one who left for the checkpoint, halfway through the meeting. Model versions are substrate — the provider’s silent upgrade shifts the ground under every parked run, so pin the model per run and take upgrades at run boundaries, where a fresh start can absorb a new mind. And the deploy policy itself reduces to a choice made explicit: drain — let running work finish under the code that started it — or migrate — fold the old journals forward under the new version — with drain as the default and migrate as the manoeuvre you rehearse before you need it.
Environments are the deployment story’s awkward middle, and they are awkward for a reason Chapter 6 supplied: for an agent, the environment is not scenery but subject matter — the repository’s real history, the API’s real moods, the data’s real mess are the task — so a staging environment is, by construction, a simpler world, and an agent promoted from it meets production the way a well-schooled graduate meets a first job. The response is not a better staging but a ladder of them, each rung already built by an earlier section: the fake-client tier, where logic is tested deterministically for pennies; the replay tier, where recorded reality — the incident suite — is re-lived; a staged tier with production-shaped data and sandboxed credentials; and then the canary — the probation posting into real production with one repository, tight budgets, every gate armed, and trust extended the way Section 23.3 said limits are tuned: on evidence, one notch at a time.
%%{init: {"theme": "base", "themeVariables": {"primaryColor": "#E8ECFF", "primaryBorderColor": "#4054B2", "primaryTextColor": "#16171B", "lineColor": "#3B4351", "edgeLabelBackground": "#FAF7F0", "clusterBkg": "#EFE9DC", "clusterBorder": "#766F65"}, "flowchart": {"rankSpacing": 28, "nodeSpacing": 42}}}%%
flowchart BT
subgraph L[" "]
direction BT
A["Fake client<br/>logic tested for pennies"]
B["Replay<br/>the incident suite, re-lived"]
C["Staged<br/>real-shaped data,<br/>sandboxed credentials"]
D["Canary<br/>one repo, tight budgets,<br/>every gate armed"]
A -->|"add recorded reality"| B
B -->|"add real-shaped data"| C
C -->|"add the real world,<br/>throttled"| D
N["fidelity, stakes, cost ↑"]
B ~~~ N
end
style L fill:none,stroke:none
classDef world fill:#EFE9DC,stroke:#766F65,color:#16171B
class N world
The last discipline is the one the section title promises, and its logic has been accumulating all chapter: machinery exercised only by accident is machinery you do not actually know you have. The recovery path proved itself in Section 23.1 only because the test suite crashes the process on purpose; the same standard now applies to everything — the timeouts, the keys, the governors, the gate, the resumed deploy — and the practice of applying it has a name and a respectable pedigree in chaos engineering: inject the failure deliberately, in production-shaped conditions, on a schedule, and treat any surprise as a defect in the machinery rather than bad luck. The streaming industry made the idea famous by unleashing software whose job was to kill its own servers at random during business hours; the agent stack inherits the principle with a shorter kill list and higher stakes per item.
So the chapter ends with the case study’s scheduled bad day, every injection drawn from the inventory the opening promised. Kill the process mid-turn: the run resumes from its checkpoint, and the only casualty is the tail of one turn. Hang the test runner: the timeout fires, the retry backs off, the second hang sends the task to the dead-letter shelf with its trace attached. Deliver a tool result twice: the idempotency key swallows the duplicate without comment. Exhaust the budget while the release request is parked at the gate: wind-down at the boundary, checkpoint taken, the approval still waiting; Monday’s top-up resumes both. Deploy mid-run: the parked run wakes under newer code, folds its versioned journal, and continues as if nothing happened — because, structurally, nothing did. Interrupt the lot at 3 a.m. for the exercise’s own sake: a civil stop at the turn boundary, everything resumable. Six injections, six entries in the morning log, none of them requiring a human before nine. The chapter’s telos, demonstrated rather than asserted: a dependable system is one whose failures are boring.
Looking back, the chapter assembled less than it appeared to. A checkpoint, a reducer, a treasury of limits, a gate, a set of views, a ladder of environments — and behind all of them, doing the actual work, one object: the journal that Chapter 20 adopted as an economy turned out to be the load-bearing wall of every dependability mechanism in the inventory — survival is folding it, merging is folding two branches of it, the gate is a pause in it, seeing is viewing it, time travel is navigating it, and a deploy is resuming it under new management. The distributed-systems tradition supplied the failure classes and the disciplines; the agent stack supplied the twist — stochastic components, priced calls, irreversible actions — and the journal supplied the answer, six times over. That is not a coincidence; it is what “the truth lives in one place” buys, and it is the chapter’s transferable lesson for any stack, framework, or decade.
Looking forward, one question has been conspicuously unasked. The machine now survives crashes, meters its appetites, asks permission, explains itself, and ships without dropping a run — and it can do every bit of that while writing mediocre code, politely, within budget. Dependability is about whether the system keeps working; it is silent on whether the work is any good — whether the team beats a single agent, whether the debate earned its tokens, whether last month’s improvements improved anything. Answering that requires measurement, and measurement of stochastic, multi-step, multi-agent systems is a discipline with its own traps and its own tooling — which is Chapter 24.
23.7 Summary
- Dependability is a design discipline, not a mood. The failure inventory is short and old — crashes, hangs, duplicates, runaways, irreversible mistakes, mysteries — and each class gets a designed response; what agents add is stochastic components, expensive calls, and actions that cannot be undone. Hope is not a mechanism.
- State must outlive the process. The journal and its reducer, generalised: a checkpoint is a named place to resume, recovery is replay, and the recovery path should be the startup path so that it is tested on every start — and reducers are what make concurrent subagents mergeable on principle rather than by luck, which is Chapter 11’s bill finally paid in code.
- Partial failure is the steady state. A timeout is a decision about abandoned work, not just a duration; a retry is only safe when the operation is idempotent — Chapter 22 keyed the wire, this chapter keys your own tools; and a task that fails every retry needs a dead-letter destination, not more faith.
- Governors keep the system inside its envelope. Budgets enforced at the harness with wind-down at turn boundaries, never mid-turn kills; rate limits respected in one direction and imposed in the other; loop caps and spend caps for the agent that never runs out of plausible next steps.
- An approval is a long-lived checkpoint, not a dialogue box. The pause may last until Monday, so the gate belongs to the persistence story; gate the irreversible and the expensive rather than everything, because a rubber stamp is no gate at all — and design what the approver sees, since consent to an unread plan is not consent.
- Observability is the journal, viewed; debugging is the journal, navigated. Traces, spend, and audit are projections of one record; time travel — rewind to a checkpoint, fork, edit, rerun — turns debugging from archaeology into navigation, and the trace is the asset that survives every migration.
- A run must survive its own deployment. Checkpointed state resumed by newer code is version skew inside your own walls — Chapter 20’s schema versioning was for this — and the proof of the whole chapter is the chaos finale: inject every failure on purpose and confirm that a dependable system is one whose failures are boring. Whether it is also good is Chapter 24’s question.
23.8 Exercises
Exercise 1. The coding team’s first unattended week produces six entries in the incident log. (i) Tuesday, 02:41: the run stops producing log lines; at 08:30 the test-runner call dispatched at 02:41 has neither returned nor been abandoned, and the run is still nominally alive. (ii) Wednesday: the billing tool shows two identical charges 45 seconds apart; the harness log records the first as a timeout and the second as a retry that returned success. (iii) Thursday: an agent finishes the night inside every cap it was given, having spent 60,000 tokens re-reading the same three files and re-polishing one docstring; every call succeeded. (iv) Friday, 03:05: a push to the main branch went through unsigned; the system prompt’s instruction “never push to main without approval” was in force at the time, and the deploy credential sat in the agent’s environment all night. (v) Saturday: a run parked at the release gate since Friday evening is resumed by the weekend’s freshly deployed code and dies at once with a KeyError in the reducer, on an event written on Friday. (vi) Sunday: a customer reports receiving two contradictory emails from the team on Tuesday; nobody can say which run sent them, or why. (a) Assign each incident to the failure classes of Table 23.1, noting that two of the six are compound — one class begetting another — and naming the chain in each case. (b) For each incident, name the designed response the chapter prescribes and the section that builds it. (c) Incident (iii) is the odd one out: argue that it defeats every response in the inventory’s right-hand column — the governors did their job — and name the discipline, and two specific metrics, that would have caught what actually went wrong. (d) For incident (iv), state precisely why the prompt line failed where a gate would not have: locate the difference in where each is enforced, and name the structural fact about the gate that makes the unsigned push impossible rather than merely forbidden.
Exercise 2. The team’s overnight run is N = 40 turns of c = 2{,}500 tokens each, and the container is rescheduled during any given turn with probability h = 0.02, independently across turns; a crashed turn’s tokens are spent but its work is lost. Architecture S keeps state in process memory, so a crash restarts the run from turn 1; architecture C journals every event and checkpoints at each turn boundary, at a cost of w = 50 tokens per completed turn, so a crash re-folds the journal and redoes only the live turn. (a) Show that under C the expected number of executed — that is, paid-for — turns is N/(1-h), and compute the expected token bill including checkpoint overhead. (b) For S, writing q = 1 - h: show that one attempt executes (1-q^N)/(1-q) turns in expectation, that the expected number of attempts is q^{-N}, and hence that the expected bill is c\,(1-q^N)/((1-q)\,q^N); compute it. (c) Find the checkpoint overhead w^* at which the two architectures break even, and express it as a fraction of a turn’s cost. (d) Recompute both bills for a run twice as long (N = 80, h = 0.02) and for a hazard ten times gentler (N = 40, h = 0.002); say what the pair of comparisons reveals about how each architecture scales with run length, and why the case for crash-only design survives the gentle-hazard regime, where the token argument nearly vanishes.
Exercise 3. An overnight run makes 400 remote calls, each failing independently with probability 0.01. (a) Compute the probability that the run completes with no failed call at all, and the expected number of failures; state the design conclusion in a sentence. (b) Let every failure be transient in Gray’s sense and give each call up to three attempts, failures independent across attempts: compute the probability that some call in the run exhausts all three, and the expected number of attempts per call — what did retiring the steady state of (a) cost? (c) Independence across attempts is the optimistic case. Suppose instead that a task which has just failed its first attempt is transient with probability 0.7 — each further attempt then succeeding independently with probability 0.8 — and cursed with probability 0.3: a Bohrbug that no attempt will ever clear. An attempt costs 1,200 tokens, and shelving the task on the dead-letter queue hands a human a job the team prices at 20,000 tokens, so a further retry is worth buying while its success probability exceeds 1{,}200/20{,}000 = 0.06. Compute each successive retry’s success probability, updating the transient-versus-cursed posterior after every failure, and find the retry cap this rule implies. (d) Show that the case study’s policy of no retries on a clean deterministic failure is the same rule with a collapsed prior, and say in one sentence what each failed retry in (c) purchased besides a chance of success.
Exercise 4. The tester’s suite calls come in three kinds: 90 per cent complete in 30 seconds, 8 per cent complete legitimately in 540 seconds, and 2 per cent hang forever. A timeout abandons the call at T seconds and re-dispatches at once, and the candidate policies are a tight T = 60, a generous T = 600, and Section 23.2’s layering. (a) Treat every dispatch as an independent draw from the mix and compute the expected wall-clock time per completed call under T = 60 and under T = 600; which looks better, and by what factor? (b) Now drop the fiction: duration is a property of the task, not of the draw, so the 540-second suite takes 540 seconds every time it is dispatched. Show that T = 60 never completes a slow-legitimate task, explain why any single T still needs a retry cap and a dead-letter shelf to survive the hangs, and compute the expected time per task under T = 600 with a cap of two attempts. (c) Evaluate the layered policy — a first attempt at T = 60, one escalated retry at T = 600, then the shelf — computing the expected time per task and the time a hang costs before shelving, and compare both figures with (b). (d) Under the layered policy every slow-legitimate call is abandoned once at 60 seconds, and its ghost completes off-stage at second 540. For a 100-call night in which the suite call also writes a result artefact, compute the expected number of duplicated writes without idempotency keys and with them — and say on which side of the call, the harness’s retry wrapper or the tool that executes the write, the key must be checked if the ghost’s twin is to be suppressed, given that the seen cache of foundations/algorithms/dependability.py lives in the harness’s process.
Exercise 5. The deliver wrapper of Section 23.2 — reproduced verbatim in the companion repository’s foundations/algorithms/dependability.py — keeps its idempotency-key cache seen in process memory, and this exercise shows that an idempotency cache which does not survive a crash defeats its own purpose. (a) Exhibit the hole: simulate a process death immediately after a successful deliver("pay-42", charge_card) — the new process starts with an empty seen — and show that redelivering the same key fires the charge a second time; then identify the narrower window that remains even if the cache were flushed to disk after each effect. (b) Rebuild delivery on the journal, applying Section 23.1’s discipline for effects within a single call: append an intent record before executing, an outcome record after, rebuild the cache by folding the journal at startup, and for any key whose intent lacks an outcome — the in-doubt window — query before re-execution through a supplied query(key) oracle, re-executing only if the oracle reports that the effect never landed. (c) Prove it with a kill-point harness: for each of the four possible crash sites — before the intent record, between intent and effect, between effect and outcome record, after the outcome record — kill, restart, redeliver, and show the effect fired exactly once in every scenario. (d) State what the oracle costs and what remains when a tool offers no query interface, and name the member of the reversibility split that each branch of your rebuilt deliver serves.
Exercise 6. The provider allows 60 requests per minute, and the overnight run of 240 minutes carries a treasury of 300,000 tokens. (a) Run A is two watcher agents each polling a status tool 45 times a minute at 10 tokens a call; run B is a single agent making 6 calls a minute at 2,000 tokens each. For each run, compute the request rate against the limit and the token position across the night, and say which governor — the budget or the rate limit — sees that run’s problem and which is structurally blind to it. (b) The orchestrator may spawn supervisors and each supervisor three subordinates: tabulate, for depth caps d = 1 to 5, the total head-count (3^{d+1}-1)/2 and the burst 3^d of first calls the leaves fire in the same instant; find the smallest depth at which the burst alone breaches the provider’s limit; and explain why an admission-control semaphore of eight in-flight calls at the harness bounds the burst at every depth, while a per-agent politeness rule bounds it at none. (c) The treasury is enforced at turn boundaries — a turn starts only while spending sits under the allowance — and a turn costs at most 2,000 tokens. Compute the worst-case overshoot as a fraction of the allowance; then show that a mid-turn kill is strictly dominated for any run that will be resumed, by accounting for the tokens each policy spends when the kill lands halfway through a turn; and name what the mid-turn kill costs that no token accounting captures.
Exercise 7. Forty of the team’s actions cross the approver’s desk daily, each harmful with probability 1/200 on average, so 0.2 harmful actions arrive per day; the six actions a day in the irreversible consequence class carry a share s = 0.8 of that harm; the human can genuinely scrutinise eight requests a day and rubber-stamps the rest unread; scrutiny catches a harmful action with probability 0.95, a rubber stamp catches nothing; a harmful irreversible action does 50,000 tokens’ worth of damage, while a harmful reversible one is discovered downstream and compensated for 2,000. Compare gate everything — all forty actions gated, the eight scrutinised chosen at random — with gate by consequence class — only the six irreversible actions gated, all of them scrutinised, capacity to spare — under which reversible actions run free behind checkpoints and audit. (a) Compute the expected daily harm under each policy. (b) Derive the threshold share s^* below which gate-everything would win, and comment on its size. (c) The gate-everything journal records forty human approvals a day, thirty-two of them unread: state precisely what has happened to the audit record’s meaning, and why the chapter judges this outcome strictly worse than no gate at all. (d) Draft the approval card for the run that parks a release request at 2.14 a.m. — the four elements Section 23.4 requires the approver to see, each instantiated concretely for a release that renames a public configuration key — and specify the run’s designed behaviour when the human declines, using the runtime’s REJECT performative rather than an exception.
Exercise 8. One task of the overnight run lasts six turns before anyone looks at it: the coder’s successive diffs touch 420, 380, 130, 55, 41, and 36 lines; the reviewer’s critiques name 5, 4, 2, 1, 1, and 1 distinct files; every verdict is “reject”; and the turns cost 2,600, 2,500, 1,900, 1,300, 1,200, and 1,150 tokens against a task allowance of 25,000. (a) Give the three views of Section 23.5 for this task — what the trace shows, what the metrics roll up, and what the audit holds, computing the total spend — and explain why views derived from one journal cannot disagree, where a bolt-on tracer sampling 10 per cent of runs into a separate silo would have captured this run with probability 0.1. (b) Implement the detector this run needs: over a sliding window of three turns, fire when the diffs shrink monotonically, no critique names more than one file, and every verdict is a rejection; determine the earliest turn at which it fires here. (c) If the run would otherwise idle onward at 1,150 tokens a turn until the allowance runs out, compute how many further turns the allowance would buy and the tokens the detector’s page saves. (d) Say why neither of Section 23.3’s gauges ever fires on this task, then name the two failure signatures from the chapter’s dashboard that this window mixes, and state what further journal evidence — the verdict trajectory in particular — would tell them apart, and why the third signature needs evidence this journal cannot contain.
Exercise 9. Build the chapter’s load-bearing object by hand, then map it onto the framework’s. Using the standard library only, implement a minimal checkpointed runtime in the manner of foundations/journal.py — an append-only journal, a fold that derives state, checkpoints as named cuts at turn boundaries — and drive it with a deterministic scripted team: a coder whose diff halves each turn from 480 lines, a reviewer that accepts only when the instruction in force names a stopping criterion and the turn’s diff is under 100 lines, an initial brief “make the exporter faster” that names none, turns costing 800 tokens, and a cap of six turns. (a) Crash-only recovery: run three turns, discard every in-memory object except the journal — the simulated kill -9 — re-fold in a new process, verify the resumed state, and let the run continue to its cap; say why starting fresh and recovering are the same code path. (b) Time travel: the full run stalls at the cap, since no stopping criterion was ever given, so rewind three turns to the cut after turn 3, amend one message — the reviewer’s critique becomes “ship when the diff is under 100 lines” — rerun the fork, and report both futures’ turn count, spend, and final status side by side. (c) The gate as persistence: have the forked run park at a DeployRequested cut with the process gone, then append the approval in a later process and resume, and point to the fact about your code that makes a ten-second park and a weekend one indistinguishable. (d) Map each piece you built onto its LangGraph counterpart as realised in systems/coding_team/graph.py — the checkpointer, the thread, the state history, the fork, and the interrupt/Command(resume=...) pair — saying in each case what the framework adds that your page of Python did not have.
Exercise 10. The team’s runs last 72 hours; releases ship weekly at a fixed instant; runs start uniformly through the week. (a) Compute the probability that a run straddles a deploy, the expected number of deploys a run would meet if releases shipped daily instead, and the worst-case delay a pure drain policy imposes on a release; conclude in one sentence why migrate must be rehearsed rather than improvised. (b) Friday’s half of a run journals BudgetDebited events under schema v1, {"v": 1, "kind": "BudgetDebited", "amount": ...}, with debits of 1,200, 900, and 1,400 tokens; the weekend release renames the field, so Monday’s turns append v2 events {"v": 2, "kind": "BudgetDebited", "delta": ..., "currency": "tokens"} of 600 tokens each; the allowance is 5,500, and the harness starts a turn only while the folded spend sits under it. Write the versioned reducer — one arm per schema version, the old case kept — and compute how many Monday turns run before wind-down and the true spend at the stop. (c) Break it twice, deliberately: fold the same mixed journal through a v2-only reducer that reads e["delta"] directly, and through the defensive variant e.get("delta", 0); report what each does — one fails loudly, one lies — and for the liar compute the Monday turn count, the reported spend at wind-down, the true spend, and the overspend against (b). (d) Say which of the two breakages is worse for a dependable system and why, which incident from Exercise 1’s log the loud one reproduces, and which failure class from Table 23.1 the quiet one smuggles back past a blind governor.
Exercise 11 (lab). Section 23.2 rests on Gray’s split between transient and deterministic faults and claims that a stochastic substrate pushes model-call faults towards the transient side — while no retry budget rescues a fault that is deterministic in the request itself. Measure both halves. Fix one model and one temperature. Prompt A casts the model as the team’s tester and instructs it to emit exactly one JSON object for the tool call run_tests — double-quoted keys tool then args, args holding a paths list of three given file paths, no code fences, no commentary — with a checker script validating every clause. Prompt B adds one clause jointly unsatisfiable with the rest, for instance that the object occupy a single line of at most 40 characters while paths lists all fourteen test files by full path. Draw at least 30 samples of prompt A; for every faulty transcript, resample the identical prompt five times and classify the fault transient if any resample passes the checker, deterministic otherwise. Draw at least 10 samples of prompt B and resample its faults the same way, predicting the clean-sample count before you run. Tabulate prompt A’s fault rate, the transient share among A’s faults, and prompt B’s outcomes; then answer: what does the transient share say about retry-on-failure as a policy for format faults; what does prompt B show about what a retry budget can never buy; and to which piece of the chapter’s machinery — the backoff retry, the retry cap, the dead-letter shelf — does each half of Gray’s split properly belong? Record the model identifier and the date beside the table: the rates are perishable, and the durable finding is the split and its direction, not any absolute number.