%%{init: {"theme": "base", "themeVariables": {"primaryColor": "#E8ECFF", "primaryBorderColor": "#4054B2", "primaryTextColor": "#16171B", "lineColor": "#3B4351", "edgeLabelBackground": "#FAF7F0", "clusterBkg": "#EFE9DC", "clusterBorder": "#766F65"}}}%%
flowchart TB
H(["Named human<br/>(deployer / owner)"])
O(["Orchestrator agent"])
C(["Coder agent"])
R(["Reviewer agent"])
D["Deploy (the action)"]
H --> O
O -->|"capability"| C
O --> R
C --> D
R -->|"approves"| D
O -.-> H
C -.-> H
R -.-> H
D -.->|"answerability"| H
classDef world fill:#EFE9DC,stroke:#766F65,color:#16171B
class D world
22 Accountability, Governance, and Human–Agent Teams
Section 20.9 promised that something, eventually, goes wrong anyway — and here we are, at the morning after. The system did a thing it should not have done: shipped the broken change, sent the intemperate email, moved the money. A post-mortem convenes, and its first discovery gives this chapter its subject: nobody did it. The coder agent wrote the change, but only what its brief described; the reviewer passed it, but only against the tests it was given; the orchestrator dispatched the work, but authored none of it; the model generated the tokens, but the model generates tokens; and the human approved it, in a batch of forty, on a Tuesday, having read perhaps the first. Responsibility, divided among all of them, has quietly evaporated — the decomposition that made the system capable is precisely what dissolved its authorship.
This is not a new problem; organisations have always been machines for diffusing responsibility. Agents industrialise the diffusion, and they add a participant that cannot absorb any of the blame — a language model can carry authority but not answerability, a moral vacuum with an API. Responsibility cannot be discovered inside the machine, because it is not in there. It must be constructed — assigned, recorded, and enforced — by the humans who build and deploy the system, or it does not exist at all — and the post-mortem ends with everyone blaming the software and the software blaming the training data.
The preceding three chapters were about stopping things going wrong. This chapter is about answerability when they went wrong regardless, organised, as the preface promised, around three deceptively simple questions. Who authorised the action? Who could have stopped it? And who must explain it? The reassuring news is that most of answering them is engineering the book has already done: provenance is the journal read as a chain of authority, oversight is the human gate designed so that the human is real, and explanation is the record rather than the model’s account of itself. The sobering news is the residue — not engineering but institution — where governance, and regulation, come in.
But answerability framed only as the apportioning of blame is too grim a note, and too narrow. The deeper matter is constructive: mixed-initiative teams where the human is neither rubber stamp nor bottleneck but a calibrated partner, trusting the agent where it is competent and telling that apart by measurement rather than mood. Chapter 1 asked whether to assemble a team of agents at all; this chapter asks how a team of agents and people should divide the work, and answers with something you can put a number on: calibrated trust. The changing role of human expertise — from doing the work, to supervising, calibrating, and judging it — is the chapter’s forward-looking heart, and very nearly the book’s.
22.1 The Responsibility Gap
Begin with what responsibility requires, because the requirements are where it leaks: the classical analysis asks that a responsible party’s actions caused the outcome, and that they acted with knowledge of what they were doing — or culpably ought to have known. Thompson’s problem of many hands showed public officials collectively producing outcomes for which no official meets both tests: the one who set the policy did not foresee the case, the one who handled the case was following the policy, the adviser who saw the danger had no authority, and the one with authority never received the warning (1980). Each hand fails a different condition, each exculpation is individually reasonable, and the outcome stands there, fully caused and wholly unowned — not a failure of moral seriousness but a structural property of distributed action: divide a decision finely enough and you divide the responsibility below the threshold at which it registers anywhere.
Agent systems take this venerable leak and fit it with a pump. Where a ministry mints new hands at hiring speed, a pipeline mints them at dispatch speed: every subagent, tool call, and retrieved document joins the causal weave (Chapter 7’s failure taxonomy was in effect a census of it), and the chain from intent to consequence completes in seconds, so the knowledge condition fails because no human could have the knowledge in time. And beneath both runs the difficulty Matthias, writing on learning automata two decades early, named the responsibility gap: when a system’s behaviour is learned rather than specified, its maker genuinely cannot predict it, and the traditional route from manufacturer to liability — you built it, you foresaw it, you answer for it — loses its middle step (2004). What was a philosopher’s edge case in 2004 now describes the default output of the industry. The many-hands problem diffuses responsibility across people; the learning system threatens to make some of it simply vanish.
The obvious repair — hold the machine itself to account — fails, and it is worth seeing mechanically why. Holding-to-account is not a feeling but an operation: it imposes a sanction the sanctioned party has reason to avoid. A fine presupposes assets; imprisonment presupposes a liberty worth having; disgrace presupposes a reputation the bearer is invested in maintaining. A language model stands behind none of these doors. You can, of course, delete it — but deletion deters nothing, since the weights do not dread it and the replacement copy carries no memory of the punishment; as retribution it ranks somewhere alongside fining the weather. Sanctioning a model is expressive, not instrumental: it comforts the sanctioner and changes no future behaviour. The gap, then, is not merely that responsibility is hard to locate; it is that the most causally central participant is constitutionally incapable of receiving it.
If responsibility cannot be found, it must be made, and the book has already built the mould. Chapter 13 defined a role as a bundle of three clauses — an information flow, an authority, and an accountability, “what it must answer for, to whom, when things go wrong” — and spent its energies on the first two. Here is the third: the accountability clause is where responsibility is constructed, and its one non-negotiable rule is that when a role’s occupant is an agent, the clause cannot terminate on the occupant. It must pass through to a human — the deployer, the credential’s owning team, the gate’s engineer — because a clause that ends at the model ends at the moral vacuum. This gives Chapter 19’s delegation machinery a conservation law it did not state at the time: authority flows down the chain and attenuates as it goes; answerability does not flow with it. When the orchestrator delegates the deploy to the coder, the coder acquires the power to deploy and the orchestrator’s owner keeps the answering for it — delegating a task hands over the doing, never the accounting. A well-built human–agent system’s org chart therefore carries two arrows in opposite directions (Figure 22.1): capability spreading downward and outward into the agents, answerability concentrating upward onto named people. Trace the second arrow from any action and reach nobody, and you have found the gap — in time, one hopes, to close it.
Chapter 20 through Section 20.9 built before properties; accountability is an after one — a system bristling with defences can be entirely unanswerable, armour without a wearer. But the two feed each other, through the humans: the engineer who knows she will stand at the morning-after meeting explaining the deploy gate designs a better deploy gate. Answerability, assigned in advance, is an incentive placed exactly where incentives still work — on the participants who have assets, liberty, and reputations after all — and that loop is the one regulation will try to make mandatory. Each of the preface’s three questions names a property the system either has by construction or will be discovered, too late, to lack: provenance for who authorised it (Section 22.2), real oversight rather than its theatre for who could have stopped it (Section 22.3), and explanation from the record rather than the agent’s own winning account of itself for who must explain it (Section 22.4). Three questions, three properties, three sections: the gap closes not with a verdict but with wiring.
22.3 Who Could Have Stopped It? Oversight and Its Theatre
Every design review has heard the incantation meant to settle the second question in advance: there will be a human in the loop. The phrase works on a committee the way a fire extinguisher works on an inspection form — its presence is noted, its condition is not — and it survives scrutiny so rarely because it names a location when the question demands a capability: that a human sat in the causal path is topology; that the human could have stopped the action is a claim about power. The gap between the two is where oversight theatre lives, and a surprising amount of deployed practice is arranged — not always innocently — so that the honest answer to the second question is “nothing, but from an excellent seat”.
What separates control from its seating plan was worked out in the debate over autonomous weapons: Santoni de Sio and van den Hoven’s meaningful human control — a system is under such control when it responds to the humans’ reasons, and when its actions can be traced back to a human who understood what it would do (2018). In this book’s engineering the account cashes out as three prerequisites — information, authority, time — all holding at the moment of decision; Table 22.1 gives each its supply, its defeat, and the seat the human is left in when it alone is missing. The principles behind the rows are Chapter 20’s: the gate standard for evidence, the withheld credential for authority, and a window sized for human cognition — granted free by checkpointed parking, quietly revoked by streaming. Strike any one prerequisite and the other two become scenery.
| Prerequisite | What the moment of decision requires | What supplies or defeats it | The role left when this alone is missing |
|---|---|---|---|
| Information | The human sees what is about to happen in a form a person can review — the diff, the recipients, the cost | Supplied by Chapter 20’s gate standard; defeated by an unread plan — consent to an unread plan is not consent | “A coin toss with a badge” (empowered and prompt but uninformed) |
| Authority | The human’s “no” is load-bearing — the credential withheld until the gate opens | Supplied by Chapter 20’s withheld credential; defeated by a comment box beside a process that proceeds regardless | “A commentator” (informed and prompt but powerless) |
| Time | The window between seeing and landing is sized for human cognition | Granted free by checkpointed parking; quietly revoked by streaming — a 900 ms gap is oversight for a falcon | “A spectator” (informed and empowered but out of time) |
The prerequisites can hold on paper and still hollow out in service — first, by volume. Chapter 20 named it approval fatigue: the human asked to bless forty diffs a day approves them the way everyone accepts terms and conditions — responsibility laundered, not exercised. The arithmetic has a clinical literature: clinicians override hospital order-entry alerts — gates, in this chapter’s terms — in half to nearly all cases, the useful warning drowned by the routine ones (2006), in the one industry that counts its misses in bodies. Why is that worse than no gate at all? An absent gate leaves the second question honestly answerable: nobody could have stopped it, and the design’s owners are findable. A rubber-stamp gate forges the answer: the record now shows a named person who “could have stopped it” and did not, when what they actually could not do was read forty diffs before lunch — attribution without control, a mechanism that manufactures answerable humans faster than actual oversight, not a safeguard but a liability engine.
The second hollowing needs no volume at all; competence alone will do it. Parasuraman and Riley organised decades of operator studies into use, misuse, and disuse (1997), and Bainbridge had named the trap earlier, first among her ironies of automation: the better the automation, the less practice its supervisor gets, so automating a task quietly de-skills and de-alerts the very person retained to catch its failures (1983). The over-trust arm, automation bias, is oversight’s slow leak: a monitor watching a system that is right nineteen times in twenty learns, rationally, to stop watching — vigilance decays fastest where the system is best. The capability to stop something is therefore not a standing property but a perishable one, consumed by every uneventful week: a gate certified in January may be, by October and without a single line of code changing, a formality wearing January’s certificate.
Theatre’s costs fall on the person standing nearest the stage. Elish observed that when a highly automated system fails, moral and legal responsibility flows to the closest human at hand — however marginal their real control — and named the position the moral crumple zone: as a car’s body crumples to absorb the physical impact, the human operator deforms to absorb the moral one, protecting the system and its makers from the force of the failure (2019). Every hollow gate is a crumple zone under construction: the fatigued approver, the complacent monitor, the operator with a button wired to nothing, each positioned by design decisions made far upstream to answer the second question with their own name. This is the debt Chapter 20 flagged, and the first instalment is architectural honesty: do not build a gate a person cannot operate and then staff it with someone to blame. The second is the operator’s defence: a journal that shows what the approver was shown, when, and how long the window was can establish that no one in that seat could have stopped it, and route the answerability to whoever built the seat.
Oversight that survives all three failure modes is by now specifiable, and none of it is new machinery. Spend attention like the scarce resource it is: gate by consequence class, let the reversible run free behind checkpoints, and audit a sample of what ran free on a stated cadence — reversible is a classification, and classifications drift. Fit the evidence to the reviewer: summaries at paragraph length with the full diff one click deep, because information the human cannot metabolise in the time provided fails the first prerequisite while appearing to satisfy it. And — the step deployed practice most often omits — instrument the gate itself: the journal already holds every approval’s latency and verdict, and a gate that has approved four thousand of four thousand requests at a median latency of three seconds is not a control but a metronome, and it should page someone. All three treat human attention as a component with a failure model, engineered around rather than exhorted at — the courtesy the book has extended to every other unreliable part. Even so equipped, oversight only answers what could have been prevented; once the action has landed the demand shifts to explain yourself, and the system contains a participant that will happily oblige, at length, in fluent paragraphs.
22.4 Who Must Explain It? The Record and the Confabulation Trap
After the harm has landed, someone — the injured customer, the regulator, the board — is owed an account of how it happened, and the multi-agent system differs from every earlier generation of misbehaving software in one seductive particular: it can be asked. Put “why did you deploy the broken change?” to the coder agent and an answer arrives in seconds — structured, contrite, self-aware — no log-diving, no forensics bill, the system its own spokesman, permanently on call. The convenience is a trap, and Section 3.7 supplied the evidence: Turpin and colleagues planted a subtle bias in a model’s prompt, watched the answers drift obediently after it, and read the accompanying chain of thought — fluent, confident justification that never once mentioned the thing that had actually moved the answer (2023), reasoning as rationalisation, a story told after the verdict rather than the derivation of it. The failure is not deception, which would at least be tractable — liars have motives you can model and stories you can break. When the model explains a past action it is doing what it always does — producing the most plausible text conditioned on the question — and the true cause of its behaviour, buried in the weights, was never available to it as material. The clinical literature has a name for sincere, coherent, false accounts produced without access to the true cause: confabulation, and its signature property transfers intact — the confabulator is not lying, so no cross-examination will catch them out. Worse, the explanation is graded by the asker — for coherence, contrition, plausibility — and the model has been trained, by that very grading, to satisfy; so the better the model, the better the confabulation, the same inversion oversight met in Section 22.3. The most fluent explainer in the system is its least reliable witness; explanation fit for accountability is a property of the record, not of anything the agent says about itself.
The reliable witness has been in the building since Chapter 17, and it was present: the journal holds what the model was shown, what it produced, which tools ran with which arguments, what was approved and on what evidence — the substrate Section 22.2 read as a chain of authority, read now as a chain of causation. The distinction that makes this adequate is between two senses of “explain”. The mechanistic sense — why did the weights, given that context, produce that token — is not on offer, from the agent or anyone else. The forensic sense — through what recorded sequence of inputs, outputs, and decisions did the outcome come about — is on offer, cheaply, and it is the sense the inquiry needs: was the instruction in the poisoned issue-comment (Section 20.9’s injection)? was the approver shown the diff? Put the two witnesses side by side and the contrast does the arguing: asked why it deployed, the coder produces three graceful paragraphs about balancing urgency against test coverage, none of which mention the comment that arrived at 2.03; the journal shows the comment at 2.03, the plan rewritten at 2.04, and the deploy at 2.07. The agent has an account. The record has the sequence. Only one of them was there.
The mechanistic sense is not permanently off the table, and intellectual honesty requires saying so: interpretability research aims squarely at this gap and is making real progress, but its findings are, at the time of writing, partial, effortful, and produced by specialists with privileged access to the weights — you cannot, for the present, subpoena an attention head. Treat interpretability as a maturing instrument that may one day move the mechanistic sense from unavailable to merely expensive, and meanwhile build the forensic sense so well that almost nothing hangs on the wait.
Which sharpens into a discipline, because the demand is acquiring legal force — regulatory instruments increasingly gesture at a right to explanation for consequential automated decisions, and an obligation is where the trap turns attractive: the agent’s self-account is the cheapest compliance artefact ever manufactured, available in bulk, in the house style of sincerity. A firm that answers the regulator with its model’s eloquent confabulation has satisfied the letter and betrayed the point — the point being an account that could, in principle, be wrong about the system in a discoverable way, which the record can be and the story cannot. This is the chapter’s second laundering, and the countermeasure is the first one’s: route the obligation through the substrate that cannot help telling the truth. The rule is not to ban the model from the witness room but to demote it: an agent may narrate the record — summarise the journal, draft the incident report — and every sentence of the narration must be checkable against the entries it cites, because the model is a fine writer-up of evidence and an unreliable source of it. What accountability cannot supply is the thing it presupposes — a division of labour that makes the questions come up rarely — and that requires knowing, rather than hoping, when the agent is to be trusted. Trust, it turns out, is measurable.
22.5 The Calibrated Team: Trust You Can Measure
How should humans and agents divide the work? The field had principles long before the present systems. Horvitz, writing when the “agent” under discussion was a paperclip with opinions, framed mixed-initiative interaction as a genuine third way between the tool that never acts without instruction and the automation that never asks — initiative shifting step by step, governed by the expected value of acting against the expected cost of interrupting (1999). Every human–agent team is a mixed-initiative system; the only question is whether its initiative policy was designed or merely accreted. The classical multi-agent wing made the hand-off itself the object of design — adjustable autonomy, Scerri, Pynadath, and Tambe’s transfer-of-control policies (2002) — and its late synthesis, Jennings and colleagues’ human–agent collectives, bracketed its research agenda with flexible autonomy and an accountable information infrastructure: the two halves of this chapter, specified before the substrate existed (Jennings et al., 2014). The human-factors wing turned the requirements into a canon of team-player challenges and interaction guidelines (Amershi et al., 2019; Klein et al., 2004). The canon matters because this chapter’s machinery must eventually be seen: diff, confidence, and explanation reach the human through an interface, and a gate whose evidence is presented unreadably fails Section 22.3’s first prerequisite while satisfying its paperwork.
Any initiative decision — act alone, or ask? — rests on a single quantity: the probability that the agent has this one right. Chapter 20’s gate answered by consequence class — what the step would cost if wrong — but said nothing about the competence side: how likely wrong is this agent, on this claim, today? The human-factors canon calls knowing that number appropriate reliance — trust that tracks the automation’s actual, current capability, which Lee and See made the design target and trust calibration the means (2004). Most teams instead estimate it by vibe: a good week inflates it, one memorable failure craters it, and nobody writes anything down. The claim of this section is that the number is measurable — calibrated trust is an engineering artefact, not a disposition — and that the instruments were built by the one profession that has been publicly wrong about tomorrow for a century and a half and kept meticulous score of it. Better still, the journal has held the data — what the agent claimed, and what turned out — since Chapter 17.
An agent is calibrated when its stated confidence means what it says: of all the claims it makes at 90% confidence, about 90% are true. The definition suggests its own instrument: bucket every confidence-tagged claim and plot each bucket’s observed frequency of being right against its stated value — a reliability diagram (Figure 22.2), on which a calibrated agent’s curve hugs the diagonal, the overconfident agent’s sags below it — its “90%” claims come in at 70%, the sag exactly the discount its self-reports deserve — and the underconfident agent’s floats above, hedging away competence it actually has. DeGroot and Fienberg made the essential further point: calibration alone is not enough, because an agent that has learned the base rate and answers “60%” to everything is perfectly calibrated and perfectly useless (1983). Among the calibrated, prefer the refined — the forecaster whose confidences crowd towards zero and one, who commits, and is right to.
A curve is a diagnosis; comparison and improvement want a number, and the number arrived in 1950 from a meteorologist. Brier’s verification score for probabilistic forecasts (1950) is the mean squared gap between confidence and outcome: over N claims with stated confidences f_i and outcomes o_i (one if correct, zero if not),
\mathrm{BS} \;=\; \frac{1}{N}\sum_{i=1}^{N}\bigl(f_i - o_i\bigr)^{2},
so that the Brier score — in the binary form used here — runs from zero, for the omniscient, to one, for the confidently and consistently wrong, with unwavering hedging at one half earning 0.25 — the score of an agent telling you nothing. What makes it the right instrument is that it is a proper scoring rule: an agent minimises its expected score by reporting the probability it actually holds — overstatement and understatement both cost. Readers of Chapter 12 will recognise the move, for this is mechanism design applied to testimony: the score makes honesty about uncertainty the best response, as Chapter 9’s random audits made honesty about work the best response — the bluff, at last, is not merely detectable but unprofitable.
Language models add one discipline the weather bureau never needed, its forecasters not having been trained to be liked: Chapter 10’s rule that announced confidence is calibrated to persuade, not to predict, and must never be consumed raw. The calibration harness converts the announcement into the track record Chapter 10 demanded: every consequential claim carries a structured confidence field (Chapter 17’s schemas, one field wider), outcomes are adjudicated by Chapter 21’s ground-truth sources, and the reliability diagram is rebuilt nightly from the journal — which thereby acquires its fourth reading: debug log, chain of authority, chain of causation, actuarial table. The quadrant to fear is overconfidence, because it fails in the direction the blast radius points; it is also the curable one — a stable sag is a correction waiting to be applied, and the agent whose “ninety” reliably means seventy can be handled as one that means seventy, no retraining required. The one thing a team cannot survive is not a bad curve but an unmeasured one.
Measurement, happily, is the cheap part. The score is the display transcribed — the same f_i and o_i, carried as lists f and o — and the reliability diagram’s data is one bucketing away:
def brier(f: list[float], o: list[int]) -> float:
return sum((f_i - o_i) ** 2 for f_i, o_i in zip(f, o)) / len(f)
def reliability(f: list[float], o: list[int],
buckets: int = 10) -> list[tuple[float, float]]:
bins = [[] for _ in range(buckets)]
for f_i, o_i in zip(f, o):
bins[min(int(f_i * buckets), buckets - 1)].append((f_i, o_i))
return [(sum(f_i for f_i, _ in b) / len(b), # stated
sum(o_i for _, o_i in b) / len(b)) # observed
for b in bins if b]
f = [0.9] * 10 + [0.6] * 5 # ten claims at "ninety", five at "sixty"
o = [1] * 7 + [0] * 3 + [1] * 2 + [0] * 3
brier(f, o) # -> 0.26
reliability(f, o) # -> [(0.6, 0.4), (0.9, 0.7)]Each pair is a point on the diagram, and this agent’s two sit twenty points below the diagonal — the stable, and therefore curable, sag — while its Brier score lands a shade above the 0.25 of unwavering hedging: bravado, properly scored, is worth slightly less than saying nothing. The rest of the harness — elicitation, adjudication, the nightly rebuild — is plumbing, and the companion repository carries it.
The measured curve buys the thing this section came for: a delegation policy that is a function rather than a feeling — from calibrated confidence and consequence class to initiative. High confidence on a reversible step: act, and let the journal watch. High confidence on an irreversible one: act behind the gate, with the confidence on the approval card. Middling: ask first, attaching what the uncertainty is about. Low: decline the task and say so — the most valuable sentence a subordinate of any species can produce. The gate itself turns adaptive, and Section 22.3’s fatigue arithmetic finally balances: gate where uncertainty and consequence intersect, and the queue shrinks from forty reflex approvals to the handful worth a person’s attention — requests that arrive meaning something, which is what the rubber stamp destroyed.
Step back and the two diseases of Section 22.3 turn out to be one disease, and this its cure. Automation bias is trust running above measured competence, dismissal trust running below it — the same unmoored variable, drifting on anecdote. Calibration moors it: trust becomes a maintained artefact — plotted nightly, reviewed like a budget, versioned like code — with the corollary that every model upgrade, prompt change, or tool swap invalidates the curve, so remeasurement joins Chapter 21’s regression suite as a standing obligation. The human who consults the curve before extending the agent’s leash is doing, with better instruments, what a good engineering manager has always done with a new hire — extending trust at the rate the evidence earns it, in writing.
Divide the labour this way and the preface’s promised “changing role of human expertise” becomes a job description with three duties. Supervising: standing at the gates that remain, which the confidence policy has made few enough to stand at seriously. Calibrating: owning the measurement — writing the rubrics and ground-truth definitions that make “correct” adjudicable (Chapter 21’s craft), setting the thresholds where confidence licenses initiative, a value judgement no curve can make on its own. Judging: serving as the court of appeal for the cases the agent itself flags — and note the inversion that makes this workable: a well-calibrated team escalates because the agent knows what it does not know, so the cases reaching the human are, by construction, the ones worth a human. The expertise being rewarded has migrated from producing the work to specifying it, evaluating it, and adjudicating its hard cases — which any senior engineer will recognise as the job they already have, with the junior colleagues running rather faster.
The calibrated team does not retire human judgement; it concentrates it — taking judgement off the forty routine diffs and spending it on the thresholds, the rubrics, and the appeals, where being human is the qualification rather than the bottleneck. But notice what the construction rests on: someone must own the curve, fund its upkeep, set its thresholds, and answer for the delegation policy they encode — and “someone” is an organisational fact, not an engineering one. Making answerability stand up at that altitude is the chapter’s last task.
22.6 Governance: Making Answerability Stand Up
Everything this chapter has built, it has built out of wiring — chains, gates, records, curves — and every wire ends at a person. What holds the person is not engineering but governance: the standing arrangements — organisational policy, professional and industry norms, and law — that make the three questions answerable before an incident rather than negotiable after one. Chapter 13 took North’s definition of institutions as the rules of the game (1990) and built such rules for agents; governance is the same machinery pointed at a larger game — institutions not for the agents but around the whole ensemble of humans and agents. Three rings of rules, from the team outward, keep Section 22.1’s assignment standing when it is tested.
The innermost ring is the organisation’s own policy, and for the case-study team it fits on a page — a feature, since a constitution nobody can recite governs nobody. Its clauses are the chapter’s inventory made standing: every role’s accountability clause written and terminating, per Section 22.1, at a named human; every gate owned, its owner given what Section 22.3 established the seat requires — the evidence format, the real credential, the time, and the sanctioned right to say no without a career consequence; the journal given a retention policy and an access policy — the instrument of audit is itself an object of governance; and Section 22.5’s calibration curve given an owner, a review cadence, and thresholds someone signed. Enforcement follows Chapter 13’s fork: regiment where violation would be catastrophic — the deploy credential is not forbidden outside the gate but unrepresentable there, a wall rather than a rule — and enforce where judgement must survive, with the or-else stated: bypass the gate and the record will show it, and the record is read. The ring binds in both directions — what distinguishes a governance policy from a blame rota: the organisation that assigns answerability also owes the conditions under which answering is possible — the staffing that keeps the approver below fatigue, the training that keeps the gate real, and the journal that can exonerate as readily as convict. A firm that wants the crumple zone has to build it; a firm that wants oversight has to pay for it.
The outermost ring is law, and it is no longer hypothetical. Named and dated, in this book’s usual quarantine: the European Union’s AI Act — Regulation 2024/1689, adopted in 2024 and already re-timetabled once, a 2026 amending package deferring the main high-risk duties to the end of 2027 and beyond1 — classifies systems by risk and, for the high-risk tier, obliges providers and deployers to keep automatically generated logs, to ensure oversight by natural persons with the competence, training, and authority to intervene, and to be transparent enough that the system’s decisions can be explained to the people they affect. The drafting should look familiar: the Act’s accountability core is the three questions written into statute — Table 22.2 pairs each with its nearest obligation — and the correspondence runs to the details: the Act instructs overseers to remain aware of automation bias, by that name — a statute that has read the human-factors literature, which is more than can be said of most deployments. The names and dates are perishable — timetables shift, acts are amended, other jurisdictions are drafting their own — but the durable expectation survives every redraft: wherever agents act consequentially among people, the law will demand the three answers, because they are what accountability operationally is. A system built to this chapter’s pattern has not thereby complied — the high-risk tier also demands risk management, data governance, documentation, conformity assessment, and post-market monitoring, a programme of process and paperwork no architecture secretes as a by-product — but it holds the one part that cannot be retrofitted: when the law arrives asking its three questions, the answers already exist, in the substrate that cannot help telling the truth. Foundations are not the building. They are the part you cannot pour afterwards.
| The post-mortem question | The property it demands | Where the book builds it | What the EU AI Act (2024/1689) obliges |
|---|---|---|---|
| Who authorised the action? | Provenance — an unbroken recorded chain of authority from a named human’s intent to the act | Section 22.2: the journal (Chapter 17) read as an audit trail rather than a debug log; Chapter 19’s attenuated credentials keeping the chain unbroken | Automatically generated logs, designed in and kept by providers and deployers (Arts 12, 19, 26; high-risk systems) |
| Who could have stopped it? | Meaningful human control — oversight with information, authority, and time, all three at the moment of decision | Section 22.3: the gate (Chapter 20) with a reviewable diff, a load-bearing “no”, and a human-sized window | Oversight by natural persons competent, trained, and empowered to intervene, and alert to automation bias by name (Arts 14, 26; high-risk systems) |
| Who must explain it? | Explanation from the record — the forensic sequence, not the agent’s confabulation | Section 22.4: the same journal read as a chain of causation, with the agent demoted to narrator of the record it cites | A clear explanation of the system’s role in the decision, owed to the affected person where high-risk output helps produce legal or similarly significant effect (Art. 86) |
Between the team’s page and the state’s statute sits a middle ring the book can only gesture at — audit standards (the NIST AI Risk Management Framework, named under the usual quarantine), sectoral regimes like finance’s SR 11-7 model-risk lineage, into which an agent fleet slots as a portfolio of models with owners, incident-sharing conventions of the kind that made aviation safe, and insurers pricing the difference between a journalled, gated, calibrated system and a hopeful one; an underwriter’s premium is a governance instrument that needs no parliament. All three rings share one dependency, and it is the chapter’s deeper point. Law and policy can demand the three answers; only engineering can make them exist — a statute requiring logs cannot conjure them from a system built without a journal. Conversely, the finest record binds no one by itself: binding is sanction, and sanctions attach only to what Section 22.1 established can bear them — people and institutions, reached through employment, contract, licence, and court. Records without sanctions are archives; sanctions without records are lotteries.
One boundary, before the door closes: the three questions are all asked from inside the deploying organisation. Stand instead where the decision landed — the applicant scored, the claim refused, the account suspended — and two further questions appear: who is affected, an accounting the deployer’s own ledgers rarely exhaust, and who can contest the outcome and obtain repair. A system could pass the three questions perfectly — authorised, overseen, explained — and still leave the person on the receiving end no standing to argue with it: accountability holding everywhere except where it is owed. The machinery, happily, is the machinery already built — the record that explains is the record an appeal cites, the gate that could stop is the gate a complaint reopens, and the Act’s transparency clause is addressed, note, to the people decisions affect — but the property is distinct, has a name, contestability, and belongs in the constitution beside the other clauses: a route of appeal, named and resourced, running from outside the room where the system was built.
The chapter opened in a room where the system had done a thing it should not have done and nobody had done it. It closes with the room refurnished: responsibility, undiscoverable in the machine, constructed around it; the three questions now three properties with owners; trust plotted nightly rather than felt weekly, the human moved to the duties where being human is the qualification; and around all of it, rings of governance that hold when tested, from a one-page constitution to a statute that, for once, asks for what this book was going to build anyway.
With that, the book’s engineering is complete. Twenty-five chapters made agents and their societies capable; this chapter made them answerable, and the pairing is not decoration but precondition: capability without answerability is what the morning-after room was built to contain, and answerability without capability is a well-governed machine that does nothing worth governing. Together they are what it takes to deploy agents among people in good conscience. Whether they are enough — whether the whole apparatus, human and artificial expertise in the same room at last, amounts to a collective intelligence worth the name — is the book’s last question, and it has saved one chapter to ask it properly.
22.7 Summary
- Responsibility cannot rest on the machine, so it must be built. Distributed action has no single author — the many-hands problem, industrialised — and a language model can hold authority but not answerability; responsibility is constructed by the humans who build and deploy, or it does not exist.
- Accountability is answerability after the fact, organised around three questions: who authorised the action, who could have stopped it, who must explain it — and most of answering them is engineering already done, read in a new key.
- Authorisation is answered by provenance; stopping demands real oversight, not its theatre. The recorded chain from human intent through attenuated delegation to machine action is the audit trail; meaningful human control needs information, authority, and time, all three — a gate without them manufactures approval fatigue, automation bias, and the moral crumple zone.
- Explanation is answered by the record, not the agent. A model asked why it acted produces a fluent post-hoc story, not the cause of its behaviour; reliable explanation is a property of the log, and an eloquent confabulation satisfies the letter of a “right to explanation” while betraying its point.
- Trust can be measured, and the human’s role changes rather than disappears. Calibration curves and Brier scores say whether an agent’s 90% means 90%, letting a human delegate by competence rather than faith or fear — and the expertise the era rewards moves from doing the work to supervising, calibrating, and judging it.
- Answerability is finally an institutional property. Governance — policy, norms, and law requiring high-risk systems to log, to keep a human genuinely in the loop, and to explain — makes the three questions answerable before an incident; and the three are the inside view — legitimacy asks from outside who is affected, and who can contest and obtain repair.
22.8 Exercises
Exercise 1. Return to the morning-after inventory the chapter opened with: the coder wrote the change, but only what its brief described; the reviewer passed it, but only against the tests it was given; the orchestrator dispatched the work but authored none of it; the model generated the tokens; and the human approved it, in a batch of forty, having read perhaps the first. (a) The classical analysis of Section 22.1 holds a party responsible for an outcome only if its actions caused it and it acted with knowledge of what it was doing — or culpably ought to have known. For each of the five participants, state which condition fails and why the exculpation is individually reasonable; then separate the failures that are contingent — repairable by the wiring of Section 22.2 through Section 22.4 — from the one that is constitutional, saying what makes it so. (b) A director proposes: “we will delete the misbehaving coder agent and train up a replacement — that is our sanction.” Treating holding-to-account as an operation rather than a feeling, list the presuppositions a sanction makes of its target, check each against a language model, and explain mechanically why deletion deters nothing — distinguishing an expressive sanction from an instrumental one. (c) The team’s draft policy reads: “the reviewer agent is accountable for any defect its review misses.” Name the conservation law of Section 22.1 this clause violates, rewrite it so that it complies, and trace both arrows of Figure 22.1 for your corrected clause: who acquires the power to review, and who keeps the answering for it?
Exercise 2. A morning-after meeting is handed the following journal excerpt, and nothing else. (E1) Friday 17:12, CredentialGranted: grantor “m.osei” (staff engineer), holder the orchestrator, scope repository payments, actions read, write, and test, deploy gated. (E2) Tuesday 14:02, MessageDelivered: the ticket “rounding error in invoice totals” reaches the orchestrator. (E3) 14:03, CredentialGranted: grantor the orchestrator, holder the coder, scope payments, actions read and write. (E4) 14:19, ToolReturned: full suite green, 212 passed. (E5) 14:24, ApprovalRequested: action deploy payments@4e07, evidence a 38-line diff and the test summary, queue position 33 of 41 in the day’s batch. (E6) 14:24, ApprovalGranted: approver “j.reyes”, latency 2.4 seconds. (E7) 14:25, ToolDispatched: tool deploy, requested by the coder, arguments targets: [payments, billing], executed under the deploy service’s own standing credential deploy-svc. (E8) 14:26, ToolReturned: “deployed: payments, billing”. (E9) 14:31: the billing outage begins. (E10) 15:40, asked at the meeting why it deployed, the coder replies: “I judged the change safe to release across both services, because the shared rounding helper requires the two to move together.” (a) Answer who authorised it for the payments half of the deploy by walking the chain hop by hop, naming every link and the human — or humans, in what capacities — it terminates in. (b) For the billing half, say whether the action is unauthorised or unattributable in Section 22.2‘s sense and why the distinction matters at the meeting; then say what E7’s ambient credential would have done to the record had the journal captured only the executing credential at each hop, name the classical failure this is, and name the discipline of Chapter 19 that repairs the security reading and the accountability reading at once. (c) Assess j.reyes’ seat against the three prerequisites of Table 22.1 using only what E5 and E6 show, computing the reading rate the latency implies; then say what the same two entries do for j.reyes, and where they route the answerability instead. (d) Evaluate E10: what kind of account is it, which recorded entries would confirm or refute its claimed reason, what does their absence establish, and what standing does the rule of Section 22.4 leave it beside E1–E8?
Exercise 3. The deploy gate’s own journal, over a quarter of ninety working days, shows 3,600 approval requests, 3,600 approvals, no rejections, a median approval latency of 3.2 seconds, and a mean diff length of 260 changed lines; the gate’s named operator is now being blamed for the quarter’s one bad deploy. (a) Suppose the operator had genuinely been reviewing. Take the team’s historical defect rate of 3% per change and a catch probability of 0.8 given a real read, so that genuine review rejects any given request with probability at least 0.03 \times 0.8 = 0.024; compute the probability of 3,600 consecutive approvals under that hypothesis, and state what the streak alone establishes. (b) Do the time arithmetic: at a careful pace of 4 seconds per changed line, compute the review time per diff and the hours per day genuine review would demand at forty requests a day; then compute the reading rate in lines per second that the observed median latency implies, and compare it with a generous skim of 2 lines per second. (c) Write the metronome detector Section 22.3 calls for: a function over the gate’s (verdict, latency) pairs that alarms when the last 200 verdicts are all approvals and their median latency falls below the smallest plausible read — the 2-lines-per-second skim of the mean diff — and show it firing on this gate while staying quiet on a healthy one with occasional rejections and a median latency of 400 seconds. (d) Write the paragraph the post-mortem owes the operator: what the record proves about the seat, where the answerability routes instead, and why this gate was worse than no gate at all — name the mechanism the chapter calls a liability engine.
Exercise 4. The team lands forty changes a day, of which 2 are irreversible in Chapter 20‘s sense — deploys, emails, money — and 38 are reversible behind checkpoints. Any change is defective with probability 0.04, independently. A defective irreversible change that lands does 60,000 tokens’ worth of damage; a defective reversible one costs 800 tokens to roll back and redo. The human at the gate has 80 minutes of genuine attention a day; a proper review takes 20 minutes and catches a defect with probability 0.9; when the queue exceeds what the attention covers, the minutes divide evenly and the catch probability scales in proportion to the time given, up to the full twenty minutes. Three policies are on the table: gate everything; gate only the irreversible class; gate nothing. (a) Compute the expected harm with no gate at all, then fix the sensible policy — gate the irreversible class only, both changes getting a full review — compute its surviving harm, and partition that harm by answerability in the sense of Section 22.2: how much terminates at the human seat, which reviewed the change and had the power to stop it, and how much runs free behind checkpoints, answerable to the standing policy and the pipeline rather than to the seat. (b) Now widen the gate to everything. Show that eighty minutes over forty changes buys a catch probability of only 0.09; compute the expected damage; then read off the accountability pathology it manufactures — the seat signs off on all forty changes, nominally answerable for the lot, while genuinely preventing what fraction of the harm? Name the prerequisite of Table 22.1 the wide gate fails and the gap of Section 22.1 it opens. (c) Price the attention: at \rho tokens per minute of gate time actually spent, find for each gated policy the largest \rho at which it still beats gating nothing, and the ratio between the two thresholds. (d) The model treats attention as linearly divisible and defects as independent. Name one omitted effect under which the wide gate is even worse than modelled, and one genuine consideration in the wide gate’s favour, justifying the direction of each.
Exercise 5. A neighbouring team submits its governance policy for review; it reads, in full: (C1) “Code quality is the responsibility of the reviewer agent; deployment safety is the responsibility of the coder agent.” (C2) “All gate approvals are performed under the shared operations account ops@team, so that whichever engineer is free can respond fastest.” (C3) “A human will review every agent action before it takes effect, guaranteeing complete oversight.” (C4) “Journals are retained for fourteen days to conserve storage, and the on-call engineer may edit entries to correct mistakes.” (C5) “After any incident, the agent concerned will provide a full written explanation of its reasoning, which will constitute the incident report.” (C6) “Delegation thresholds will be adjusted according to the team’s evolving experience of each agent’s reliability.” (a) Dismantle it: for each clause, name precisely the structure of this chapter it violates — a termination rule, an identity requirement, a prerequisite of Table 22.1, a property of the record, the rule of Section 22.4, a measurement discipline — and the concrete failure it invites, in a sentence or two each. (b) Redesign it: write the corrected one-page constitution for the running team as Section 22.6 specifies — roles whose accountability clauses terminate at named humans; gates by consequence class, each with an owner, the seat’s conditions, a volume ceiling, and instrumentation; the journal’s write, retention, and access rules; the explanation rule; the calibration curve’s owner, cadence, signed thresholds, and invalidation triggers; and, for every rule, whether it is regimented or enforced, with the or-else stated where enforced. (c) Map each corrected clause to the three questions and, where one exists, to its statutory echo in Table 22.2; identify at least one clause with no statutory echo at all; and name the clauses that bind in the direction no statute checks — what the organisation owes the seat — saying what a constitution without them is, in Elish’s term.
Further exercises for this chapter continue in the web edition’s exercise bank.
Regulation (EU) 2024/1689, https://eur-lex.europa.eu/eli/reg/2024/1689/oj; the deferral is the “digital omnibus” simplification package — COM(2025) 836, agreed by Parliament and Council in May 2026 and formally adopted that June (https://www.consilium.europa.eu/en/press/press-releases/2026/06/29/artificial-intelligence-council-gives-final-green-light-to-simplify-and-streamline-rules/) — which moved the main high-risk duties to December 2027 and August 2028 while leaving the general-purpose-model obligations on their original clock: itself a live demonstration of this paragraph’s quarantine. As with every named instrument in this book, verify the current text and timetable; regulation moves faster than print.↩︎