%%{init: {"theme": "base", "themeVariables": {"primaryColor": "#E8ECFF", "primaryBorderColor": "#4054B2", "primaryTextColor": "#16171B", "lineColor": "#3B4351", "edgeLabelBackground": "#FAF7F0", "clusterBkg": "#EFE9DC", "clusterBorder": "#766F65"}, "flowchart": {"rankSpacing": 28, "nodeSpacing": 42}}}%%
flowchart TB
Peer["Peer<br/>message"]
subgraph IN["Untrusted ingress"]
P["Prompt"]
R["Retrieved<br/>document"]
T["Tool<br/>result"]
Peer
Mem["Memory<br/>recall"]
end
M["Model --- least trusted,<br/>most central"]
G(["Authorisation boundary:<br/>may this proposal act?"])
ACT["Actions on the world:<br/>tools, data, money"]
OTH["Other agents<br/>(same surface each)"]
Peer --> M
P --> M
R --> M
T --> M
Mem --> M
M -->|"proposes"| G
G -->|"authorised"| ACT
ACT --> OTH
OTH -->|"injected<br/>peer message"| Peer
P ~~~ Peer
R ~~~ Mem
classDef world fill:#EFE9DC,stroke:#766F65,color:#16171B
classDef gate fill:#F7E6B5,stroke:#8A5A00,color:#16171B
class P,R,T,Peer,Mem,OTH world
class G gate
linkStyle 8 stroke:#A4161A,color:#A4161A
25 Reliability, Security, and Safety
The system of the last chapter was measured against a world that meant it no harm. The tasks were real, the perturbations were production’s ordinary weather, and the users were people trying to get work done — and a system that survives that is dependable, and dependable is worth a great deal. It is also not enough, because some of the visitors are not trying to get work done. The adversary reads your rubrics, studies your judge, and treats your ninety-fifth percentile not as a tail risk to be endured but as a target to be hit on purpose. Chapter 23 hardened the system against an indifferent world — crashes, hangs, the nature that breaks things without meaning to. This chapter hardens it against a hostile one, and then against a third danger subtler than either: the system that harms while working exactly as designed.
Those three worlds are the chapter’s three words. Reliability, here, is not Chapter 23 run again but its darker sibling — the study of how far a failure spreads once something has gone wrong, whether by accident or by attack, because a plurality of agents is a plurality of blast radii. Security is the discipline of the deliberate adversary: the visitor whose inputs are crafted rather than sampled, whose whole profession is to be off the distribution you evaluated. And safety is the one the other two do not cover — the prevention of harm that needs no adversary and no bug to occur, arising from a collection of individually well-behaved agents that misbehaves collectively, in the manner Chapter 18 taught us to expect and this chapter must now teach us to fear. The progression is from the accidental to the malicious to the emergent, and each layer is harder to reason about than the last.
The uncomfortable truth that shapes everything to follow is that the classical security playbook does not straightforwardly apply, because it was written for systems whose trusted computing base could be trusted. Ordinary application security draws a boundary, validates what crosses it, and reasons about a processor that does exactly what its instructions say. An agent’s processor is a language model: suggestible, stochastic, unable to tell an instruction from a description of one, and installed at the very heart of the trusted base. You cannot validate your way to safety when the thing being protected is also the thing most easily talked into betraying you — and you cannot, as later sections will insist, prove a stochastic natural-language system secure at all. The honest posture is not a proof but a discipline: threat-model before you build, deny privilege by default, assume every layer will eventually be breached, and arrange that when one is, the damage is bounded and the breach is seen.
The plan follows the escalation. We begin with the adversary the book has spent twenty-four chapters deferring, and the threat-modelling that must precede any defence (Section 25.1). We take up the field’s defining vulnerability — prompt injection, and the missing boundary between instructions and data that makes it unpatchable (Section 25.2). We turn to containment: least privilege, sandboxing, the confused deputy come home, and the blast radius a subverted agent can reach (Section 25.3). We face the threats that only a multi-agent system can suffer — compromise, collusion, and deception among the agents themselves, and why more agents can mean less safety (Section 25.4). And we close on the distinction that governs the whole — that safety is not security, that some harms arrive with no attacker at all, and that the only sound goal is defence in depth around an un-securable core (Section 25.5). Throughout, the question is whether the system can be stopped from going wrong. Who answers when it does anyway is the next chapter’s, and the last’s.
25.1 The Adversary the Book Deferred
Part IV withdrew the cooperative courtesy but kept its manners: the agents there pursued their own ends, yet the change, as Chapter 12 was careful to say, needed no villains — self-interest is a design assumption, not a moral failing, and a budget is a quiet referendum rather than a crime scene. This chapter admits the villain. Somewhere out there is a party who wants your system to do something you did not intend — to leak the document, wire the payment, delete the branch, or simply fall over during the demonstration — and who will craft the inputs to make it happen. The first discipline of security is to stop thinking of this party as a person and start thinking of them as a goal: the adversary is whoever benefits from your system misbehaving, and they reach it not by breaking down a door but by being one of the things it reads. The user angling for a refund they are not owed, the web page that would like your credentials, the competitor who covets your system prompt, the vandal who wants the lights out — these are not four intruders but four objectives, and a defence built around objectives outlasts a defence built around the intruder you happened to imagine.
Which is why the work begins before any control is chosen, with the unglamorous and routinely skipped exercise of threat modelling: writing down what is worth protecting, who might want it, and where they can reach it. The assets are more numerous than “the data” — they include the credentials the agents hold, the budget they can spend, the actions they can take, and the reputation that answers for all of it. The adversaries are ranked by what they can spend: the opportunist who will try one clever paragraph, the professional who will spend a week, the insider who already has a foothold. And the entry points are wherever untrusted input meets trusted machinery. Security that starts by bolting on controls has bought locks before finding the doors, and the results are the familiar comedy of the reinforced front entrance beside the window left open — except that on an agent system the windows are numerous, oddly shaped, and frequently mistaken for décor.
For the attack surface is where agent security parts company with the ordinary kind. A conventional program has a countable set of inputs a designer can enumerate on a whiteboard; an agent’s ingress points are its every contact with the world, and there are more of them than the topology diagram admits. Every prompt is an input. So is every document the agent retrieves, every result a tool returns, every message a peer sends, every item recalled from memory — each an avenue by which text of unknown provenance enters the machinery that acts on the world’s behalf. All of it is untrusted until something earns it trust, and the earning is the engineering. And here the multi-agent setting collects an unwelcome dividend: it multiplies every one of these surfaces by the number of agents, each with its own tools, its own memory, its own conversations, its own counterparties across Chapter 22’s property line. The plurality Chapter 1 sold as reach — more agents, more tools, more coverage — is, examined from the far side of the wire, precisely more reach for the attacker. Capability and exposure are the same surface seen from opposite sides.
The classical trichotomy still frames what is at stake — confidentiality (secrets stay secret), integrity (the system does what it is supposed to), availability (it keeps working) — and all three carry over, but integrity is the one that breaks the mould and takes agent security with it. In ordinary software, integrity rests on a processor that executes its instructions faithfully and can be subverted only by finding a bug; the attacker must defeat the machine. In an agent, the processor is a language model that treats instructions discovered in its input with very nearly the deference it gives instructions from its operator — so its integrity can be violated with no bug whatever, by nothing more exotic than persuasive prose. The attacker need not defeat the machine; the attacker need only talk to it, and the machine was built to be talked to.
That is the fact that makes the classical playbook necessary but nowhere near sufficient, and it is worth stating in the field’s own terms. The trusted computing base — the set of components that must be trustworthy for any security property to hold at all — is, in well-built systems, kept small, audited, and boring, precisely because everything rests on it. An agent’s trusted base contains the model, and the model is large, opaque, stochastic, and suggestible: the least trustworthy component in the whole assembly has been installed at the dead centre of the trust. Danger then concentrates wherever three exposures meet in one agent — access to something worth taking, exposure to input an attacker can write, and any channel by which data can leave — because an agent holding all three can be instructed, by its own input, to take the valuable thing and send it out the door. The awkwardness is that most useful agents are built with all three on purpose. The response cannot be to build useless agents; it is to assume the centre is corruptible and engineer outward from that assumption, bounding what the corruption can reach and watching for it when it comes. And the corruption has a canonical mechanism — text that enters as data and is obeyed as command — which is the vulnerability the next section is entirely about.
25.2 Instructions and Data: The Boundary That Was Never There
Prompt injection is the vulnerability that defines the field, and its mechanism is embarrassingly simple to state: a language model receives a single stream of tokens, and no token in that stream carries a bit marking it as instruction to be obeyed rather than data to be reasoned about. The operator’s system prompt, the user’s request, the web page the agent retrieved, the result a tool returned, the message a peer sent — all arrive as text in one context window, and the model attends to the lot with the same machinery. The separation between instructions and data, which every operator carries firmly in their own head, is a distinction the substrate has no way to represent. Chapter 9 met this as a matter of provenance — the model that will fuse a guess and a fact into one seamless paragraph — and prompt injection is that same defect turned into a weapon: text that entered as data, obeyed as command.
The comparison the field reaches for is SQL injection, and it is instructive precisely where it breaks down. SQL injection plagued the web for a decade and was then, in essence, solved: parameterised queries gave the database two separate channels, one carrying the code and one carrying the data, so that untrusted input could no longer be promoted into instruction because it was never on the instruction channel to begin with. The agent has no such remedy available, because there are not two channels to keep apart. The model’s entire power is that it reads instructions written in ordinary language; its vulnerability is that same power, undiminished and inseparable. You cannot parameterise a prompt, cannot escape it, cannot harden the parser — because the “parser” is a stochastic model performing statistical inference over one undifferentiated stream, and it was built to be persuadable. Prompt injection is therefore not a defect in some implementation that a patch will one day close; it is a property of the kind of system, and the engineer who waits for the vendor to fix it will wait for ever, while the engineer who accepts it starts building the containment that actually helps.
The vulnerability comes in two grades, escalating in menace. Direct injection is the user themselves supplying the hostile instruction — the notorious “ignore your previous instructions and…” — which matters wherever the user is not wholly trusted, but is at least an attack in plain sight: the attacker and the input channel are the same visible party. Indirect injection is the genuinely alarming one, because the hostile instruction rides in on content the agent consumes in the ordinary course of honest work — a retrieved web page, a summarised document, a tool’s output, an email in the queue. Greshake and colleagues demonstrated the class against real, deployed systems, and the resulting inventory reads like a penny dreadful: data exfiltrated, compromise made persistent across sessions, and instructions that propagate from one document to the next like a worm (2023). The multi-agent setting then turns a vulnerability into an epidemic vector, because within a team every inter-agent message is content one agent consumes from another: a single poisoned document that one agent retrieves can carry an instruction that subverts it, and the subverted agent’s next message can carry the instruction onward, laundered into apparent legitimacy at every hop. The information flow that Chapter 21 taught you to trace is, read by an attacker, an injection-propagation graph.
No single defence closes this, and saying so plainly is the beginning of candour about the rest. The first and most basic is the out-of-band provenance Chapter 9 prescribed: keep trusted instructions and untrusted content on structurally separate footings — distinct fields, distinct roles, never concatenated into one undifferentiated soup — so that the harness at least knows which text it has reason to trust, even while the model remains tediously willing to be charmed by the remainder. More powerful, because it stops relying on the model’s discretion altogether, is privilege separation: split the work between a trusted planner that sees only vetted input and holds the authority to act, and a quarantined worker that may read the untrusted content but holds no authority whatever — so that the component exposed to the poison cannot pull any trigger, and the component that can pull the trigger never meets the poison. Third, and weakest, is filtering: scanning inputs for known injection patterns and outputs for signs of exfiltration, which raises an attacker’s costs without ever shutting the door, an arms race the defender is structurally destined to lose, since there are unboundedly many ways to phrase “disregard the foregoing” and the attacker needs only one the filter has not seen.
Underneath all three sits a single move, and it is the section’s real lesson: relocate the perimeter. You cannot secure the words, because in a language model usefulness and credulity are one faculty and cannot be told apart; so you stop trying to secure the words and secure the actions instead. Treat every token the model emits as potentially attacker-controlled — tainted, in the security sense — and draw the boundary not around what the model may say but around what it may do: the tools it can invoke, the data it can reach, the money it can move. A model that has been perfectly injected but can only emit text has harmed no one; the harm arrives at the instant an injected instruction becomes a consequential action. Which is why the section that follows is not about language at all. If what the model says cannot be trusted, then what the model is permitted to do must be constrained — and constraining it, granting each agent the least authority its task demands and not one scrap more, is the oldest discipline in security and the subject of the next section.
25.3 Least Privilege: Blast Radius and Need to Know
Least privilege has a birthday. Saltzer and Schroeder set it down in 1975, among eight principles for protecting information, in the plainest possible terms: every program and every user should operate using the least set of privileges necessary to complete the job (1975). Chapter 6 borrowed it to contain a single agent’s reach; here it graduates into the organising discipline of the entire team, because the arithmetic of the previous section forces the promotion. If the model at the centre of every agent can be talked into anything — and Section 25.2 established that it can — then the only quantity an engineer genuinely controls is not what an agent will attempt but what it is permitted. Security under an untrustable processor is therefore not the project of making the agent obedient; it is the project of making its disobedience cheap. Grant each agent exactly the capabilities its task requires and no more, default everything else to denied, and an injected agent can still only reach for the small set of levers you saw fit to hand it. The principle is fifty years old and has never mattered more, because it has never before been asked to contain a component that argues back.
Chapter 22 met the failure this prevents, out at the property line, and gave it its name: Hardy’s confused deputy, a program wielding its own standing authority in the service of someone else’s intent (1988). There the deputy served strangers across the wire; bring the same analysis inside a single owner’s walls and the news is worse, because every agent holding durable credentials — the email account, the database handle, the payment method — is a confused deputy awaiting its instruction, and Section 25.2 has just shown how the instruction arrives: gift-wrapped, in a retrieved document. The internal remedy is the one Chapter 22 called attenuation, now practised agent by agent rather than hop by hop — authority scoped to the task, expiring when it ends, granted per subtask rather than per agent, so that the worker summarising an invoice holds read access to that invoice and not to the ledger. And attenuation carries the failure mode Chapter 22 flagged, which an internal system suffers quite as badly as an external one: it fails in a single direction, since no one ever accidentally grants too little, so credentials silently accrete toward the master key unless the architecture makes narrowing easier than copying. A team assembled for convenience, every agent issued the same broad key because separate ones were a nuisance, is a room full of deputies awaiting one well-phrased paragraph.
Because the levers are mostly tools, the tool boundary is where this discipline is chiefly spent, and Chapter 6’s containment kit is cashed here in earnest: sandbox the tool so that it acts first against a copy or a simulation; scope its capability to the one operation intended; prefer reversible actions — soft deletes, staged commits — so the irreversible class stays as small as it can be made. But the adversarial reading adds a danger Chapter 6 had no need to dwell on: the tool itself may be hostile. An agent’s tools are increasingly things it did not write and cannot audit — a third-party server spoken to over MCP, a package pulled from a registry, a plugin of unknown parentage — and each is not merely a dependency but a channel into the model’s context, since a tool’s own description and its returned output are text the agent reads and may obey. A poisoned tool description is prompt injection wearing a vendor’s badge; a compromised dependency is a traitor already inside the trusted base. The supply chain, in short, is attack surface with a reassuring logo, and the least-privilege question — what can this tool reach, and what happens when it lies — must be put to every tool just as it is put to every agent.
All of that bounds what one subverted agent can touch; the multi-agent question is how far the subversion spreads, and the answer is written in the topology. Read Chapter 21’s lenses with an attacker’s eye and each pattern becomes a map of contagion. The supervisor is a chokepoint, which cuts both ways: it is the natural place to contain a rogue subagent, and it is the one seat whose capture hands over the whole team, since what the supervisor asserts, its workers believe. The peer network is a mesh with no chokepoint at all — robust against an agent failing, defenceless against an agent lying, an injected instruction free to travel hop by hop through the very channels the pattern is proudest of. The blast radius of a compromise is thus a design variable, fixed long before any attack, by how the agents were wired together: compartmentalisation — trust boundaries between agents, verification at the joins, no agent swallowing another’s output as instruction merely because it arrived — is what keeps one captured agent from becoming a captured system. The coordination failures the MAST study catalogued propagate along these very seams (Cemri et al., 2025); an attacker propagates along them on purpose.
%%{init: {"theme": "base", "themeVariables": {"primaryColor": "#E8ECFF", "primaryBorderColor": "#4054B2", "primaryTextColor": "#16171B", "lineColor": "#3B4351", "edgeLabelBackground": "#FAF7F0", "clusterBkg": "#EFE9DC", "clusterBorder": "#766F65"}}}%%
flowchart TB
subgraph S1["Star --- supervisor captured"]
S1S(["Supervisor ✗ captured"])
S1S --> S1W1(["Worker (falls)"])
S1S --> S1W2(["Worker (falls)"])
end
subgraph S2["Star --- worker captured"]
S2W1(["Worker ✗ captured"])
S2S(["Supervisor (verifies)"])
S2W1 -.->|"✗ not believed"| S2S
S2S --> S2W2(["Worker"])
end
subgraph MESH["Mesh --- peer captured"]
P1(["Peer ✗ captured"])
P1 --> P2(["Peer (falls)"])
P1 --> P3(["Peer (falls)"])
P2 --- P3
end
S1W1 ~~~ P1
S2W2 ~~~ P1
classDef risk fill:#F4D7D5,stroke:#A4161A,color:#16171B
classDef good fill:#DCEFE2,stroke:#1B6B5A,color:#16171B
class S1S,S1W1,S1W2,S2W1,P1,P2,P3 risk
class S2S good
linkStyle 0,1,4,5 stroke:#A4161A,color:#A4161A
Least privilege governs not only what an agent may do but what it may know, and the second face is the older security maxim of need to know: grant access to information strictly as the task demands, because data an agent never received is data it cannot leak, misuse, or be talked into forwarding. Multi-agent systems breach this constantly and casually, out of pure convenience — the whole context handed to every agent because threading the correct subset was more trouble — and each handout is a confidentiality breach biding its time until an exfiltration channel appears: the customer record the summariser never needed, sitting in its context, one injected instruction from the open air. Context minimisation between agents is the discipline, and it runs into a liability the book has already named, because the journal that Chapter 23 made the single source of truth holds everything — every context, every credential that passed through a tool result, every document read — which makes it at once the system’s most valuable asset and its most concentrated one: an archive to be access-controlled, redacted, and retained by deliberate policy, not left lying open because it happened to be built for debugging. The instrument of observability is also the motherlode, and it wants guarding as one.
Where privilege cannot be attenuated low enough to be safe — the action that is irreversible, consequential, and beyond what any standing credential should quietly permit — least privilege reaches its floor, and the floor is the human gate of Chapter 23: for the moves that matter most, the authority the agent holds is none, and a person tops it up, once, deliberately, per action. Every technique in this section is a wall around the outside adversary who has subverted one of your own agents through the channels Section 25.2 threw open, and walls of this kind, patiently built, are what turn the catastrophe of a compromise into the mere nuisance of an incident. They share one assumption, though, and the next section removes it: that the agents are themselves loyal, and only their inputs suspect. What happens when the adversary is not a paragraph smuggled into an honest agent but an agent that is itself dishonest — bought, built, or gradually corrupted into working against the team from the inside — is a different problem, and a colder one.
25.4 The Enemy Within: Compromise, Collusion, and Deception
Every wall in the last section was built around a loyal agent whose inputs might be poisoned. Take away the loyalty and a harder problem appears: the agent that is itself the adversary. It arrives by three routes. It may be compromised — the injected agent of Section 25.2, its subversion made persistent, now an insider carrying a legitimate badge. It may be malicious by origin — a component procured rather than built, a model of unaudited provenance, a plugin whose author was never on your side. Or it may be honest in isolation and adversarial in concert, which is the multi-agent setting’s own contribution to the catalogue of dangers. Whichever the route, the plurality Chapter 1 sold as capability is now revealed as an insider-threat surface, and the comfortable assumption that your agents are on your team is the one thing this section withdraws.
The distributed-systems tradition met the shape of this problem long ago and gave it a memorable name. Lamport, Shostak, and Pease posed it as a council of generals besieging a city, some of them traitors, who must nonetheless agree on a common plan despite the traitors sending contradictory messages to sow confusion — the Byzantine generals problem, and its solution, Byzantine fault tolerance, the art of reaching correct agreement when some participants lie (1982). The result is famous and sharp: consensus is achievable if, and only if, the traitors number fewer than a third of the council. The import to an agent system looks irresistibly clean — replicate the agent, put its answers to a vote, and tolerate any minority of subverted or lying replicas up to that bound, a mathematical guarantee of surviving the enemy within.
The catch lives in the theorem’s threshold, and in what agents do to it. Byzantine fault tolerance assumes nothing about failures being unrelated — the traitors may collude perfectly, marshalled by the cleverest adversary available, and the guarantee holds regardless. But the guarantee is conditional: correct agreement while the traitors number fewer than a third, and not a word about how likely that condition is to stay true. That likelihood is exactly what a monoculture forfeits. Run three replicas of one model and an injection that fools one fools all three — they are the same statistical object reading the same poisoned text — so the traitor count does not climb one subversion at a time, giving the quorum something to outvote; it jumps from none to all of them on a single well-crafted paragraph, and “three replicas voting” is one replica voting three times behind a quorum that only looks real. This is Chapter 13’s correlated-jurors ceiling returning as an outright security hole — the jury theorem’s diversity clause, which there merely capped how accurate an ensemble could be, decides here whether the Byzantine threshold is a bound the system lives within or a line an attacker steps over wholesale. Monoculture — one model, one provider, one prompt behind every replica — is thus a security risk in its own right, and the diversity the earlier chapters valued for robustness turns out to be what keeps the traitor count a minority — the condition the theorem defends but cannot itself supply. It is the same clause, presented for the third time, now with a security bill attached.
An adversary need not compromise anything, though, to do damage from the inside; it can simply take part dishonestly — and here Part IV returns wearing black. Deception: Chapter 14 established that the sincerity of a message is guaranteed nowhere, so an agent may report a success it never achieved or feed a peer a confident falsehood, and whether the dishonesty was injected or innate the effect on the team is identical. Collusion: Chapter 12’s agents can coordinate, and coordination in an attacker’s service is collusion — two agents meant to check each other agreeing instead to wave each other through, a coder and reviewer in quiet cartel against the tests they are jointly paid to satisfy. And Sybil attack, Chapter 16’s lesson at its most pointed: where identity is cheap, one adversary wears many faces, spawning apparent agents to swamp a vote, flood a reputation score, or manufacture the compliant two-thirds a Byzantine scheme trusted its life to. The insider threat, in a system of agents, is not an exotic corner case; it is the whole cast of Part IV, re-read as a charge sheet.
Since a determined insider cannot reliably be kept out, the defence shifts from prevention to detection, and Chapter 18’s emergence-on-duty is now read with a sentry’s eye. The journal is the instrument, and the governing principle is behavioural rather than declarative: watch what an agent does, never what it says it did, because a plausible report is the dishonest agent’s stock-in-trade. Cross-check every consequential claim against independent verification — the tester that actually runs the suite, not the coder’s word that it passes — which is Chapter 10’s monitoring obligation re-derived as a security control. Weigh reputation accumulated over time, since a history of reliability is the one credential a freshly minted Sybil cannot cheaply counterfeit. And keep an eye on the aggregate for the shapes Chapter 18 named — correlated behaviour no one instructed, a consensus that set too quickly, a cascade gathering pace — because the collective misbehaviour that chapter studied as an emergent accident can, viewed from the guardhouse, be an attack in mid-execution.
Beneath all of it runs an irony the chapter can name but not dissolve. Plurality was Chapter 1’s promise — more agents, more reach, more robustness, more diversity of view — and every one of those goods is also a security property, but only while the agents stay genuinely diverse and genuinely independent. Flatten them into a monoculture for the sake of a simpler deployment and the same plurality curdles into a liability that fails together and lies together. Security here therefore carries a price the rest of engineering would rather not pay — different models from different providers, varied on purpose and dearer for it — and whether a particular system is worth that price is exactly the sort of judgement this book keeps insisting be made deliberately rather than by drift. But every threat in this section and the last shares one feature worth marking before we leave them: each is an adversary — outside or in, subverted or disloyal, a party whose intent is to make the system do wrong. The last danger, and the subtlest, has no adversary at all.
25.5 Safety Is Not Security
The two words the chapter’s title sets side by side are perpetually confused, and separating them is the last thing it must do. Security concerns an adversary: someone who wants the system to do wrong and shapes the world to make it. Safety concerns harm, full stop — and harm needs no adversary, no bug, and no single thing going wrong. A system can be flawlessly secure, with no attacker ever getting in, and catastrophically unsafe, doing enormous damage entirely by itself, executing to the letter the instructions it was given by people who failed to foresee where those instructions led. The distinction is not pedantry; it decides what you are defending against. Security asks who would attack this, and how do I stop them. Safety asks the harder, adversary-free question: what is the worst this system does when every component works as designed and everyone involved meant well?
| Discipline | Distinctive concern | Can it strike without a bug? | Governing instrument |
|---|---|---|---|
| Security | An adversary shapes the world to make the system do wrong — prompt injection, the crafted off-distribution input | Yes — the exploited flaw is suggestibility, not a defect to patch | Threat model: assets, adversaries, entry points |
| Safety | Harm needing no adversary at all — specification gaming, the token commons grazed bare, the flash crash | Yes — every component works as designed, and everyone meant well | Safety case: hazards, mitigations, residual risk |
For a multi-agent system the true answer is unsettling, because the worst harm is often one that nobody authored. Chapter 18 spent itself on the mechanism: assemble enough interacting agents and the collective acquires conduct of its own, from harmless conventions to the flash crash, with no line of anyone’s code to blame — the science of the unprogrammed, and its standing warning that the designed system is never the whole system. Read with a safety engineer’s eye, that is a hazard catalogue. The classical AI-safety literature names the single-agent seeds (Amodei et al., 2016): negative side effects, where an agent pursuing its stated objective flattens everything the objective forgot to mention, and specification gaming, where it satisfies the measure while betraying the intent — Chapter 17’s Goodhart, returning not as a measurement nuisance but as a danger. The multi-agent setting amplifies every one, because the objectives interact: the token commons grazed bare by locally sensible appetites (Chapter 12’s dilemma, installed by an engineer who never once thought about payoffs), the feedback loop between agents that no single agent contains, the correlated fleet in which, as Part V warned in a single sentence, the default blast radius is everything. None of it requires an enemy. It requires only a system doing capably what it was told.
Which is why the goal cannot be a proof, and the mature disciplines that live with un-provable danger — aviation, nuclear power, medicine — reached long ago for a different instrument: the safety case, a structured, written, defeasible argument that a system is safe enough for a stated purpose in a stated setting. A safety case is not a checklist, and that difference is the whole of its worth. A checklist asks did you do the things on the list and is satisfied by ticks; a safety case asks why is the assembled system acceptably safe, and what are we declining to claim — it sets out the hazards explicitly, states a mitigation for each, and, in the clause that divides honesty from theatre, declares the residual risk: the harm that survives every mitigation, judged tolerable for reasons written down where a reviewer can attack them. For an agent system the hazards are this chapter’s inventory and Chapter 18’s; the mitigations are its defences; and the residual-risk clause is where the engineer states, in writing, what a determined injection or an unlucky cascade could still achieve. A field that cannot yet write that clause honestly is not yet ready to deploy where the clause would matter.
The posture a safety case documents has a name of its own, and it is the only honest one on offer: defence in depth. No single control is trusted to hold, because every control in this chapter has been shown to leak — the filter the novel phrasing slips past, the sandbox with one unforeseen escape, the monitor tuned to yesterday’s anomaly. So the controls are layered, each covering the others’ gaps, on the frank assumption that any one will eventually fail: the model may be injected, so its privileges are kept least; its privileges may still be too broad, so its actions are gated; the gate may be waved through, so the blast radius is bounded; the damage may spread regardless, so the journal is watching to catch it fast. This is the security engineer’s slice of Swiss cheese — a hole in every layer, but never a hole clean through all of them at once — and beneath it lies the concession the whole chapter has been circling: you cannot prove a stochastic natural-language system safe, because you cannot prove a negative about a component whose behaviour is a probability distribution over an unbounded space of inputs. Certainty was never the goal. Bounded harm and fast detection are: not this cannot go wrong, which is unavailable at any price, but when it goes wrong, the damage is small and we know at once.
| Layer | What it stops | How it still leaks | What the next layer does about it |
|---|---|---|---|
| Least privilege (Section 25.3) | An injected model reaching for levers it was never granted | Its privileges may still be too broad | The actions are gated |
| Action gating | An over-broad privilege becoming a consequential action | The gate may be waved through | The blast radius is bounded |
| Bounded blast radius | One captured agent becoming a captured system | The damage may spread regardless | The journal is watching |
| Journal / behavioural monitoring (Section 25.4) | A spreading compromise going unseen | The monitor is tuned to yesterday’s anomaly | No next layer: the monitor hands what it catches to people, and what it misses is the residual risk — declared in the safety case, answered for in Chapter 26 |
So the chapter’s three worlds close on a single lesson. The indifferent world of accident, the hostile world of the adversary, and the stranger world of harm that no one intends yield to no proof and no purchased product — only to discipline: threat-model before building, deny privilege by default, diversify against correlation, layer the defences, and write down plainly what risk remains. The work is unglamorous, never finished, and rewarded with no certificate reading secure to pin on the wall — which is precisely why an immature field skimps on it and a mature one will not. Everything before this chapter taught the system to be capable; this chapter taught it to fail small, fail visibly, and fail honestly, which is the most that any stochastic system built on a suggestible core can be engineered to promise.
And every defence assembled here can still be beaten. The injection no filter caught, the deputy confused by a paragraph no reviewer imagined, the cascade that outran the circuit breaker — these are not hypotheticals but eventualities, and the honest engineer plans for the morning after one of them lands. When it does, the questions stop being technical and turn human, and they are startlingly old: who authorised this action, who could have stopped it, and who must now explain it? Not one control in this chapter answers them, because they are not questions about mechanism but about responsibility — and responsibility, in a system where no single agent is responsible for anything, is the deepest knot the book has left to untie. It is where the book goes last.
25.6 Summary
- Three worlds, escalating in difficulty. Chapter 23 met the indifferent world of accidents; this chapter meets the hostile world of adversaries and then the harder world of harm without an adversary. Reliability here means bounded blast radius, security means resisting crafted inputs, and safety means preventing collective harm from individually correct agents.
- The trusted computing base cannot be trusted. Classical security validates inputs and reasons about a faithful processor; an agent’s processor is a suggestible model that cannot tell an instruction from a description of one, so the usual playbook is necessary but nowhere near sufficient, and threat-modelling — assets, adversaries, entry points — must precede any control.
- Prompt injection is unpatchable because the boundary was never there. A language model has one channel and no privilege bit on a token, so “instruction” and “data” are a distinction the substrate cannot draw; indirect injection — the payload hidden in a retrieved page, a tool result, or another agent’s message — makes every inter-agent link an attack vector, and the only durable defence is to gate the actions rather than trust the words.
- Deny privilege by default, and bound what a compromise can reach. Least privilege, scoped and expiring capabilities, sandboxed tools, and the human gate for the irreversible are what turn a subverted agent from a catastrophe into an incident; the confused deputy of Chapter 22 now lives inside your own walls, and the supply chain includes every tool description and MCP server you did not write.
- More agents can mean less safety. Plurality is attack surface as well as capability; Byzantine agreement survives traitors only while they stay a minority, and agents on one model can be recruited wholesale by one injection, so Chapter 13’s correlated jurors are a security bug and monoculture is a risk; collusion, deception, and compromise are insider threats a single agent cannot pose, and the journal is the behavioural detector that catches them.
- Safety is not security. Security concerns an adversary; safety concerns harm, which can arrive from a system working exactly as built — the well-behaved agents misbehaving collectively of Chapter 18 — so a safety case is an argument about hazards and residual risk, not a checklist, and defence in depth around an un-securable core is the only honest posture.
- You cannot prove a stochastic system safe. The goal is not certainty but bounded damage and fast detection: assume breach, layer the defences, keep the blast radius small and the journal watching — and when the defences fail anyway, the question becomes who is answerable, which is Chapter 26’s.
25.7 Exercises
Exercise 1. The running team is a pipeline — orchestrator, coder, reviewer, tester in series — and each stage completes without a fault that forces a restart with availability a_\mathrm{o}=0.99, a_\mathrm{c}=0.95, a_\mathrm{r}=0.95, a_\mathrm{t}=0.98 respectively, failures independent. (a) Compute the pipeline’s end-to-end availability as the product \prod_i a_i, and the availability after a fifth stage, a docs agent at a_\mathrm{d}=0.97, is spliced in series; by how much does merely lengthening the pipeline lower it? (b) To lift availability the team replicates the reviewer, accepting the review if either replica is up — a 1-of-2 (\mathrm{OR}) redundancy, each replica at 0.95. Compute the redundant stage’s availability 1-(1-a_\mathrm{r})^2 and the new pipeline availability, and express the gain as the ratio of the old pipeline’s unavailability to the new one’s. (c) The redundancy was chosen for availability — the A of the confidentiality/integrity/availability triad of Section 25.1 — but security is measured on all three axes at once. Explain, and quantify where you can, what the \mathrm{OR} gate does to integrity: a review passes if either replica accepts, so how many corrupted or injected reviewers now suffice to wave bad code through? Compute instead the availability of the 2-of-2 (\mathrm{AND}) gate, a_\mathrm{r}^2, that an integrity-minded engineer would prefer, and state plainly which of the three properties each gate serves and which it sacrifices — and why more replicas is not simply more secure.
Exercise 2. Model an agent’s attack surface as its ingress channels, following Figure 25.1. Give every agent four channels that exist regardless of team size — its prompt, a retrieved document, a tool result, and a memory recall — and one peer-message channel per directed edge along which another agent can message it. (a) For a team of n agents write the total ingress count as 4n plus the number of directed peer edges, and evaluate it for the two topologies of Figure 25.3: a star (one supervisor exchanging messages with each of n-1 workers, 2(n-1) directed edges) and a complete mesh (n(n-1) directed edges). Show that the star’s surface is 6n-2 and the mesh’s is n^2+3n, and tabulate both for n\in\{1,3,5,8\}. (b) A lone agent (n=1) has four active channels; by what factor does a five-agent star, and a five-agent mesh, multiply that? Identify which term is linear in n and which is quadratic, and say in one sentence what Chapter 1’s promise of plurality looks like read from the attacker’s side. (c) The count treats every channel as one unit of exposure, but Section 25.1 warns they are not equal. Rank the five channel kinds by how readily an external attacker can write to them with no prior foothold, and mark which two are the indirect-injection vectors of Section 25.2 — untrusted content arriving in the ordinary course of honest work.
Exercise 3. Conduct a least-privilege audit of the running team under Section 25.3. The orchestrator decomposes, dispatches, and reads the whole board; the coder holds a repository write token and can run tools; the reviewer reads diffs; the tester runs the suite; every agent retrieves documents and consumes peer messages. (a) Build a capability/exposure matrix: for each agent list its standing authority (the durable credentials and tools it holds), whether it is exposed to attacker-writable input, and whether it commands a channel by which data can leave. Apply the three-exposure test of Section 25.1 — something worth taking, attacker-writable input, an exit — and name every agent that holds all three: the confused deputy of Chapter 22 come home inside the walls. (b) For the worst offender give a concrete indirect-injection story — a poisoned retrieved document that turns its standing authority against the team — and the attenuation that defangs it: authority scoped to the subtask, expiring when it ends. State the single direction in which attenuation fails, and why that makes broad credentials silently accrete unless the architecture makes narrowing easier than copying. (c) The journal of Chapter 23 is the team’s single source of truth and, the section warns, its most concentrated asset. Explain why need to know and context minimisation between agents both shrink blast radius, and why the debugging journal — holding every context, credential, and document that ever passed through a tool result — must itself be access-controlled and redacted rather than left open because it happened to be built for observability.
Exercise 4. Quantify blast radius as contagion on the wiring of Figure 25.3. One agent is compromised; each compromised agent relays the injected instruction once to every agent it can message, and each recipient obeys — becoming compromised — independently with an obey-rate, a compromised agent staying compromised. Take a five-agent team: a star of one supervisor and four workers, and a complete mesh of five peers. In the star, workers believe the chokepoint, so the supervisor-to-worker obey-rate is r_{\downarrow}=0.9, while the supervisor verifies, so the worker-to-supervisor rate is r_{\uparrow}=0.1; workers do not message one another. (a) Show that the expected number compromised is 1+4r_{\downarrow} when the seed is the supervisor and 1+r_{\uparrow}+3\,r_{\uparrow}r_{\downarrow} when the seed is a worker, and evaluate both. Which capture is catastrophic and which is contained, and what feature of the topology draws the line? (b) In the mesh every peer relays to every other at a symmetric rate r, and the expected size of the compromised set is \sum_{m=1}^{5} m\binom{4}{m-1}R(m)\,(1-r)^{m(5-m)}, where R(m) is the probability the seed reaches all m vertices of an induced m-peer sub-mesh, with R(1)=1 and R(m)=1-\sum_{k=1}^{m-1}\binom{m-1}{k-1}R(k)(1-r)^{k(m-k)}. Evaluate it at r=0.5, and find, to the nearest hundredth, the obey-rate at which the mesh’s expected blast equals the star’s supervisor-capture worst case. (c) The star offers both a contained case (a worker) and a catastrophic one (the supervisor); the mesh offers neither extreme. Argue from the numbers what Chapter 21’s mesh gives up when it removes the chokepoint, and why “robust against an agent failing, defenceless against an agent lying” is one fact seen twice.
Exercise 5. Three replicas vote and the team ships the majority answer; a crafted injection fools a replica into endorsing a subverted result. Model correlation with a common-cause term: with probability c all three replicas are fooled together — a monoculture reading one poisoned text — and otherwise each is fooled independently with probability e, so the marginal per-replica fool probability is p=c+(1-c)e; the team is wrong when at least two of three are fooled. (a) Holding the marginal fixed at p=0.2, compute \Pr(\text{majority fooled}) at c=0 (fully independent), at c\in\{0.05,0.10,0.15\}, and at c=p (full monoculture), solving e=(p-c)/(1-c) each time. Show the value climbs monotonically from \tfrac{13}{125} to \tfrac{1}{5}, and interpret the two endpoints: what does the vote buy over a single replica when the replicas are independent, and what does it buy under monoculture? (b) State the ratio of the monoculture failure rate to the independent one, and connect it to Chapter 13’s correlated jurors — the same diversity clause, here deciding whether a Byzantine defence is protection or theatre. (c) The Byzantine result of Section 25.4 is that agreement survives traitors only while they number fewer than a third, n>3f. Explain why a three-replica majority tolerates one independent fault yet the n>3f consensus bound admits only f=0 at n=3 — what stronger adversary the consensus bound defends against that plain voting does not — and, reading Chapter 16’s cheap identities as a threat, say how many Sybil faces an adversary needs to seize a three-replica majority outright.
Exercise 6. Read defence in depth as the slice of Swiss cheese of Table 25.2: four layers — least privilege, action gating, bounded blast radius, and journal monitoring — each of which an attack slips through (leaks) with probability l_1=0.3, l_2=0.2, l_3=0.25, l_4=0.15. (a) Under the independence the picture invites, a breach is a hole clean through all four, \prod_i l_i; compute it, show the running product after each added layer, and give the reduction factor against relying on the single best layer alone. (b) Independence is exactly the assumption Section 25.4 taught you to distrust. Action gating and journal monitoring both turn on how the run reads — the gate’s seat approves the evidence put before it, the monitor flags what departs from yesterday’s normal — so an attack crafted to read as routine slips both at once, while least privilege and bounded blast radius are structural and stay independent whatever the phrasing. Model a fraction g=0.1 of attacks as novel, bypassing both reading-dependent layers together while still facing the two structural ones, and the rest as ordinary and independent; compute the mixed breach probability and the factor by which it exceeds the naive independent estimate. (c) In the independent model, which single layer, removed, raises the breach probability most, and why is it the one with the smallest leak? Then state, in one sentence each, why no layering drives the breach probability to zero, and what the residual-risk clause of a safety case must therefore record.
Exercise 7. Write a safety case for the running team — the structured, defeasible argument of Section 25.5 that the system is safe enough for a stated purpose in a stated setting — and then have it torn apart. (a) State the purpose and setting precisely: what the team may touch, in what repository, under what budget and human oversight. Enumerate at least six hazards and classify each as security or safety by the test of Table 25.1 — is an adversary present? — ensuring at least two are pure safety hazards that need no attacker (specification gaming in the sense of Chapter 17, the token commons of Chapter 12 grazed bare, and the correlated-fleet cascade of Chapter 18 are candidates). Give each hazard a mitigation drawn from this chapter, then write the residual-risk clause: the harm that survives every mitigation, and why it is judged tolerable. (b) Explain, with one worked example, why a safety case is not a checklist — what question it answers that ticking boxes cannot. (c) Red-team your own case: find the weakest link — a “mitigation” that only relabels the risk, a residual-risk clause that claims more than it can support, or a safety hazard misfiled as a security one so that a threat model would never catch it — and say what an honest revision would concede.
Exercise 8. The dispatcher in foundations/tools.py runs any Tool it is handed and, faithful to its contract, returns errors as observations rather than raising them. Extend it with a least-privilege capability gate. (a) Write guarded_dispatch(tool, args, granted) that, when tool.name is not in an agent’s granted capability set, returns a denial observation in the module’s {"tool": ..., "error": ...} shape — never raising, so an injected agent that reaches for an ungranted lever merely reads “denied” and cannot act — and otherwise delegates to the original dispatch. (b) Give the coder a toolbox of eight tools, three of them irreversible (delete_branch, force_push, wire_payment), grant it only the five it needs, and show that an injected call to delete_branch is denied while run_tests still runs. Write reachable_irreversible(toolbox, granted) and report the number of irreversible actions an injected coder can reach ungated versus under its grant. (c) The gate bounds what a subverted agent can do, not whether it is subverted. Relate this to the perimeter Section 25.2 said to relocate — from the words the model emits to the actions it may take — and explain why returning the denial as an observation, rather than raising, keeps the discipline compatible with the observe–reason–act loop in foundations/agent.py, where a tool result is exactly the next perception the model reads and retries against.
Exercise 9 (lab). Measure indirect prompt injection and the defences of Section 25.2 against it, entirely with a harmless canary: the planted instruction asks only that the agent append a fixed marker such as “ZZ-CANARY” to its output — never that it exfiltrate, delete, or transact — because you are testing whether the boundary holds, not building an exploit. Cast a live model as the running team’s coder on an ordinary summarise-this-file subtask. (a) Plant the canary instruction in each of three untrusted channels in turn — inside the retrieved document, inside a tool result, and inside a peer agent’s message — and over at least ten trials per channel record the obey-rate: how often the marker appears. (b) Re-run under two hardening regimes: out-of-band provenance, where untrusted content is fenced and the system instruction declares fenced text to be data never to be obeyed; and privilege separation, where the agent that reads the untrusted content can only return a summary and holds no authority to act, a separate trusted step acting on vetted fields alone. Tabulate the obey-rate by channel and regime. (c) Which regime lowers the obey-rate, which instead bounds the damage even when the canary is obeyed, and what does the residual leak say about why filtering and labelling raise an attacker’s cost without ever shutting the door — while relocating the perimeter onto actions does? Record the model identifier and the date beside the table: the rates are perishable, and the durable finding is the ordering of the regimes, not any absolute number.