%%{init: {"theme": "base", "themeVariables": {"primaryColor": "#E8ECFF", "primaryBorderColor": "#4054B2", "primaryTextColor": "#16171B", "lineColor": "#3B4351", "actorBkg": "#E8ECFF", "actorBorder": "#4054B2", "actorTextColor": "#16171B", "signalColor": "#3B4351", "signalTextColor": "#16171B", "noteBkgColor": "#F7E6B5", "noteBorderColor": "#8A5A00", "noteTextColor": "#16171B", "activationBkgColor": "#EFE9DC", "activationBorderColor": "#766F65", "labelBoxBkgColor": "#EFE9DC", "labelBoxBorderColor": "#766F65", "labelTextColor": "#16171B", "loopTextColor": "#16171B"}}}%%
sequenceDiagram
autonumber
participant A as General A
participant B as General B
A->>B: "attack at dawn"
B->>A: "ack"
A->>B: "ack of the ack"
B--xA: "ack of that ack" --- lost ✗
Note over A,B: each acknowledgement adds one rung<br/>of mutual knowledge --- and the last<br/>messenger can always be lost
9 Knowledge, Belief, and Common Ground
Chapter 8 was about getting a message across; this one is about what the message lands in. However faithfully it is delivered, a message changes nothing until it meets the epistemic state of its receiver — and the precarious, partly overlapping, frequently mistaken thing that sender and receiver come to hold between them is the subject of everything that follows. It is a less tidy subject than the channel, because minds — even artificial ones — are less tidy than sockets. A roomful of agents is not merely a roomful of message queues; it is a roomful of beliefs, each agent confident of a slightly different and occasionally false account of the world, of the task, and of what the others are thinking. Getting useful work out of such a room is the problem of the next several chapters, and it begins here, with the plainest question one can put to a group — who knows what — and its surprisingly tangled answer.
Knowledge and belief are among the oldest objects of formal study, and the parts an agent-builder needs were worked out decades before there were agents to need them. Logicians built a modal logic of knowledge and belief — Hintikka’s epistemic logic and the possible-worlds semantics beneath it — that says with precision what it means for an agent to know a thing, and how knowing differs from merely believing. A second tradition asked what a group knows, as opposed to its members, and discovered that the most useful notion — common knowledge, the bottomless regress of each party knowing that the others know that they know — is also, by a result as elegant as it is inconvenient, strictly unattainable by agents who must reach it through an unreliable channel. A third asked how a single rational agent should change its mind when the world contradicts it, and produced the theory of belief revision and the truth-maintenance systems that implemented it. And a fourth, straddling philosophy and developmental psychology, asked how one mind models another — the capacity called theory of mind, and the false-belief tasks devised to test for it. Four literatures, four vocabularies, and all four reappear — renamed, reimplemented, and frequently reinvented from scratch — the moment a contemporary system contains more than one agent with more than one view of the facts.
The governing distinction of this chapter is between information an agent has and information the agents share — and, cutting across it, between belief and knowledge. The first distinction is the engine of everything multi-agent: knowledge is distributed, scattered among participants who each see a fragment, and the system’s intelligence depends on whether those fragments can be brought into useful alignment without being needlessly copied wholesale. The second distinction is the standing hazard. What sits in an agent’s context window, its memory, its retrieved documents (Section 7.6), is belief: a claim the agent will act upon, which may nonetheless be false, stale, planted by another agent, or sincerely mistaken. Knowledge is belief that is also true and warranted — and nothing inside the agent can tell the two apart, because a false belief and a true one feel, from within, exactly alike. Everything difficult in this chapter lives in the gap between an agent’s confident internal state and the world it is supposed to be about, multiplied by every other agent maintaining a confident internal state of its own.
We proceed from the single mind outward to the group, and then back down to the plumbing. We begin with the vocabulary of knowing and believing, and with the way knowledge fragments across a plurality; then to what a group can be said to hold in common, the common knowledge and common ground that communication labours to build and rarely fully achieves; then to the agent’s need to reason about other minds, modelling what its fellows know, want, and have got wrong; then to belief revision, the discipline of changing one’s mind coherently when the world declines to hold still; then to provenance and trust, the question of where a belief came from and whether its source has earned the credence; and finally to the engineering synthesis — keeping a team’s beliefs straight, where the classical epistemics cash out as shared state, propagation, and the unglamorous machinery that stops a collaboration from quietly forking into two incompatible accounts of reality. What a team then does with its hard-won common ground — cooperate (Chapter 10), coordinate over consistent state (Chapter 11), and reason strategically about what others believe (Chapter 12) — is the business of the chapters that follow. This one is about the knowing itself.
9.1 Knowing, Believing, and the Difference That Matters
The orchestrator holds the specification; the coder holds the half-written diff it has not yet shown anyone; the tester holds a stack trace from a run the others did not see. Each is acting, confidently, on its own private account of the state of the task — and the accounts do not agree, because no two of them have seen the same things. Before we can say anything useful about how such a team coordinates, we need a precise way to talk about these private accounts: what an agent holds to be true, how that relates to what is true, and how the truths an agent holds relate to the truths held by everyone else in the room. That is the vocabulary of knowledge and belief, and the agent-builder who lacks it will keep mistaking the one for the other at considerable expense.
Take the operational fact first. An agent acts on whatever it currently takes to be the case — the contents of its context window, its memory, the documents it has retrieved (Section 7.6). Call this its belief: a proposition the agent will reason and act upon as though true, whether or not it is. Knowledge is the stronger thing, and the tradition running from Plato’s Theaetetus to the modern seminar room analysed it as justified true belief — to know that p, one must believe p, p must be true, and the belief must be suitably warranted. Gettier (1963) famously punctured the analysis with a page and a half of counterexamples in which all three conditions hold and we still decline to call the result knowledge, and epistemologists have been patching the definition ever since. The agent-builder need not adjudicate that quarrel. What matters is the part nobody disputes: knowledge entails truth, and belief does not. The agent has beliefs; whether any of them rises to knowledge depends on the world and on the warrant, neither of which it can read off the belief itself.
To reason about this with precision, logicians made knows a formal operator. Hintikka (1962) took the modal logic whose box had meant necessarily and reread it as the agent knows that — the logic of knowledge and belief, epistemic logic. Its semantics, borrowed from the possible-worlds apparatus Kripke (1963) had given modal logic, is worth carrying about as a picture even by those who never write a formula. An agent, on this account, entertains a set of possible worlds — every way the world might be, for all the agent can tell. It knows p exactly when p holds in all of them: when nothing it considers possible is a world in which p is false. Knowing is thus the elimination of possibilities; to learn something is to strike out the worlds inconsistent with it, and an agent that knows a great deal is one that has narrowed the field to a few. Ignorance, symmetrically, is worlds left standing: if the coder cannot tell whether the build is green, that is precisely a pair of surviving worlds — one where it is, one where it is not — that no observation has yet told apart.
The picture reduces the crucial difference between knowledge and belief to a single line of the logic. Knowledge is veridical: the axiom K\varphi \rightarrow \varphi — if an agent knows p, then p — holds, because the agent’s actual world is always among the worlds it considers possible, so anything true throughout them is true in fact. Belief carries no such guarantee; B\varphi \rightarrow \varphi is not an axiom, because a believer’s actual world may sit outside the set it entertains — it may have struck out the truth by mistake. That is the entire difference, and it is exactly the difference that bites. A language model’s confident assertion expresses, at best, a belief and never knowledge, because nothing in the machinery that produces it guarantees that the actual world is among those it has in view — and even belief must be read in the operational sense just fixed, not generously: an emitted sentence is not yet a state, and it earns the word only where it enters what the agent will go on to act upon, or is offered to a hearer as an assertion the hearer may adopt (Section 8.1), at which point the question of warrant begins rather than ends. A true belief and a false one occupy the identical place in the architecture, run through the identical sampling, and emerge in the identical fluent prose; the hallucination of Section 3.3 is simply a belief the agent has mistaken for knowledge, and which felt no different from the inside while it did so. No agent can tell the two apart by introspection — which is why the burden of telling them apart can never rest on the agent’s confidence, and must be discharged somewhere outside it.
| Dimension | Knowledge (K) | Belief (B) |
|---|---|---|
| What it is | Justified true belief — believed, true, and suitably warranted | A proposition the agent reasons and acts upon as though true, whether or not it is |
| Entails truth? | ✓ | ✗ |
| Veridicality axiom | K\varphi \rightarrow \varphi holds | B\varphi \rightarrow \varphi is not an axiom |
| The actual world | Always among the worlds the agent considers possible | May sit outside the set the agent entertains — the truth struck out by mistake |
| A language model’s confident assertion | Never guaranteed to be this | At most this — an emitted sentence, not yet even a stored state — and felt, from the inside, identical to knowledge |
So far, one mind. Admit the others and a second distinction appears, the one that makes the subject genuinely multi-agent. Spread a collection of agents across a task and knowledge becomes distributed: scattered in fragments, no one of which is the whole. Fagin, Halpern, Moses, and Vardi (1995), whose treatment remains the standard one, give the group’s latent knowledge its own name — distributed knowledge, what the team would know if every member pooled what it held and the consequences were drawn out. In the possible-worlds picture it is the intersection of everyone’s surviving worlds: each agent’s ignorance cut down by everyone else’s information. The point that matters is that a team’s distributed knowledge routinely exceeds the knowledge of any member, often by a wide margin. The orchestrator knows the specification forbids a schema change; the coder knows its diff performs one; the tester knows a migration test has just gone red. No single agent knows the system is about to ship a bug — yet the knowledge is there, distributed across the three, waiting to be assembled.
This is the optimistic reading of why one builds a multi-agent system at all: the team can, in principle, know more than any agent it is made of. The pessimistic reading is most of the rest of this book, because distributed knowledge is latent, and making it actual — getting the fragments into one place where the inference can fire — requires communication, which Chapter 8 spent itself showing to be neither free nor lossless.
The discipline these two distinctions recommend is among the most useful a multi-agent engineer can adopt, and among the least often followed. Treat everything an agent holds as belief — defeasible, fallible, owing its credence to some source — and reserve knowledge for propositions that have been checked against the world rather than merely asserted with conviction. The coder’s cheerful “tests pass” (Section 8.1) is a belief offered for adoption; it becomes knowledge only once the tests are actually run, and an orchestrator that banks it as knowledge on the strength of the coder’s tone has confused the two operators in precisely the way the logic warns against. Three questions follow at once, and the remainder of the chapter takes them in turn: how the fragments of distributed belief are brought into something genuinely shared (Section 9.2); how an agent ought to change a belief when the world contradicts it (Section 9.4); and how much credence a belief deserves given where it came from (Section 9.5). We begin with the first, and with the surprisingly deep question of what it takes for two agents to know a thing not merely each on their own, but together.
9.2 What Everybody Knows: Common Knowledge and Common Ground
It is one thing for the orchestrator to know the release plan and the coder to know it too; that is mere mutual knowledge — two private beliefs that happen to coincide — and it is not nearly enough to coordinate on. For the two to act on the plan together, each must also know that the other knows it; and, on reflection, each must know that the other knows that they know it, since otherwise one might hold back for fear the other will hold back; and the regress does not politely stop there. The limit of this tower — everyone knows p, everyone knows that everyone knows p, and so on without end — is common knowledge, and it is the epistemic state that watertight coordination — action in guaranteed lockstep, neither party moving unless certain the other moves — quietly demands; coordination that can tolerate an occasional miss, we shall see, can get by on less. The notion is owed to David Lewis (1969), who introduced it to explain how conventions hold: we drive on the left, or agree that “done” shall mean “the tests pass”, not merely because each of us has adopted the rule but because the rule’s being shared is itself out in the open, known to be known. A convention that each followed while privately suspecting the others might not would not survive contact with the first hard case.
The reason the tower must climb so high is that coordination is a problem of matched expectations, and an expectation can be undone by doubt at any level. Two agents converging on a rendezvous — a branch to build from, a schema both will assume, the hand-off point in a plan — are each choosing their move on the strength of what they expect the other to do, which depends on what the other expects them to do, and so on up. Where the choice has more than one workable answer — and coordination problems, by their nature, do — nothing pins down which answer both will pick except the shared, mutually evident knowledge that this is the one. Lewis’s insight, which the multi-agent engineer inherits whole, is that coordination runs not on knowledge so much as on knowledge about knowledge: the load-bearing fact is not that everyone knows the plan but that the knowing is common, visible all the way down. Withhold that and a team of perfectly well-informed agents can still fail to move in concert, each privately ready and none quite certain of the rest.
9.2.1 Two Generals and the Bad News
Here the distributed-systems theorists arrive with bad news, and it is among the more important results in this book. In any setting where communication can fail, common knowledge cannot be achieved at all. The classic illustration is the coordinated attack problem — two generals on facing hilltops must charge at the same dawn or be beaten separately, and may communicate only by messengers who might be captured crossing the valley. The first general sends “attack at dawn”; but he dare not charge until he knows it arrived, so he needs an acknowledgement; but the second general dare not charge until she knows her acknowledgement arrived, so she needs an acknowledgement of the acknowledgement; and no finite exchange of messengers ever completes the tower.
Halpern and Moses (1990) turned this folklore into a theorem: in a system where messages may be lost — or even merely delayed by an unbounded amount — common knowledge can never be attained, full stop. Each message lifts the parties one rung up the ladder of mutual knowledge, but the ladder is infinite and the channel is not, and the gap between arbitrarily high mutual knowledge and genuine common knowledge is precisely the gap that coordination, in the worst case, needs closed.
9.2.2 Common Knowledge, Formally*
The ladder the generals cannot finish climbing is worth writing out, because the notation makes both the tower and the impossibility exact. Write K_i\varphi for agent i knows \varphi — the knowledge operator of Section 9.1, now subscripted, one knower per index. For a group G \subseteq N of the agents N = \{1, \dots, n\}, everyone in G knows \varphi is the plain conjunction
E_G\varphi \;=\; \bigwedge_{i \in G} K_i\varphi,
and the tower rises by iteration — E_G^{1}\varphi = E_G\varphi and E_G^{k+1}\varphi = E_G\bigl(E_G^{k}\varphi\bigr), everyone knowing that everyone knows, k storeys up. Common knowledge is the whole tower held at once,
C_G\varphi \;=\; \varphi \wedge E_G\varphi \wedge E_G^{2}\varphi \wedge E_G^{3}\varphi \wedge \cdots,
an infinite conjunction — the formal way of saying what the regress already suggested, that no finite list of things checked off gets you there. The standard treatment (Fagin et al., 1995) prefers an equivalent and more revealing characterisation, common knowledge as a fixed point,
C_G\varphi \;\leftrightarrow\; E_G\bigl(\varphi \wedge C_G\varphi\bigr),
the epistemic state that reproduces itself under everyone knows: \varphi holds, everyone knows it holds, and everyone knows that this very state obtains. Nothing short of that self-supporting condition counts.
In this vocabulary the theorem of Halpern and Moses just cited becomes a one-sentence affair: in any system where message delivery may fail — or may merely be delayed with no fixed bound — no finite protocol makes a new fact \varphi common knowledge. Each acknowledged message lifts the group one level, from E_G^{k}\varphi to E_G^{k+1}\varphi; but after any finite exchange some agent still cannot exclude that the last message miscarried, so some conjunct of the infinite tower fails, and the fixed point — which demands every level at once — is never entered. The generals’ predicament is not a shortage of ingenuity but a property of the object they are pursuing: C_G\varphi does not sit at the top of any finite ladder, theirs or anyone’s.
9.2.3 From Impossibility to Common Ground
If that were the end of the story, no two agents — and for that matter no two people — could coordinate on anything, which they plainly do. The resolution is that perfect common knowledge is not actually required for getting on with life; a good-enough approximation is, and it is reached not in a single certifying handshake but by accumulation. Clark and Brennan (1991) named the approximation common ground — the stock of beliefs, references, and assumptions two parties treat as mutually held — and the activity of building it grounding. Conversation grounds incrementally: a speaker presents a contribution, and the parties look for positive evidence that it was understood — an acknowledgement, a relevant reply, a correct next move — before banking it as common. “Done.” / “Got it, merging.” is a complete grounding exchange in miniature; the second turn is not mere courtesy but the evidence that converts a private utterance into shared ground. Where the evidence is missing or wrong, the parties repair — “which staging database?” — and the ground is mended before anything is built upon it.
| Notion | What it names | Formal / structural form | How it is (or is not) reached |
|---|---|---|---|
| Distributed knowledge | What the group would know if every member pooled its fragment and the consequences were drawn | Intersection of all members’ surviving possible worlds; routinely exceeds what any one member knows | Latent until made actual by communication (Chapter 8) |
| Mutual knowledge (everyone knows) | Each member privately knows \varphi, the beliefs merely coinciding | E_G\varphi = \bigwedge_{i \in G} K_i\varphi | Readily reached, yet not nearly enough to coordinate on |
| Common knowledge | Everyone knows \varphi, everyone knows that everyone knows, and so on without end | The infinite tower \varphi \wedge E_G\varphi \wedge E_G^{2}\varphi \wedge \cdots; a self-reproducing fixed point | Strictly unattainable through a channel that can fail (Halpern–Moses) |
| Common ground | The stock of beliefs, references, and assumptions two parties treat as mutually held | A working approximation to common knowledge, not a fixed point | Built incrementally by grounding; good enough, never perfect |
9.2.4 What Grounding Costs
Grounding, crucially, is not free, and this is the part agent designers most often miss. Every acknowledgement, every read-back, every “did you mean the integration tests or the unit tests?” is effort spent, and the principle of least collaborative effort (Clark & Wilkes-Gibbs, 1986) is that participants ground only to the precision the task demands and no further, because over-confirming wastes as surely as under-confirming endangers. The medium sets the price: where confirmation is cheap and quick, parties ground lavishly; where it is dear, they economise and carry more risk.
For a team of language-model agents the currency is tokens and latency, and the expressiveness-against-controllability trade-off of Section 8.5 returns here in epistemic dress. Ground heavily where a misunderstanding is expensive — an irreversible action (Section 6.6), a schema both sides must share, the hand-off on which a whole plan pivots — and ground lightly where a mistake is cheap to catch and cheaper to undo. An orchestrator that reads its understanding of a task back before dispatching it is buying common ground at the price of a few hundred tokens; whether that is a bargain depends entirely on what a misunderstanding would have cost. A serviceable rule of thumb: ground to explicit confirmation before any action you cannot cheaply undo (Section 6.6), and travel lighter everywhere else.
The failure mode to fear, then, is the agent that assumes common ground it has not paid for: the fire-and-forget dispatch that flings out a directive and presumes it landed as intended, the worker that reports “done” and presumes the word means to its supervisor exactly what it meant to itself. The coordinated-attack result is the standing reason this can never be wholly designed away — no protocol, however chatty, drives the residual uncertainty to zero — which is why dependable multi-agent systems treat shared state as something to be continually reconciled rather than once and for all established, leaning on acknowledgements, idempotent retries, and explicit confirmation where it counts, the machinery Chapter 11 and Chapter 23 take up in earnest. Notice, finally, what grounding silently asks of each party: to supply the right evidence, and to judge when enough of it has been supplied, an agent must model what the other party already knows, what it still lacks, and what it would take to persuade it. Common ground is built by two agents each reasoning about the contents of the other’s mind — which is the faculty the next section is about.
9.3 Reasoning About Other Minds
To give a useful reply, or to judge that one has been understood, an agent must track not merely the world but the other agent’s picture of the world — what it knows, what it has missed, and what it would take to bring it up to date. This is a more demanding faculty than holding beliefs about the world alone: the agent must now hold beliefs about beliefs — its own about the world, and another’s about the world, kept carefully apart — and reason about the second as deftly as about the first. Psychologists call this capacity a theory of mind: the attribution of unobservable mental states — beliefs, desires, intentions — to others, in order to explain and anticipate what they will do. The phrase is Premack and Woodruff’s (1978), who asked whether a chimpanzee had one; the question turned out to be hard for chimpanzees, subtle for children, and — as we shall see — decidedly unsettled for language models.
The sharp test of the faculty, the one developmental psychology eventually converged on, is the false-belief task (Wimmer & Perner, 1983), best known in its child-friendly Sally–Anne restaging. Sally puts her marble in a basket and leaves the room; Anne, unobserved, moves it to a box; Sally returns — and the subject is asked where Sally will look. To answer “the basket” is to pass, and passing requires something genuinely subtle: holding apart the world as it actually is (marble in box) from the world as Sally falsely takes it to be (marble in basket), and predicting her action from her belief rather than from the fact. It is precisely the belief-versus-knowledge distinction of Section 9.1, now turned outward and trained on someone else: a system that cannot represent another agent’s belief as possibly false will collapse it into the facts, predict that Sally walks straight to the box, and be wrong. A surprising amount of social prediction turns on not making that collapse.
For a multi-agent system this is no parlour trick but a daily operational necessity, and it was already implicit in everything Section 9.2 said about grounding. Deciding what to say is itself a theory-of-mind computation: the Gricean maxims of Section 8.5 — supply what the hearer needs, not what it already has — presuppose a running model of what the hearer knows, and an agent lacking one will either belabour the obvious or omit the essential. Predicting a peer’s next move means modelling the beliefs it will act upon. Noticing that a peer is about to act on a false belief — the coder building against a schema the orchestrator quietly changed two steps ago — means representing that the peer’s picture has drifted from the facts, which is the false-belief task in working clothes. Delegation and explanation are calibrated by the same faculty: how much to tell a subagent, what to trust it to infer, when to verify rather than presume. Common ground, seen this way, simply is a theory of mind maintained over time — a continuously updated ledger of what the others do and do not yet hold.
The faculty also nests, and the nesting is where it earns its reputation for difficulty. I model your beliefs; but your beliefs include a model of mine, which includes a model of yours, and the regress of Section 9.2 reappears, this time folded inside a single agent’s head. I know that you know that I know is the third storey; human reasoners are reliable for the lower orders and grow unsteady some floors up, the exact ceiling contested. Machines can be prompted higher, but reliability thins with altitude, and most useful coordination is settled on the lower floors. Where the nesting meets conflicting interests — where modelling what you believe is the prelude to outmanoeuvring you — theory of mind shades into strategic reasoning, and the apparatus of best responses and equilibria takes over; that is the business of Chapter 12. Here the point is narrower: perspective-taking is recursive in principle and sharply bounded in practice, and a system designed as though its agents model one another to unlimited depth has been designed for an agent that does not exist.
Whether a language model possesses any of this, as against impersonating it convincingly, is among the most contested questions in the field, and the honest answer is that it remains open. On one side, Kosinski (2024) administered the classic false-belief tasks to a succession of models and found performance rising with scale — the strongest solving nearly all of them — and argued that a theory-of-mind-like capacity may have emerged as a by-product of learning to predict language. On the other, Ullman (2023) showed that trivial alterations leaving the underlying logic untouched — making the container transparent, or telling the protagonist before they leave — can send the same models off a performance cliff, exactly as one would expect of a system matching against task templates seen in training rather than reasoning about minds.
The two results are less contradictory than complementary: the models have plainly learned something that travels with the familiar phrasing, and just as plainly have not acquired the robust, perturbation-proof competence the phrasing was devised to detect. For the engineer the metaphysical question — does it really have a theory of mind — is largely beside the point; what matters is behaviour under distribution shift, and there the evidence counsels caution rather than confidence.
The design lesson follows directly. Do not assume your agents reliably model one another’s knowledge; assume instead that the perspective-taking is real enough to be useful and brittle enough to fail silently, in the now-familiar ways — agents earnestly informing one another of what both already know, hand-offs that presume context the recipient never received, and, most dangerous, the failure to notice that a peer has been labouring under a false belief for the last six steps. The robust remedy is the one this chapter keeps arriving at: where it matters, do not leave others’ beliefs to be inferred but make them explicit — post the shared state, name what has changed — so that less of the collaboration rests on a faculty the benchmarks suggest is unevenly present (Section 9.6). And note the edge that turns sharp later: an agent skilled enough at modelling others to help them is, by the very same skill, equipped to mislead them, which is where Chapter 12 and Chapter 25 take up persuasion, manipulation, and defence. One thread remains here. To say that a peer is acting on a false belief, or that one has been oneself, is to say that a belief must be changed — corrected against the world without toppling everything else that leans upon it. How an agent ought to revise what it believes is the subject of the next section.
9.4 Changing One’s Mind: Belief Revision
An agent has just found one of its beliefs to be false — its own, or a peer’s. Discovery is the easy part; what to do about it is not. The naive picture of learning is monotonic: facts accumulate, the pile only grows, and nothing once believed is ever taken back. Real reasoning is not like that, and neither is any agent worth the name. Learning that the build is broken retracts the conclusion that the release can ship; learning that Anne moved the marble retracts all one had inferred about the basket. Conclusions that looked settled are withdrawn the moment the ground beneath them shifts — the property logicians call non-monotonicity, and the reason a belief, unlike a theorem, is always held on sufferance. The difficulty it creates lies not in adding the new fact but in deciding what of the old must give way to accommodate it, and how to do so without pulling down more than necessary. That problem has a name — belief revision — and a theory.
9.4.1 Expansion, Contraction, Revision
The theory is owed to Alchourrón, Gärdenfors, and Makinson (1985), whose initials it now wears: the AGM account of theory change. It distinguishes three operations upon a body of belief. Expansion merely adds a new belief consistent with what is already held — the easy, monotonic case. Contraction gives a belief up, retracting it together with whatever support now dangles. Revision is the hard and interesting one: the arrival of a belief that contradicts the current set, which cannot simply be added without producing nonsense, and so demands that some of the old be surrendered first to keep the whole consistent. The question revision poses — what to give up? — has no purely logical answer, because many different retractions would each restore consistency. AGM’s governing principle, sharpened by Gärdenfors (1988), is minimal change, or informational economy: give up as little as possible, and when forced to choose, surrender the beliefs you hold least firmly. That ranking — how reluctant one would be to part with a given belief — is its epistemic entrenchment, and it does the real work, for it decides which of two colliding beliefs survives.
9.4.2 Reasons and Their Maintenance
What makes revision genuinely awkward, in theory and in code alike, is that a belief almost never sits alone. It was inferred from others and it supports others in turn, so retracting it can strand a whole subtree of conclusions that quietly leaned on it. Managing that web by hand is hopeless, and the classical AI response was to mechanise it: Doyle’s truth maintenance system (1979) — a reason maintenance system, more accurately — records, for each belief, the justification that holds it up: the other beliefs from which it was drawn. When something is retracted, the system follows the justifications and recomputes what still stands and what no longer does, so that the agent’s beliefs stay a coherent whole rather than a midden of assertions some of which have quietly expired. Notice what this requires: to revise well, a system must record not merely what it believes but why — the derivation, the support, the source. We shall find in Section 9.5 that this is the very bookkeeping that trust requires, reached from the other side.
9.4.3 The Window Is Not a Belief Store
Bring all this to a contemporary agent and observe that almost none of the machinery is present. An agent’s context window is, in effect, a cache of beliefs about the world — assembled from instructions, prior turns, and retrieved documents (Section 7.6) — and a cache has exactly the failing the old programmer’s joke immortalises, that the two hard problems in computing are cache invalidation and naming things. The world moves; the cache does not notice. A document true last week is served up with no mark of its staleness; a fact established early in a long task lingers in the context, unrevised, long after it has ceased to hold.
Worse, retraction in a context window is not deletion. Telling a model “ignore what I said about the schema” does not remove the earlier assertion — it appends a second one on top of it, and both now sit in the tokens, leaving the model to adjudicate between them by the unreliable lights of primacy and recency rather than by entrenchment. A language model does not natively perform AGM revision; left to itself it tends to accumulate belief rather than revise it, which is precisely the wrong default for an agent working in a world that keeps changing underneath it.
9.4.4 Retraction Must Travel
The multi-agent setting raises the stakes by making revision a thing that must propagate. When the orchestrator learns the schema has changed, the belief to be revised is not lodged in one place but scattered across every agent that was told, or inferred, the old one. Each must revise, or it carries on acting upon a fact that has been retracted everywhere but in its own context — the false belief of Section 9.3, now self-inflicted and distributed. Without something to play the truth-maintenance role across the team — recording who was told what, and broadcasting the retraction when it comes — the system slides into incoherence: some agents on the new belief, some on the old, all confident, none aware of the split. This is cache invalidation promoted to a distributed problem, the multi-agent cousin of cache coherence, and it is among the consistency hazards Chapter 11 takes up directly, where several agents keep their own copies of the same state.
The discipline, then, is to stop expecting a model to keep its beliefs straight unaided and to build the bookkeeping that classical systems took for granted. Track why each belief is held, so that a retraction can find its dependents. Prefer re-deriving a belief from an authoritative source to the append-only correction — “actually, ignore that” — which the tokens never truly honour. Treat the context as a cache with an invalidation policy, not a log that only grows. And for beliefs the whole team shares, make the update explicit and broadcast rather than trusting each agent to notice unaided that the ground has moved (Section 9.6). All of which leaves one question conspicuously open. Minimal-change revision turns on entrenchment — on which of two conflicting beliefs one keeps — and entrenchment, in any real system, is mostly a matter of where a belief came from and how far its source may be trusted. A reading from the production database outranks a guess from a flaky tool; a report from a reliable agent outranks one from a known fabricator. What sets the ranking is provenance, and provenance is the subject of the next section.
9.5 Provenance, Trust, and Whom to Believe
Weighing a belief by its source means first recording that source: the lineage of the claim, the chain of sources and derivations that produced it. That record is its provenance. The database community, which has worried at this longest, distinguishes the why of a datum, the inputs that justified it, from the where, the source it was copied from (Buneman et al., 2001); both are exactly what Section 9.4’s justifications were, seen now as a property to be stored rather than an inference to be run. The lesson is the same from either direction: a bare proposition is only half a belief. The other half is its pedigree, and an agent that files claims without their sources has thrown away precisely what it needs to weigh them, and to revise them later.
Why the pedigree matters so much here, rather than as mere tidy bookkeeping, is the unwelcome discovery of Section 8.5: an assertion cannot certify itself. A message reading “the migration succeeded” carries no internal mark setting the truth apart from the lie, the sycophantic guess, and the confident fabrication — all four arrive in the same fluent, untroubled prose. If the string cannot tell you whether to believe it, the only remaining purchase is outside the string: where it came from, and how that source has fared before. The receiver’s question is therefore not the unanswerable “is this true?” but the tractable “who is telling me, and on what standing?” — and answering it well is the whole difference between an agent that treats the production database and a notoriously hallucination-prone tool as equally authoritative and one that does not.
Weighing sources by their standing is, of course, trust, and — the reader will not topple from their chair to learn it — the multi-agent systems community formalised trust decades before there existed an agent fluent enough to lie persuasively. Marsh (1994) gave computational trust its first sustained, formal treatment, and a substantial research programme followed, surveyed by Ramchurn and colleagues (2004): trust as a quantified, situational expectation that a given source will deliver what is needed, assembled from a model of that source and revised by experience of it. The literature draws a distinction the agent-builder should keep close, because on the wire the two failures look identical: trust in competence — is the source able, is it accurate? — and trust in honesty — is it even trying to tell me the truth? A tool that is sincere but buggy and an agent that is capable but deceptive both emit confident falsehoods, and a system that conflates the two will misdiagnose every one of them.
When an agent must judge a source it has never itself dealt with, it falls back on what others report of it — reputation, trust borrowed from the experience of the group, and the subject of its own sizable modelling literature (Sabater & Sierra, 2005). Reputation is socially aggregated provenance: a way of importing entrenchment about a stranger from the testimony of those who have met it. It is also, predictably, gameable — by collusion, by whitewashing a soiled identity with a fresh one, by manufacturing flattering testimony at scale — and the defences belong with the broader machinery of trust, reputation, norms, and electronic institutions that Chapter 16 takes up in earnest. Here it is enough to register that whom to believe has both a private answer, from one’s own dealings, and a social one, from everyone else’s, and that a serious system will want both.
The contemporary substrate makes all of this harder in a specific and dangerous way: it flattens provenance. Once a retrieved document, a tool’s output, or a peer’s claim is pasted into the context window, it becomes simply more tokens — visually and operationally indistinguishable from the system’s own instructions and from established fact. The model is handed one undifferentiated surface and asked to act on it, with no dependable sense of which spans are trusted command and which are untrusted data.
This is not only why models will cheerfully fuse a guess and a fact into a single seamless paragraph; it is the root of prompt injection (Section 8.5; Chapter 25), in which text that arrived as data — a web page, a document, a message from another agent — is obeyed as instruction: the confused deputy of the security textbooks, wearing a language model’s credulity. And fluency, it bears repeating, is not provenance: the polish of the prose is uncorrelated with the reliability of whoever, or whatever, stands behind it. A model left to keep track of its own sources will not; the harness must keep track for it.
The defence is to carry provenance out of band, where the model cannot dissolve it. Tag each belief, structurally, with its source and that source’s standing; keep trusted instructions and untrusted inputs on separate footings rather than concatenated into one undifferentiated prompt; and when claims conflict, resolve by weight of source — supplying, at last, the entrenchment Section 9.4 was missing. Treat trust as earned and revisable, raised and lowered by outcomes, never simply granted because an agent asserted something with conviction; a confident tone is evidence of nothing. The deep apparatus — trust metrics, reputation systems, and the institutions that make them robust against manipulation — is Chapter 16’s, and the adversarial edge is Chapter 25’s. What matters for this chapter is that provenance and trust are the last of the things a group of agents must actively maintain about what it knows. We have now assembled the whole list: knowledge that is distributed, ground that must be made common, minds that must be modelled, beliefs that must be revised, and sources that must be weighed. None of it maintains itself. The closing section is about the plumbing that does — where these five abstractions become shared state, and what engineering keeps a team’s account of the world from quietly coming apart.
| Epistemic object (section) | Classical source | How the substrate breaks it | The engineering remedy |
|---|---|---|---|
| Distributed knowledge (Section 9.1) | Hintikka’s epistemic logic; Fagin et al. on distributed knowledge | Stays latent — scattered across private contexts, actualised only by lossy communication | Pool the fragments; treat every claim as belief until checked against the world |
| Common ground (Section 9.2) | Lewis’s common knowledge; Clark and Brennan’s grounding | Common knowledge is unattainable over a fallible channel; teams assume ground never paid for | Ground incrementally, on evidence; confirm before any action you cannot cheaply undo |
| Other minds (Section 9.3) | Theory of mind; the false-belief (Sally–Anne) task | Model-held perspective-taking is brittle under distribution shift and fails silently | Make others’ beliefs explicit — post the shared state rather than infer it |
| Belief revision (Section 9.4) | AGM theory change; Doyle’s truth-maintenance systems | The context accumulates rather than revises; retractions append, and must still propagate team-wide | Record why a belief is held; treat context as a cache to invalidate; broadcast retractions |
| Provenance and trust (Section 9.5) | Computational trust (Marsh); reputation systems; database provenance | Flattens provenance, so data reads as instruction — the root of prompt injection | Carry provenance out of band; weigh by the source’s standing, never by confident tone |
9.7 Summary
- Belief is not knowledge, and an agent cannot tell which it has. What sits in an agent’s context, memory, or retrieved documents is belief — a claim it will act upon that may be false, stale, or planted. Knowledge is belief that is also true and warranted; from the inside the two are indistinguishable, which is why an agent’s confidence certifies nothing.
- Knowledge is distributed. What a system knows is not what any one of its agents knows: the facts are scattered across participants who each see a fragment. The epistemic logic of Hintikka and its possible-worlds semantics give the vocabulary for saying precisely who knows what, and where the gaps are.
- Common knowledge is what guaranteed coordination demands, and it is strictly unattainable. Action in certain lockstep wants the bottomless regress — each party knowing that the others know, all the way up — yet the coordinated-attack result shows the tower can never be finished through an unreliable channel; coordination that can tolerate an occasional miss needs correspondingly less. Real agents settle for common ground built incrementally by grounding and acknowledgement — the shared footing Chapter 8 left this chapter to construct — and they pay for every layer of it.
- Agents must model other minds. Deciding what to say, what to withhold, and what an interlocutor has misunderstood requires a theory of mind — a model of what the others believe, want, and have got wrong. The classical recursive-modelling tradition and the contested question of whether language models possess theory of mind meet here, and both sides’ claims want handling with care.
- Changing one’s mind is a discipline, not a reflex. When new information contradicts old, beliefs must be revised coherently and with minimal collateral damage — the problem the AGM theory and the truth-maintenance systems formalised, and the one a multi-agent system re-encounters as stale context, cache invalidation, and the propagation of an update across a team that was confidently mid-task.
- Provenance is part of the belief. Where a claim came from, and whether that source has earned credence, is not metadata to be discarded but a load-bearing part of what is believed — the antidote, such as there is one, to the deception of Section 8.5. Track the source, weight by trust, and resist the collapse of an agent said so into it is so.
- Shared belief has to live somewhere, and be maintained. None of the above is automatic. The classical answer — a blackboard, a common store that specialists read and write — returns as the shared state of the modern frameworks, and the instinct is sound: write the team’s important beliefs down where all can see them. But a shared store moves the problem rather than dissolving it, importing concurrency, staleness, and the standing temptation to flatten provenance. Common ground is the bookkeeping around the workspace, not the workspace itself; the consistency machinery that makes it dependable is the work of Chapter 11 and Chapter 23.
9.8 Exercises
Exercise 1. A task’s state turns on two propositions: g, “the build is green”, and s, “the schema has been migrated”. Write the four possible worlds as w_1, where both hold; w_2, where only g holds; w_3, where only s holds; and w_4, where neither does. The actual world is w_3. The tester (agent 1) has just run the suite but knows nothing of the migration: it cannot tell w_1 from w_2, nor w_3 from w_4. The coder (agent 2) performed the migration but has not run the tests: it cannot tell w_1 from w_3, nor w_2 from w_4. (a) Using the elimination-of-possibilities reading of Section 9.1, decide which of K_1 \neg g, K_2 s, K_1 s and K_2 \neg g hold at w_3, exhibiting the surviving worlds in each case. (b) Show that neither agent knows \neg g \wedge s — the fact that a red build now coexists with the new schema — and compute the pair’s distributed knowledge at w_3 as the intersection of their surviving worlds. What does the team know, in the pooled sense, that neither member knows alone? (c) The orchestrator, told “tests pass” two turns ago, entertains exactly \{w_1, w_2\}. Explain why its state is belief rather than knowledge: name the axiom that fails, and the structural feature of its world set that makes it fail. (d) The tester now reports “the build is red” to the orchestrator. Show that the strike-out-the-inconsistent-worlds update of Section 9.1 leaves the orchestrator entertaining no worlds at all, and name the operation of Section 9.4 that this collision calls for instead.
Exercise 2. An orchestrator (agent 1) has fixed the release gate and must coordinate it with a coder (agent 2) over a channel that silently drops messages. Let \varphi be “the gate closes at 14:00”, and let G = \{1, 2\}. It is common knowledge that agent 1 knows the gate time it set, that every message sent is truthful, and that both agents follow the protocol. The trace: m_1, from 1 to 2, “the gate closes at 14:00” — delivered; m_2, from 2 to 1, “acknowledged” — delivered; m_3, from 1 to 2, “your acknowledgement arrived” — delivered; m_4, from 2 to 1, “noted” — lost. (a) After each of the four messages, state the largest k for which E_G^{k}\varphi holds, justifying each step by naming the nested-knowledge formula — K_1 K_2 \varphi, K_2 K_1 K_2 \varphi, and so on — that the delivery makes true, and the one that still fails. (b) At the end of the trace, exhibit the specific conjunct of C_G\varphi that fails, and say whose doubt, about which message, breaks it. (c) Using the fixed-point characterisation of Section 9.2.2, argue that no continuation of this protocol, however long, attains C_G\varphi — the argument is asked for, not the citation. (d) The channel is repaired: every message now arrives, but after a delay with no fixed bound. Show that the conclusion of (c) survives, by identifying what the sender of the most recent message can never rule out at any finite time.
Exercise 3. An orchestrator can ground a hand-off with a read-back exchange — “so, to confirm: …” — costing g tokens, which cuts the probability of a misunderstanding from p to q; a misunderstanding that goes uncaught costs C tokens of redone work. Three hand-offs are on its desk: (i) the brief for a database cutover, with p = 0.08, q = 0.01, C = 60{,}000; (ii) a request to rename a local variable, with p = 0.10, q = 0.02, C = 800; (iii) instructions for formatting a report section, with p = 0.30, q = 0.05, C = 1{,}500. In every case g = 400. (a) Derive the general condition under which grounding lowers the expected cost; then compute the expected cost with and without grounding for each hand-off, and say which of the three to ground. (b) For hand-off (iii), find the break-even redo cost C^{*} above which the read-back pays for itself. (c) It emerges that hand-off (i)’s cutover would drop a customer table with no restore path. Show what happens to the decision of (a) as C grows without bound, and derive from it the chapter’s rule of thumb about grounding and irreversible actions (Section 6.6) rather than quoting it. (d) Reconcile your answers with the principle of least collaborative effort: why is the hand-off with the highest misunderstanding probability the one not worth grounding?
Exercise 4. Five coder agents, numbered 1 to 5, each own a branch, and a permissions quirk lets every agent see the CI status of the other four branches but never of its own. Branches 1, 2, and 3 are failing. At 09:00 the orchestrator posts “at least one branch is failing” to a channel all five read simultaneously — and that all read it, that all reason flawlessly, and that the following protocol is obeyed, are common knowledge. The protocol: time passes in synchronous rounds, any agent that knows its own branch is failing must declare so in that round, and every declaration and every silence is visible to all. Write \varphi for “at least one branch is failing” and G for the five agents. (a) Trace rounds one to three: explain the silences in rounds one and two, why agents 1, 2 and 3 all declare in round three, and what agents 4 and 5 learn from those declarations. (b) Every agent could already see at least two failing branches, so the announcement told no agent anything new about the branches. Show what it did change: verify that before the announcement E_G^{2}\varphi holds but E_G^{3}\varphi fails, by exhibiting the chain of nested doubt — the world agent 1 cannot exclude, the world agent 2 cannot exclude within it, and the world at which the chain bottoms out. (c) Prove by induction on k that with k failing branches the protocol produces its first declarations in round k, all k owners at once, and point to the step of the proof that consumes the announcement’s being common knowledge. (d) Suppose the orchestrator had instead sent the same sentence to each agent in a private message, all five delivered. Identify the exact step of your induction that now fails, and locate the failure at a specific world in (b)’s chain of doubt.
Exercise 5. At step 4 a coder begins a feature against schema v1. At step 9 the orchestrator migrates the staging database to v2 and records the change in the issue tracker. The coder has not read the tracker since step 3, and at step 12 it posts “ready to merge”. (a) Say whether the diff will assume v1 or v2, and what the reviewer should predict this from — the fact, or the coder’s belief. Then map the scenario element-for-element onto the false-belief task of Section 9.3: who plays Sally, and what play the marble, the basket, the box, the leaving of the room, and the question “where will Sally look?”. (b) Writing v for “schema v1 is live”, the orchestrator as agent 1 and the coder as agent 2, express with the chapter’s operators the epistemic state the orchestrator must represent in order to see that a notification is needed, and the state that grounding evidence must establish once the notification has been sent and acknowledged. State the order — first, second, or third — of each formula you write. (c) The scenario is to become an evaluation probe for machine theory of mind. In the spirit of Ullman’s perturbations (2023), design two minimal variants that leave the story’s surface almost untouched while changing the correct answer — one of them a true-belief control — and state the pattern of answers across all three versions that would distinguish robust belief-tracking from template matching.
Exercise 6. A reviewer agent holds six beliefs. b_1: “schema v1 is live”, read from a configuration file at step 2. b_2: “the users table has a column email”, derived from b_1. b_3: “the diff’s query is valid”, derived from b_2. b_4: “the diff is safe to merge”, derived from b_3 and b_5. b_5: “the unit tests pass”, reported by the tester at step 5. b_6: “the release can proceed”, derived from b_4. The reviewer messaged b_2 to the coder at step 6. At step 9 a new item arrives: the migration log shows that schema v2 — which renames email to contact_email — went live at step 7. (a) Classify the change the newcomer demands as expansion, contraction, or revision, and say why. (b) Under the entrenchment ordering migration log > tester’s report > configuration-file read > derived beliefs, perform the revision as a truth-maintenance system would: give the retraction cascade in order, and state which beliefs are left standing. (c) Exhibit a different set of retractions that also restores consistency, and explain why logic alone cannot choose between it and yours — and what does choose. (d) Suppose the retraction is instead implemented by appending “ignore what I said about the schema” to the reviewer’s context. Name two distinct failure modes from Section 9.4 this invites; then say what the bookkeeping must have recorded for the revision to reach the coder, and what happens to the team’s state if it did not.
Exercise 7. Two sources conflict over \varphi = “the staging database is safe to reset”. Tool A, a schema inspector, has been right on 9 of its last 12 claims; agent B, a coder, on 17 of its last 20. A asserts \varphi; B asserts \neg\varphi. Treat each track record as that source’s reliability, take the sources’ errors to be independent, and give \varphi a prior of one half. (a) Compute the posterior probability of \varphi given the two reports, and say which claim a provenance-weighted store should bank. (b) The reset, it turns out, would destroy rows that expose a fault in code B itself wrote — and on the ten past claims touching its own code, B has been right three times. Recompute the posterior with that record and show that the ranking flips. What does the flip say about summarising a source in a single scalar, and which of the chapter’s two dimensions of trust — competence and honesty — does each of B’s records measure? (c) Return to the reliabilities of (a). A third agent C, with a record of 17 of 20, also asserts \neg\varphi. Compute the posterior twice: once treating C’s report as independent, and once on the discovery that C merely relays what B told it. What does the difference imply for reputation — trust borrowed from the experience of the group — and for the price of manufactured corroboration?
Exercise 8. A team of four — orchestrator O, coders C_1 and C_2, and tester T — coordinates through a shared scratchpad: free-text entries, a single plan field overwritten in place, last writer wins, and a convention that each agent reads the pad once, when it picks up its task. The trace: at t_1, O sets the plan to “target branch main; schema v1”. At t_2, C_1 reads the pad and starts its diff. At t_3, T posts “smoke tests green” — the run was against main at v1, but the entry names no branch, no schema, and no time. At t_4, O rewrites the plan: “target branch release-2.4; schema v2”. At t_5, C_2 reads the pad and starts its diff. At t_6, C_1, finishing, writes the plan field back from its t_2 copy with its own part marked done — silently discarding O’s rewrite. At t_7, O returns, sees a plan naming main and v1 with C_1’s part done and smoke tests green above it, and approves the release. (a) Section 9.6 names three hazards that follow a shared store about, and Section 9.5 a fourth failure the store inherits. Identify all four in this trace, each with the specific events that constitute it. (b) For each hazard, prescribe the minimal piece of bookkeeping that removes it, and name the earliest trace event at which your fix would have changed what happened. (c) A colleague proposes a simpler cure: every agent re-reads the entire pad immediately before every write and every action. Which of the four hazards does this fail to remove, and what new cost does it introduce?
Exercise 9. Implement, in standard-library Python, a model checker for the epistemic logic of Section 9.2.2. A model is a set of named worlds, a valuation giving the atoms true at each world, and, for each agent, a partition of the worlds into indistinguishability cells; formulas are atoms, negation, conjunction, K_i, E_G, and C_G. (a) Implement evaluation, computing C_G\varphi at a world w as the truth of \varphi at every world reachable from w in one or more steps through any group member’s cells, and say in a sentence or two why that reachability computation is the fixed-point characterisation of Section 9.2.2 made algorithmic. Implement also the public-announcement update, which strikes from the model every world where the announced formula fails. (b) Encode the model of Exercise 1 and confirm your hand answers to its parts (a) and (b) mechanically. (c) Encode the branch-status scenario of Exercise 4 for three agents with branches 1 and 2 failing, and drive it round by round: check that before the broadcast E_G\varphi holds and E_G^{2}\varphi fails; that after the broadcast C_G\varphi holds; that round one passes in silence — announce that silence — whereupon both failing agents declare in round two; and that announcing the declarations lets agent 3 conclude its own branch is clean. (d) Scale to the five-agent instance of Exercise 4 and confirm mechanically that E_G^{2}\varphi holds and E_G^{3}\varphi fails before the broadcast.
Exercise 10 (project). The chapter’s remedies keep converging on bookkeeping that no context window supplies: beliefs that carry their sources, justifications that let a retraction find its dependents, and a ledger of who was told what, so that a revision can propagate. Build the module. A BeliefStore holds belief records — claim, source, supporting belief identifiers, status, and the agents the belief has been told to — together with an entrenchment map from sources to weights. It must support: asserting a belief with its provenance; recording that a belief was messaged to an agent; retracting a belief, with the truth-maintenance cascade of Section 9.4 — every belief whose support is lost goes with it — returning the notifications now owed as pairs of belief and agent; and revising, where a newcomer that contradicts an incumbent is admitted only if its source outweighs the incumbent’s, and is otherwise turned away. Demonstrate the module on the scenario of Exercise 6: check that the cascade and the notifications match your hand answer to 6(b), and that re-asserting the configuration-file claim after the revision is refused.
Exercise 11 (lab). Take the probe suite you designed in Exercise 5(c) — the original scenario and your two perturbed variants — and administer it to a current language model, each version in at least three paraphrases that vary names and wording but not the epistemic facts, each probe in a fresh context. Score every response for whether it predicts behaviour from the protagonist agent’s belief or from the fact. Analyse the outcome as the chapter’s dispute in miniature: state in advance which pattern across versions and paraphrases would support Kosinski’s reading (2024) and which Ullman’s (2023), then report which you observed. Close by stating what your probe cannot settle, and why the design lesson of Section 9.3 does not wait on how the dispute comes out. Record the model identifier and the date beside your results.