17  Implementing a Multi-Agent System from Scratch

Every systems curriculum has its rite of passage: the course where you write the compiler, the toy operating system, the database engine with one table and no excuses. The point was never that the world needed another compiler. The point was that afterwards, no compiler could mystify you again — you had seen where every behaviour comes from, and error messages became navigation rather than omens. Agent systems have reached the age where they deserve the same rite, and this chapter conducts it: a complete multi-agent runtime — loop, tools, context, state, persistence, messages, and a coordinating harness — built in plain Python, with a model client as the only dependency, finishing with the book’s running team completing a real task on a real repository, end to end, on infrastructure whose every line you will have watched being justified.

The build makes one principle literal, and it is the principle this book has repeated since Chapter 2: the model is a component of the agent, not a synonym for it. In the runtime we construct, that sentence becomes architecture — the model sits behind a single narrow interface, swappable by editing one file, while everything that makes the system an agent system lives outside it, in code we write: the loop that drives action, the journal that remembers, the messages that coordinate, the budget that constrains. Chapter 16 priced this road and found the folklore wrong about its cost; this chapter pays the bill in public and itemises the receipt — the six services of Section 16.1, each at its honest minimum scope, none skipped, in a runtime measured in hundreds of lines rather than thousands.

A word on what minimal means, because it does not mean toy. The runtime is a paper aeroplane by Section 16.1’s distinction — it will not carry production passengers, and Chapter 20’s hardening is explicitly deferred — but it is a complete paper aeroplane: it persists, it resumes after a crash, it replays yesterday’s failure deterministically, it enforces a token budget, and it fails in ways you can read. Completeness, not polish, is the pedagogical load-bearer, since the parts most often omitted from tutorials — persistence, resumption, failure paths — are exactly the parts whose absence separates a demonstration from a system, and what Chapter 16 said you were paying frameworks to have opinions about. The prose names every module, type, and decision, and sets the load-bearing listings in place as it goes; the full modules live in the companion repository, which carries the burden of currency so that this chapter need not.

17.1 The Design Brief: Minimal but Complete

A build begins with a brief, and this one fits in three words that pull in opposite directions: minimal but complete. Completeness is defined by Chapter 16’s anatomy — all six services present: loop, state and persistence, tool dispatch, context assembly, topology, observability — because a runtime missing any of them is not a smaller runtime but a different and lesser kind of thing — and because a tutorial that omits a service is usually flattering itself. Minimality is the counter-pressure: each service at its honest smallest scope, so that everything in the runtime must earn its lines and nothing is present because it might be clever later. The two constraints discipline each other — completeness forbids the dishonest omission, minimality forbids the speculative flourish — and the space between them is exactly one paper aeroplane wide.

The completeness contract, itemised. The loop runs single-threaded, one turn at a time, with three stop conditions: a done-signal from the agent, a token budget exhausted, a turn cap reached — no parallelism, no streaming. State and persistence are a handful of typed objects plus an append-only journal, with resumption after a crash a requirement, not a stretch goal. Tool dispatch is a schema per tool, validation, execution, and errors returned as observations. Context assembly is an explicit, readable policy: what enters the window, in what order, truncated how. Topology is mailboxes, a spawn function, and one coordinator — no peer-to-peer exotica. And observability comes free, by a small economy the chapter is quietly proud of: the journal that gives us persistence is the trace — one design decision serving two services, which is the sort of bargain minimality exists to hunt. One licence in that list should be named, because the digital setting grants it silently: one turn at a time presumes an environment content to hold still between turns, which a repository is and the physical world of Section 4.6 is not — an embodied runtime cannot be turn-based, and Chapter 20’s layered timeouts and interrupts are where this book’s machinery begins, though only begins, to meet a world that will not wait.

Four principles discipline every choice, and each has a concrete justification. First, the model behind one interface: a single ModelClient — give it messages and tool schemas, receive a response — is the only place any vendor’s API is spoken. The obvious dividend is the vendor swap in one file; the quieter and larger one is testability: substitute a scripted fake for the client and the entire runtime can be exercised deterministically without spending a token, which is how infrastructure should be tested and almost never is. Second, own the loop: Chapter 16’s inversion-of-control question, answered for control — the runtime calls the model, never the reverse. Third, typed everything: messages, state, events, and tool results all carry explicit schemas, because in a system whose participants improvise, the types are the parts that do not, and they serve as documentation, validation, and the contract between agents at once. Fourth, no magic: every behaviour traces to a line that can be read — no decorators doing invisible work, no global registries, no convention-over-configuration. The debugging dividend Chapter 16 promised begins here, as policy.

What the runtime is for fixes its final shape. The build target is the book’s running team — orchestrator, coder, reviewer, tester — completing a bounded repository task with a checkable outcome: make a failing test pass and leave the suite green, so that success is a fact the repository itself certifies rather than a judgement anyone renders. The cast, in this runtime, is a single construction repeated four times: one ModelClient, wrapped in four harnesses that differ in their system prompts, their toolsets, and their standing in the message protocol — and the differences are Chapter 13’s three clauses, implemented rather than implied. The reviewer sees diffs and test output but not the orchestrator’s planning; it may approve or reject but not edit; and its approval is a typed message the harness requires before integration proceeds — a jurisdiction in code, which is what Chapter 13 said a role must be and what a backstory field is not.

The exclusions, finally, are contractual rather than embarrassed. Not built: parallel agent execution, streaming output, a human-approval interface, distributed deployment, retry sophistication beyond the basics, sandbox hardening, authentication — each a real need, each deferred to the chapter that treats it properly (hardening to Chapter 20, security to Section 20.9), and none of them architecturally foreclosed: the journal and the typed messages are precisely the seams such upgrades attach to, which is the difference between building small and building yourself into a corner. This is the aircraft you can understand completely, not the one that carries passengers — the brief says so on its face, and Chapter 20 holds the passenger licence. The build starts where every multi-agent system starts, whether its builders notice or not: with one agent, alone, in a loop.

17.2 One Agent: Loop, Tools, Context

The smallest complete thing in this build is one agent: four small modules whose names the companion code keeps — model.py, the client interface behind which every vendor detail hides; tools.py, the registry and dispatcher; context.py, the assembly policy; and agent.py, the loop that composes the other three. One claim is worth stating in advance so the reader can hold it to account: the loop itself — the part with the mystique, the part the frameworks wrap in the most machinery — fits on a page, and its page is mostly the honest handling of the three ways a turn can end.

model.py first, because it is the boundary of everything we do not own. The interface is one method: complete(messages, tools) -> ModelResponse, where the response carries whatever the model produced — text, one or more tool calls, or both — and, non-negotiably, the token usage of the call, because Section 17.4’s budget enforcement is only as good as its raw data. Inside this interface lives everything vendor-specific: authentication, request shapes, retry-on-rate-limit, the translation between the vendor’s message format and ours. Outside it, no other module knows which laboratory’s model is running — the vendor swap of the design brief, honoured structurally. And beside the real client sits the build’s most useful ten lines: a FakeClient that replays scripted responses. Every test in the runtime runs against the fake — loop, dispatch, journal, harness, all exercised deterministically, at no cost, in milliseconds — which converts “does the infrastructure work?” from an expensive empirical question about a model into a cheap logical question about our code, where it belongs.

In code, the module’s whole public face — the response, the interface, and the fake — fits in one look:

# foundations/model.py (excerpt)
@dataclass
class ModelResponse:
    text: str = ""
    tool_calls: list[ToolCall] = field(default_factory=list)
    usage: int = 0        # prompt + completion, one figure; never omitted


class ModelClient:
    """One method; everything vendor-specific lives behind it."""

    def complete(self, messages: list[dict],
                 tools: list[dict]) -> ModelResponse:
        raise NotImplementedError


class FakeClient(ModelClient):
    """Replays a script: deterministic tests, no tokens spent."""

    def __init__(self, script: list[ModelResponse]):
        self.script = list(script)

    def complete(self, messages, tools):
        return self.script.pop(0)  # exhaustion IS the test failure

(ToolCall, referenced by the response, is the obvious two-field record — a name and its arguments — defined a few lines up.)

Now the loop. A turn begins by assembling context (the policy of context.py, below) and calling client.complete. The response branches three ways. If it contains tool calls, each is dispatched, and the results — successes and failures alike — are appended as observations for the next turn. If it signals completion — a designated done-tool, so that finishing is an explicit, typed act rather than a stylistic inference from the prose — the loop returns the agent’s result, and any further calls the same response carried go undispatched: models do emit work after the full stop, and a reply that both finishes and keeps working is a contradiction the harness resolves in favour of finishing, because the harness owns the verdict. If it is plain text with neither, the loop applies a stated policy: append it and nudge, once, toward the done-tool; a second consecutive drift ends the run as inconclusive. After each turn, three stop conditions are checked: the done-signal, the token budget, and a turn cap. The cap deserves its sentence of justification, because it looks crude and is load-bearing: it is Section 3.7’s metacognition at its bluntest — the commitment that no agent, however confident it feels about turn forty-one, gets a turn forty-one — and an agent that cannot stop is not persistent but broken. Every event of the turn — request, response, dispatch, observation, verdict — is appended to the journal, of which more in the next section.

Here is the page in question:

# foundations/agent.py (excerpt)
def run(agent_id: str, client: ModelClient, tools: list[Tool],
        system: str, task: str, journal_path: str, budget: Budget,
        max_turns: int = 12) -> dict:
    toolbox = {t.name: t for t in tools}
    schemas = [t.schema for t in toolbox.values()]
    if "done" not in toolbox:         # the loop's own verb, always on offer
        schemas.append({"name": "done", "description":
                        "Finish the run; your arguments are the result.",
                        "required": []})
    history: list[str] = []
    observations: list[str] = []
    nudged = False
    append(journal_path, "RunStarted", agent=agent_id, task=task)
    for turn in range(max_turns):
        messages = assemble(system, task, history, observations)
        observations = []
        append(journal_path, "ModelCalled", agent=agent_id, turn=turn)
        response = client.complete(messages, schemas)
        append(journal_path, "ModelResponded", agent=agent_id,
               usage=response.usage)
        append(journal_path, "BudgetDebited", agent=agent_id,
               amount=response.usage)
        budget.debit(response.usage)
        for call in response.tool_calls:
            if call.name == "done":   # finishing is an explicit, typed act
                append(journal_path, "AgentFinished", agent=agent_id)
                result = {**call.args, "status": "done"}
                return result  # harness owns the verdict; later calls drop
            append(journal_path, "ToolDispatched", agent=agent_id,
                   tool=call.name)
            tool = toolbox.get(call.name)
            observation = (dispatch(tool, call.args) if tool else
                           {"tool": call.name, "error": "unknown tool"})
            append(journal_path, "ToolReturned", agent=agent_id,
                   **observation)
            observations.append(json.dumps(observation))
        drifted_again = not response.tool_calls and nudged
        if response.tool_calls:
            history.append(response.text or f"(turn {turn}: tool calls)")
            nudged = False
        elif not nudged:              # plain prose: nudge once, firmly
            history.append(response.text)
            observations.append("Reply with a tool call; "
                                "call done when finished.")
            nudged = True
        if budget.exhausted():        # wind down at the turn boundary
            append(journal_path, "BudgetExhausted", agent=agent_id)
            return {"status": "halted", "turns": turn + 1}
        if drifted_again:             # second consecutive drift: stop
            return {"status": "inconclusive", "turns": turn + 1}
    return {"status": "out_of_turns", "turns": max_turns}

tools.py holds the dispatcher, and its design is three decisions. A tool is a name, a schema, and a callable; registration is an explicit list passed to the agent at construction — no decorators quietly populating a global registry, per the brief’s no-magic clause, because “which tools does this agent have?” should be answerable by reading one line. Arguments are validated against the schema before anything executes, and the minimal sandboxing — a working directory, a timeout — is present because its absence would be dishonest even in a paper aeroplane. The decisive convention, though, is what happens on failure: errors are observations. A failed call does not raise through the loop; it returns to the context as structured information — the exception, the stderr, the exit code — because Chapter 4’s grounding argument applies with full force: to an agent, a stack trace is data about the world, and frequently the best data it will receive all day. Whether to retry is the loop’s policy, not the tool’s; the tool’s job is to fail informatively.

context.py is the smallest module and the most consequential, which by now is the book’s least surprising sentence. The assembly function builds each turn’s window from four parts: the system prompt (the agent’s role — the cast differences of Section 17.1 live here), the task instructions, the rolling history, and the latest observations; and it applies one explicit truncation policy — keep the system prompt and task always, keep the recent tail, and compress or drop the middle, oldest first. That is Section 4.8’s entire problem, decided in roughly thirty lines — and the point of writing them yourself is not that your thirty lines will beat a framework’s. It is that they are thirty lines you can read: the single most consequential hidden opinion identified in Chapter 16 — what does the model see? — is, in this runtime, a short function with your name on the commit. Its two sharpest opinions deserve saying aloud. Observations are news, not memory: a tool result is shown to the model exactly once, on the turn after it arrives, and never joins the rolling history — an agent that will need a file again must read it again, or restate what mattered in its own prose, which does persist. The budget, meanwhile, is counted in characters, a deliberately crude proxy for tokens that keeps the vendor’s tokeniser out of the module; Exercise 1 prices the crudeness and names the honest repairs.

The entire policy:

# foundations/context.py (excerpt)
def assemble(system: str, task: str, history: list[str],
             observations: list[str], limit: int = 8000) -> list[dict]:
    # The system prompt and the task are always kept; the rolling
    # history keeps its recent tail, oldest dropped first; the latest
    # observations always arrive. This is the whole policy.
    context = [{"role": "system", "content": system},
               {"role": "user", "content": task}]
    news = [{"role": "user", "content": o} for o in observations]
    budget = limit - sum(len(m["content"]) for m in context + news)
    tail: list[str] = []
    for step in reversed(history):    # newest first, until the budget
        if budget < len(step):
            break
        tail.insert(0, step)
        budget -= len(step)
    dropped = len(history) - len(tail)
    if dropped:
        tail.insert(0, f"[{dropped} earlier steps compacted]")
    context += [{"role": "assistant", "content": s} for s in tail]
    return context + news

What exists at the end of this section is a complete single agent: it acts, observes, recovers information from its own failures, stops for three well-defined reasons, accounts for every token, and can be tested end to end without a model in the room. What it cannot yet do is remember — kill the process and the run is gone, along with everything the agent learnt by doing it. Our agent is complete and amnesiac, and the cure — the one design decision this chapter most wants the reader to steal — is the next section.

17.3 The Journal Is the Truth: State and Persistence

Here is the decision, stated baldly so it can be argued with: the runtime’s source of truth is not a state object but a journal — an append-only sequence of events, one typed, timestamped, immutable record for everything that happens: a model called, a response received, a tool dispatched, a result returned, a message sent, a budget debited, an agent finished. Concretely it is a JSONL file, one event per line, readable by a human with less and by a program with ten lines — and nothing in the runtime ever mutates it, only appends. State still exists, but demoted: it is a derived view, computed by folding over the journal, and it can be thrown away and recomputed at will, because it was never the truth — only a convenient summary of it. The pattern has respectable names and a long lineage — the software literature calls it event sourcing,¹ databases have run on its sibling, the write-ahead log, for half a century — and accountants worked it out several centuries before software existed: you do not erase the ledger; you append the correction.

The decision is load-bearing because three capabilities that tutorials treat as features — and frameworks as selling points — fall out of it as corollaries, requiring no further design at all. Persistence is free: events are appended as they happen, so there is no “save” operation to design, schedule, or forget — the journal on disk is never more than one event behind reality. Resumption is a re-read: after a crash, the runtime folds the journal and stands exactly where it stood, mid-task, balances intact — and because this is a corollary rather than a feature, it can be tested brutally: the companion suite kills the process, unceremoniously, mid-run, and asserts that the resumed run completes identically; crash-recovery as a unit test, not an aspiration. Replay is deterministic debugging: feed the journal’s recorded model responses back through Section 17.2’s FakeClient and yesterday’s run is re-lived exactly — same responses, same dispatches, same failure — under a debugger, with breakpoints, as many times as the investigation needs — a promise Exercise 3 prices against Table 17.1 and funds with a one-field enrichment. The fake client, bought for testing, pays a second dividend as a time machine.

The event vocabulary is small and typed, per the brief: ten record kinds in all (Table 17.1), each with an explicit schema, each carrying the identifiers that stitch a run together: run, agent, turn, and (once Section 17.4 arrives) task. The state view is then a pure function — a reducer that folds events into a TeamState of tasks, statuses, artefacts, and balances — and the purity buys a freedom worth noticing: one journal supports many views. The budget view sums the debits; the conversation view threads the messages; the audit view answers “who approved this and on what evidence?” — one truth, many projections, and no projection ever disagrees with another, because none of them is primary. Chapter 6’s shared-state anxieties and Chapter 8’s consistency machinery are not abolished by this — they are localised: the journal is the single point of write, the runtime is single-threaded by the brief, and the day either assumption is relaxed is the day Chapter 8’s locks earn their keep, at a location the design has already marked.

Table 17.1: The journal’s event vocabulary — the ten typed record kinds the runtime appends, when each is written, and the payload that distinguishes it. Every event carries a schema version v, a timestamp t, and the emitting agent; the two terminal markers need nothing beyond that identity, and new kinds join by adding a line, not architecture.

Event	Appended when	Distinctive payload
RunStarted	An agent’s run begins	The task
ModelCalled	Before each call to the model	The turn number
ModelResponded	The model returns	The token usage of the call
BudgetDebited	Immediately after the response	The amount debited
ToolDispatched	Before a tool executes	The tool name
ToolReturned	A tool returns	The observation — result or structured error
MessageSent	An agent posts a message	Performative, sender, recipient, task thread
MessageDelivered	A message reaches an inbox	Recipient and task thread
AgentFinished	The done tool is called	—
BudgetExhausted	The budget empties at a turn boundary	—

The whole apparatus — the append, the read-back, and the reducer — is smaller than the paragraphs that described it:

# foundations/journal.py (excerpt)
def append(path: str, kind: str, **data) -> dict:
    event = {"v": 1, "t": time.time(), "kind": kind, **data}
    with open(path, "a") as f:
        f.write(json.dumps(event) + "\n")
    return event


def events(path: str) -> list[dict]:
    out: list[dict] = []
    with open(path) as f:
        for line in f:
            try:
                out.append(json.loads(line))
            except json.JSONDecodeError:  # a torn final line: died mid-write
                break
    return out


def fold(evts: list[dict]) -> dict:
    """The reducer: state is a view, recomputable at will."""
    state = {"spent": 0, "finished": [], "messages": 0}
    for e in evts:
        match e["kind"]:
            case "BudgetDebited":
                state["spent"] += e["amount"]
            case "AgentFinished":
                state["finished"].append(e["agent"])
            case "MessageDelivered":
                state["messages"] += 1
            # ...one arm per event kind the views care about
    return state

Honesty requires the costs, and there are two. Append-only grows without limit, and the standard remedy is the standard one: periodic snapshots — a cached fold, stored beside the journal, so that resumption reads snapshot-plus-tail rather than the whole history — with the journal remaining the truth and the snapshot remaining, like all state here, disposable. And events are forever, which makes the event schema the one interface in the runtime that must be versioned from day one: a journal written in June must still fold in December, so events carry a schema version and the reducer keeps its old cases. This is Section 16.4’s own-your-schema discipline arriving as a concrete artefact — the journal’s schema is the asset that outlives the runtime, the framework, and quite possibly the team — and it is also Section 17.1’s promised economy delivered in full: the trace that Chapter 21’s evaluation will read and the persistence the runtime relies on are the same file. Observability was never a separate service to build. It was the journal, viewed.

If the reader takes one transplantable idea from this chapter, the authors nominate this one, and it travels: even inside a framework — especially inside a framework — journal your own events, in your own schema, from day one, whatever the framework’s state machinery is doing. It costs an append call at each significant moment and buys replay, audit, and an exit route measured in views rather than migrations. Within our build, meanwhile, the amnesia is cured: the agent acts, and what it does persists, resumes, and replays. What it cannot yet do is talk to anyone — and a multi-agent runtime with one agent is not yet earning the adjective.

17.4 Many Agents: Messages and the Harness

The step from one agent to many is smaller than the folklore suggests, and the code makes the point bluntly: a second agent is the same construction as the first — one ModelClient, one loop, one toolset — with a different system prompt and a different tool list, and constructing it is one function call. What is genuinely new is not the agents but the space between them, and that space is three modules: messages.py, the typed envelope everything travels in; mailbox.py, the delivery machinery; and team.py, the coordinating harness that makes four loops into one system.

A message, in this runtime, is a typed object with five fields: a performative, a sender, a recipient, a task identifier that threads conversations, and a typed payload. The performative is an enum — REQUEST, INFORM, PROPOSE, ACCEPT, REJECT, DONE, ERROR — and the reader will recognise what has just happened: Chapter 5’s speech acts, and behind them three decades of agent communication languages, have arrived as a dozen or so lines of Python. The alternative — agents parsing one another’s prose in a shared transcript, the group-chat pattern — is not simpler but older: it reinstates every ambiguity the performatives were invented to remove, and Chapter 5’s silent-referent failures with them; when the coder tells the reviewer something, the runtime should know whether it was a report, a request, or a promise without running sentiment analysis on the question. A mailbox and a schema beat a group chat, and the margin is widest precisely when things go wrong. Every message, naturally, is journalled — MessageSent, MessageDelivered — so a conversation is an auditable thread in the same file as everything else.

The arrival, in full:

# foundations/messages.py (excerpt)
class Performative(Enum):
    REQUEST = "request"
    INFORM = "inform"
    PROPOSE = "propose"
    ACCEPT = "accept"
    REJECT = "reject"
    DONE = "done"
    ERROR = "error"


@dataclass
class Message:
    performative: Performative
    sender: str
    recipient: str
    task_id: str
    payload: dict

Delivery is deliberately dull. Each agent owns an inbox — a queue; sending appends the journal events and enqueues; and agents receive their mail at the top of their turn, where the context-assembly policy of Section 17.2 folds it into the window — the modules composing as advertised, with no new mechanism required for mail to become context. Scheduling honours the brief’s single-threadedness: the team runs on a coordinator-driven schedule, one agent’s turn at a time, which forecloses whole categories of Chapter 8 misery — races, lost updates, deadlock — by construction rather than cleverness. The concurrency this gives up is real, and the seam is marked: parallel turns would enter through the scheduler, one module, and would bring Chapter 8’s machinery with them as the admission price.

The coordinating harness, team.py, contains the chapter’s best joke, which is that it contains almost nothing new. The orchestrator is just another agent — same loop, same journal — whose toolset happens to operate on the team: a decompose step that turns the task into a typed task list; a dispatch step that sends each subtask as a REQUEST to a chosen agent; a collect step that awaits DONE or ERROR; a verify step; an integrate step. This protocol made its debut in 1980 (Smith, 1980) — announce, award, collect, report: the Contract Net, minus the competitive bidding round, which our minimal version replaces with direct assignment and marks as the obvious upgrade (the bidding is an exercise, and Chapter 12 explained what it buys and what it costs). The verify step is where Section 17.1’s promise about roles is kept in full: integration is a harness operation that requires a prior ACCEPT from the reviewer on the task thread — not a convention, not a prompt suggestion, but a precondition in code. Chapter 13 called this regimentation; here it is five lines, and the difference between a reviewer role and a reviewer costume is now something the reader can point to in a diff.

Here they are, hostage clause and all:

# foundations/team.py (excerpt)
def integrate(inbox: list[Message], task_id: str,
              apply: Callable[[], object]):
    approvals = [m for m in inbox
                 if m.task_id == task_id
                 and m.performative is Performative.ACCEPT]
    if not approvals:
        raise PermissionError(f"no ACCEPT on task {task_id}: not merging")
    return apply()

Money, finally — or its local currency. The budget is an accounting object: an allocation of tokens, debited on every ModelCalled event with the usage figures Section 17.2’s interface insisted upon, subdividable so that the orchestrator can allocate per agent or per task and the journal’s budget view can say, at any moment, who has spent what on which. Exhaustion is handled the way everything in this runtime is handled: as a typed event with a clean consequence — BudgetExhausted triggers a graceful wind-down at the next turn boundary, never a mid-turn kill, so a halted run is still a consistent, resumable, explicable one. The allocation policy itself is fiat — the orchestrator as Chapter 12’s central planner, which at four agents is exactly the right institution, since the planner can still read every status message. The market upgrade — agents bidding for budget — enters through this one object, and Chapter 12 has already supplied both the design and the warnings.

Tally the build. Four modules for one agent, three for the space between, one journal underneath (Figure 17.1) — and the whole runtime sits in the low hundreds of lines, a number the companion repository prints for audit. The smallness is the chapter’s thesis made measurable, but it wants stating carefully: the lines are few because the plumbing is genuinely small, not because the problems are — everything Section 16.2 said no framework provides is equally not provided by these hundreds of lines, and the specification, decomposition, and evaluation still cost what they cost. What the smallness buys is that the plumbing can no longer hide anything from you. The team is assembled: an orchestrator, a coder, a reviewer, and a tester, each a jurisdiction, all of it journalled. It remains to fly the aeroplane — and then, because this book is what it is, to crash it on purpose.

%%{init: {"theme": "base", "themeVariables": {"primaryColor": "#E8ECFF", "primaryBorderColor": "#4054B2", "primaryTextColor": "#16171B", "lineColor": "#3B4351", "edgeLabelBackground": "#FAF7F0", "clusterBkg": "#EFE9DC", "clusterBorder": "#766F65"}}}%%
flowchart TB
    subgraph team_tier["The space between agents"]
        team["<code>team.py</code><br/>the coordinator"]
        mailbox["<code>mailbox.py</code><br/>inbox delivery"]
        messages["<code>messages.py</code><br/>typed<br/>performatives"]
    end
    subgraph agent_tier["Four modules for one agent"]
        agent["<code>agent.py</code><br/>the loop"]
        model["<code>model.py</code><br/>vendor boundary"]
        tools["<code>tools.py</code><br/>dispatch"]
        context["<code>context.py</code><br/>window assembly"]
    end
    journal[("<code>journal.py</code><br/>append-only<br/>source of truth")]
    team ~~~ agent
    agent ~~~ tools
    model ~~~ context
    tools ~~~ journal
    team_tier -->|"imports, calls"| agent_tier
    agent_tier -->|"writes"| journal
    team_tier --> journal
    classDef world fill:#EFE9DC,stroke:#766F65,color:#16171B
    class journal world

Figure 17.1: The runtime as three layers: the team modules that furnish the space between agents, the four that compose a single agent, and the one journal beneath into which everything writes — the chapter’s own 4+3+1 tally made spatial, and the reason persistence and observability were never two services to build but a single file seen twice.

17.5 First Flight: Running the Team

The flight plan: a repository with one failing test; the task, make it pass and leave the suite green; the cast of four; a budget the orchestrator allocates. What follows is the successful run, narrated through the only witness that matters — the journal. RunStarted. The orchestrator’s first turn decomposes the task into three: diagnose and fix, review, verify — each a typed entry with a boundary. A REQUEST goes to the coder, and the coder’s thread begins: it reads the failing test, reads the implicated module, proposes an edit, runs the tests — and fails, because first attempts do, at which point Section 17.2’s convention earns its keep in public: the stack trace returns as an observation, the next turn’s context contains exactly what went wrong, and the second edit passes. The coder signals DONE, with the diff and the test output as its payload.

The review thread is shorter and, institutionally, the more interesting read. The reviewer receives the diff and the test output — and only those, per its jurisdiction; the orchestrator’s planning chatter is not in its context, by policy rather than accident. It replies ACCEPT on the task thread; the harness, holding integration hostage to that very message, proceeds; the tester runs the full suite against the integrated result and confirms green; AgentFinished, and the budget view prints the bill — who spent what, on which subtask, to the token. The whole run is one file, and the file reads like minutes (Figure 17.2 traces the handshake). The preface remarked that a committee of language models is still a committee and minutes should be kept; this runtime is, among other things, a machine for keeping them.

%%{init: {"theme": "base", "sequence": {"width": 110, "actorMargin": 40, "messageMargin": 30, "diagramMarginX": 10}, "themeVariables": {"primaryColor": "#E8ECFF", "primaryBorderColor": "#4054B2", "primaryTextColor": "#16171B", "lineColor": "#3B4351", "actorBkg": "#E8ECFF", "actorBorder": "#4054B2", "actorTextColor": "#16171B", "signalColor": "#3B4351", "signalTextColor": "#16171B", "noteBkgColor": "#F7E6B5", "noteBorderColor": "#8A5A00", "noteTextColor": "#16171B", "activationBkgColor": "#EFE9DC", "activationBorderColor": "#766F65", "labelBoxBkgColor": "#EFE9DC", "labelBoxBorderColor": "#766F65", "labelTextColor": "#16171B", "loopTextColor": "#16171B"}}}%%
sequenceDiagram
    autonumber
    participant O as Orchestrator
    participant C as Coder
    participant R as Reviewer
    participant T as Tester
    Note over O,C: decompose: fix, review, verify
    O->>C: REQUEST "fix the test"
    Note over C: tests fail, then pass
    C->>O: DONE "diff + tests"
    O->>R: REQUEST "review the diff"
    R->>O: ACCEPT on the thread
    Note over O,R: gate: integrate only after ACCEPT
    O->>T: REQUEST "verify the suite"
    T->>O: DONE "suite green"
    Note over O,C: budget view prints the bill

Figure 17.2: The successful first flight as a message trace: the orchestrator dispatches the fix as a REQUEST, collects the coder’s DONE, and cannot integrate until the reviewer returns an ACCEPT on the task thread — the Contract Net of 1980 with its bidding round removed, and the reviewer’s jurisdiction rendered as a precondition the handshake must clear rather than a suggestion in a prompt.

# journal.jsonl --- the successful first flight (abridged)
{"kind": "MessageSent", "sender": "orchestrator", "recipient": "coder", "task_id": "T1", "performative": "request"}
{"kind": "MessageDelivered", "recipient": "coder", "task_id": "T1"}
{"kind": "RunStarted", "agent": "coder", "task": "Make the failing parser test pass."}
{"kind": "ModelResponded", "agent": "coder", "usage": 120}
{"kind": "ToolDispatched", "agent": "coder", "tool": "run_tests"}
{"kind": "ToolReturned", "agent": "coder", "tool": "run_tests", "result": {"passed": 1, "failed": 1, "trace": "AssertionError: parse('') != []"}}
{"kind": "ModelResponded", "agent": "coder", "usage": 95}
{"kind": "ToolDispatched", "agent": "coder", "tool": "run_tests"}
{"kind": "ToolReturned", "agent": "coder", "tool": "run_tests", "result": {"passed": 2, "failed": 0}}
{"kind": "ModelResponded", "agent": "coder", "usage": 60}
{"kind": "AgentFinished", "agent": "coder"}
{"kind": "MessageSent", "sender": "coder", "recipient": "orchestrator", "task_id": "T1", "performative": "done"}
{"kind": "MessageDelivered", "recipient": "orchestrator", "task_id": "T1"}
{"kind": "RunStarted", "agent": "reviewer", "task": "Review the change.\ndiff:\n+ if not s: return []\ntests:\n2 passed"}
{"kind": "ModelResponded", "agent": "reviewer", "usage": 40}
{"kind": "AgentFinished", "agent": "reviewer"}
{"kind": "MessageSent", "sender": "reviewer", "recipient": "orchestrator", "task_id": "T1", "performative": "accept"}
{"kind": "MessageDelivered", "recipient": "orchestrator", "task_id": "T1"}
{"kind": "RunStarted", "agent": "tester", "task": "Verify the integrated change.\ndiff:\n+ if not s: return []\ntests:\n2 passed"}
{"kind": "ModelResponded", "agent": "tester", "usage": 90}
{"kind": "ToolDispatched", "agent": "tester", "tool": "run_tests"}
{"kind": "ToolReturned", "agent": "tester", "tool": "run_tests", "result": {"passed": 2, "failed": 0}}
{"kind": "ModelResponded", "agent": "tester", "usage": 30}
{"kind": "AgentFinished", "agent": "tester"}

Now the promised crashes, administered one at a time to a healthy system. First, the vague subtask: instruct the decompose step to be terse, and the boundary between fix and verify blurs — the journal shows the coder and the tester editing the same file within three turns of each other, a duplicate-work collision visible as two ToolDispatched entries no one has to hunt for. Nothing crashed; the specification failed, and the run’s inefficiency traces to one terse entry in the task list. Second, the silent dropout: replace the reviewer with a stub that accepts its REQUEST and never replies. The naive collect step waits forever — so the run demonstrates why deadlines are not a refinement but a requirement, and the fix (a timeout on awaited replies, a REQUEST re-issued or escalated) is Chapter 7’s failure-and-repair machinery, arriving as roughly ten lines.

Third, the premature “done”: prompt the coder to signal completion without running the tests. The DONE arrives, confident as ever — and integration does not proceed, because the reviewer’s ACCEPT never comes: the diff carries no test output, the reviewer rejects, and the harness routes the rejection back as a new turn for the coder. The five lines that made the reviewer’s role a precondition rather than a costume pay for themselves in this one trace. Fourth, the tool error mid-turn: a flaky test runner that dies with a cryptic exit code — absorbed as an observation, retried by loop policy, life goes on. Step back and tally what these four runs re-enacted: a specification failure, an inter-agent coordination failure, a verification-and-termination failure, and an infrastructure hiccup — the MAST taxonomy’s three categories, reproduced on demand in a runtime with no framework anywhere in it (Cemri et al., 2025). Chapter 16 argued from the study that the fourteen failure modes were never in anyone’s stack. We can now say it from the inside, having built the stack and watched the failures arrive anyway — with the one difference that matters: here, every failure is a readable thread in a file we own.

# crash 1 --- the vague subtask: the coder and the tester both edit the same file
{"kind": "ToolDispatched", "agent": "coder", "tool": "edit_file"}
{"kind": "ToolReturned", "agent": "coder", "tool": "edit_file", "result": {"edited": "parser.py"}}
{"kind": "ToolDispatched", "agent": "tester", "tool": "edit_file"}
{"kind": "ToolReturned", "agent": "tester", "tool": "edit_file", "result": {"edited": "parser.py"}}

# crash 2 --- the silent dropout: the request is delivered, no reply is ever appended
{"kind": "MessageSent", "sender": "orchestrator", "recipient": "reviewer", "task_id": "T2", "performative": "request"}
{"kind": "MessageDelivered", "recipient": "reviewer", "task_id": "T2"}
# ...  (a naive collect blocks here; a deadline is the ~10-line fix)

# crash 3 --- the premature done: no ACCEPT, so integration never runs (outcome: rejected)
{"kind": "MessageSent", "sender": "orchestrator", "recipient": "coder", "task_id": "T1", "performative": "request"}
{"kind": "AgentFinished", "agent": "coder"}
{"kind": "MessageSent", "sender": "coder", "recipient": "orchestrator", "task_id": "T1", "performative": "done"}
{"kind": "AgentFinished", "agent": "reviewer"}
{"kind": "MessageSent", "sender": "reviewer", "recipient": "orchestrator", "task_id": "T1", "performative": "reject"}

# crash 4 --- the tool error: the cryptic exit code returns as an observation, and loop policy retries
{"kind": "ToolReturned", "agent": "coder", "tool": "run_tests", "error": "RuntimeError: runner exited 137"}
{"kind": "ToolReturned", "agent": "coder", "tool": "run_tests", "result": {"passed": 2, "failed": 0}}

What the exercise taught can be checked against Chapter 16’s three claims. Small: the line count stands, printed in the companion repository for audit. Complete: the crash-resume test passes, the replays are deterministic, the budget balances. Yours: the duplicate-edit collision above was found by reading the journal, understood by replaying the run against the fake client, fixed by one change to the decompose prompt, and verified by re-running the same recorded scenario — exactly the debugging dividend Chapter 16 promised. And one exercise from the chapter’s set deserves its trailer here, because it turns the book’s own instrument on the book’s own build: replace each agent, in turn, with a mindless stub and measure what survives — the Gode–Sunder test of Section 12.4, applied to ourselves, to discover how much of the team’s success was harness all along. The authors decline to spoil the result.

The wish list that accumulates by the end of first flight is the point of the next chapter. Parallel dispatch, once tasks are independent; a retry-and-escalate policy richer than the timeout; a router that sends task types to specialist agents; a reflection loop that lets the coder critique its own diff before the reviewer sees it; a planner that revises the decomposition mid-run — every item is a named pattern with a known trade-off sheet, and the runtime just built is the substrate on which each can be added as a visible, testable increment rather than a framework toggle. The rite of passage is complete: the reader has a runtime, a set of failures they have caused and can therefore recognise, and the standing asset this chapter set out to purchase — the certainty that nothing a framework does is magic, because they have done all of it themselves, with the line count printed and the receipts journalled. Chapter 18 now names the shapes worth building on top.

17.6 Summary

• From-scratch is pedagogy first and an option second. The build teaches what every framework does on your behalf, and the understanding is permanent even if the runtime is not; afterwards, you debug a framework knowing where every decision lives.
• The model is a component, and the build makes it literal. One narrow interface, one file to edit for a vendor swap; everything agentic — loop, memory, coordination, budget — lives outside the model, in code you own. Agent = Model + Harness, compiled.
• The loop is small and its ambitions are incremental. The core cycle fits on a page; retries, interruption, budgets, and resumption each add lines rather than architecture — which is why the six services at minimum scope stay measured in hundreds of lines.
• The journal is the load-bearing decision. An append-only event log as the source of truth makes state a derived view — and persistence, crash-resumption, and deterministic replay stop being features to build and become corollaries to observe. If you take one design idea from the chapter, take this one.
• Messages are typed and performative. A mailbox and a schema beat a group chat: Chapter 5’s speech acts arrive as an enum, and the coordinating harness that dispatches on them is the Contract Net, recognisably, a few dozen lines long and older than the web.
• When it breaks, it breaks in MAST’s ways — with no framework to blame. The classic failures reproduce faithfully in a runtime you wrote every line of, now visible, replayable, and yours; and the upgrades you would add next — routers, reflection loops, supervisors — are named patterns with known trade-offs, Chapter 18’s catalogue approached from below.

17.7 Exercises

Exercise 1. The runtime’s entire perception policy is assemble in foundations/context.py (Section 17.2). Take one call with a 40-character system prompt, a 100-character task, a single fresh observation of 60 characters, limit=300, and a rolling history of five steps of 80, 70, 60, 50, and 40 characters, oldest first. (a) Trace the policy by hand — the character budget left for history, which steps survive, what marker is inserted — predict the total length of the returned window, then run assemble and check every figure. (b) State the invariant assemble actually enforces — a bound on the surviving history tail, not on the window — and prove it from the loop; then exhibit the two distinct ways the returned window can exceed limit, with a verified minimal example of each, saying which of the two is bounded (and by how much) and which is not. (c) The budget is counted in characters while the model’s real constraint is tokens, and the exchange rate between the two is vendor property. Say where a tokeniser-faithful limit belongs in this architecture, why importing a vendor’s tokeniser into context.py would breach the first principle of Section 17.1, and give the two honest alternatives.

Exercise 2. The budget is enforced only at turn boundaries, and a done return is not checked at all. (a) Prove the overshoot bound: for a run that ends halted after calls of usages u_1, \dots, u_k against an allowance B, the spend obeys \sum_{i=1}^{k} u_i \le B - 1 + u_k — an excess bounded by one call, and by nothing else; confirm both faces of the bound on the runtime, with two calls of 900 against an allowance of 1,000 and a single call of 5,000 against the same allowance, and say why the design prefers this overshoot to a mid-turn kill. (b) The team shares one pot. Fly foundations/demo_team.py’s scripted flight through run_team on Budget(allowance=300) and read the outcome dictionary with care: identify which agent halted and at what spend, quote the returned status and suite fields, name the institutional failure in the terms Section 17.1 used for the reviewer — which role turns out to be a costume, and on which line of foundations/team.py? — and give the one-line repair. (c) The budget is “subdividable so that the orchestrator can allocate per agent or per task” (Section 17.4), yet the shipped Budget is a single pot. Implement SubBudget, whose debits hit both itself and its parent; fund coder, reviewer, and tester at 200, 100, and 150 under a 5,000-token parent and show all four ledgers after the flight; then show containment — a coder funded at 100 halts on its own share while the parent and every sibling stand untouched.

Exercise 3. Section 17.3 promises deterministic replay: feed the journal’s recorded model responses back through FakeClient and yesterday’s run is re-lived exactly. (a) Show that the promise is not yet funded by Table 17.1: name the event that must carry more than it does, and state precisely what a replayer cannot reconstruct from the journal as written. (b) Fund it: enrich the loop’s ModelResponded append into a version-2 event carrying the response’s text and tool calls, establish that foundations/journal.py needs no edit at all (why not?), and write replay_client(evts, agent); then explain why recorded model responses alone still leave replay non-deterministic, and write replay_tool, which replays recorded ToolReturned observations — including reproducing a recorded error as a raised exception with the recorded name and message. (c) Prove determinism the journal’s own way: record crash four under the enriched loop, re-run it from replay_client and replay_tool alone, and show the two event streams identical but for timestamps. (d) “Fixed from the trace alone” can be mechanised: write diagnose(evts), returning one finding per crash signature of Section 17.5 — duplicate work on one artefact, a delivered REQUEST with neither reply nor run, an unevidenced done, a transient tool error later absorbed — and run it over all five of that section’s journals: the first flight must come back clean, and each crash must yield exactly its own finding and no other’s.

Exercise 4. Section 17.5 trailered this exercise and declined to spoil its result; run it and spoil it. The Gode–Sunder test of Section 12.4 replaced human traders with zero-intelligence bidders to measure how much of a market’s performance was the institution rather than the participants; turn the same instrument on the build. (a) Write a ZIClient — one fixed done reply at a nominal 5 tokens, whatever the context says — with stub payloads for a coder (an empty diff, no tests run), a reviewer (“accept”, always), and a tester (“green”, always); and write one discriminating reviewer that is still not a model: a single 8-token rule that accepts exactly the submissions carrying test evidence. (b) Run the 2 \times 2 — competent scripted coder against ZI coder, rule reviewer against ZI reviewer, the ZI tester throughout — through run_team, tabulating each cell’s outcome, total spend, and whether an ACCEPT appears in its journal. (c) Read the table as Gode and Sunder read theirs: list the properties that held in every cell, including the one where an empty, untested diff shipped, and attribute them; name the single property the harness supplied in no cell; and close with the spoiler, one sentence each — what fraction of the first flight’s success was harness, and in what sense; and what the integrate gate is worth when the provenance of its ACCEPT is a rubber stamp.

Further exercises for this chapter continue in the web edition’s exercise bank.

“I’m not saying you were wrong in 1998. I’m appending a correction.”

Cemri, M., Pan, M. Z., Yang, S., Agrawal, L. A., Chopra, B., Tiwari, R., Keutzer, K., Parameswaran, A., Klein, D., Ramchandran, K., Zaharia, M., Gonzalez, J. E., & Stoica, I. (2025). Why do multi-agent LLM systems fail? Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track. https://arxiv.org/abs/2503.13657

Smith, R. G. (1980). The contract net protocol: High-level communication and control in a distributed problem solver. IEEE Transactions on Computers, C-29(12), 1104–1113. https://doi.org/10.1109/TC.1980.1675516

---

1. Martin Fowler’s write-up is the standard reference: https://martinfowler.com/eaaDev/EventSourcing.html.