Exercise Bank — Multi-Agent Architectural Patterns

The printed Chapter 18 keeps Exercises 1–4; the bank continues from Exercise 5.

Exercise 5. The running team places an intake router in front of three specialist coder pools; a routed specialist run costs W = 8{,}000 tokens. The router’s self-reported confidence is calibrated: on a fraction h = 0.2 of incoming tasks it is uncertain and its dispatch accuracy is 0.6; on the remaining tasks it is confident and its accuracy is 0.98. A misrouted task completes confidently — Section 18.2’s healthy run of the wrong kind — and the wrongness surfaces downstream at a cost of D = 30{,}000 tokens of expert reading and redo. Two policies: always dispatch, which routes every task to the chosen specialist; and none-of-the-above, which sends the uncertain fraction to a generalist instead, at 14{,}000 tokens a run, the generalist’s own misroutes rare enough to ignore. (a) Compute the always-dispatch misroute rate and the expected token cost per task under each policy. (b) Find the break-even downstream cost D^* at which the escalation route pays for itself, and express it as a multiple of W. (c) The task mix drifts: the uncertain fraction climbs to h = 0.5 over a quarter while both accuracies hold. Recompute both policies and the gap between them, and say what the movement demonstrates about Section 18.2’s warning that last quarter’s routing accuracy is a fact about last quarter. (d) The companion repository’s frontier/patterns/signatures.py gives its misroute detector the signature is_misroute(r, correct_specialist) — alone among the module’s detectors it takes an argument the journal does not carry. Say why no journal-only detector could replace it, and name the one line the router could nonetheless write into the journal that would make the dispatch auditable after the fact.

Exercise 6. Chapter 17’s wish list promised the router and the reflection loop as visible, testable increments on the runtime, and this chapter’s opening priced several such shapes at about a page apiece; make both good, working in your own copy of the companion repository’s foundations/ layer. (a) The router: write route(task, menu, classify, journal) — one classification call against a menu of specialists plus the none-of-the-above escape of Section 18.2, escalating to a generalist, and journalling the dispatch with its rationale, the entry Section 18.2 notes almost never survives into the trace. (b) The reflection loop: write reflect(brief, draft, actor, check, cap, journal) — the actor revises against objections held by an external check (the armed critic of Section 18.3), the loop bounded twice, by a clean check and by a hard cap, and every round writing a Reviewed journal entry carrying the round number, the diff size, and the count of open objections. (c) Demonstrate both on scripted stubs in the runtime’s FakeClient manner: a correct dispatch, a none-of-the-above escalation, and a loop that converges in two rounds; then show that the journal your loop writes is legible to is_placation in the companion repository’s frontier/patterns/signatures.py, and state which single field makes it so. (d) Say where each increment plugs into foundations/team.py’s run_team — the router in front of which dispatch, the loop between which two seats — and price each addition in extra model calls per task, in Section 18.1’s every-arrow-is-a-bill terms.

Exercise 7. The journal census in the companion repository’s frontier/patterns/signatures.py covers seven failure signatures, but the hand-off’s baton-drop is not among them; supply it. (a) Design the journal vocabulary: a handoff: detail marking the seam — sender, receiver, timestamp — read against the module’s existing dispatch: entries and an ask-user: entry for questions put to the user. (b) Implement is_baton_drop(journal) for the benign grade Section 18.2 describes — after the seam, the receiver re-derives what the sender already held: a tool step repeated across the seam, or the user re-asked an answered question — and exhibit it firing on a dropped-baton trace and staying silent on a typed-payload hand-off, in the injections module’s broken-and-healthy manner. (c) Now write the expensive-grade trace: the receiver proceeds without the load-bearing fact, and the artefact is wrong for reasons no one downstream can reconstruct. Show it is journal-indistinguishable from your healthy trace, and name the shipped detector that already lives with this exact predicament, together with the module’s standing reminder it encodes. (d) The chapter’s structural mitigation is the typed payload — what is done, what is pending, what is odd. Say what typing the payload changes about detectability, not just prevention: which absence a detector could flag at the seam itself, before any duplicated work has had time to appear.

Exercise 8. A detector is the operational definition of its signature, so probe three of the shipped definitions in frontier/patterns/signatures.py at their boundaries, verifying every claim by running the detectors. Construct: (a) the cosmetic replan — a planner–executor journal in which a tool return falsifies step s2, a replan: entry follows that names no step, and s2 is then dispatched anyway; show is_stale_plan stays silent, and say which behaviour of the detector launders the contradiction. (b) The interleaved stampede — four dispatches off one trigger onto one board region, each separated from the next by its own ToolReturned; show is_stampede stays silent, and explain which property of the book’s stampede definition the detector’s reset is enforcing. (c) The unlogged placation — a review sequence ending in an accept whose entry carries no open= count; show is_placation stays silent however agreeable the critique was. (d) Adjudicate the three misses under Section 18.1’s definitions: which is a detector bug, which a defensible tolerance — argue from what the chapter says a stampede is — and which an instrumentation gap whose fix belongs in the logging discipline rather than the detector; then patch exactly the one you called a bug, and show the patch catches the offending trace while the module’s shipped healthy runs stay quiet.

Exercise 9. Composition voids the warranties; produce the receipts. (a) The running team’s supervisor is given w = 4 workers, each upgraded to a reflection loop with round cap R = 3, a round costing one actor pass and one critic pass; the supervisor itself spends one decomposition call and one integration call, and every call averages 1{,}800 tokens. Compute the composed system’s worst-case model calls and token bill against the flat supervisor whose workers make one pass each, and state the multiplier the composition applies to the workers’ share of the bill — Section 18.5’s integer multiples, exhibited. (b) Read the two specimens in the companion repository’s frontier/patterns/compose.py and reconcile an apparent contradiction: Section 18.5 says the supervisor of reflection loops cannot tell politeness from verification, yet the repository’s is_placation catches the laundered placation in supervisor_of_reflection’s journal. Who reads what, and when — and which rule of Section 18.2 about run-level reading does the reconciliation instantiate? (c) Build the third specimen: a supervisor whose verification gate is a debate, wired exactly as Section 18.2’s mitigation prescribes — and let the debate chorus. Write the composed journal, show is_chorus still fires on it, and state the new failure precisely: what the supervisor’s certification is worth when the gate that saw the whole returned a verdict that pooled nothing. (d) Generalise from the three specimens: which component failures remain detectable after composition, which class of failure is new and detector-less, and what one-line journal discipline at the join — what must the certifier log — would make the new class legible.

Exercise 10 (lab). The pattern-swap, first hermetically and then against a live model. Hermetic: from the companion repository, run python -m frontier.patterns.run and the suite tests/test_frontier_patterns.py; confirm the census flags all eight injected failures with every healthy counterpart silent, and state why the pattern-swap table this run prints demonstrates the instruments rather than measuring any pattern. Live: pick one small task from the running team’s world with a planted, checkable defect — a function whose failing test is given — and hold one model, one temperature, and one budget fixed across a single-agent baseline and three topologies: a supervisor that decomposes into two briefs and integrates behind a verifier; a debate of three seats over two rounds with a judge; and a peer network of three peers messaging directly under two conventions — address one peer at a time, and post “AGREED” with the patch hash when satisfied — the run ending when all three have agreed or a hard cap of fifteen messages strikes. Run at least five trials per condition; journal every run in the detail grammar of frontier/patterns/signatures.py, so that the shipped detectors can read your traces; score the four columns — quality as the fraction of trials whose patch passes the planted test, tokens, latency, and failures as detector flags plus cap strikes. Then report: each topology’s token multiplier over the baseline; whether the debate chorused, by is_chorus on your own journals; and whether the peer network’s runs ended by convention or by cap — Section 18.4’s observation that asking a peer network whether it is finished is a research question, here given an empirical answer. Record the model identifier and the date beside the table; the durable finding is the ordering of the multipliers and the failure signatures, not any cell.