%%{init: {"theme": "base", "themeVariables": {"primaryColor": "#E8ECFF", "primaryBorderColor": "#4054B2", "primaryTextColor": "#16171B", "lineColor": "#3B4351", "edgeLabelBackground": "#FAF7F0", "clusterBkg": "#EFE9DC", "clusterBorder": "#766F65"}, "flowchart": {"rankSpacing": 28, "nodeSpacing": 42, "subGraphTitleMargin": {"top": 2, "bottom": 20}}}}%%
flowchart TB
subgraph OUT["Facing outward"]
TOOL["Tool dispatch<br/>schemas, results<br/>(Ch. 6)"]
TOP["Topology and<br/>messaging<br/>(Parts III, IV)"]
OBS["Observability<br/>traces, replays<br/>(Ch. 24)"]
end
subgraph KNOW["What the system knows"]
CTX["Context assembly<br/>what enters the<br/>window (Ch. 7)"]
STATE["State, persistence<br/>checkpoint, replay<br/>(Ch. 9)"]
end
subgraph HEART["The heart"]
LOOP["The loop<br/>call, parse, dispatch,<br/>observe (Part II)"]
MODEL["The model<br/>bought in; the harness<br/>never improves it"]
end
TOP ~~~ CTX
TOP ~~~ STATE
CTX ~~~ LOOP
STATE ~~~ LOOP
LOOP -->|"calls"| MODEL
classDef world fill:#EFE9DC,stroke:#766F65,color:#16171B
class MODEL world
19 Frameworks and the Art of Choosing One
Part VI opens with the decision most teams make first, fastest, and with the least evidence: which framework to build on. It is usually settled in an afternoon — a demonstration watched, a quick-start followed, a colleague’s enthusiasm caught — and then lived with for years, shaping every architectural choice downstream of it. The disproportion between how casually the decision is made and how long its consequences last is the reason this chapter exists, and its ambition is stated in the preface’s own terms: not a survey that will age like fruit, but a decision procedure that will outlast the current version of everything it mentions. The catalogue — names, idioms, version-dated comparisons — lives in Appendix B, accurate at the time of going to press, which is the most any book can honestly promise. This chapter keeps the part that does not expire.
Writing durably about frameworks requires facing the perishability problem squarely, since the preface has already compared the half-life of an agent framework unfavourably with that of the mayfly and the prawn sandwich. The strategy here is the book’s usual one, applied to software: teach the anatomy — what any agent framework must provide, which is fixed by the nature of the problem rather than by any vendor’s roadmap; teach the design space — the axes along which frameworks genuinely differ, which have been stable across two generations of tooling; and teach the decision procedure, which operates on questions about your task and your team rather than on feature matrices. Product names appear as instances and live mostly in footnotes, where their inevitable renamings will do the least damage.
There is also a heritage here, and it is a memento mori. The multi-agent field has built frameworks before — JADE with its agent lifecycles, mailboxes, and directory services; Jason compiling the belief–desire–intention loop into a programming language; the FIPA platforms with their interoperability dreams — an entire generation of infrastructure, rigorously engineered, internationally standardised, and now, for the most part, archaeology. The lesson is not that framework-building is futile; it is that the survivals are instructive. The platforms died; the ideas in them — the agent loop, the typed message, the capability directory, the lifecycle — are all present in today’s stacks, usually under new names and without the citation. A framework is a temporary container for durable ideas, and the engineer who knows the ideas can change containers without grief. That, in one sentence, is the chapter.
The organising decomposition is the one Chapter 3 installed and Chapter 20 will implement: Agent = Model + Harness. A framework is, first and last, a set of opinions about the harness — about the loop, the state, the tools, the context, the topology, and the telemetry — and evaluating a framework means evaluating its opinions against your needs, not its demonstrations against your hopes. One further discipline is announced now and honoured in Section 19.5: this chapter measures. Its lab runs one task across three frameworks and scores the results — tokens, latency, code, failure behaviour — because the alternative to measurement is choosing infrastructure by charisma. We proceed from what a framework actually provides (Section 19.1); to what no framework can provide, which is the more important list (Section 19.2); through the design space and its two load-bearing axes (Section 19.3); to the price of dependence — lock-in, churn, and the discipline of depending on unstable software (Section 19.4); and finally to the decision procedure itself, including the option, demonstrated in the next chapter, of choosing none at all (Section 19.5).
19.1 What a Framework Is Buying You
Strip away the demonstration video and the quick-start, and ask the buyer’s question: what, concretely, is in the box? Chapter 3’s decomposition gives the answer its frame. The model arrives from a laboratory, priced separately, and no framework improves it by a single parameter; a framework’s entire product is the harness — the apparatus that turns a model that can only ever complete text into an agent that acts. And because the harness’s contents are fixed by the nature of the problem rather than by anyone’s roadmap, every agent framework, of any generation, must answer the same six questions. They differ — sometimes profoundly — in their answers. They do not differ in the questions, which is what makes the anatomy worth learning once.
The first service is the loop, and it is the heart. Something must drive the cycle the book met in Chapter 4 — assemble context, call the model, parse the response, dispatch the action, observe the result, repeat — and doing so in production means owning everything the tidy diagram omits: retries and backoff when the model call fails mid-turn, timeouts, cancellation and interruption, streaming partial output to whoever is watching, pausing for a human’s approval and resuming a week later without amnesia. A loop is a while-statement with ambitions, and the ambitions are where the engineering lives: the difference between a demo loop and a production loop is roughly the difference between a paper aeroplane and an aircraft, in that both fly and only one may carry passengers.
The second and third services manage what the system knows. Typed state and persistence make the team’s situation a first-class object — what has been decided, produced, and promised, held in a schema, checkpointed so that a crashed run resumes rather than restarts, replayable so that yesterday’s failure can be re-lived deterministically today; the reader will recognise Chapter 9’s shared state and Chapter 11’s consistency worries arriving as API surface. Context assembly is Chapter 7 operationalised: something must decide, each turn, what enters the window — which instructions, which memories, which tool results, truncated how, ordered how — and Simon’s point from Chapter 16 gives the service its true weight: deciding what each mind attends to is most of what an organisation does, and in an agent system that decision executes in code, every single turn.
The remaining three services face outward. Tool dispatch is Chapter 6 as infrastructure: schemas validated, calls executed, sandboxes enforced, results routed back into context — the perception–action loop’s plumbing. Topology and messaging are Parts III and IV as configuration: spawning subagents, routing hand-offs, holding mailboxes, supervising failures — the machinery of teams, made available as objects with methods. And observability — traces, token accounting, replayable transcripts — is the service most casually skipped in quick-starts and least dispensable in production, because Chapter 18 established that a multi-agent system’s behaviour lives at a level no single component reveals: if you cannot see the run, you cannot debug the society. Set the six services beside the book’s table of contents and the correspondence is exact — loop to Part II, state and context to its chapters, tools to Chapter 6, topology to Parts III and IV, observability to Chapter 24. A framework is the book’s first five parts, shrink-wrapped, with opinions.
One distinction cuts across all six services and does more work than any feature comparison: inversion of control — the old software-engineering question of who calls whom. With a library, you own the loop and call its parts: your code decides when the model runs, what enters the context, when an agent spawns; the library supplies competence, you retain control flow. With a framework in the strict sense, the relationship inverts: it owns the loop and calls you — your code lives in the slots it provides, as nodes in its graph, tools in its registry, roles in its crew. Neither is wrong; but the choice determines, more than any other, three things the buyer will care about later — how the system is debugged (your stack trace or theirs), how it is extended (anywhere, or where a slot exists), and what leaving costs (Section 19.4’s subject). The question to ask of any candidate is therefore not “what can it do?” — the demonstrations all do everything — but “who holds the loop?”
| Dimension | Library | Framework |
|---|---|---|
| Who holds the loop | You do — your code calls its parts | It does — it owns the loop and calls you |
| What it supplies | Competence; you retain control flow | The control flow itself; you supply the behaviour |
| Where your code lives | Wherever you choose to call it | In the slots it provides — nodes in its graph, tools in its registry, roles in its crew |
| How you debug | Your stack trace | Its stack trace |
| How you extend | Anywhere | Only where a slot exists |
| What leaving costs | Set by how far its idioms and types wove through your code | Set by how much of your logic lives in its slots (Section 19.4) |
What you are buying, in sum, is opinions about the harness — and the point of this anatomy is not that opinions are bad. Good opinions, bought from people who have made the mistakes already, are exactly what a strong framework sells, and building opinion-free infrastructure is neither possible nor desirable. The point is that the purchase should be itemised: for each of the six services, know whether you have delegated the decision or retained it, and know what the framework’s opinion actually is — because several of those opinions, particularly about context assembly and topology, are decisions this book has spent five parts showing to be consequential. The model, meanwhile, sits inside all of this as a component — the framework never makes it smarter; it only decides what the model sees and when, which Chapter 7 and Chapter 16 argued is most of the ballgame. That is the inventory of what is in the box. The next section lists what is not — and it is the more important list.
19.2 What No Framework Provides
Buyers compare what candidate frameworks provide; the wiser comparison runs the other way, over what none of them can. The previous section’s inventory was honest but flattering — six services, all real, all valuable — and it invites a fatal inference: that a system built from good services will be a good system. The claim of this section is that nearly everything which decides whether a multi-agent system succeeds sits upstream of all six services, in territory no framework can reach, because it is not plumbing at all. The box contains the harness. It does not, and cannot, contain the judgement.
The evidence has appeared twice in this book already, and its third reading is the one this chapter needs. The MAST study — fourteen failure modes, catalogued from over a thousand annotated traces — was conducted across seven different frameworks, and the taxonomy holds across all of them (Cemri et al., 2025): the failure modes do not cluster by framework, and not one of the fourteen is a plumbing failure. They are failures of specification, of inter-agent coordination, of verification and termination — Chapter 10 read them as missing teamwork machinery, Chapter 16 as missing organisation, and this chapter reads the finding the study itself is too polite to headline: the framework column did not matter. Systems built on rival stacks, with different loops, different state models, and different topological primitives, failed in the same fourteen ways, because the ways they failed were never in the stack.
So take the inventory of what is not in the box, and notice that it is the book’s own syllabus. First, specification and decomposition. No framework can tell you what the task actually is, where its boundaries lie, or how to divide it — and the division is the decision: Chapter 10’s delegation, Chapter 11’s dependency structure, Section 16.1’s coalition-structure problem, and Chapter 15’s warning about complementary subtasks are all live at the moment you sketch the graph or cast the crew. The framework will execute your decomposition with perfect obedience. It will not devise it, will not warn you that the parser and its tests belong together, and will not notice that two of your subtasks are the same subtask wearing different names.
Second, coordination discipline and organisation. The framework supplies topology primitives — supervisors, hand-offs, mailboxes — but whether the topology suits the task is Chapter 21’s question, and the primitives answer none of it. The sharper version concerns roles, and Chapter 16’s three-clause test applies with full force to every crew abstraction on the market: declaring an agent “the Reviewer” changes its prompt, and changes nothing else unless you wire the information flow, the authority, and the accountability that make a role real. The costume rack is included; the jurisdictions are sold separately. The same goes for the institutional layer entire — norms with sanctions, identity made scarce, reputation that carries — none of which arrives by import statement, all of which Part IV showed to be exactly where multi-agent systems are won and lost.
Third, evaluation and the model itself. No framework knows what “better” means for your task, owns your ground truth, or ships your evaluation suite — Chapter 24’s machinery is always bespoke — and, per the previous section, none improves the model by a parameter. Assembled, these three absences explain a pattern every practitioner has watched: the framework compresses the time from idea to running system, which means it compresses the time from idea to running mistake. Frictionless assembly moves the bottleneck — swiftly, invisibly — onto precisely the things no framework provides, and a team that measures its progress by how quickly the demo ran has measured the one thing the framework guaranteed anyway.
None of this is an argument against frameworks; plumbing is real work, and buying it from specialists is usually right. It is an argument about where to spend the savings. The engineering effort a framework releases should be reinvested upstream — in specifying the task, designing the decomposition, wiring real roles, and building the evaluation that will catch the fourteen failure modes the stack cannot — because the alternative allocation, spending the savings on more agents and more features, is how a team re-derives the MAST taxonomy at its own expense. With the box’s contents and its absences both inventoried, the remaining question about the box itself is which one to buy — and for that, the market’s noisy variety reduces to a design space with two axes.
19.3 The Design Space: Abstraction, Control, and the Layered Stack
The market’s variety is noisier than its structure. New frameworks launch weekly, each with a manifesto, and the practitioner who tries to track them as individuals is playing a losing game against the release calendar. Tracked as positions in a design space, however, the population collapses to something teachable: two axes and a layering pattern, stable across two generations of tooling, onto which any newcomer — including the ones released after this book went to press — can be placed in about five minutes of reading its documentation. Learning the space is the durable act; the placements are perishable, and this section will quarantine them accordingly.
The first axis is abstraction against control, and it runs from declaration to construction. At the declarative end sit the role and crew systems: describe your agents — names, goals, backstories — declare a task, and the framework decides the rest: who speaks when, what each sees, when the work is done. In the middle sit the workflow and graph runtimes: you draw the control flow explicitly — nodes, edges, state transitions — and the framework executes your drawing. At the far end sit the bare SDKs and libraries: you own the loop outright and buy parts. The trade along the axis is the oldest one in software: the declarative end reaches a running demonstration in an afternoon and reaches its ceiling shortly afterwards — when the framework’s hidden decisions turn out wrong for your task, there is nowhere to put the fix — while the constructive end starts slower, and every decision that matters has an address in your own code. Prototypes, uniform tasks, and internal tools favour declaration; production systems with specific coordination needs migrate, in practice and with grumbling, toward construction.
The placements, then — offered as a dated snapshot, quarantined in this one paragraph, and superseded by Appendix B the moment the two disagree. The crew and role systems — CrewAI is the type specimen1 — occupy the declarative end, as did the conversational AutoGen lineage before its absorption into a successor framework.2 LangGraph is the type specimen of the graph runtime,3 with Google’s ADK occupying similar middle ground.4 And the vendor SDKs — the Claude Agent SDK,5 the OpenAI Agents SDK6 — sit toward the library end: loops you may own, parts you may compose. Every name in this paragraph will drift; the axis they are pinned to will not.
The second axis is opinionation — how much is decided for you — and the reason it earns separate billing is that it does not reduce to the first. A graph runtime can be radically unopinionated: it executes whatever you draw and ventures no view about what you should have drawn. A vendor SDK, meanwhile, can sit at the low-abstraction end and still be intensely opinionated — about context assembly, tool-result handling, retry behaviour, sub-agent budgets — because it encodes one laboratory’s hard-won view of how its own models are best harnessed. That combination, low abstraction with high opinion, is not a contradiction; it is arguably the most valuable quadrant on the map, since Section 19.1’s verdict was that good opinions are what is worth buying, and opinions about the harness are best bought from whoever trains the model. The practical reading of the axis: for each of the six services, ask who decides — and remember that an undecided question is not freedom, it is homework.
%%{init: {"theme": "base", "themeVariables": {"quadrant1Fill": "#FAF7F0", "quadrant2Fill": "#DCEFE2", "quadrant3Fill": "#FAF7F0", "quadrant4Fill": "#F3EFE5", "quadrantPointFill": "#1E2EB8", "quadrantPointTextFill": "#16171B", "quadrantXAxisTextFill": "#16171B", "quadrantYAxisTextFill": "#16171B", "quadrantTitleFill": "#16171B", "quadrantInternalBorderStrokeFill": "#766F65", "quadrantExternalBorderStrokeFill": "#766F65"}}}%%
quadrantChart
x-axis You hold the loop --> The framework holds the loop
y-axis Unopinionated --> Opinionated
quadrant-1 Declare and delegate
quadrant-2 Buy the opinions
quadrant-3 Own everything
quadrant-4 Draw the flow yourself
Crew and role systems: [0.82, 0.80]
Vendor agent SDKs: [0.20, 0.82]
Graph runtimes: [0.55, 0.28]
Bare libraries: [0.15, 0.15]
The third structure is not an axis but a pattern: the layered stack, and one vendor’s ecosystem makes the cleanest worked example. The LangChain family decomposed, over successive years, into components (models, tools, retrievers), an orchestration layer (the graph runtime), a memory layer, and an observability-and-evaluation layer — four strata, separately adoptable, jointly marketed. The details are one company’s product history; the lesson is structural, because the strata correspond, almost line for line, to Section 19.1’s six services — which is to say that every mature stack converges on this layering, not because vendors copy each other (though they do) but because the anatomy demands it. The buyer’s corollary is worth the price of the section: evaluate stacks layer by layer, not as monoliths — a strong orchestration layer does not oblige you to adopt its sibling memory layer, and clean layer boundaries are precisely what make Section 19.4’s exit disciplines possible.
The opening’s archaeology can now be given its citations, and its point sharpened. JADE provided agent lifecycles, mailbox messaging, and a directory of capabilities (Bellifemine et al., 2007); Jason compiled the belief–desire–intention loop into an executable language (Bordini et al., 2007), with AgentSpeak as its theoretical core (Rao, 1996). Map them onto Section 19.1’s services and the exercise is sobering: lifecycle management, typed messaging, topology, capability discovery — four of the six services, engineered to international standards, twenty years early. What the classical platforms lacked was not architecture but a substrate: the slot where behaviour goes was filled by hand-written Java, and hand-written behaviour is exactly as capable as its author’s foresight. The modern framework is, structurally, the old anatomy with that one slot filled by a foundation model — which is the book’s substrate thesis (Section 3.4) told as product history, and a useful humility for any buyer convinced that this generation’s scaffolding, at last, is the permanent one.
The map, then: one axis for who abstracts, one for who decides, a layering that every stack converges on, and a family tree with more dead branches than live ones. What the map deliberately omits is the fourth dimension every buyer discovers eventually and few price in advance: commitment. Positions in this design space are cheap to enter and expensive to leave, the expense varies enormously by position, and the next section is about reading the price tag before, rather than after, the purchase.
19.4 The Price of Dependence: Lock-in, Churn, and Exit
Dependence on a framework is paid in two currencies, and both invoices arrive whether or not anyone read the terms. The first is lock-in, denominated in space: how deeply the framework’s types, idioms, and assumptions have penetrated your codebase, and therefore how much of your system would have to change if it did. The second is churn, denominated in time: how fast the framework changes underneath you, and therefore how often you pay a migration tax merely to stand still. The preface has already offered its actuarial estimate of framework half-lives, involving a mayfly and a prawn sandwich, and recorded that a rename, a merger, and a 1.0 all occurred among the major frameworks during its own composition. This section takes that comedy seriously: dependence on unstable software is not a misfortune to lament but an engineering condition to design for, with known disciplines and a price that can be read before purchase.
Lock-in rarely arrives by decision; it arrives by accretion. Nobody resolves to couple their system to a framework — instead, the framework’s state type appears in a function signature because that was convenient, its callback shape bends a module’s design because that was required, its graph idiom becomes the team’s mental model because that was the tutorial — and eighteen months later the framework is not a dependency but a load-bearing wall. Two pieces of practitioner lore explain why this is the default outcome rather than a lapse. The law of leaky abstractions holds that every non-trivial abstraction leaks,7 which for frameworks has a sharp corollary: the abstraction saves you learning the harness only until the first serious bug, at which point you learn the harness anyway — through the framework’s wrapper, in its vocabulary, at two in the morning. And Hyrum’s law holds that with enough users, every observable behaviour of an interface will be depended upon, documented or not8 — and you are among the users: some part of your system now relies on framework behaviour its authors never promised, and the next minor version is under no obligation to preserve it.
Churn compounds the position, because a framework’s rate of change is set by its market, not by your roadmap. Agent frameworks are competing for developer attention in the fastest-moving corner of the industry, which makes rewrites, renamings, and deprecation cycles not failures of stewardship but rational strategy — the stillness you want from infrastructure is, for them, a competitive liability. Nor is declining to upgrade a genuine option for long: model APIs move, security patches land on new versions only, and the ecosystem’s tutorials, hires, and integrations all migrate forward — so standing still is also a treadmill, merely one with worse footing. The cost of dependence is thus a product: the depth of the coupling multiplied by the frequency of the shifts, and the design goal is to shrink the first factor, since the second is not yours to control.
Shrinking it has a known discipline — the habits of teams that depend on unstable software and remain cheerful — and it fits in a paragraph. Wrap, don’t weave: the framework lives behind thin adapters that speak your interfaces, so its types stop at the boundary instead of percolating through the codebase. Own the state schema: the team’s shared state — Chapter 9 and Chapter 11’s crown jewels — is defined in your types and serialised in your format, with the framework seeing projections of it, never holding the original. Own the traces and the evaluations: transcripts, token accounts, and test suites are the assets that outlive every stack, and Chapter 24’s evaluations should score tasks, not frameworks, so they transfer intact across a migration. Keep prompts in your repository as first-class artefacts, not woven into framework configuration. And pin versions, upgrading deliberately on your schedule rather than continuously on theirs. The posture, in one line: treat the framework as a contractor, not a co-owner — valued, paid, and replaceable.
The final move is to promote exit cost from post-mortem lament to selection criterion, priced at the moment of adoption — which is the cheapest possible moment to think about leaving, since nothing has yet been built and no sunk cost has yet learned to argue. The estimate is rough but clarifying: sketch what migrating the pilot task off the candidate would touch, and notice that the answer does not reduce to the design-space axes — a high-abstraction crew framework can be cheap to leave precisely because so little of your logic lives inside it, while a low-level library can be expensive to leave because its idioms wove through everything. What predicts the price is Section 19.1’s question — who holds the loop — and how far the types travel. Appendix B applies the same honesty to its own contents by version-dating everything it says. And with that, the chapter’s materials are assembled: what a framework provides, what it cannot, where the candidates sit, and what commitment costs. What remains is to put them in order and choose — the procedure the preface promised, and the section that ends with permission to choose nothing at all.
19.5 Choosing: A Decision Procedure
A decision procedure earns the name only if it operates on inputs you actually possess, and the failure of most framework advice is that it operates on the wrong ones: feature matrices, benchmark leaderboards, and adoption statistics — facts about the candidates, when the discriminating facts are about you. The procedure this chapter has been assembling asks five questions, in an order that matters because each answer prunes the space the next question must search. They are questions about your task, your team, and your tolerance for commitment; the candidates only enter at the end, and by then most of them have already left.
First: what topology does the task actually need? Chapter 21 catalogues the patterns, and its lesson can be borrowed in advance: most tasks need less topology than the showcases suggest, and a task whose shape you cannot yet state is itself an answer — it says choose for changeability, which points down-spectrum toward construction and away from frameworks whose topology is a commitment. Second: which of the six services must you own? Walk Section 19.1’s anatomy and sort each service into decided-for-me-gladly or must-control; then check the must-control list against each candidate’s opinions, because a must-own service that the framework has already decided is not a convenience but a scheduled fork — the day your requirement and its opinion diverge, you will be patching around your own infrastructure.
Third: what does the team already know? Existing skills and stack gravity are legitimate inputs, not biases to be ashamed of — a familiar framework operated well beats a superior one operated badly — but weigh the asset honestly: churn depreciates it, and hiring against a framework is hiring against its half-life. Fourth: what is the observability and evaluation story? Can you see every run, export every trace in a format you own, and point Chapter 24’s suite at the system without absorbing the framework’s types into your ground truth? A framework that cannot answer this cheerfully is proposing that you fly on its instruments, which is a proposal to decline. Fifth: what does exit cost? — priced per the previous section, at adoption, in writing.
Then measure, because the questions shortlist candidates and only evidence separates them. The chapter’s lab is the template: take one task — the book uses its running team’s pipeline, a bounded coding job with a checkable outcome — and build it three times, at three positions in the design space: a graph runtime, a vendor SDK, a declarative crew system, with the same model underneath, because the substrate must be controlled or the comparison measures nothing. Score four columns: tokens, latency, lines of code, and — the column that repays the whole exercise — failure behaviour: inject a tool error mid-turn, a model timeout, a malformed response, and record what each candidate actually does. The numbers this book obtained will be stale before the ink is; the lab’s deliverable is the method, which is Chapter 24’s method in miniature, and the reader is expected to re-run it against whatever the market looks like on the day it matters.
What such comparisons teach has a stable shape even as the entries change, and the shape is worth knowing in advance. The token and latency columns usually spread modestly — same model, same task; the harness differs at the margins. The lines-of-code column spreads dramatically and misleads reliably: declaration wins it, and the victory measures exactly the decisions you did not get to make. The failure-behaviour column is the one that separates production candidates from demonstration candidates — retried silently, surfaced cleanly, or swallowed whole? — and it is, without fail, the column no feature matrix publishes. The lab’s standing headline: the interesting column is the one nobody advertises.
And then there is the option the procedure must price honestly, because for some readers it wins: no framework at all. The folklore holds that building a runtime from scratch is a quarter’s work and a maintenance millstone; the truth, which the next chapter demonstrates by doing it, is that a minimal, complete harness is small — the six services at honest minimum scope are each modest pieces of engineering: Section 19.1’s paper aeroplane rather than its passenger aircraft, admittedly, but complete, flying, and yours — because the cognitively hard part is the model’s job and the coordination-hard part is yours either way (Section 19.2’s lesson from the other side). From-scratch is the right answer when the must-own list is long, the task is narrow, and longevity matters more than ecosystem; and it pays a dividend no purchase can: you know where every decision lives, which converts the worst debugging sessions from archaeology into navigation. It costs what independence always costs — you maintain it, and nobody writes integrations for your runtime but you. It is not the default. It is a candidate, priced like the others.
The procedure, then, in one breath: topology, control, team, observability, exit — then measure, then choose, and let the choice be shallow enough to revisit. The chapter’s whole posture compresses to three maxims: choose late, because commitment made before the task is understood is commitment to a guess; choose shallow, because Section 19.4’s disciplines keep the exit lit; and keep the receipts, because your traces and evaluations are the assets that survive every migration. The next chapter takes the road this one has kept open, and pays the from-scratch tax in full — a complete runtime in plain Python — partly as a live option, and mostly because there is no better way to learn what every framework is actually doing on your behalf. Whichever container you choose, what you build inside it will be assembled from a small set of recurring shapes, and those are Chapter 21’s subject.
19.6 Summary
- A framework is a set of opinions about the harness. The loop, typed state and persistence, tool dispatch, context assembly, multi-agent topology, and observability are what every agent system needs and what every framework sells; the load-bearing distinction is inversion of control — whether you call the library or the framework calls you — and the model, throughout, is a component of the agent, not a synonym for it.
- No framework installs the hard parts. Task specification, decomposition quality, coordination discipline, organisational machinery, and evaluation are upstream of all plumbing — the MAST failure taxonomy holds across every framework it studied, which is the finding. A crew abstraction supplies costumes; the jurisdictions are still your job.
- The design space has two load-bearing axes. Abstraction versus control separates declarative role systems from graph runtimes and bare SDKs; opinionation measures how much is decided for you. Layered ecosystems are one vendor’s answer to a decomposition every stack must make — learn the decomposition and the vendors become interchangeable.
- The classical generation is a memento mori worth reading. JADE, Jason, and the FIPA platforms were rigorous, standardised, and are mostly archaeology; their ideas — loops, mailboxes, directories, lifecycles — run in every modern stack uncredited. Frameworks are temporary containers for durable ideas.
- Price the exit before the entrance. Lock-in arrives through leaky abstractions and compounds with churn; the discipline is thin adapters at the boundary, ownership of your state schema, your traces, and your evaluations — the data outlives the framework — and an exit cost estimated at adoption time, not discovery time.
- Choose by procedure, not charisma. Topology first, control second, team third, observability fourth, exit cost fifth — then measure: one task, several candidates, scored on tokens, latency, code, and failure behaviour. A framework chosen from its demonstration has been chosen by its marketing department.
- Choosing none is a live option. The from-scratch tax is lower than folklore claims, the understanding it buys is permanent, and the next chapter pays the tax in full, in plain Python, to show exactly what it purchases.
19.7 Exercises
Exercise 1. Six excerpts from the documentation of stacks the running team is evaluating. (i) “Annotate any Python function with @tool; when the model requests it, we validate the arguments against the generated schema, execute the call in a sandbox, and splice the result into the next turn.” (ii) “step(state) performs exactly one model turn and returns the parsed action; call it whenever you are ready.” (iii) “Declare your state as a typed schema; after every node we checkpoint the latest state, and a crashed run resumes from the last checkpoint.” (iv) “Subclass Agent and override on_message; the runtime delivers mail and schedules your handler.” (v) “Each turn, our relevance policy assembles the window; to change what the model sees, implement rank(candidates) and register it.” (vi) “Set trace=True to stream span-level telemetry to our hosted dashboard; raw traces are retained on our servers for thirty days.” (a) Name the service, among Section 19.1’s six, that each excerpt is selling. (b) For each, answer the section’s load-bearing question — for this service, do you call it or does it call you? — and read the consequences off Table 19.1: whose stack trace at debugging time, and where your code may live. (c) The team operates under two hard constraints: every token must be accounted in the team’s own exportable ledger, and yesterday’s failed run must be replayable deterministically today. Sort the six services into decided-for-me-gladly and must-control for this team, and identify which excerpts are scheduled forks in Section 19.5’s sense — a must-own service the candidate has already decided — saying for each what the divergence will look like on the day it arrives.
Exercise 2. After a pilot on one framework, the running team’s post-mortem lists five incidents: (i) the reviewer approved a patch whose tests were never run; (ii) two coders independently implemented the same parsing subtask under different names; (iii) a worker discovered its subtask was impossible as specified, went silent, and the team merged around the gap; (iv) a tool call failed mid-turn and the whole run died with a stack trace from inside the framework; (v) token spend came in at three times budget because every agent saw the entire history every turn. The team lead proposes migrating to a rival framework whose site advertises “production-grade multi-agent support”. (a) Classify each incident: does it sit upstream, in one of Section 19.2’s three absences — specification and decomposition, coordination discipline and organisation, evaluation and the model — or inside the box, in one of the six services; and where it is inside the box, say whether it is missing plumbing or a present opinion that suits this team badly. (b) State what the MAST finding — fourteen failure modes, catalogued across seven frameworks, none clustering by framework and none a plumbing failure — predicts the migration will fix and will not fix, and what the post-mortem would have to contain instead for the migration hypothesis to survive it. (c) Design the experiment that would settle the question before the migration is paid for, using Section 19.5’s lab as the template: what is held fixed, what is varied, which columns are scored, which incident classes are deliberately re-injected, and which confound the same-model control removes.
Exercise 3. Four frameworks, described by their own documentation. Ensemble: “describe your agents in a YAML manifest — names, goals, personalities — declare the task, and the runtime decides who speaks, in what order, and when the work is done.” Circuitry: “you draw the nodes and edges; we execute exactly the graph you drew and venture no view about what you should have drawn; the state schema is yours.” Foundry SDK, from a model laboratory: “you own the loop and call our parts; context packing, tool-result truncation, retry backoff, and sub-agent budgets follow our defaults, which encode what we know about harnessing our own models.” Kit: “a bag of parts — an HTTP client, a schema validator, prompt templates; bring your own everything else.” (a) Place each on Section 19.3’s two axes and name its quadrant in the vocabulary of Figure 19.2. (b) Use Circuitry and Foundry SDK to demonstrate that the opinionation axis does not reduce to the abstraction axis, and say why the chapter argues Foundry SDK’s corner is arguably the most valuable position on the map. (c) For each candidate, answer “who holds the loop?” and state one consequence each for debugging, extension, and exit. (d) The classical platforms sit on the same map: set JADE’s inventory — lifecycles, mailbox messaging, a capability directory — and Jason’s compiled belief–desire–intention loop against the six services, name the service that had no classical counterpart at all, and explain, in the substrate thesis’s terms, why filling the one empty slot with a model made that particular service load-bearing overnight.
Exercise 4. The running team’s codebase touches its framework at d = 60 call sites where the framework’s types appear directly. The framework ships breaking releases at a rate \lambda = 3 per year, and declining to upgrade is not an option — model APIs move and security patches land on new versions only. Each release forces a fix at every exposed site, at 15 minutes per site. The alternative is Section 19.4’s wrap-don’t-weave: an adapter layer of a = 6 modules behind which the framework’s types stop at the boundary, so that a release touches only the adapters, at 45 minutes each; building the layer costs A = 40 engineer-hours up front. (a) Compute the annual maintenance bill under each regime and the break-even time T^* at which the adapter layer has paid for itself. (b) The churn rate is set by the framework’s market, not by the team’s roadmap: recompute T^* at \lambda = 6 and at \lambda = 1, and state which factor of the depth-times-frequency product the discipline shrinks and which it cannot touch. (c) Hyrum’s law: suppose ten per cent of the woven sites depend on observable but unpromised behaviour, and each such breakage costs triple (45 minutes) because it is discovered at run time rather than at review; recompute the woven bill and T^*. (d) An exit migration touches every site where the framework’s types travel: 1 hour per woven site, or 2 hours per adapter module. Compute both exit invoices; then show that at \lambda = 1 over a two-year horizon the adapter layer fails to pay for itself on churn savings alone but pays comfortably once the exit option is priced in, and say what this teaches about when the disciplines are worth their cost.
Exercise 5. A run of the team’s pipeline costs R = 20{,}000 tokens when clean, and a mid-run fault — a tool error, say — strikes any attempt independently with probability f = 0.08. Three dispositions, in the vocabulary of Section 19.5’s census: surface — the faulted attempt is abandoned visibly and rerun from scratch until an attempt is clean; silent retry — the fault is recovered inside the run at an extra 5,000 tokens on faulted runs, which then ship correct output over clean-looking columns; swallow — the run ships on schedule with silently wrong output, and a wrong artefact that reaches the repository costs D = 150{,}000 tokens of downstream detection and repair. (a) Derive the expected token cost per correct outcome under each disposition — price every surfaced attempt at the full R, and let the silent retry always recover — and rank the three at the given numbers. (b) Find the downstream cost D^* below which swallowing would undercut surfacing, and the stricter bound below which it would undercut the silent retry too; express both in units of R, and say what a system in which swallowing were genuinely the economy would imply about the value of the artefact being produced. (c) Now let the environment regress so that f jumps to 0.3 while nobody is watching. Under surfacing, every fault is a visible event; under silent retry, a run fails visibly only when the fault and its retry both fail, with probability f^2. Compute, for each disposition at both the healthy and the regressed rate, the expected number of runs until the first visible signal; compute the probability of a signal within five runs at the regressed rate; and compute the extra spend per run that the retry regime accrues silently meanwhile. (d) Your arithmetic in (a) makes silent retry the cheapest disposition, yet the census in frontier/framework_lab/scorecard.py orders its verdicts surfaced, retried, swallowed, best to worst. Reconcile the two using (c), name the rig artefact in which a silent retry remains legible when all four columns are clean, and state the condition under which silent retry is genuinely acceptable rather than a slow-motion swallow.
Exercise 6. The companion repository’s frontier/framework_lab/scorecard.py records a crashing framework in two contradictory ways: _guarded books it as a run with status="crashed" and a crashed: tag in failures, while classify_response maps every exception — the injected TimeoutError and an unhandled TypeError alike — to the verdict surfaced_cleanly. Meanwhile the judge-free fallback _default_quality in frontier/runners.py scores any output containing the string return at 1.0, and the swallow branch of inject replaces a run’s output with a wrong artefact after its quality has been computed. (a) Demonstrate the quality column’s double blindness on the swallow path: show that a swallowed run reads 1.0 for two independent reasons — the pipeline freezes quality before the fault fires, and even re-judging the mutated output changes nothing — and explain why this is Section 19.2’s third absence arriving as a measurement bug. (b) A crash is not a clean surface: reimplement the classifier with a fourth verdict, crashed — an exception of the injected fault’s own type still reads as surfaced; any other exception reads as a crash — and exhibit both verdicts on the same faulted run. (c) Replace the string heuristic with an output oracle that executes the artefact against the failing test it was meant to fix, and show that the oracle separates the clean run from the swallowed one where the heuristic scores both at 1.0. (d) Rank crashed among the census’s three existing verdicts, best to worst, and defend the ranking in the chapter’s terms: visibility, control, and whose stack trace you will be reading at two in the morning.
Exercise 7. Read frontier/rig.py and frontier/runners.py as a specimen of Section 19.4’s disciplines. (a) Audit wrap-don’t-weave: locate every framework import in frontier/runners.py and say what its placement buys; name the single type that crosses the seam between the adapters and the instruments; and match the file’s habits to the disciplines they implement — whose schema is RunResult, whose vocabulary is the journal, and where the versions are pinned and dated. (b) Classify each position’s model seam for a vendor swap — a change of model supplier, not a version bump: the graph runtime and the plain baseline take an injected brain object; the declarative crew takes a provider-prefixed string through a translation layer; the vendor agent SDK’s supplier is constitutive. Say which position cannot swap at all, why that is a position on the opinionation axis rather than a defect, and what the buyer is buying when they accept it. (c) Write a thin adapter of your own: wrap any framework you have at hand — or a deliberately awkward toy stand-in — behind the same Runner seam, so that scorecard and failure_behaviour_table accept it unchanged, and keep a leak ledger as you go: every fact your adapter needed that the seam’s contract does not promise (token-accounting shape, status vocabulary, termination signal), taking as your worked specimen the usage dict-or-object defence inside the repository’s Claude Agent SDK adapter and naming the law from Section 19.4 that the defence instantiates. (d) Price the exits: using the lab’s dated lines-of-code table (crew 12, plain baseline 15, agent SDK 30, graph runtime 90) and your audit from (a), estimate what migrating the lab task off each position touches; then answer the chapter’s claim that exit cost does not reduce to the design-space axes — in the lab as built, which position is dearest to leave and why is the highest-abstraction one the cheapest; and in the woven counterfactual, where the crew’s task and output types had become the lab’s own representation, what would the invoice look like instead?
Exercise 8. The declarative crew adapter in frontier/runners.py declares its one agent with three fields: role “Coder”, goal “Fix the bug with the smallest possible patch”, backstory “You reply with a unified diff and nothing else.” (a) Apply Chapter 16’s three-clause role test to the declaration as shipped: say exactly what the three fields change about what the model sees; then, for the crew as a whole, establish who routes information, who decides when the work is done, and what checks the diff — identifying which clauses are carried by prompt text, which by the framework’s own loop, and which by nothing at all. (b) A vendor demonstration extends the crew with a second agent, role “Reviewer”, and a second task, “review the patch”, run sequentially so that the coder’s output feeds the reviewer’s prompt. State what the added declaration genuinely provides, then specify the three pieces of wiring that would make the Reviewer real: the information it must receive (name the artefacts), the authority a rejection must exercise over shipping (name the mechanism and its bound), and the accountability record the rig must carry (name the journal event in frontier/rig.py that exists for exactly this). (c) Name the absence from Section 19.2 that this exercise instantiates and the quadrant of Figure 19.2 whose temptation it dramatises, and reconcile your findings with the chapter’s verdict that the costume rack is included while the jurisdictions are sold separately.
Exercise 9. Run Section 19.5’s five questions, in order, over two briefs, recording at every step what the answer prunes. Brief A: two engineers, one weekend, an internal bot that labels incoming bug reports; the task’s final shape is unclear (deduplication and routing may follow); the team already operates a declarative crew framework on another project; nothing is regulated; the tool will be reviewed for retirement in a quarter. Brief B: the running team productionised for a regulated client on a five-year contract: topology known and fixed (an orchestrator over coder, reviewer, and tester); every model call must pass a client-mandated redaction layer; a complete audit trail must be exportable in a format the client’s auditors, not any vendor, dictate; no framework skill can be assumed to survive in the hiring pool for the contract’s life. For each brief: (a) answer the five questions — topology, which services must be owned, team knowledge, observability, exit cost — in a sentence each; (b) produce the must-own sort of the six services and name which positions of Figure 19.2 each answer eliminates; (c) name the surviving candidate class — a quadrant, or none at all — and state explicitly whether the from-scratch option qualifies under the chapter’s three conditions: the must-own list is long, the task is narrow, and longevity matters more than ecosystem; (d) for the brief where a framework survives, name the two disciplines from Section 19.4 you would impose on day one, and defend the pair.
Exercise 10 (lab). Re-run the chapter’s lab against the market as it stands on the day you read this. First, hermetically: from the companion repository, run python -m frontier.framework_lab.run (no key, no network) and the suite tests/test_frontier_framework.py, and confirm the rig is sound — the scripted scorecard should show the token column in a narrow band and the lines-of-code column spread wide, and the hermetic failure table enacts the module’s dated disposition assumptions, so it demonstrates the mechanism rather than measuring any framework. Then live: with one model held fixed under every position — the same-model control is the comparison’s entire validity — run python -m frontier.framework_lab.run --live with every position that installs on your machine, adding or repairing adapters as the frontier’s drift demands. Record the version of every framework package, the model identifier, and the date beside the results. Score the four columns and the failure-behaviour table, and compare the shape, not the cells, with Section 19.5’s predictions: tokens and latency spreading modestly, lines of code spreading dramatically and misleading, failure behaviour separating the production candidates from the demonstration candidates. Treat every obstacle as data: a position that no longer installs, an import that has been renamed, an adapter that needs an afternoon’s repair — each is Section 19.4’s churn, measured, so log the minutes each repair cost and report them beside the columns; and a framework that crashes on the probe is a failure-behaviour data point, not a reason to stop (the scorecard’s _guarded records it as exactly that). The durable finding is the method and the shape of the spread, not any cell of the table: your numbers will disagree with the book’s, and that disagreement is the chapter’s thesis working.
https://github.com/microsoft/autogen; the lineage’s ideas continue in Microsoft’s unified agent framework, the merger the preface alluded to.↩︎
Joel Spolsky’s coinage: https://www.joelonsoftware.com/2002/11/11/the-law-of-leaky-abstractions/.↩︎