Exercise Bank — Frameworks and the Art of Choosing One

The printed Chapter 16 keeps Exercises 1–4; the bank continues from Exercise 5.

Exercise 5. Four frameworks, described by their own documentation. Ensemble: “describe your agents in a YAML manifest — names, goals, personalities — declare the task, and the runtime decides who speaks, in what order, and when the work is done.” Circuitry: “you draw the nodes and edges; we execute exactly the graph you drew and venture no view about what you should have drawn; the state schema is yours.” Foundry SDK, from a model laboratory: “you own the loop and call our parts; context packing, tool-result truncation, retry backoff, and subagent budgets follow our defaults, which encode what we know about harnessing our own models.” Kit: “a bag of parts — an HTTP client, a schema validator, prompt templates; bring your own everything else.” (a) Place each on Section 16.3’s two axes and name its quadrant in the vocabulary of Figure 16.2. (b) Use Circuitry and Foundry SDK to demonstrate that the opinionation axis does not reduce to the abstraction axis, and say why the chapter argues Foundry SDK’s corner is arguably the most valuable position on the map. (c) For each candidate, answer “who holds the loop?” and state one consequence each for debugging, extension, and exit. (d) The classical platforms sit on the same map: set JADE’s inventory — lifecycles, mailbox messaging, a capability directory — and Jason’s compiled belief–desire–intention loop against the six services, name the service that had no classical counterpart at all, and explain, in the substrate thesis’s terms, why filling the one empty slot with a model made that particular service load-bearing overnight.

Exercise 6. The running team’s codebase touches its framework at d = 60 call sites where the framework’s types appear directly. The framework ships breaking releases at a rate \lambda = 3 per year, and declining to upgrade is not an option — model APIs move and security patches land on new versions only. Each release forces a fix at every exposed site, at 15 minutes per site. The alternative is Section 16.4’s wrap-don’t-weave: an adapter layer of a = 6 modules behind which the framework’s types stop at the boundary, so that a release touches only the adapters, at 45 minutes each; building the layer costs A = 40 engineer-hours up front. (a) Compute the annual maintenance bill under each regime and the break-even time T^* at which the adapter layer has paid for itself. (b) The churn rate is set by the framework’s market, not by the team’s roadmap: recompute T^* at \lambda = 6 and at \lambda = 1, and state which factor of the depth-times-frequency product the discipline shrinks and which it cannot touch. (c) Hyrum’s law: suppose 10% of the woven sites depend on observable but unpromised behaviour, and each such breakage costs triple (45 minutes) because it is discovered at run time rather than at review; recompute the woven bill and T^*. (d) An exit migration touches every site where the framework’s types travel: 1 hour per woven site, or 2 hours per adapter module. Compute both exit invoices; then show that at \lambda = 1 over a two-year horizon the adapter layer fails to pay for itself on churn savings alone but pays comfortably once the exit option is priced in, and say what this teaches about when the disciplines are worth their cost.

Exercise 7. The companion repository’s frontier/framework_lab/scorecard.py records a crashing framework in two contradictory ways: _guarded books it as a run with status="crashed" and a crashed: tag in failures, while classify_response maps every exception — the injected TimeoutError and an unhandled TypeError alike — to the verdict surfaced_cleanly. Meanwhile the judge-free fallback _default_quality in frontier/runners.py scores any output containing the string return at 1.0, and the swallow branch of inject replaces a run’s output with a wrong artefact after its quality has been computed. (a) Demonstrate the quality column’s double blindness on the swallow path: show that a swallowed run reads 1.0 for two independent reasons — the pipeline freezes quality before the fault fires, and even re-judging the mutated output changes nothing — and explain why this is Section 16.2’s third absence arriving as a measurement bug. (b) A crash is not a clean surface: reimplement the classifier with a fourth verdict, crashed — an exception of the injected fault’s own type still reads as surfaced; any other exception reads as a crash — and exhibit both verdicts on the same faulted run. (c) Replace the string heuristic with an output oracle that executes the artefact against the failing test it was meant to fix, and show that the oracle separates the clean run from the swallowed one where the heuristic scores both at 1.0. (d) Rank crashed among the census’s three existing verdicts, best to worst, and defend the ranking in the chapter’s terms: visibility, control, and whose stack trace you will be reading at two in the morning.

Exercise 8. Read frontier/rig.py and frontier/runners.py as a specimen of Section 16.4’s disciplines. (a) Audit wrap-don’t-weave: locate every framework import in frontier/runners.py and say what its placement buys; name the single type that crosses the seam between the adapters and the instruments; and match the file’s habits to the disciplines they implement — whose schema is RunResult, whose vocabulary is the journal, and where the versions are pinned and dated. (b) Classify each position’s model seam for a vendor swap — a change of model supplier, not a version bump: the graph runtime and the plain baseline take an injected brain object; the declarative crew takes a provider-prefixed string through a translation layer; the vendor agent SDK’s supplier is constitutive. Say which position cannot swap at all, why that is a position on the opinionation axis rather than a defect, and what the buyer is buying when they accept it. (c) Write a thin adapter of your own: wrap any framework you have at hand — or a deliberately awkward toy stand-in — behind the same Runner seam, so that scorecard and failure_behaviour_table accept it unchanged, and keep a leak ledger as you go: every fact your adapter needed that the seam’s contract does not promise (token-accounting shape, status vocabulary, termination signal), taking as your worked specimen the usage dict-or-object defence inside the repository’s Claude Agent SDK adapter and naming the law from Section 16.4 that the defence instantiates. (d) Price the exits: using the lab’s dated lines-of-code table (crew 12, plain baseline 15, agent SDK 30, graph runtime 90) and your audit from (a), estimate what migrating the lab task off each position touches; then answer the chapter’s claim that exit cost does not reduce to the design-space axes — in the lab as built, which position is dearest to leave and why is the highest-abstraction one the cheapest; and in the woven counterfactual, where the crew’s task and output types had become the lab’s own representation, what would the invoice look like instead?

Exercise 9. The declarative crew adapter in frontier/runners.py declares its one agent with three fields: role “Coder”, goal “Fix the bug with the smallest possible patch”, backstory “You reply with a unified diff and nothing else.” (a) Apply Chapter 13’s three-clause role test to the declaration as shipped: say exactly what the three fields change about what the model sees; then, for the crew as a whole, establish who routes information, who decides when the work is done, and what checks the diff — identifying which clauses are carried by prompt text, which by the framework’s own loop, and which by nothing at all. (b) A vendor demonstration extends the crew with a second agent, role “Reviewer”, and a second task, “review the patch”, run sequentially so that the coder’s output feeds the reviewer’s prompt. State what the added declaration genuinely provides, then specify the three pieces of wiring that would make the Reviewer real: the information it must receive (name the artefacts), the authority a rejection must exercise over shipping (name the mechanism and its bound), and the accountability record the rig must carry (name the journal event in frontier/rig.py that exists for exactly this). (c) Name the absence from Section 16.2 that this exercise instantiates and the quadrant of Figure 16.2 whose temptation it dramatises, and reconcile your findings with the chapter’s verdict that the costume rack is included while the jurisdictions are sold separately.

Exercise 10 (lab). Re-run the chapter’s lab against the market as it stands on the day you read this. First, hermetically: from the companion repository, run python -m frontier.framework_lab.run (no key, no network) and the suite tests/test_frontier_framework.py, and confirm the rig is sound — the scripted scorecard should show the token column in a narrow band and the lines-of-code column spread wide, and the hermetic failure table enacts the module’s dated disposition assumptions, so it demonstrates the mechanism rather than measuring any framework. Then live: with one model held fixed under every position — the same-model control is the comparison’s entire validity — run python -m frontier.framework_lab.run --live with every position that installs on your machine, adding or repairing adapters as the frontier’s drift demands. Record the version of every framework package, the model identifier, and the date beside the results. Score the four columns and the failure-behaviour table, and compare the shape, not the cells, with Section 16.5’s predictions: tokens and latency spreading modestly, lines of code spreading dramatically and misleading, failure behaviour separating the production candidates from the demonstration candidates. Treat every obstacle as data: a position that no longer installs, an import that has been renamed, an adapter that needs an afternoon’s repair — each is Section 16.4’s churn, measured, so log the minutes each repair cost and report them beside the columns; and a framework that crashes on the probe is a failure-behaviour data point, not a reason to stop (the scorecard’s _guarded records it as exactly that). The durable finding is the method and the shape of the spread, not any cell of the table: your numbers will disagree with the book’s, and that disagreement is the chapter’s thesis working.