Exercise Bank — Tools, Actions, Memory, and Context
The printed Chapter 4 keeps Exercises 1–6; the bank continues from Exercise 7.
Several of the remaining exercises use the retrieval scoring rule of Section 4.12, \mathrm{score}(m) = w_{\mathrm{rel}} \cdot \mathrm{rel}(m, q) + w_{\mathrm{rec}} \cdot \mathrm{rec}(m) + w_{\mathrm{imp}} \cdot \mathrm{imp}(m), with every component normalised to [0, 1]; unless an exercise says otherwise, the weights are the chapter’s defaults, w_{\mathrm{rel}} = 1.0, w_{\mathrm{rec}} = 0.5, w_{\mathrm{imp}} = 0.5. The agents throughout are the book’s software-engineering team.
Exercise 7. The dispatcher printed in Section 4.2 — reproduced in the companion repository as foundations/algorithms/tool_schema.py — promises that a malformed call comes back as an observation, an error dictionary, not a stack trace. It does not keep the promise. (a) Exhibit two calls, each a well-formed argument dictionary for run_tests_tool, that make the printed dispatch raise an exception instead of returning: one that slips through a gap in the validation loop, and one that fails inside the tool function itself. Name the exception each raises, and say where the gap in the validation loop lies. (b) Repair the dispatcher — fifteen lines suffice — so that both calls, and any like them, come back as error dictionaries: undeclared arguments must be rejected, and a tool that raises must have its failure reported as an observation. Show your dispatcher’s output on both calls from (a). (c) In the vocabulary of the perception–action loop, state what the agent loses when an exception escapes instead: what happens to its next perception, and why an error observation lets the loop continue where a crash cannot.
Exercise 8. Three of the coder agent’s tools are declared with these descriptions: read_file(path) — “read a file into the context; call this before editing a file”; apply_patch(path, diff) — “apply a unified diff to a file; only call this once the file has been read”; run_tests() — “run the suite; use it to check the tree after a change”. Work over the propositions \mathrm{read}(f), \mathrm{patched}(f), and \mathrm{tested}. (a) Write each tool as a STRIPS operator in the sense of Section 3.9: preconditions, add list, delete list. Exactly one operator warrants a non-empty delete list; identify it and justify the deletion. (b) From the empty initial state, give the plan a classical planner returns for the goal \mathrm{patched}(f) \wedge \mathrm{tested}, where f is src/auth.py, showing at each step the state before, the precondition check, and the state after. Point to the entry in your operators that forces run_tests to come last. (c) At run time the model emits apply_patch on a file it has never read, and the dispatcher of Section 4.2 duly executes it. State precisely which classical guarantee has been lost, then restore it in the harness: write a wrapper around dispatch, a dozen lines, that records which paths read_file has returned and answers any premature apply_patch with an error observation — enforcement moved from the model’s judgement into deterministic code.
Exercise 9. The coder agent runs an edit–test loop on a broken module. After each edit the tree is actually correct with probability 0.6, independently across rounds. The harness truncates the test runner’s output to fit the window, and the truncation misreports a failing run as a pass with probability e = 0.1; a passing run is always reported as a pass. The agent stops at the first reported pass. (a) Compute the probability that, when the agent stops, the tree is in fact still broken. (b) Show that this probability does not depend on how many rounds the loop happens to run. (c) The budget stretches to exactly one improvement: a stronger model that raises per-edit correctness to 0.75, or a repaired truncation policy that cuts the misreport probability to 0.02. Compute the false-stop probability under each, say which wins, and connect the outcome to what Section 4.1 says about an agent’s perception being built rather than given.
Exercise 10. The team’s release agent proposes merges to the main branch. A proposed merge is wrong with probability w = 0.02, and a wrong merge that lands costs L = 5000 units to put right. An approval gate — a human confirming each merge before it executes — costs g = 8 units per merge and catches a wrong merge with probability d = 0.9; a caught merge is abandoned at no further cost. (a) Compute the expected cost per proposed merge with and without the gate. Does the gate pay? (b) Derive the break-even loss L^* below which the gate stops paying, and evaluate it. (c) Staged deployment now makes any landed merge recoverable at a cost of \rho = 40 units in place of L. Compute the expected cost per merge of all four designs — neither safeguard, gate only, staging only, both — rank them, and state what the ranking says about where safeguard effort should go first.
Exercise 11 (project). Build the dispatcher the chapter’s toy gestures at. Starting from foundations/algorithms/tool_schema.py in the companion repository and the runtime dispatcher in foundations/tools.py, write a module — standard library only, in the style of the foundations layer — whose dispatch (a) validates arguments against a JSON-Schema subset covering the types string, integer, boolean, and number, arrays with typed items, nested objects carrying their own required and properties, and enum constraints; (b) rejects undeclared arguments at every nesting level, of which your repair in Exercise 7 is the flat case; (c) returns every failure, whether of validation or of execution, as an error observation and never raises; and (d) appends to a caller-supplied audit log one record per dispatch — tool name, arguments, outcome, wall-clock time — whether the call succeeded or not: the raw material of the accountability that Chapter 22 will demand. Ship a test battery with at least one test per failure class, including a boolean supplied where a string or an integer is declared (mind what Python’s bool is a subclass of).
Exercise 12 (lab). The chapter claims that a tool the model cannot tell when to use is, to the agent, no tool at all. Measure it. Fix five tools for the running team — read a file, edit a file, run the tests, search the code, run a shell command — and write twenty one-sentence requests, each with one intended tool. Using any function-calling model, present every request under three menus: menu A, the five tools crisply described (what each does, and when to use it); menu B, the same five schemas with vague, overlapping descriptions; menu C, menu A plus five plausible near-duplicates (search_files beside search_code, and so on). Score each menu by the fraction of requests whose first emitted call names the intended tool. Report the pattern across the three menus, say which mechanism from Section 4.3 each degradation isolates, and explain why the two degradations you have separated — description quality and menu width — are the two halves of the chapter’s expressiveness-against-controllability trade-off. Note the model and the date; the pattern, not the number, is the result.
Exercise 13. The team’s reviewer agent is preparing to review a change to the payments module, and its episodic store offers three candidates, components already computed: m_1, “the payments module has a known race condition in its retry logic”, with \mathrm{rel} = 0.8, \mathrm{rec} = 0.1, \mathrm{imp} = 0.9; m_2, “this morning’s CI run failed two payments tests”, with 0.6, 0.8, 0.5; and m_3, “the maintainer prefers small, single-purpose commits”, with 0.2, 0.5, 0.8. (a) Compute the three scores under the default weights and give the ranking. (b) Holding w_{\mathrm{rel}} = 1.0 and w_{\mathrm{imp}} = 0.5, find the exact value of w_{\mathrm{rec}} above which m_2 overtakes m_1. (c) Determine whether any choice of w_{\mathrm{rec}} \ge 0 and w_{\mathrm{imp}} \ge 0, still with w_{\mathrm{rel}} = 1.0, puts m_3 first. Derive the conditions and either exhibit such weights or prove none exist; then say in a sentence what a builder who shipped those weights would have done to the agent.
Exercise 14. A long-running orchestrator computes recency from a decaying clock: \mathrm{rec}(m) = \lambda^{\Delta t}, where \Delta t is the memory’s age in hours and \lambda = 0.9. Two memories of equal importance compete for one recall slot, so the importance term cancels: a, “the schema migration is still half-applied”, with \mathrm{rel}(a, q) = 0.9, laid down \Delta t hours ago; and b, “the linter configuration changed”, with \mathrm{rel}(b, q) = 0.6, laid down this instant, so \mathrm{rec}(b) = 1. (a) With w_{\mathrm{rel}} = 1.0 and w_{\mathrm{rec}} = 0.5, find the age at which b overtakes a. (b) Recompute with w_{\mathrm{rec}} = 1.0, and say in a sentence what doubling the recency weight did. (c) Show in general that, for a relevance gap g = \mathrm{rel}(a, q) - \mathrm{rel}(b, q) > 0 and a freshly laid-down rival, the crossover age is
\Delta t^{*} \;=\; \frac{\ln \bigl( 1 - w_{\mathrm{rel}} \, g / w_{\mathrm{rec}} \bigr)}{\ln \lambda},
and that no crossover exists at all when w_{\mathrm{rec}} \le w_{\mathrm{rel}} \, g. What does the second finding imply about how much staleness the default weights can ever punish?
Exercise 15. Say that, for the present query q, memory a dominates memory b when \mathrm{rel}(a, q) \ge \mathrm{rel}(b, q), \mathrm{rec}(a) \ge \mathrm{rec}(b), and \mathrm{imp}(a) \ge \mathrm{imp}(b), with at least one inequality strict. (a) Show that if a dominates b, then \mathrm{score}(a) > \mathrm{score}(b) for every strictly positive weight vector. (b) An engineer proposes speeding up recall by discarding every dominated candidate before the sort. Show that for strictly positive weights this never changes which memory is recalled top-1, but can change top-k recall for k \ge 2: with components (\mathrm{rel}, \mathrm{rec}, \mathrm{imp}) of a = (0.9, 0.9, 0.9), b = (0.8, 0.8, 0.8), and c = (0.3, 0.2, 0.1), compute the top-2 under default weights before and after the discard, and say what the agent would now be shown. (c) The engineer next proposes deleting dominated memories from the store outright. Construct a concrete pair of memories and two queries showing that domination under today’s query need not survive tomorrow’s, and identify which of the three components is solely responsible for the reversal, given that all memories age under the same clock.
Exercise 16. The coder agent makes one model call per turn and runs for T = 40 turns. Its context carries a fixed preamble of s = 1{,}000 tokens (instructions and the issue text), and each turn appends h = 500 tokens of history — tool output and the model’s reply. Uncompacted, the context at turn t is s + h(t-1) tokens, and every token of it is paid for again on every call. (a) Compute the total input tokens paid for across the run — the growing-context arithmetic series that Chapter 2’s Exercise 2 derived for the stateless coder, here with the memory chapter’s constants. (b) Now the harness compacts: at the end of turns 10, 20, and 30, the accumulated history — any earlier précis included — is replaced by a précis of c = 800 tokens. Give the context size at turn t under this policy, recompute the total, and express the saving as a percentage. (c) The window holds W = 8{,}000 tokens. If the harness instead compacts at the end of every k-th turn, the context peaks at s + c + h(k-1) tokens; find the largest k that never overflows the window. (d) Give the leading term of the total cost as a function of T under each policy, and state in one sentence what compaction buys asymptotically and what it irrecoverably costs.
Exercise 17. Extend the chapter’s Memory record with two fields: key, the subject an entry is about, and written, the turn on which it was laid down. Against tonight’s query “does the orders table need an index?”, the tester agent’s store holds three entries: key orders-index, “orders table lacks an index on user_id”, with \mathrm{rel} = 0.85, \mathrm{rec} = 0.30, \mathrm{imp} = 1.00, written at turn 3; key orders-index, “index added on orders.user_id in commit f3a2”, with 0.80, 0.90, 0.40, written at turn 9; and key retry-policy, “payments retries use exponential backoff”, with 0.20, 0.60, 0.50, written at turn 7. (a) Rank the unconsolidated store under default weights, and name the life-cycle failure of Section 4.12 that the top-1 recall exhibits. (b) Implement consolidate(store), which keeps only the newest entry per key, and show that it repairs the recall. (c) In a few sentences: for which of the two long-term stores of Section 4.10 is newest-per-key the right rule, and what does it destroy in the other?
Exercise 18. A reflection pass in the style of Section 4.12 periodically synthesises new memories: each pass-k reflection reads b memories from pass k-1, their soundness independent, and is unsound if any of its inputs is unsound or — with probability \varepsilon, independently — through fresh misreasoning even from sound inputs. Pass-0 memories are all sound. Let q_k be the probability that a pass-k memory is unsound. (a) Derive the recurrence
q_k \;=\; 1 - (1 - \varepsilon)\,(1 - q_{k-1})^{b},
and compute q_1, q_2, and q_3 for \varepsilon = 0.05 and b = 3. (b) Show that q_k is strictly increasing and converges to 1 for every \varepsilon > 0 and b \ge 1. (Hint: (1-q)^{b} \le 1-q on [0, 1].) (c) Find the smallest k with q_k > 0.9; then say in two sentences which assumption of the model a repair must break for the limit to change, and what machinery would break it.
Exercise 19 (project). Starting from foundations/algorithms/retrieval.py in the companion repository — its Memory, score, and top_k — build a recall benchmark for the team and tune the weights against it, as Section 4.12 instructs. (a) Construct a store of at least eight memories, each with an age in hours and an importance, and at least six queries, each naming one gold memory; include one query whose gold memory is older than every distractor, and one phrased so that it shares no informative word with its gold. (b) Implement \mathrm{rel} as Jaccard overlap of lowercased word sets, and \mathrm{rec} = 0.9^{\Delta t} with \Delta t the age in hours; measure top-1 accuracy under the default weights and list the failures. (c) Grid-search (w_{\mathrm{rel}}, w_{\mathrm{rec}}, w_{\mathrm{imp}}) over \{0, 0.25, \dots, 1.5\}^{3}; report the best accuracy and characterise where the maximising weights lie. (d) Diagnose every residual failure as either a weight failure — the blend misweights sound components — or a component failure, where a component itself is the wrong proxy; and relate what you find to the chapter’s advice on which agents want which weights.
Exercise 20 (lab). Test the lost-in-the-middle finding of Liu and colleagues (2024) on a model you can call; the companion repository’s setup guide describes the set-up. (a) Synthesise a context of about sixty invented key–value facts of the form “the deploy code for service-17 is QRV-482”; plant one probe fact at relative depths 0, 0.25, 0.5, 0.75, and 1.0; ask the model for the probe’s value; run at least twenty trials per depth with fresh fillers each trial; and tabulate accuracy by depth. Describe the shape you find. (b) Repeat with the probe duplicated at the end of the context, as in Exercise 4(b), and estimate whether the two placements fail independently by comparing the observed joint failure rate with the product of the marginal failure rates. What do your measurements say about the duplication trick, and about the independence assumption Exercise 4 rested on? Record the model identifier and the date beside the table.