Exercise Bank — Evaluation and Experimental Method

The printed Chapter 21 keeps Exercises 1–6; the bank continues from Exercise 7.

Exercise 7. Ground truth in the case study is executable, but not everything the team ships is a test suite’s business, and not everything that matters is an outcome. (a) The pull-request description the team attaches to each fix — the paragraph that explains the defect, justifies the approach, and flags risks for whoever merges it — is open-ended work: decompose “the description is good” into at least five rubric clauses in the manner of Section 21.2, assign each clause the cheapest adequate grader (mechanical check, judge model, or human), and say what genuinely subjective residue survives your decomposition and what the two honest options for it are. (b) Two configurations of the team, P and Q, are timed over twenty resolutions each; wall-clock minutes, P: 9, 10, 10, 10, 11, 11, 11, 11, 12, 12, 12, 12, 12, 13, 13, 13, 13, 14, 14, 21; Q: 6, 6, 7, 7, 7, 8, 8, 8, 8, 9, 9, 9, 10, 10, 10, 11, 12, 28, 32, 39. Any resolution over twenty minutes trips the on-call gate — it is the run that pages someone. Compute each configuration’s mean latency, its ninety-fifth percentile (take the 19th of the twenty sorted values), and its empirical rate of tripping the gate. (c) Compute, for each configuration, the probability that a working day of twelve resolutions pages someone at least once, using the empirical rates from (b); then say which configuration a mean-only report recommends, which one the robustness family of Table 21.1 recommends, and why the deployed experience sides with the latter.

Exercise 8. The ablation series of Section 21.5, run on the case-study team: three ablated variants are each compared with the full team on the same twelve tasks, paired, with d_k the full team’s score minus the ablated team’s; the summaries are — remove the reviewer: \bar d = 0.058, s_d = 0.062; remove the tester: \bar d = 0.021, s_d = 0.080; remove the debate gate: \bar d = 0.004, s_d = 0.015. Of the team’s 260,000 tokens per task, the reviewer accounts for 60,000, the tester for 40,000, and the debate gate for 130,000. (a) Compute the three paired t statistics and classify each result as evidence or its absence. (b) Compute each ablation’s minimal detectable effect at K = 12 (80% power, two-sided 5%) and explain why the tester’s t \approx 0.91 and the gate’s t \approx 0.92 — nearly identical statistics — license opposite conclusions. (c) Deliver the verdicts with the invoice attached: which component earns its keep and at how many tokens per quality point, which component “changes nothing but the invoice”, and which verdict must wait. (d) For the verdict that must wait, compute the K at which its minimal detectable effect falls to 0.02, price that experiment in tokens, and say — given that ablation grids multiply costs and priority follows price — which action comes first, the bigger tester experiment or the gate’s removal, and why Chapter 18’s subtraction heuristic only becomes knowledge at this level of ceremony.

Exercise 9. The substrate’s training data has a cutoff, and the team’s favourite public benchmark predates it; a fresh twin — same task genre, same difficulty mix, born after the cutoff — has just been released. The team’s system scores 0.78 on the old benchmark’s 500 tasks and 0.67 on the fresh one’s 300. (a) Taking the fresh score as the system’s true capability and assuming a contaminated task is passed by recall with probability 0.95, estimate the fraction of the old benchmark that is inside the examinee from the mixture 0.78 = f \times 0.95 + (1 - f) \times 0.67. (b) Before blaming contamination, check the cliff against sampling noise: compute the standard error of each score, the standard error of their difference, and the number of standard errors the eleven-point cliff spans. (c) A quarter of leaderboard-targeted tuning follows: the public score rises from 0.78 to 0.87 while the home suite — distilled from the team’s own traffic — moves from 0.74 to 0.76. Compute the fraction of the leaderboard gain that transferred, name the decay at work and the link of the inference chain it corrupts per Table 21.2, and say what the untransferred residue consists of. (d) Answer Section 21.6’s three diagnostic questions — how old, how targeted, how far from home — for the proposal “adopt the tuned configuration for our backlog on the strength of the 0.87”.

Exercise 10. Last quarter the team distilled a home suite of 100 tasks from the journal — 70 bug-fix, 30 refactor, matching the production mix of the day — and the per-stratum pass rates have been stable ever since: 0.90 on bug-fixes, 0.60 on refactors. (a) This quarter the production mix has drifted to 40% bug-fix, 60% refactor: compute what the frozen suite still reports and what production now experiences, and state the precise sense in which the suite’s figure is a fact about last quarter. (b) Reweighting rescues the estimate without new tasks: compute the reweighted suite score, state the assumption under which reweighting is valid and one concrete way the drift itself could break it, then compute the price of the thrift — the effective sample size n_{\mathrm{eff}} = 1 / \sum_h W_h^{2}/n_h of the reweighted 100-task suite against a fresh 100-task suite drawn at the new mix, and the factor by which the standard error widens. (c) The incident suite of Chapter 20 contributes 40 regression tasks drawn from failure classes that make up 6% of live traffic and pass at 0.55; the other 60 tasks are stratified traffic passing at 0.85. Compute the blended suite score and the current production pass rate, explain why the suite should score below production — the oversampling is the point — and state the reporting discipline that stops the gap from being read as a regression. (d) Write the suite’s maintenance policy as Section 21.6 prescribes: for each of re-stratification, retirement, addition, and label top-up, give a measurable trigger and the action it fires — no trigger may be a calendar date alone.

Exercise 11 (lab). Goodhart, measured. Take the single-agent runner on six of your Exercise 6 tasks and a development judge — the LLMJudge of systems/evaluation/judge.py given a rubric prompt you record verbatim. Optimise against it: for at least five rounds, generate three variants of the coder’s system prompt, score each variant’s outputs with the development judge, and keep the champion. Then grade the first and final champions three ways: (i) the development judge; (ii) an evaluation judge held out as Figure 21.1 draws it — a different model, or at minimum an independently worded rubric, never consulted during the rounds; (iii) the executable checks. Report the three before-to-after deltas; the gap between (i)’s delta and (iii)’s is the measure-become-target, measured. Inspect the winning prompts for judge-shaped adaptations — verbosity, rubric echoing, confident summaries — and note which survive into (ii) and (iii). Record both model identifiers and the date; the durable finding is the ordering of the three deltas, not their sizes — though if (i) rose while (iii) fell, frame the transcript.