21  Evaluation and Experimental Method

Chapter 20 left one question standing, and it is the expensive one. The team demos beautifully; the engineers believe in it; the transcript reads like a well-run meeting, the dashboard is green, and the invoice is four times what a single agent would have cost. Is it better? Around that innocent word an entire discipline hangs, because the field routinely answers it by anecdote, adrenaline, and comparison with an opponent assembled principally to lose — the lone model with no tools, no iteration, and a prompt written by someone rooting for the other side. The preface warned that you cannot unit-test a vibe; this chapter is about what you can do instead.

The honest answer is hard to obtain, for reasons that deserve respect. The system is stochastic, so a single run is an anecdote and the same configuration will beat itself on Tuesdays; the tasks are long-horizon, so errors compound; the workers are plural, so when quality moves nobody knows which agent moved it; every sample costs real money, so the statistics must run on a budget that would make an agronomist laugh; and the substrate will not hold still — the model under everything upgrades on a vendor’s schedule, and this quarter’s improvement may be the provider’s. Evaluation under these conditions is not quality assurance with extra steps; it is experimental method — the discipline agriculture and medicine spent a century learning, arriving at software that argues back.

The book’s contribution is that its systems have been building the instruments all along: the journal holds every decision with its full context, Chapter 20’s replay makes any incident a deterministic test, and the incident suite is an evaluation set growing itself from production. Over the whole chapter stands the emphasis the preface promised: the baseline. Chapter 1 asked whether to build a team at all; this chapter hands that question instruments, and their first duty is to keep the comparison honest.

At the end the system will be measured, compared, and honestly accounted — good today, against these tasks, on this substrate. What an adversary can make it do tomorrow is another matter, and Section 20.9 has already treated it; who answers when it happens anyway is the next chapter’s.

21.1 The Question, and the Opponent Built to Lose

“Is it better?” is not yet a question; it is a question-shaped noise with two blanks in it. Better than what — and at what cost? Evaluation of a multi-agent system is never absolute, because nobody deploys a system against nothing: the alternative is always something — a single agent, a pipeline, the intern, the status quo — and the only claim worth making is comparative. The cost blank is not an accounting afterthought but half the verdict: a team that scores 5% higher at four times the price has not won an argument; it has stated a trade, and whether the trade is good depends on facts — how much a marginal point of quality is worth, and to whom — that no benchmark knows. Fill in both blanks and the question becomes answerable; leave them blank and any answer will do — which is how the field’s weakest claims are constructed.

The commonest failure is the one the section title names: the opponent built to lose. It is rarely fraud; it is enthusiasm plus asymmetric effort. The team gets six weeks of prompt-tuning, hand-picked tools, an iteration loop, and a champion who stays late; the baseline gets an afternoon, a default prompt, and nobody’s love. Then the paper — or the pitch deck — reports that the team wins. Most riggings are invisible in the headline: the baseline ran a weaker model (the team on the frontier model, the “single agent” on something cheaper); the baseline was denied the tools the team’s members each enjoy; the baseline was denied iteration (one shot, against a team that critiques and revises for twenty turns); the baseline was denied the budget (a fraction of the tokens, then judged on quality alone as though the tokens were free). A result of this shape does not show that the team beats an agent; it shows that effort beats neglect — a finding of no great novelty. Chapter 11’s audit of the debate literature met the same pattern from the other side: gains that shrink, sometimes to nothing, once the single-model baseline gets the same compute and a competent prompt.

So the discipline is to specify the honest opponents in advance, and the book’s distinctions say what they are. The first is the strong single agent: the same frontier model, the same tools, the same token budget — all of it, the full multiple the team would spend, available as iteration, self-critique, retries, and self-consistency — in one context, with no coordination losses and no whisper game, because there is nobody to whisper to. One exclusion must be stated in the protocol because it looks like a rigging and is the experiment: “the same tools” does not include a tool that spawns subagents. Delegation is the variable under test — the moment the baseline can hire ephemeral workers it stops being the control arm and becomes Section 1.3.4’s delegating single agent, a third architecture worth an arm of its own wherever it is the incumbent — so the line is the locus of control: one context, one journal, everything else equal. Chapter 18 taught why this opponent is formidable: everything the team pays in joins, the single agent keeps. The second is the fixed workflow, Chapter 1’s most practically important look-alike: the predetermined pipeline with model calls at fixed points — cheap, deterministic, testable, and very often sufficient — as Chapter 16 put it, most tasks need less topology than the showcases suggest. Both opponents deserve what the team had: a champion who has genuinely tried to make each alternative win — the critic’s instinct from Chapter 11 applied to the experiment itself.

What does an honest result look like once the opponents are real? The book has been citing one all along — a vendor’s account of its own system, cited for the shape of the claim rather than a referee’s stamp it never had. Anthropic’s evaluation of its research system reported a genuine win for the orchestrator over a strong single agent — and, in the same breath, the price (many times the tokens) and the boundary (tasks that parallelise across independent facets; tasks that fit one context showed no such gain) (2025). That triple — better at this class of task, at this cost, and not elsewhere — is the canonical shape of a defensible claim, and its third clause is the one enthusiasm always omits: a system evaluated on research synthesis has been evaluated on nothing else, and the reader who deploys it on contract review has left the evidence behind.

All of which turns Chapter 1’s decision framework from a judgement call into an experiment. The questions asked there in the armchair — does the task genuinely decompose, do the objectives diverge, does one context suffice — the lab answers with instruments. Build all three systems — the team, the strong single agent, the workflow; run them on the same task set, many times each; record everything, because the journal makes recording free. That triple experiment is this chapter’s running spine, with the case-study’s coding team as the defendant — and before it can run, one deceptively trivial question must be settled: what gets written down? Outcomes are the obvious answer, and they are perhaps a quarter of it.

21.2 What to Measure: Outcomes, Process, Cost, Robustness

The seduction to resist is the single number: leaderboard culture runs on one score per system, and one score discards the two things that make the exercise multi-agent at all — how the agents worked together, and what the working-together cost. A complete record answers four questions — did it succeed, how did it get there, at what price, and how much does the answer wobble — and Table 21.1’s four families are those questions made operational. All four, as a Chapter 20 dividend, are views over the journal: measurement, on this architecture, is not added to the system but asked of the record.

Table 21.1: The four families of measurement that replace the single leaderboard number — each a different question made operational and, as the Chapter 20 dividend, a near-free view over the journal. A system reported on one family has been reported on a quarter of itself.
Family Question it answers What you measure Where the number comes from
Outcomes Did it succeed? Pass/fail against ground truth; rubric clauses for open-ended quality Executable truth, human labels, or downstream acceptance
Process How did it get there? Duplicated work per task, messages per resolution, conflict and escalation rates, dead-letter counts, idle time, the divergence signatures of Chapter 18’s failure catalogue Views over the journal; MAST-style trace annotation (Cemri et al., 2025)
Cost At what price? Tokens and pounds per resolved task, wall-clock latency per resolution, gate latency and approver attention, cost per marginal quality point Unit economics read off the journal
Robustness How much does the answer wobble? Distributions rather than points, the tail (the ninety-fifth percentile), repeats across seeds and temperatures, survival of perturbations Repeated runs; Chapter 20’s chaos injections read as experimental treatments

Outcomes first, because they anchor everything else, and the operative question is where ground truth comes from — three sources, on very different terms. Executable truth: the tests pass, the build compiles, the answer equals 42 — cheap, objective, and available only in the lucky domains. Human labels: the gold standard, at gold prices, budgeted like a scarce reagent — and assayed like one, because the gold is an alloy: raters disagree, tire, and drift, so a label campaign carries its own agreement statistics (Section 21.4 supplies the coefficients) or asks to be taken on trust. And downstream acceptance: the pull request merged, the report used, the customer not writing in — slow, noisy, nearly free, and very hard to fake. Hard to fake, though, is not hard to inflate: acceptance has a person in its denominator, and Chapter 22’s automation bias flows through it — an approver working in batches launders convenience into apparent quality — so a rising acceptance rate is evidence only alongside Section 21.4’s reliance discipline. The coding case study lives in measurement’s luxury quarter, where truth is executable — and the field’s benchmarks cluster there for the same reason: we measure best where measuring is easiest, then generalise with a confidence the evidence does not travel with. The embodied evaluator lives at the other end of the street: every sample costs wall-clock and hardware wear, so executable truth is bought from a simulator — cheap, repeatable, and biased, Section 4.6’s sim-to-real gap the price — and the scarce physical runs are spent on paired comparisons, sized in advance. It is no accident that the community so constrained built the refereed annual benchmark culture — RoboCup above all (Kitano et al., 1997) — Section 21.6’s exemplar.

Where ground truth runs out, the vibe problem begins: is the research summary good? has no test suite. The preface’s quip — you cannot unit-test a vibe — is cashed here as a technique, because what you can do is decompose it. A rubric breaks open-ended quality into clauses checkable one by one: every claim carries a source; all requested facets addressed; no internal contradictions; within length; correct register. Each clause is assigned the cheapest adequate grader — some mechanical (length, structure, citation presence), some a judge model (the next section’s subject), a few a human. What survives decomposition undecomposed — elegance, judgement, taste — is the genuinely subjective residue, and the honest options are two: pay humans to measure it, or declare it unmeasured and say so in the report. What is not on offer is the field’s favourite third option, measuring the checkable clauses and quietly presenting the total as though it covered the vibe entire.

Process is the family that makes the evaluation multi-agent, because outcomes are compatible with wildly different insides: two teams can resolve the same task, one by clean decomposition and crisp hand-offs, the other by flailing into success — and the difference is predictive, because luck does not survive a change in the task mix. The metrics are the dependability dashboard read as instruments — Table 21.1’s process row runs from duplicated work per task to the divergence signatures of Chapter 18’s failure catalogue. And the methodological exemplar the field already owns is MAST — whose deepest contribution here is not the fourteen failure modes but how they were obtained: over a thousand real traces, annotated against a defined taxonomy, agreement between annotators checked and reported (Cemri et al., 2025). That is trace annotation as ethnography, and any team with a journal owns the corpus to do the same — most simply by reading a stratified sample of traces on a schedule, the single highest-yield evaluation practice nobody budgets for.

Cost is the family Section 21.1 made half the verdict, and the discipline is unit economics rather than meter-reading. Tokens and pounds per resolved task — not per call, not per run — because a system cheaper by the token and less likely to resolve is dearer where it counts; latency as wall-clock per resolution, where the parallelism that justified the team must show up or stand exposed as decoration; and the human line item the ledgers omit: gate latency and approver attention are costs, and a team that resolves tasks cheaply by escalating every judgement to a person has outsourced its budget to the org chart. Cost per marginal quality point — what the last 5% of quality costs — is the number executives actually need, computable from the same records.

Robustness, finally, respects the opening premise: the system is stochastic, so any single number is a sample and variance is a result, not noise to be averaged into silence. Report distributions; mind the tail, because the deployed experience is dominated by the bad runs — the ninety-fifth percentile is the run that pages someone — and a system chosen on its mean has been chosen on its best behaviour. Repeats across seeds and temperatures separate the system’s variance from the substrate’s; and perturbation closes the Chapter 20 loop: the chaos-day injections double as experimental treatments — does quality survive a crash-and-resume, a tool outage, a mid-run interrupt? — so robustness is measured under the conditions production will supply. Four families, then, and a recurring dependency: outcomes beyond the executable, rubric clauses beyond the mechanical, and process judgements beyond the countable all need a grader — and the grader of choice is a model, either an excellent idea or a circular one, depending on details that deserve their own section.

21.3 The Judge Problem

Grading at scale is the judge model’s job — the field’s load-bearing shortcut, a model set to mark the work of models — and this book has already tried the arrangement in court: Chapter 10 read the judge as Arrow’s permitted dictatorship, reported the study that licensed it — verdicts agreeing with humans about as often as humans agree with each other (2023) — and catalogued the dictator’s tastes — position, verbosity, self-preference — with their constitutional remedies and the office’s standing audit, on pain of running “a preference pipeline with delusions of objectivity”. That constitution is taken here as ratified. What evaluation practice adds is a change of stance: Chapter 10 treated the judge as a constitution to be designed; an experimenter must treat it as an instrument to be maintained — and instruments drift.

Calibration is the maintenance schedule. A judge is calibrated the way a thermometer is: against a reference — a stratified sample of human labels, per task type, not one global figure — and the agreement rate is not a private comfort but part of the result: a comparison graded by a judge at 92% agreement with humans is a different piece of evidence from the same table at 74%. Agreement also has a base rate: where 90% of outputs pass, a judge that always says “pass” reaches 90% agreement while measuring nothing, so the honest figure is chance-corrected — Cohen’s \kappa (1960) rather than the raw percentage.1 Recalibration has triggers: the judge’s own model upgrades — an upgraded judge is a different examiner with the same name; the task mix drifts; and, most easily missed, the system under test changes style — calibration holds against a distribution of outputs, and a system that starts writing differently has quietly walked out from under its judge’s reference sample.

None of this maintenance is new; the field that has practised it longest repays the citation with the judge problem’s most useful theorem. Information retrieval has graded machine output by human judgement since the 1960s, when Cyril Cleverdon’s Cranfield tests fixed the form — shared task, shared corpus, hired humans judging relevance — that search evaluation has run ever since as the Cranfield paradigm (1967). The tradition’s standing embarrassment was that its assessors disagree, substantially and irreducibly; Voorhees measured what the disagreement costs: swap one assessor’s relevance judgments for another’s and every system’s absolute score moves, but the ranking barely stirs (2000). That stability is the licence Chapter 10’s study invoked, and it sharpens what calibration protects: comparative verdicts survive an imperfect judge; absolute grades are denominated in the judge that issued them — one more reason Section 21.5’s designs will be paired. The licence’s boundary sits where disagreement stops being honest noise: Cranfield’s assessors erred unsystematically, and unsystematic error is what rank stability forgives; a judge model’s tastes are systematic, and a bias correlated with one of the systems under comparison is exactly what reorders rankings — why Chapter 10’s remedies are constitutional rather than statistical, and why the audit belongs on the maintenance schedule.

The failure mode evaluation adds — beyond the tastes Chapter 10 catalogued — is circularity, organisational rather than statistical. A development loop needs constant judgement — selecting prompts, filtering outputs, deciding whether Tuesday’s change helped — and the temptation is to let the same judge serve both loop and final evaluation, at which point the team has spent weeks optimising against its examiner and then submitted the examiner’s grades as evidence. Chapter 10 showed the mechanism (a dictator’s stable tastes are a target); Chapter 14’s Goodhart supplied the epitaph (the measure that became a target ceased to be a measure); evaluation practice supplies the remedy, the oldest rule in machine learning: hold the judge out, exactly as one holds out data — a development judge the team may consult hourly, an evaluation judge that development never sees, and a report that says which was which. The same separation resolves a subtler conflation: Chapter 11’s critic is a component, working for the team to improve the artefact; the moment its approval is reported as the evaluation, the examiner is on the payroll of the examined, and the office, not the model, is what has been corrupted. One mind may be capable of both jobs; no constitution lets it hold both offices at once — Figure 21.1 separates them.

%%{init: {"theme": "base", "themeVariables": {"primaryColor": "#E8ECFF", "primaryBorderColor": "#4054B2", "primaryTextColor": "#16171B", "lineColor": "#3B4351", "edgeLabelBackground": "#FAF7F0", "clusterBkg": "#EFE9DC", "clusterBorder": "#766F65"}, "flowchart": {"rankSpacing": 28, "nodeSpacing": 42, "subGraphTitleMargin": {"top": 4, "bottom": 14}}}}%%
flowchart TB
    HUM["Human labels · stratified, per task type"]
    subgraph DEV["Development loop"]
        direction TB
        SUT["System under test"]
        DJ(["Development judge"])
        SUT --> DJ
        DJ -->|"consulted hourly"| SUT
    end
    FRZ(["freeze · snapshot<br/>the final system"])
    subgraph EV["Held-out evaluation"]
        direction TB
        EJ(["Evaluation judge<br/>development never sees it"])
        REP["Reported result"]
        XT["✗"]
        EJ --> REP
        EJ -.->|"no tuning feedback"| XT
    end
    SUT -->|"final system"| FRZ
    FRZ -->|"fresh outputs"| EJ
    HUM -.->|"calibrate"| DJ
    HUM -.->|"calibrate"| EJ
    classDef gate fill:#F7E6B5,stroke:#8A5A00,color:#16171B
    classDef world fill:#EFE9DC,stroke:#766F65,color:#16171B
    classDef risk fill:#F4D7D5,stroke:#A4161A,color:#16171B
    class FRZ gate
    class HUM world
    class XT risk
Figure 21.1: Two offices, one kind of judge: a development judge inside the loop, consulted hourly, and an evaluation judge development never sees, grading fresh outputs beyond the freeze gate with no tuning feedback crossing back. The geometry is the safeguard: it stops a team tuning itself against the marker that later certifies it.

Humans, then, are not the alternative to the judge but its reserve currency: the calibration samples, the subjective residue Section 21.2 set aside, the high-stakes verdicts too consequential to delegate, and the judge’s periodic audit. The economics are a pyramid — mechanical checks for the many, the judge for the scale, humans for the anchor — and the discipline is monetary: the judge’s verdicts are paper money backed by human labels, and an evaluation programme that prints grades without buying labels is running an unnoticed inflation. But the marker’s anchor is not the only place a person stands in this laboratory; the other place changes the rules.

21.4 The Human in the Denominator

So far this chapter has employed people in two roles: the instrument’s reference — the labels that anchor the judge — and a line item, the gate latency and approver attention Section 21.2 booked under cost. But Chapter 1 declared the running team’s fifth member on day one: the person approving at the gate, correcting the drafts, accepting or declining the output. The deployed unit, in most deployments that matter, is a mixed human–agent team — and then some of the chapter’s own metrics quietly change species. The rule deserves the book’s usual bluntness: the moment a metric’s denominator contains a human action — an approval granted, a correction made, a suggestion accepted — the evaluation is an experiment on people, and it inherits, whole, the methods of the sciences that run them. Nothing about journals and judges is retracted; those instruments measure the agents — they do not, alone, measure the team.

Measuring the team is better charted than the field’s optimism assumes, because human–computer interaction ran the experiments early. The target has a name, complementary performance: the mixed team beating both of its members working alone — Chapter 23’s collective-intelligence bar applied to the smallest mixed unit — and the empirical record is usefully deflating. Bansal and colleagues, surveying the early studies and running their own, found the bar cleared far less often than assumed — prior teams beat their best member mainly when the machine was the weaker party — and found the era’s favourite instrument backfiring: adding explanations to the AI’s recommendations made people more likely to accept them whether they were right or wrong (2021). Acceptance, in other words, is not accuracy. The instrument that detects the difference is reliance conditioned on correctness: split the human’s agreement rate by whether the agent was actually right — acceptance-when-right is the collaboration working; acceptance-when-wrong is Chapter 22’s automation bias arriving as a coefficient, invisible in the raw agreement rate that averages the two into comfort. The pair moves under intervention: forcing a beat of thought before the acceptance — the cognitive forcing designs — measurably reduces acceptance-when-wrong, at a price in speed and satisfaction the participants disliked (2021): exactly the trade an evaluation exists to price.

The design kit is classical; the parts that matter are the ones agent-only evaluation never needed. Assignment first: comparing conditions within the same participants buys the paired test’s variance discount, but people, unlike task sets, carry state — they learn the system, tire of it, and anchor on whichever condition came first — so orders must be counterbalanced and carry-over reasoned about; the between-subjects design pays more participants for cleaner inference. Validity second, and it bites hardest here: a consented volunteer, fresh, in an hour-long session, is not the operator on their fortieth diff of the Tuesday — the vigilance decay Chapter 22 dissects is a phenomenon of weeks, not sessions — so a lab result about oversight generalises to production precisely as far as the boredom generalises — stated in the report rather than hoped past. Workload third: the person is the system’s scarcest component, so their expenditure is an outcome, not an externality — the standard instrument is the NASA-TLX self-report scale (1988) — and a team that wins its accuracy by consuming the person is running down the one budget nobody meters. And agreement last: wherever humans supply the labels, the campaign reports chance-corrected agreement — Section 21.3’s \kappa for two raters, Krippendorff’s \alpha where raters, categories, or missing judgements multiply (2018) — which is MAST’s “agreement between annotators checked and reported” handed its actual coefficients.

One obligation is institutional rather than statistical: an experiment on people needs informed consent, the freedom to withdraw, a harm ceiling — and where participants are recruited, or colleagues observed at their gates, an ethics board’s answer before the first trial, not the write-up’s. The point is worth a sentence in a book whose laboratory is otherwise silicon: every (lab) exercise in these pages is computational — models measuring models, no person in any denominator — and the day a course extends one to human participants, the review board comes before the API key. With the human’s side given its own methods, the remaining questions are arithmetic: how many runs make a fact, what counts as a real difference, and which agent, if any, is earning its keep.

21.5 Method: Variance, Ablations, and the Cost of Knowing

The first law of method here is blunt: multiple runs or it did not happen. A single run is a sample of size one, a comparison of two such samples a coin toss with a write-up; any conclusion that would not survive a re-run was never a conclusion. The wobble has more sources than the temperature dial — production serving is nondeterministic even at nominally deterministic settings, task difficulty varies enormously, and the long horizon compounds small early divergences — so the unit of evidence is a distribution per configuration per task set, and the question of record is whether two distributions are distinguishable, not whether two anecdotes differ.

Classical statistics answers by collecting more data, and here every datum is an invoice, so the method must be chosen for thrift. The cheapest variance reduction is pairing: run both systems, A and B, on the same K tasks, with quality scores q_k^{A} and q_k^{B} on task k, and compare differences per task,

d_k \;=\; q_k^{A} - q_k^{B},

reporting the mean \bar d and its standard error s_d / \sqrt{K} (s_d the standard deviation of the d_k), so that task difficulty — usually the largest noise source by far — cancels in each d_k instead of drowning the signal. The ratio of \bar d to that standard error is the paired t statistic, and the whole computation fits in the standard library — here on toy quality scores for K = 6 tasks:

from statistics import mean, stdev

q_a = [0.82, 0.74, 0.91, 0.63, 0.78, 0.85]  # team, per task
q_b = [0.79, 0.70, 0.88, 0.66, 0.71, 0.80]  # strong single agent
d = [a - b for a, b in zip(q_a, q_b)]
K = len(d)
s_d = stdev(d)
t = mean(d) / (s_d / K**0.5)

round(mean(d), 3)       # -> 0.032
round(s_d / K**0.5, 3)  # -> 0.014
round(t, 2)             # -> 2.3

The raw scores wander across a quarter of the scale as the tasks change; the d_k scatter by only a few hundredths — the cancellation at work — and a t of 2.3 on K - 1 = 5 degrees of freedom is suggestive rather than settled: evidence, not yet a fact. Report the interval rather than the verdict — \bar d \pm 2\,s_d/\sqrt{K} here runs from roughly 0.004 to 0.059, somewhere between negligible and worthwhile, which says more than an asterisk — and mind the structure of the task pile: tasks arrive in families — ten variations on one repository — and near-copies carry the information of far fewer, so the family, not the variant, is the unit the standard error should count (Miller, 2024). Sequential designs stop spending when the answer is clear — provided the looks are paid for in advance: a threshold peeked at after every run is the selection problem wearing a lab coat. And three pieces of statistical honesty cost nothing and are routinely skipped. First, an underpowered null is not a finding: with twenty tasks and high variance, “no significant difference” means we could not afford to know, not they are equal; if parity is the actual claim, it must be tested as such: fix in advance a margin too small to care about and show the whole interval sits inside it — the equivalence test, a different and harder demonstration. Second, selection inflates: run eight configurations, report the best, and you have re-invented Section 12.2’s winner’s curse in your own lab — the maximum of noisy estimates is biased upward, and the winning configuration’s score will sag on fresh tasks. The remedy is the auction world’s too: re-run the winner on tasks it has never seen, and report that figure. Third, the institutional form of the other two: commit the design — metrics, task set, configurations, stopping rule — before the first token is spent — preregistration, which converts a dozen daily temptations into one signed promise. (The arithmetic — the paired test, power, the inflation of a selected maximum — is collected in Section A.2.)

Attribution — which agent earns its keep? — is the question multi-agent evaluation adds, and it has one honest instrument: the ablation. Remove the reviewer and measure defect rates; drop the debate gate and watch whether verdict quality moves or merely latency; take away a tool; collapse two roles; swap the whole pattern, supervisor for peers for debate, on the same task set — Chapter 18’s pattern-swap experiment, now run with the statistics above. Ablation grids multiply costs, so priority follows price: the component that doubles the budget proves its keep first. And it closes a loop: Chapter 18’s test for the gratuitous agent was subtraction — remove it and see what changes — offered there as a design heuristic; the ablation is that heuristic promoted to an experiment, and quietly one of the most cost-effective things a team can do, since the commonest finding is an agent whose removal changes nothing but the invoice.

Evaluation is not an event but a standing appointment: the system, the substrate, and the task mix all change, and each change re-asks every answered question. The suite — benchmark tasks, incident-suite regressions from Chapter 20, rubric checks, judge calibrations — re-runs on every material change: prompt tweak, model upgrade, new tool, new pattern; the alternative is discovering last month’s improvement made things worse from whoever pays for the consequences. Results attach to configurations, pinned and versioned per the dependability discipline, or the finding floats loose of anything reproducible. Where offline evaluation runs out — effects that appear only under real users, real load, real adversaries — A/B tests on live traffic take over, the journal once again supplying the measurements — and with Section 21.4’s obligation attached, because live traffic is people: a harm ceiling, a kill switch, and someone outside the team empowered to ask whether the users in the worse arm would have agreed to be there.

One last distinction separates the two experimental substrates the dependability chapter built. Replay — recorded responses through the fake client — is deterministic, instant, and free: the right court for infrastructure claims (the retry fires, the merge is orderly, the gate parks and resumes). It is the wrong court for behavioural claims, and the error is a category mistake worth naming: a replayed model cannot respond to your improvement — the recorded transcript answers as it answered, whatever you changed — so replaying a “better prompt” tests nothing but your patience. Plumbing is judged on replay; minds are judged live, at live prices. Which leaves the task set itself: every claim so far has quietly assumed a pile of tasks worth measuring on — where benchmarks come from, how they decay, and why the one that matters most is grown at home.

21.6 Benchmarks, Drift, and the Living Evaluation

A public benchmark is a genuine public good, and the case study of the genre has already appeared in this book: SWE-bench took the question “can agents fix real software?” and made it a number computed the same way by everyone — thousands of genuine GitHub issues, executable ground truth, a shared yardstick across laboratories and years (Jimenez et al., 2024). That is what benchmarks buy: comparability, a common language for progress, community effort concentrated on a measurable target. The genre has speciated since: GAIA sets general-assistant tasks whose whole difficulty is tool use, browsing, and multi-step composition (Mialon et al., 2023); \tau-bench wires a simulated user into the loop and scores pass^k — the same task passed on all k of k attempts — pricing reliability alongside capability — this chapter’s variance lesson built into the metric (Yao et al., 2024); and its successors were out within the year — a churn that will outpace any printed census. Every one of those virtues, though, has a decay product: a benchmark’s life cycle runs from public good to public hazard on a schedule set by its own success.

The decays are three, and each corrupts a different link in the chain of inference — Table 21.2 tabulates the mechanisms and their symptoms. Contamination corrupts the measurement: benchmarks live on the web, models are trained on the web, and sooner or later the exam is inside the examinee — the tell is the cliff between performance on a benchmark older than the training data and one born yesterday. Goodhart corrupts the target: Chapter 14 supplied the law — the measure that becomes a target ceases to be a measure (Strathern, 1997) — and a leaderboard is that law run as a business model. And unrepresentativeness corrupts the transfer itself, no corruption required: the benchmark’s task distribution was never yours, so even an honest, uncontaminated score is evidence about a neighbouring planet. The reflex the three decays should install is not cynicism but chronology: ask of any benchmark result how old, how targeted, and how far from home.

A fourth hazard arrives the moment the yardstick is other systems. Pit agents against agents — pairwise win rates, arena-style ladders — and the score inherits Chapter 14’s relativity, because “better” was opponent-indexed all along: win-rate tables are non-transitive in exactly Section 14.4’s spinning-top way, so an average win rate rewards beating the particular crowd that entered, and a crowd padded with near-duplicates tilts every average toward whatever beats the duplicates. The population-level instruments exist — estimating a game from simulation records and analysing the estimate has been empirical game-theoretic analysis since Wellman codified it (2006): Nash averaging scores each entrant against the population’s own equilibrium mixture and is by construction indifferent to duplication (Balduzzi et al., 2018); \alpha-rank ranks by evolutionary stability under perturbation (Omidshafiei et al., 2019) — for the working engineer the pointer suffices: it is this chapter’s refrain wearing game theory’s clothes — whom you measured against is part of the measurement.

Table 21.2: The three ways a public benchmark decays, each corrupting a different link in the chain from score to real-world competence — the success that makes a benchmark a public good is what starts its decay into a hazard.
Decay product What it corrupts How it corrupts Diagnostic question
Contamination Measurement The exam — tasks, solutions, the discussion threads dissecting them — ends up inside the examinee, so scores measure recall, not capability How old? Public before the model’s training cutoff — and any evidence of exposure?
Goodhart Target A leaderboard is a measure become a target as a business model, so systems accrete benchmark-shaped adaptations that transfer to nothing (Strathern, 1997) How targeted? Adapted to this leaderboard?
Unrepresentativeness Transfer The benchmark’s task distribution was never yours, so even an honest, uncontaminated score is evidence about a neighbouring planet How far from home? Far from your task distribution?

The book’s own field shows the alternative culture. The multi-agent reinforcement learning community diagnosed its reproducibility crisis in print and answered it with infrastructure rather than exhortation: VMAS standardised the experimental worlds themselves — vectorised multi-robot scenarios, cheap enough to run in bulk, defined precisely enough to replicate (Bettini et al., 2022) — and BenchMARL standardised the comparisons, one harness where algorithms, tasks, and configurations meet under identical conditions, reporting built in (2024). The lesson that transfers is not the simulators — MARL’s worlds replicate because they are simulations, a control the open-ended agent task will never grant — but the institutional form: reproducibility as tooling, with pinned configurations, standard harnesses, and variance reported by default — Section 21.5‘s discipline made too convenient to skip. The language-model world, still reporting single runs on contaminated leaderboards, has a neighbour worth visiting — and an elder one, which answered contamination decades before the decay had a name: the Text REtrieval Conference (TREC) has issued a fresh test collection every year since 1992 — new tasks, new documents, new human judgments, the judging economics solved by pooling only the entrants’ top-ranked returns for assessment (Voorhees & Harman, 2005) — so that the exam is always younger than the systems sitting it, and leakage is outrun rather than audited. The institution quietly teaches the form of the cure: a benchmark is not an artefact but a periodical.

The benchmark that actually governs your decisions is the one grown at home, and this chapter’s systems have been growing it all along. The journal is the ore: real production tasks, sampled and stratified by type; the incident suite is the seed — every bottled failure a regression task with its expected behaviour attached; rubrics and calibrated judges do the grading. What makes it a living evaluation is the maintenance the word implies — Chapter 18 wrote the reason on the wall: the task mix drifts, and last quarter’s accuracy is a fact about last quarter. So the suite refreshes as the mix moves, stale tasks retire, new failure classes join as discovered, and the human labels are topped up on a schedule — the reserve currency kept convertible. This is also Section 16.4’s ledger closed at last: the receipts that survive every migration were never the framework or the prompts — they are the traces and the evaluation suite, the portable, accumulating definition of what good means for your task, and the only artefact in the stack that appreciates.

Which leaves the chapter’s own spine, and a debt to settle. Chapter 1 promised that the team-versus-single-agent question would be put to instruments; the instruments now exist, and the companion repository’s evaluation harness wires all three arms to them. What this page will not do is print the verdict — this section’s argument applied to the book itself: any number set in type here would be a lab reading — one model, one date, one task set — a fact about last quarter by the time it reached you, promoted to load-bearing print exactly as this chapter warns against. The durable deliverable is the instrumented comparison, so the verdict is run rather than quoted: Exercise 6 hands over the trigger, and its first discovery is the chapter in miniature — the single-agent arm ships as an opponent built to lose, and no verdict is worth recording until the reader has strengthened it.

What measurement cannot do supplies the closing caution. Everything in this chapter evaluated the system against conditions sampled from the world — real tasks, honest baselines, production-shaped perturbations: users trying to get work done. It certifies nothing about the visitor with different intentions, whose inputs are not sampled but crafted, who has read your rubrics and probed your judge, and for whom your ninety-fifth percentile is not a tail risk but a target. Good, as established here, means good against the distribution; the adversary’s whole profession is to be off it — which is why Section 20.9’s defences stand outside this chapter’s verdicts, and no benchmark ever retires them. And when the system — evaluated, defended, and wrong anyway — does the thing it should not have done, the question stops being technical altogether; the next chapter convenes at the morning after.

21.7 Summary

  • The question is comparative, and the comparison is rigged by default. “Better” means better than the strongest honest alternative — the single agent with the same model, tools, and budget, or the fixed workflow — not an opponent built to lose; the multi-agent tax is real and must be earned in the open.
  • Measure four families, not one number — and decompose the vibe. Outcomes against ground truth; process, with the MAST categories as measurable trace properties; cost as per-outcome unit economics; robustness as variance and tail behaviour. Rubrics turn open-ended quality into checkable clauses; the genuinely subjective residue is measured by humans and priced accordingly.
  • The judge is part of the experiment. A model grading models inherits Chapter 10’s aggregation theorems and adds its own biases — position, verbosity, self-preference; calibrate against human labels, re-calibrate when models change, and never let the system optimise against its own examiner.
  • A metric with a human action in its denominator is an experiment on people. The deployed unit is usually a mixed human–agent team: reliance split by whether the agent was right, workload beside accuracy, agreement statistics for the raters — and an ethics answer before the first participant, since complementary performance is a bar mixed teams clear less often than their transcripts suggest.
  • Multiple runs or it did not happen, and your own evaluation outlives every benchmark. Variance is a result, not noise; ablations are the only honest answer to which agent earns its keep; public benchmarks rot by contamination and Goodhart, while the suite distilled from your own traces compounds.
  • Measured is still not safe. Evaluation certifies what the system does under the conditions you tested — today, on these tasks, on this substrate; what a motivated adversary can make it do is a different question, and it is Section 20.9’s.

21.8 Exercises

Exercise 1. The team’s champions circulate a memo: “the coding team resolved 34 of 40 backlog tasks; the single-agent baseline resolved 22 of 40; the case is closed.” The appendix discloses the conditions: (i) the team ran the frontier model, the baseline a cheaper mid-tier one; (ii) every team member had the repository tools — test runner, code search — and the baseline had none; (iii) the team’s review loop iterated up to twenty turns, the baseline answered in one shot; (iv) the team spent a mean of 260,000 tokens per task against the baseline’s 15,000, and the memo compares quality alone; (v) the team’s prompts had six weeks of tuning, the baseline’s an afternoon; (vi) a follow-up note adds that an improved reviewer prompt was “validated with no regressions” by replaying forty recorded transcripts through the fake client. (a) Name the rigging in each of (i)–(v) and the property of an honest comparison from Section 21.1 that it violates. (b) Specify the two honest opponents for this evaluation as Section 21.1 defines them, precisely enough that an engineer could build both without further decisions — what each is given, and who champions it. (c) The honest re-run comes back: the team resolves 31 of 40 at its mean 260,000 tokens per task attempted; the strong single agent resolves 27 of 40 at a mean 75,000 tokens per task attempted. Fill in both blanks of “is it better?”: compute each system’s tokens per resolved task, the extra tokens the team spends per point of resolution rate and per additional resolved task, and write the result as a defensible claim in the canonical triple shape — better at what, at what cost, and where the evidence stops. (d) Explain why (vi) is the category mistake Section 21.5 names, state what replay remains the right court for, and name the court in which the reviewer-prompt claim must be tried.

Exercise 2. The team’s judge model grades each resolution’s write-up pass or fail against the rubric, and calibration follows Section 21.3: a stratified reference sample of fifty human-labelled outputs — thirty-five bug-fix, fifteen refactor — on which the judge agrees with the humans on all thirty-five bug-fixes and on eleven of the fifteen refactors. (a) Compute the global agreement rate and its 95% Wilson confidence interval, then the per-stratum rates; the agreement helper in systems/evaluation/judge.py returns exactly one global figure — say what the stratified table shows that the global figure hides, and compute the agreement to expect if next quarter’s mix moves to 30% bug-fix and 70% refactor with the per-stratum rates unchanged. (b) The judge’s underlying model upgrades on the vendor’s schedule — a recalibration trigger of Section 21.3 — and the fresh fifty-label calibration agrees on only 39. Test whether the drop is distinguishable from sampling noise at the 5% level (Fisher’s exact test), and say what the verdict implies about the size of the calibration sample relative to the drops it exists to catch. (c) Compute how many labels a calibration round needs to detect a drop in agreement from 0.92 to 0.82 with 80% power at two-sided 5%, and reconcile the answer with the chapter’s line that human labels are the judge’s reserve currency. (d) Model the judge as an instrument that flips a true pass-or-fail verdict with probability e, symmetrically and independently: derive the observed pass rate of a system whose true pass rate is p, show that the observed gap between two systems is (1 - 2e) times the true gap, and compute what a stable true eight-point gap between the team and the single agent reads as under the old judge (e = 0.08) and the upgraded one (e = 0.22). Say why judge drift can masquerade as system convergence, and why none of this is Chapter 10’s aggregation problem — what is corrupted here is not the vote but the instrument.

Exercise 3. The team (A) and the strong single agent (B) each run once on the same eight tasks, and the judge returns, task for task, q^{A} = 0.36, 0.45, 0.61, 0.65, 0.75, 0.83, 0.87, 0.98 and q^{B} = 0.31, 0.42, 0.55, 0.63, 0.70, 0.79, 0.86, 0.92. (a) Analyse the runs unpaired: compute the two means, the two standard deviations, the two-sample standard error \sqrt{s_A^{2}/K + s_B^{2}/K}, and the resulting t — what is the verdict? (b) Analyse them paired, per the display in Section 21.5: the d_k, \bar d, s_d, the standard error, and the paired t — what is the verdict now? (c) Reconcile the two verdicts with the decomposition q_k^{S} = \mu_k + \beta^{S} + \varepsilon_k^{S}, where \mu_k is task k’s difficulty, \beta^{S} the systematic effect of system S, and \varepsilon_k^{S} run-to-run noise: which term does pairing cancel, and how many tasks would the unpaired design need to match the paired standard error (compute K (s_A^{2} + s_B^{2}) / s_d^{2})? (d) Pairing is not magic: suppose the run-to-run noise within a task has standard deviation 0.15 for each system while task difficulty contributes only 0.05; compute the standard deviation of d_k with one run per system per task, compare it with the unpaired difference’s, then recompute with the mean of m = 4 runs per system per task paired instead — and state the design rule this teaches about where pairing earns its keep.

Exercise 4. Two pieces of statistical honesty, priced. The team wants to A/B the debate gate: configuration A keeps it, configuration B drops it, a pilot puts the standard deviation of the paired per-task difference at s_d = 0.16, and one paired evaluation — one task through both configurations — costs 520,000 tokens. (a) Using the normal approximation, compute the number of paired tasks needed to detect a true difference of 0.05 with 80% power at two-sided 5%, and its token price. (b) The evaluation budget is 25 million tokens: compute the affordable K and the minimal detectable effect at that K, and write the sentence an honest report prints if the comparison comes back non-significant — the chapter’s “we could not afford to know”, made quantitative. (c) Selection inflates: eight configurations are tried, all in truth identical with per-task pass probability 0.7, each measured independently on 25 tasks, and any configuration whose pass count clears the one-sided 5% exact binomial threshold is declared an improvement. Find the threshold count and its exact level, the probability that at least one of the eight clears it, and then the Bonferroni-corrected threshold and the family-wise rate it buys. (d) Compute the expected reported pass rate of the best of the eight — the exact distribution of the maximum is within reach of the standard library — the probability that the best clears 0.80, and the expected sag when the winner is re-run on fresh tasks; state the remedy, which the chapter borrows from Section 12.2.

Exercise 5. The companion repository’s foundations/algorithms/evaluation.py computes the paired t and stops — and at small K the t reference distribution leans on a normality assumption the d_k have not earned. Extend it. (a) Implement sign_flip_p(q_a, q_b), the exact two-sided sign-flip test: under the null hypothesis that neither system is systematically better, each d_k is as likely to have been -d_k, so the 2^{K} sign assignments are equally probable and the p-value is the fraction whose mean is at least as extreme as the observed \bar d; mirror paired_t‘s zero-variance nil convention. (b) Run it on the toy scores of Section 21.5’s listing and report the exact p; compare it with the roughly 0.07 the t reference assigns to t = 2.3 on five degrees of freedom, and state which assumption the exact test replaces normality with. (c) Quantify the price of exactness at small K: the smallest attainable two-sided p is 2/2^{K} — compute it at K = 6, and find the smallest K at which a p below 0.01 is attainable at all. (d) Wire the new test into compare in systems/evaluation/harness.py and state exactly what changes in the report printed by systems/evaluation/run.py — and why the module’s scripted mode, in which the two systems’ outputs are textually identical, must come back with p = 1 rather than an error.

Exercise 6 (lab). Run Section 21.1’s triple experiment on the companion repository’s own harness. Extend _TASKS in systems/evaluation/run.py to at least ten self-contained tasks of the same shape — the defective function and its required property on the page — so that every task has executable ground truth; note that single_agent_runner in systems/evaluation/harness.py is, as shipped, an opponent built to lose (one pass, no iteration, no tests), and strengthen it per Section 21.1 — iteration, self-critique, retries, all within its one context, never a spawned subagent, which would change the arm — until it may spend what the team spends; replace workflow_runner’s canned patch with a genuine fixed pipeline (one model call at a fixed point, one mechanical retry on a failing check). Run team, strengthened single agent, and workflow at least five times each on the full task set with a live model; grade outcomes twice, once by executable check and once with the LLMJudge of systems/evaluation/judge.py, and tabulate where the two graders disagree. Report all four families of Table 21.1: outcomes as per-task pass distributions rather than means alone; process read from the traces (review cycles, messages per resolution); cost as tokens per resolved task and the marginal tokens per quality point of team over single agent; robustness as the spread across the five repeats and the worst run. Analyse team versus single agent with paired_t and Exercise 5’s sign_flip_p on per-task mean scores, and deliver the verdict in the canonical triple shape — better at what, at what cost, and where the evidence stops. Record the model identifier and the date beside every table: the rates are perishable; the durable finding is the shape of the comparison, and what the honest baseline did to the headline.

Further exercises for this chapter continue in the web edition’s exercise bank.

“I find myself persuasive.”

  1. For two raters over nominal categories, \kappa = (p_o - p_c)/(1 - p_c): the observed agreement rate p_o, less the agreement p_c that chance alone would produce given each rater’s base rates, rescaled so that 1 is perfect agreement and 0 no better than chance — the judge that always says “pass” scores exactly 0. Where raters multiply, categories acquire order, or judgements go missing, the standard generalisation is Krippendorff’s \alpha, which Section 21.4 brings to the label campaign.↩︎