27 Collective Intelligence and Open Problems
The book has been building, by degrees, a small society: agents that reason and act, speak and listen, plan together, coordinate without colliding, vote, bargain, trade, organise, learn, survive the night, withstand attack, and answer for themselves afterwards. Twenty-six chapters of machinery, and one question deferred at every step of the assembly, always with a promissory note, always payable later. It is later. The question is whether the assembled whole is ever more intelligent than its parts — not more productive, which is bookkeeping, nor more impressive, which is marketing, but better at the thing intelligence is for: reaching truer conclusions, better decisions, and sounder designs than any member could have reached alone. The preface called a committee of language models still a committee, and demanded minutes; the final chapter must now say when a committee is worth convening at all.
Two answers are on offer in the surrounding culture, and both are wrong in instructive ways. The enthusiast’s answer is that intelligence accumulates like headcount — that ten agents are ten times the mind, and a swarm is a superintelligence with better branding. The book has spent too long among the failure taxonomies to believe it, and the honest ledger reads otherwise: multiplied tokens for the same result, teams that fail at the joins, and the standing discovery that a great many multi-agent systems would be faster, cheaper, and sturdier as one agent with a good tool. The sceptic’s answer — that the whole is therefore always less than its parts and the entire enterprise decorative — fails against evidence the book has also weighed: crowds that outguess their experts, juries provably wiser than their jurors, parallel teams that beat strong single agents on the tasks that genuinely divide. The truth, as usual, has conditions attached. Collective intelligence is real, and it is neither free nor automatic: it holds when diversity, independence, and sound aggregation hold, and it fails — quietly, expensively, and at scale — when they are absent. The conditions are checkable, and most of this book has been, without always saying so, a manual for engineering them.
Behind the engineering question stands an older and grander one, and the final chapter is the place to look at it directly. The field’s founding intuition — older than its algorithms, older than the frameworks by decades — is that intelligence was never a solo phenomenon to begin with: that a mind is itself a society of lesser agents, that cognition in the wild is distributed across people and instruments, and that the human intelligence now building these systems is the product of a collective process no individual brain could have run. If that intuition is right, then the question of the present era — where does greater capability come from, one ever-larger mind or a better-organised many? — is not a choice between rivals but a question about composition, and composition is precisely what this book has a theory of. Whether multi-agent systems are a road to more general intelligence, or merely a way of spending more on the intelligence we have, is the sharpest open question the field owns, and it deserves better than confident answers in either direction.
And open questions, at the end of a contemporary treatment, are not an embarrassment to be hidden but the inheritance to be handed on. A field that were finished would not need new readers. The book therefore closes the way a well-run team closes a sprint: by writing down, honestly, what was attempted and not achieved — the problems across theory, learning, engineering, economics, safety, and governance that these pages have sharpened without solving — and delegating them, with a scoped brief and the accounting kept, to the only agent the book has left: the reader. It is, the authors feel, the properly multi-agent way to end.
We proceed by putting the central claim under discipline: what collective intelligence is, and the conditions under which a group of agents actually exhibits it (Section 27.1). We then ask how the property scales — what happens to capability, and to risk, as population and connectivity grow, and how to measure both rather than assume either (Section 27.2). We widen the lens to the oldest version of the question: whether intelligence itself is collective, and what that implies about where more of it will come from (Section 27.3). We open the ledger of unfinished business (Section 27.4). And we end where the preface began — with a thesis that can now be given its verdict (Section 27.5).
27.1 Is the Whole Smarter? The Conditions of Collective Intelligence
A term this attractive needs a definition before the marketing department gets to it. Say that a group exhibits collective intelligence when its performance on cognitive tasks reliably exceeds what its best member could have achieved alone — reliably, because any committee can beat its best member on a lucky Tuesday, and best member, because beating the average is too cheap a bar to clear: a group that merely dilutes its strongest mind with company has arranged an expensive way to be worse. The definition is deliberately stricter than “the team shipped a lot”, which is productivity, and stricter than “the transcript was impressive”, which is theatre. Chapter 13 proved the narrow case in full: on a binary question with a fact of the matter, a majority of competent jurors whose errors are independent beats any one juror, and the whole result balances on the independence clause. But a working team is more than a jury convened at the end. It decomposes, allocates, works in parallel, checks, argues, and only then aggregates — the full pipeline this book has spent its length assembling — and collective intelligence, if the term is to earn its keep, is a property of that entire loop. The question of this section is what the jury theorem’s fine print looks like when it is scaled up from the verdict to the whole enterprise.
The first thing to say is that the property exists and can be measured, and the measurement vindicates a sentence this book committed to on page one. Woolley and colleagues put small human groups through batteries of varied tasks — brainstorming, moral judgement, negotiation, puzzles — and found what psychometrics finds in individuals: a single statistical factor, a group’s general ability, that predicts its performance across tasks (2010). The finding with teeth is what that factor did not track. It was only weakly predicted by the members’ average intelligence, and only weakly by the smartest member’s — the two quantities a recruiter would buy; it was predicted instead by properties of the interaction: the members’ social sensitivity, and how evenly the conversational turns were distributed, with groups dominated by one voice measurably stupider than groups that took turns. The preface asserted that intelligence is a property of interactions and not only of individuals; here is that assertion as a coefficient. And for the builder of agent teams the implication is direct, because every quantity in the finding is something this book has treated as a design surface: who speaks, when, in what order, with what right of reply is not conversational etiquette but Chapter 21’s control lens — and it appears to be where a good deal of a group’s intelligence actually lives.
Diversity’s claim on the property deserves equally careful handling, because it comes with a famous theorem and the theorem comes with a health warning. Hong and Page modelled problem solvers as bundles of heuristics searching a rugged landscape, and showed conditions under which a diverse collection of solvers outperforms a collection of the individually best (2004). The mechanism deserves to be understood, because it is the interesting part: the best solvers, selected on a common yardstick, tend to be similar, so they stall at the same local optima — a team of them is one search, repeated; a diverse group stalls in different places, and each member’s dead end is somewhere another member can restart from. The result holds under real conditions — a hard problem, a large and varied pool, solvers able to build on one another’s positions — and the slogan it spawned, that diversity trumps ability, promptly outran the mathematics and has been under dispute ever since. The durable residue: diversity is not a substitute for competence but a second, independent axis of group design — the ensemble literature reached the identical verdict from the opposite direction, accurate and diverse, wrong in different places — and its value is precisely as large as the difference between the members’ blind spots.
Assemble the pieces and the conditions for collective intelligence can be written out like a pre-flight check, each item with a chapter behind it. Competence: every contributor better than chance at its station, a measurable, per-domain quantity — Chapter 24 measures it, Chapter 26 calibrates it. Diversity: members wrong in different places, so that one agent’s blind spot falls within another’s field of view. Independence: the errors kept uncorrelated at the moment of judgement, which is a property of process — of who saw whose answer before committing to their own. Aggregation: a combining rule matched to the species of question, majority and median for facts, fairness-aware rules for preferences, with Chapter 13’s impossibility taxes paid knowingly. And two conditions the jury frame is too small to show. Decomposability: the task must actually divide, or the perspectives complement — Chapter 1’s first question, still the first question. Verification: cheap, trustworthy checking at the joins — Chapter 24’s harness — because diversity without verification merely produces disagreement, and disagreement is raw material, not intelligence. Six conditions; not one of them is headcount.
For teams of language-model agents, the checklist has a distinctive failure mode, and the book has already named it: samples from one model share its training wholesale, and a thousand such jurors are one juror, photocopied. It is worth being blunt about the popular remedies. Turning up the temperature buys surface variety, not independence — the paraphrases differ, the blind spots are identical, and the errors that matter remain correlated at exactly the moments they matter, since the questions a model gets confidently wrong are not the ones its sampler dithers over. Prompting one model to role-play five experts convenes five costumes around one mind. Real decorrelation has to be bought where the correlation lives: different base models, trained by different hands on different data; different evidence — each agent given a different window on the problem, a different tool, a different slice of the corpus, so that even identical minds are reasoning from different premises; different methods, enforced by role and harness rather than requested by adjective. And independence, once bought, must be protected by process: separate contexts, verdicts committed blind, aggregation before deliberation — the warning that letting the jury talk pools its information and correlates its errors in the same breath. None of this is free, which is rather the point: in this substrate, diversity is a procurement decision, and a team’s error correlation is a measurable quantity sitting in the journal — Chapter 24 can estimate it from disagreement rates — rather than a hope.
What, then, does the ledger show, when the conditions are respected? The best-documented result the field has is the one this book has cited from three chapters already, because it has the right shape: a parallel, orchestrator-led team beat a strong single agent decisively on research tasks that divide into independent facets — and cost many times the tokens, and showed no advantage on tasks that fit in one context (Anthropic, 2025). Better at this, at this price, not elsewhere: Chapter 24’s canonical triple, and a clean instance of every condition above — decomposable task, diverse evidence windows, aggregation by synthesis, verification by the orchestrator. Against it stands the equally well-documented failure census: systems that collapse not because members lacked competence but at the joins — specification, coordination, verification — which is to say, at the conditions (Cemri et al., 2025). Read together, the two results say what this section has been building towards. Collective intelligence is not what you get by convening agents; it is what you get by convening them correctly, and “correctly” has technical content — six conditions, each checkable, each buildable, none automatic. The whole can be smarter than its parts. Whether it goes on getting smarter as the parts multiply — whether intelligence scales with population, and what scales with it — is the next question, and the answer will want a graph rather than a slogan.
27.2 More Agents, More Wire: How Capability and Risk Scale
The intuition this section exists to retire is additive: if one agent is useful, ten agents are ten times as useful, and a hundred are a research programme. The refutation was published in 1975, about human software teams, and it has aged into one of engineering’s few reliable laws. Brooks’s The Mythical Man-Month observed that adding people to a late software project makes it later, and located the mechanism in the wiring rather than the workers: n contributors hold n(n-1)/2 potential channels of communication, so the coordination burden grows quadratically while the hands grow linearly, and past a modest size the new hire consumes more of the team’s attention than they contribute (1975). His deeper point named the fallacy in the title: men and months are interchangeable only when a task partitions perfectly among workers who need never speak — which is to say, almost never — and the “man-month” is mythical exactly to the extent that the task is coupled. The reader will recognise Section 27.1’s decomposability condition arriving from a different century. Substituting agents for programmers changes none of the structure and sharpens the accounting, because agents communicate in tokens and tokens are the budget: Chapter 11 already itemised coordination in the two currencies the system actually spends, and its warning — the team that spends its whole allowance staying in step — is Brooks’s law with a meter attached. The man-month was mythical; the agent-token is billable.
So capability does not scale with population; it scales with the task’s parallelism, and population merely sets a ceiling on how much of that parallelism can be harvested. The regimes follow. Where the work divides into independent facets, gains are real and can approach linear up to the number of true subtasks — the research-synthesis win Section 27.1 has just weighed sits here, many hands on many independent documents (Anthropic, 2025). Where the work couples, the curve flattens fast: each additional agent brings less uncovered work and more wire, and the orchestrator’s context — the place the joins are managed — becomes the serial bottleneck that caps every parallel speed-up. And past the flat maximum the returns turn negative, for a reason the failure census makes concrete: the errors that kill multi-agent systems live at the interfaces — specification, hand-off, verification — and interfaces are exactly what a marginal agent adds (Cemri et al., 2025). An agent added to a coupled task contributes its work minus its wiring, and the subtraction changes sign quietly, with no alarm beyond a bill growing faster than a benchmark score. Diminishing returns, it should be said plainly, is the good outcome; the bad one is a team that gets worse while getting bigger and dearer, which the joins data says is not a hypothetical.
Risk, meanwhile, is running up a different axis, and the asymmetry between the two ledgers is the section’s second lesson. Capability scaling needs the task’s cooperation; risk scaling needs only the wiring. Add a channel between two agents and you acquire its dangers unconditionally — whether or not the task ever profits from it — and the book has already audited what those dangers are. The channel is attack surface: Chapter 25 found that plurality multiplies every ingress point, and that capability and exposure are the same surface seen from opposite sides. The channel is a contagion path: the poisoned paragraph, the corrupted belief, and the fashionable error propagate along the very connections installed for coordination, and Chapter 18‘s cascade arithmetic showed how quickly a densely connected population converts two early mistakes into a thousand-agent consensus — a pipeline in which each agent sees its predecessors’ verdicts is a cascade machine, and dense wiring builds such pipelines by accident. And the channel is a correlation engine: agents that share a model, a context, or a retrieval corpus fail together, so the more thoroughly a population is knitted, the less its redundancy is worth — a security finding already delivered: replicas on one substrate are not replicas. Connectivity, in short, taxes the very independence that Section 27.1 identified as collective intelligence’s scarcest input. The wire carries the signal and the disease at the same rate, and only the signal needs an invitation.
Population and connectivity are therefore design variables with different signs and different slopes, and the lever that sets both at once has been in the book’s hands since Chapter 11: topology. A topology, that chapter said, is a placement of coordination cost; the risk ledger now adds that it is equally a placement of exposure. The star bounds contagion — spokes that cannot address one another cannot infect one another — and concentrates both bottleneck and blast radius at the hub. The peer mesh distributes the load and maximises the channels, buying resilience to any one failure at the price of a superb propagation fabric for the correlated kind. The hierarchy bounds the wiring per node and inserts firebreaks at its boundaries, at the usual cost in delay and rigidity. None of this changes Chapter 11’s conclusion; it doubles it: the cheapest dependency is still the one designed out, and the safest channel is the one never installed. The practical rule is an audit discipline the book’s diagrams have been quietly enforcing for some time — every arrow is a token bill (Chapter 11), an attack path (Chapter 25), and a contagion channel (Chapter 18), so every arrow should correspond to a dependency the task actually has. Wiring installed because connection feels like collaboration is paying three taxes to deliver a feeling.
A book that has spent a chapter insisting on evidence over vibes (Chapter 24) owes its own central claims the same treatment, and this section’s claims are eminently testable — so the companion repository ends on an experiment rather than an exhortation. The capstone lab takes one task from the running example — a batch of repository issues of graded difficulty — and runs it across a grid: team sizes of one, two, four, and eight; topologies from the single agent with good tools, through the orchestrator’s star and the sequential chain, to the blackboard and the peer mesh; every configuration under the same token accounting, against the strong single-agent baseline Chapter 24 insists upon, with enough runs for the statistics to mean something. Four curves come out — quality, tokens, latency, and the failure-mode census — and the hypotheses this chapter has argued for are then exposed to refutation: a flat maximum at small team sizes for coupled tasks and later for parallel ones; cost rising faster than quality past it; failures migrating from competence errors to join errors as the population grows. The reader may find the curves land elsewhere — tasks differ, models improve, and the flat maximum moves — and that would not embarrass the lab but vindicate it, because the portable finding is the method: the scaling question is empirical, per task, and cheap to answer with a harness you already own. Scaling by assumption is the one configuration the lab forbids.
What the two ledgers say together is easily compressed: more is not smarter; structured is smarter, and the structure earns more than the size. The era’s reflex — when in doubt, scale it — meets, at the end of this book, the field’s oldest result: that the intelligence of a collective lives in its organisation, its independence, and its joins, all of which degrade with careless growth and none of which headcount supplies. Small, well-wired, well-verified teams keep winning the honest evaluations not because ambition is vulgar but because the conditions of Section 27.1 are easier to hold at four agents than at forty. Which leaves the question that has been waiting behind the engineering all along. If piling on agents is not the road to more intelligence, what is a collective’s contribution to intelligence in the first place — and might the answer explain where intelligence, including the reader’s, came from?
27.3 Societies of Mind: What Collectives Contribute to Intelligence
Chapter 1 opened this book with a wager it borrowed from Minsky: that intelligence is not a possession but an organisation — the society of mind picture, in which a single mind is itself a society of mindless components and the cleverness lives in the arrangement (1986) — and that in the multi-agent setting this would prove not a suggestive metaphor but a literal engineering reality. Twenty-six chapters later the wager can be assessed, and the preceding two sections are the assessment: what made the difference between a collective that thinks and an expensive committee was never the parts — it was the conditions, the topology, the joins, the independence, the aggregation rule; organisation, at every turn. But if the wager has paid out, its logic should be pushed to the end. Minsky’s claim was about the inside of one head; this book’s was about systems of agents; and between and beyond the two lies a larger version of the same inversion, worked out by people who studied minds and cultures rather than machines. If intelligence is organisation, then there is no privileged scale at which it must live — and the evidence is that it never did live at the scale we flatter ourselves it does.
| Scale | Framework | Where the intelligence lives | Lesson for agent teams |
|---|---|---|---|
| One mind | Society of mind (Minsky, 1986) | In the arrangement of mindless components | Agent teams make the picture a literal engineering reality |
| One crew (a warship’s bridge) | Distributed cognition (Hutchins, 1995) | In the system — people, instruments, and procedure together | The journal, schema, protocol, and verifier are constituents of the system’s cognition, not accessories to it |
| One species | Cumulative culture / the collective brain (Henrich, 2016) | In innovations retained, transmitted, and recombined across generations | Agents still have no cross-run cumulative culture; building its full loop — the test and the ledger included — is open |
The first piece of evidence is a warship entering San Diego harbour. Hutchins, an anthropologist embedded on the navigation bridge, asked the obvious question — who is doing the navigating? — and found that no answer at the level of an individual survives contact with the data (1995). The fix that lands on the chart is computed by a system: bearing-takers on the wings, a plotter at the table, alidades, gyrocompass, the chart itself with three centuries of method compressed into its projection, and a choreography of who reports what to whom — with the crucial observation that the system’s cognitive properties differ from any member’s. No officer holds the fix in their head; several hold pieces; the instruments hold more; the procedure holds the rest. Hutchins called the framework distributed cognition, and its unit-of-analysis lesson transfers to this book’s systems without modification: asking which agent in a well-built team is the intelligent one is the same category error as asking which sailor is the navigation. The reader has, in fact, already built Hutchins’s bridge — the journal that remembers, the schema that disciplines, the protocol that choreographs, the verifier that checks are not accessories to the agents’ intelligence but constituents of the system’s — and the book’s engineering chapters can be reread, in this light, as the instrumentation of a cognitive system rather than the housing of several.
The second piece of evidence is the reader. Henrich assembled the case that human intelligence itself is the product of a collective process rather than an individual endowment (2016): stripped of their culture, individual humans are unimpressive apes — the history of exploration is punctuated by well-equipped European expeditions starving in landscapes where local populations had flourished for millennia, because the local food required preparation knowledge that no individual, however clever, could re-derive on the spot. What fills the gap is cumulative culture: innovations retained, transmitted, recombined, and improved across generations, so that each mind starts where the last generation finished rather than where the species did. Henrich’s name for the resulting engine is the collective brain, and his scaling claim should make the reader of this chapter sit up: its power grows with the size of the group and the interconnectedness of its members — larger, better-connected populations hold more innovations, lose fewer to the deaths of their carriers, and recombine them faster. Our intelligence, on this account, is not what built the collective; it is what the collective built. The existence proof for composition creating intelligence is the species reading this sentence.
But Henrich’s scaling claim and Section 27.2’s appear to point in opposite directions — connectivity as the collective brain’s fuel there, as a triple-taxed liability here — and resolving the tension yields this section’s most practical insight. Cultural transmission is not a mesh forwarding everything; it is ferociously selective. Learners copy the successful rather than the adjacent; teachers filter; practices that fail die with their practitioners or, better, before them; and institutions — apprenticeships, guilds, journals in both senses — retain what survived the filtering. Culture, in this book’s vocabulary, is connectivity plus verification plus memory: the wire, the test, and the ledger, and stripped of either of the latter two it degenerates into exactly the cascade machinery Chapter 18 warned of — fashion is what cumulative culture looks like without the test. This names, more sharply than the engineering chapters could, what today’s agent systems are still missing. Within a run, the book has built the organs: the journal remembers, the verifier selects. Across runs, across teams, across organisations, there is as yet no cumulative culture of agents worth the name — each deployment re-derives its working practices, and what one team’s expensive failure taught is retained, if at all, in its humans. The asset ledger of Chapter 19 and the incident suite of Chapter 23 are first artefacts of such a culture; the open engineering question — it will reappear in the next section — is whether agent collectives can be given the full loop, selective transmission included, without also being given the fashions.
All of which allows the era’s loudest question to be put with some discipline. Does the road to more general intelligence run through scaling one mind or composing many? The question is durable even though every current answer is perishable, and three things can be said at textbook temperature. First, the alternatives are complements, not rivals: every composed system improves when its members do, and every large mind is deployed, in practice, into a system of tools, checks, and colleagues — the pure cases exist only in arguments. Second, the one instance we possess of general intelligence actually arising — ours — arose by the collective route, with individual brains as the product of the process rather than its designer; an existence proof is not a roadmap, but it is a standing rebuke to the assumption that composition is the derivative programme. Third, composition has properties that scale, by itself, does not supply and that this book’s later chapters have argued are non-negotiable in deployment: division of cognitive labour, independent verification, auditability of the joins, and institutions that keep capability answerable — Chapter 17’s rubric, compose the skeleton and learn the joints, restated at civilisational scale. What multi-agent systems contribute to the pursuit of general intelligence is therefore not a shortcut and not a rival bet; it is the science of organisation that any outcome of that pursuit will need on the day it is deployed — because the destination, whatever it turns out to be, will not be a mind in a vat. It will be a participant in a collective that includes us, and the collective is where intelligence has always lived. What we do not yet know about building such collectives well is, fittingly, the longest list in the book — and it is next.
27.4 The Open Problems
A place on this list has to be earned twice over: each problem below is open — unsolved by the classical tradition, the modern practice, and every chapter of this book — and each is sharpened, meaning the book has given it a precise enough form to be attacked rather than merely admired. The provenance is the text itself: these are the places where the preceding chapters had to write “at the time of writing”, “remains the frontier”, or “nobody yet knows”, and the reader who felt the hedges accumulating may treat this section as their consolidated statement of account. Two warnings before the ledger opens. Lists of open problems date in both directions — some entries will be solved embarrassingly soon after the ink dries, others will dissolve under scrutiny into questions that were never well posed — and both outcomes retire an entry honourably; what should outlast the list is the habit of stating problems sharply enough to suffer either fate. And the six headings that follow, inherited from the preface’s promise, are a filing system rather than a taxonomy: the best problems, as will become obvious, live at the seams between them.
Theory. The book’s formal machinery runs on idealisations that the substrate violates daily, and the violations are now the interesting objects. First: what is the right solution concept for agents that only approximately optimise — that err, satisfice, and can be moved by the phrasing of an offer? Chapter 12 flagged the gap between an equilibrium’s existence and its reachability; the deeper gap is between the player game theory models and the player we deploy, and a strategic theory of approximately rational, language-conditioned agents — with equilibria that are robust to paraphrase — does not yet exist. Second: a predictive, compositional theory of team capability. Chapter 24 and Section 27.2 can measure when n agents beat one; nothing yet lets an engineer derive it — given a task’s dependency structure, the optimal team size and topology, computed rather than discovered at four times the token bill. The field wants its own Amdahl’s law, and Chapter 1’s opening question — should this be a multi-agent system at all? — deserves, eventually, an answer that is a calculation. Third: the theory of juries as they actually convene. Chapter 13’s guarantees assume independence, competence, and sincerity; its own later pages broke each assumption in turn, and the mathematics of the real case — jurors partially correlated, miscalibrated, and trained to persuade — is patchy where it exists at all.
Learning. Chapter 17’s four ideas each leave a successor problem, and the substrate has added one of its own. Non-stationarity was defined as agents adapting to one another; the industrial version is harsher — the cognitive substrate itself is periodically replaced by its provider, so every coordination habit a team has learned is an asset of uncertain survivorship, and transfer of collective competence across substrate change has no theory and barely a benchmark. Credit assignment grows a new face when the actions are paragraphs: which utterance, in a forty-turn collaboration, won or lost the task is a question that training needs answered forwards and Chapter 26’s accountability needs answered backwards, and current methods do neither well at transcript scale. Emergent communication poses its bargain at production scale: teams permitted to compress their own codes will likely beat teams confined to ours, and every increment of that efficiency is paid in legibility — the efficiency–auditability frontier is real, and nobody has mapped it. And Section 27.3’s cheque falls due here: the cumulative culture problem — retention, selective transmission, and recombination of what works, across runs, teams, and organisations — is learning at the level of the collective, and building its full loop, the test and the ledger included, without also building the fashions, may be the most consequential open problem on this page.
Engineering. Three problems, each the residue of a chapter that did everything it could with what exists. End-to-end attenuation: Chapter 22 named it the thinnest treaty in the consolidation — authority that provably narrows across autonomous hops spanning different owners — and it remains a protocol problem, an implementation problem, and a deployment problem all at once. Verification of open-ended outputs: the book’s whole dependability story leans on cheap, trustworthy checking, and for tasks with a test suite the lean is sound; for prose, designs, plans, and judgement calls the ground-truth ladder of Chapter 24 runs out at a judge model holding an office Chapter 13 proved troublesome — a science of verification for unverifiable-looking work, with known error bars, would strengthen nearly every pattern in Chapter 21 simultaneously. And the regression science of substitution: when a member’s brain is swapped overnight, the prompts, the learned coordination, and Chapter 26’s trust curves are all invalidated together, and the discipline for re-certifying a team after a substrate change is, at the time of writing, folklore plus vigilance.
Economics. Mechanism design earned its guarantees against solitary deviation, and Chapter 15 was candid about the cartel-shaped hole: VCG is hospitable to collusion, false-name bids defeat per-bidder guarantees, and both problems sharpen viciously when the bidders share a language to scheme in and possibly a substrate to think with — the cartel may be implicit, learned, or one mind in several seats, and collusion-resistant mechanism design for natural-language agents is wide open at every level from detection to prevention. Behind it stands a market-structure question the book could only gesture at: when the marginal cost of a unit of cognition approaches its token bill, what happens to the pricing of thought — who captures the surplus from delegated work, and what does an economy of agent labour equilibrate to? And between strangers’ agents there is still no commercial layer worthy of the name: Chapter 22 built the conversation, but binding commitment, escrow, liability, and reputation that travels across platforms — Chapter 16’s institutions, rebuilt for an open web of transacting agents — remain unbuilt, and until they exist every cross-boundary delegation is a handshake without a court.
Safety. The composition problem is Chapter 25’s residue stated as a research programme: every agent passes its evaluations and the ensemble still colludes, cascades, or drifts, because alignment is currently certified per agent and needed per system — under what conditions does safety compose, what must be re-proved at each level of aggregation, and what is the safety case for a collective? Scalable oversight keeps its standing as the field’s hardest supervision problem: Chapter 14’s debate proposal remains the sharpest known attack, Chapter 26’s calibration instruments the gap without closing it, and the asymmetry — work produced at machine speed and volume, reviewed by human-speed judgement — grows with every capability increment. And substrate monoculture wants its systemic theory: an economy of agents running on a handful of model families is a correlated-failure machine of exactly the kind financial regulators learned to fear in banking networks, and the contagion mathematics of cognitive infrastructure — stress tests, exposure limits, diversity requirements — has yet to be written.
Governance. Chapter 26 built answerability inside one organisation, and the open problems begin at its property line. The chain of authority that crosses companies — your agent, calling my tool, running on their model — currently has no owner: provenance federation, liability allocation, and the three questions asked across organisational boundaries are unsolved in law and unimplemented in protocol. Institutions at machine speed: norms, sanctions, adjudication, and appeal all assume human-tempo enforcement, and violations now complete in milliseconds — governance that runs at the violator’s clock without itself becoming ungoverned automation is a genuinely new institutional design problem, not a faster copy of an old one. And the oversight regress asks where supervision grounds: the auditors are acquiring agents too, judge models are reviewed by judge models, and Chapter 26’s answer — the chain must terminate at a named human — collides with the volume problem that made delegation attractive in the first place; reconciling the two is governance’s version of scalable oversight, and it is nobody’s solved problem.
| ID | Open problem | What remains open | Rooted in |
|---|---|---|---|
| T1 | A solution concept for approximately rational agents | A strategic theory of approximately rational, language-conditioned agents, with equilibria robust to paraphrase | Chapter 12 |
| T2 | A compositional theory of team capability | Deriving optimal team size and topology from a task’s dependency structure — the field’s own Amdahl’s law | Chapter 24, Section 27.2, Chapter 1 |
| T3 | Juries as they actually convene | The mathematics of jurors partially correlated, miscalibrated, and trained to persuade | Chapter 13 |
| ID | Open problem | What remains open | Rooted in |
|---|---|---|---|
| L1 | Transfer of collective competence across substrate change | Preserving learned coordination when the provider replaces the underlying model | Chapter 17 |
| L2 | Credit assignment at transcript scale | Which utterance won or lost the task — forwards for training, backwards for accountability | Chapter 17, Chapter 26 |
| L3 | The efficiency–auditability frontier | Mapping the trade between agents’ compressed private codes and human legibility | Chapter 17 |
| L4 | The cumulative-culture problem | Retention, selective transmission, and recombination of what works — across runs, teams, and organisations, without the fashions | Section 27.3, Chapter 18 |
| ID | Open problem | What remains open | Rooted in |
|---|---|---|---|
| E1 | End-to-end attenuation | Authority that provably narrows across autonomous hops spanning different owners | Chapter 22 |
| E2 | Verification of open-ended outputs | A science of verification, with known error bars, for prose, designs, plans, and judgement calls | Chapter 24, Chapter 21, Chapter 13 |
| E3 | The regression science of substitution | A discipline for re-certifying a team after its substrate is swapped overnight | Chapter 26 |
| ID | Open problem | What remains open | Rooted in |
|---|---|---|---|
| Ec1 | Collusion-resistant mechanism design for natural-language agents | Detection and prevention when bidders share a language, and possibly a substrate, to scheme with | Chapter 15 |
| Ec2 | The pricing of thought | Who captures the surplus from delegated work, and what an economy of agent labour equilibrates to | none — only gestured at |
| Ec3 | A commercial layer between strangers’ agents | Binding commitment, escrow, liability, and reputation that travels across platforms | Chapter 22, Chapter 16 |
| ID | Open problem | What remains open | Rooted in |
|---|---|---|---|
| S1 | The composition problem | When safety composes, what each aggregation level must re-prove, and the safety case for a collective | Chapter 25 |
| S2 | Scalable oversight | Reviewing machine-speed, machine-volume work with human-speed judgement | Chapter 14, Chapter 26 |
| S3 | Substrate monoculture | The systemic contagion mathematics of a few model families — stress tests, exposure limits, diversity requirements | none — only gestured at |
| ID | Open problem | What remains open | Rooted in |
|---|---|---|---|
| G1 | The cross-company chain of authority | Provenance federation, liability allocation, and the three questions asked across organisational boundaries | Chapter 26 |
| G2 | Institutions at machine speed | Norms, sanctions, adjudication, and appeal at the violator’s clock, without becoming ungoverned automation | Chapter 26 |
| G3 | The oversight regress | Where supervision grounds when the auditors and judge models are themselves acquiring agents | Chapter 26 |
Nineteen problems, six headings, and a fair question from the reader: which one? The engineer’s criterion is the one this book has trusted throughout — pick the problem whose absence you have personally paid for, because the pain is a proof of consequence and the incident report is a head start on the related work. The researcher’s criterion is visible in the list’s own structure: the strongest entries live where two chapters collide — economics with natural language, safety with composition, governance with speed, learning with culture — and the book’s parting methodological advice is to hunt at exactly those seams, since a problem with two parents inherits two literatures’ tools. The final exercise of the book, accordingly, asks not for an implementation but for a research proposal. None of this list will survive the decade intact, and that is the correct fate for such a list; what should survive is what it demonstrates — that the field is alive, that its difficulties are specifiable, and that the distance between a working system and an open problem is, in multi-agent systems, roughly one deployment. What remains is to say what the book believes it has shown.
27.5 Coda: No Longer Premature
The preface stated a thesis plainly, so that the reader could decide early whether to argue with it: the classical theory of multi-agent systems was not wrong, nor obsolete, nor superseded — it was premature. A verdict is now owed, and the evidence has been the book’s own method. Every time the modern practice ran into a wall, a result was found waiting on the other side of it, usually with the dust of decades on the cover: the supervisor pattern stood revealed as the Contract Net; the tool protocols re-derived the speech acts; the judge model walked into an office Arrow had described to the syllable; the ensemble convened Condorcet’s jury and inherited its fine print; the injected agent re-enacted the confused deputy; and the deployed fleet demanded, on schedule, the norms, institutions, and answerability that the theory of agent societies had drafted for occupants it never met. Not every wall, and not always the whole way through — the open problems of the last section are the honest remainder — but often enough, and centrally enough, for a verdict whose scope can be stated exactly: where the book could put a correspondence to work — run the lab, price the assumptions, watch the theorem’s failure mode arrive on cue — the wager has paid out as a finding; where it could only exhibit the correspondence, it stands as a programme, now posed sharply enough to fail. The theory was a map drawn before the territory could be reached. The territory has arrived, the map has been substantially right wherever the book could survey it, and premature is a word the field may now retire.
The preface’s other image can be retired with it. Two tribes, it said, had sailed past each other in the fog — the theorists with the vocabulary and no substrate, the engineers with the substrate and no vocabulary — and the book was offered as a bridge. But the bridge metaphor undersells what twenty-seven chapters of crossings keep suggesting, which is that there was only ever one landmass: not two subjects, but one field with a forty-year gap between its specification and its hardware. And the traffic runs in both directions. The classical results discipline the modern practice, as advertised — that was the toll the engineering tribe paid at every chapter. What the theory tribe receives in return is the thing it lacked from the beginning: experiments. The canonical text of the classical tradition was written for agents its author could only specify (Wooldridge, 2009); its successors can now run the specifications — game theory with subjects that actually play, social choice with reproducible juries, institutions whose sanctions execute — and every deployed fleet is, seen from the theory side, the largest multi-agent laboratory ever accidentally assembled. A field reunited with its own experiments does not merely apply its old results; it finds out which of them were wrong, which is the part of science the classical era had to postpone. That, and not nostalgia, is why the old literature deserves the reader’s attention: it is finally falsifiable.
Which leaves only the ending, and the book proposes to end in the one style it trusts: as a delegation, executed by its own rules. Chapter 10 set the requirements, and they are met. The brief is specified clearly enough to commit to — nineteen problems, six domains, one field; or, for the reader whose ambitions are a system rather than a thesis, simply this: build the smallest team that works, measure it against the strongest loner, keep the journal, and gate what cannot be undone. The commitment must be taken on rather than merely received, and that part was never the book’s to perform. And there must be a channel for reporting back — which exists, for this book is written in public and its margins are open, and beyond the margins lie the workshops and journals of a field that has spent forty years being politely early and would be glad of the company. The accounting, meanwhile, follows Chapter 26 to the letter: authority is hereby delegated, and answerability stays where it belongs — whatever you build is yours to answer for, and whatever this book got wrong remains, entirely and cheerfully, ours.
The preface asked for a cup of tea and a terminal. If you have followed the whole way, the tea is long cold and the terminal has history worth keeping — so put the kettle on again, and go and build some societies. Mind the conditions. Measure the whole. And do keep the minutes: as the book has been arguing since page one, they were never really minutes at all — they are where the intelligence lives.
27.6 Summary
- Collective intelligence is conditional, not automatic. A group outperforms its best member when its members are diverse, their errors independent, and the aggregation sound — the jury theorem’s fine print, which is the entire theory. Headcount is not a condition; an ensemble of clones sharing one context is a crowd of one, at chorus prices.
- The conditions are engineering targets. Diversity can be built — different models, prompts, tools, and evidence; independence can be protected — separate contexts, blind judgements, aggregation before discussion; competence and error correlation can be measured from the journal; and every one of these is a design decision this book has already equipped the reader to make.
- The whole is smarter only where the task divides and the errors decorrelate. The honest evidence shows real wins on genuinely parallelisable work, at a multiple of the cost, and no gain where one context suffices — better at this, at this price, not elsewhere; teams still fail overwhelmingly at the joins.
- Capability scales sublinearly; risk can scale faster. Coordination overhead grows with the wiring rather than the work — Brooks’s law survives the substitution of agents for programmers intact — while connectivity breeds correlated failure, cascades, and behaviour nobody chose. Population and connectivity are design variables with different signs, and topology is the lever that trades them.
- Measure the scaling; never assume it. The capstone experiment runs one task across team sizes and topologies, counting quality, tokens, latency, and failure modes with Chapter 24’s discipline — the book’s own central claim, put to its own standard of evidence.
- Intelligence was collective all along. A mind as a society of agents, cognition distributed across crews and instruments, human capability itself the dividend of cumulative culture: the existence proof that composition creates intelligence is us. Whether the next increment of general capability comes from scaling one mind or organising many is the field’s sharpest open question — and organising many is what this book has a theory of.
- The remainder is delegated. Open problems across theory, learning, engineering, economics, safety, and governance — each sharpened by a chapter of this book and solved by none of them — are handed to the reader with a scoped brief and the accounting kept. The classical theory was not wrong, and it is no longer premature; the field it was waiting for is the one you are now equipped to build.
27.7 Exercises
Exercise 1. The running team’s review gate currently draws five verdicts by having the one reviewer model post them in sequence in a single transcript, each verdict able to see those before it, and takes the majority. Six configurations are on the operator’s desk: (i) the current gate as described; (ii) one call in which the reviewer is prompted to role-play five named specialists and return five verdicts; (iii) five separate calls to the same model on the same evidence, each verdict committed before any is shared; (iv) five reviewers on five different base models trained by different hands, same evidence, verdicts blind, majority; (v) five calls to the same model, each given a different evidence window — the diff alone, the diff with the tests, with the specification, with the call sites, with the file’s recent history — blind, majority; (vi) the five reviewers of (iv), but sharing one discussion thread and converging on a consensus verdict, which is what gets reported. (a) For each configuration, say which of Section 27.1‘s six conditions it purchases, which it merely imitates, and which it damages, each verdict grounded in the section’s mechanics in a sentence or two. (b) Order the six by the error correlation \hat{\rho} that a disagreement-rate estimate from the journal would be expected to report, ties permitted, justifying every inequality — and say which configuration makes \hat{\rho} unmeasurable and why that is itself the diagnosis. (c) The journal shows the current gate’s majority verdicts scored 0.78 over the last forty decisions, its strongest member alone 0.81 on the same forty, and the members’ average 0.66: deliver the section’s verdict — does the gate exhibit collective intelligence, which comparison is the binding one and why is the other too cheap a bar, and what does a standard error of roughly \sqrt{0.8 \times 0.2 / 40} \approx 0.063 say about the word reliably, and about what Chapter 24’s paired design would have to establish before the panel’s five-fold token bill is justified?
Exercise 2. Hong and Page’s mechanism — the best solvers, selected on a common yardstick, stall at the same local optima (Section 27.1) — can be reproduced at desk scale. A tuning problem is a ring of 200 positions, each carrying a quality drawn uniformly from [0, 100); a solver’s heuristic is an ordered triple of distinct step sizes from \{1, \dots, 12\}, giving a pool of 1{,}320; from any position a solver tries its steps in order, moves to the first strictly better position, and rests where none improves; a solver’s ability on a landscape is its mean rest quality over all 200 starts; a team searches by relay — from the common position, members take turns running their own search to rest, until a full round moves no one. (a) Implement the model with a fixed seed and, over twenty landscapes, report the mean team score of the best team (the ten ablest solvers), a diverse team (ten drawn at random from the pool), and the worst team (the ten least able), alongside each team’s ablest member’s solo ability. (b) Apply the chapter’s definition of collective intelligence: which of the three teams reliably exceeds what its own best member achieves alone, and does the diverse team’s edge over the best team survive the lucky-Tuesday clause — check the direction of the gap across at least five seeds before ruling. (c) Diagnose the mechanism with one statistic — the mean number of step sizes a pair of teammates shares — explain what ranking on a common yardstick did to the best team’s composition, and name the failure of Section 27.1 that is the same event on the language-model substrate. (d) The worst team turns out to be more alike than the best team and weaker besides, and finishes last: state what the three scores jointly establish about the slogan that diversity trumps ability, and square the verdict with the chapter’s durable residue — diversity as a second, independent axis, worth exactly the difference between the members’ blind spots.
Exercise 3. The diversity ablation in the companion repository’s frontier/scaling_lab/jury.py decorrelates a seven-juror panel of competence p = 0.7 rung by rung — from one shared context to fully split evidence — and at each rung measures the panel’s pairwise disagreement and estimates the error correlation back from it: exactly the audit Section 27.1 says the journal makes possible. Where Chapter 13‘s Exercise 9 built the correlated jury — a copy mechanism and the design-effect n_{\mathrm{eff}} — and took the correlation as given, this exercise recovers it from what a panel visibly does. (a) Derive the instrument: under the module’s photocopying mixture — with probability \rho a single shared draw is copied to every juror, otherwise all seven vote independently, each correct with probability p — show that the correlation between two jurors’ correctness indicators is exactly \rho, that the expected pairwise disagreement rate is d = 2p(1-p)(1-\rho), and hence that \hat{\rho} = 1 - d/\bigl(2p(1-p)\bigr) recovers the correlation from an observed disagreement rate; then show that forcing n_{\mathrm{eff}} = n/\bigl(1 + (n-1)\rho\bigr) to equal a target m requires \rho = (n-m)/\bigl(m(n-1)\bigr), and evaluate the design rungs for n = 7 at m = 1, 3, 5, 7. (b) Run the ablation (python -m frontier.scaling_lab.run prints it) and check its columns: reproduce the accuracy column analytically from the jury sum P_m(0.7) at m = 1, 3, 5, 7, and set the \hat{\rho} column against the design values from (a), explaining why the estimates land near but not on them — including why the split-evidence rung reports \hat{\rho} \approx 0.003 rather than zero. (c) Price the correlation: compute the wisdom-of-crowds gain at full independence, the fraction of it the panel keeps at the m = 3 and m = 5 rungs as the instrument reads it, and compare each fraction with 1 - \rho at that rung — which way does the inequality run? Then derive, from (a)’s mixture itself, the panel’s exact accuracy \rho p + (1 - \rho) P_n(p) and hence the exactly retained fraction 1 - \rho, and say what that makes of the instrument’s sub-proportional readings — and of any single formula trusted for a correlated panel’s accuracy. (d) Map the ladder onto the section’s procurement menu — temperature, role-play, separate contexts, split evidence, different base models — saying which purchases genuinely move a panel down a rung and which only redecorate the top one, and why the estimator’s input being a disagreement rate makes a team’s diversity an auditable number rather than a hope.
Exercise 4. Run the capstone lab: python -m frontier.scaling_lab.run sweeps team sizes \{1, 2, 4, 8\} across five topologies in a coupled regime (p = 0.2) and a parallel one (p = 0.9), hermetically — no key, no network — with a scripted stand-in calibrated so the sweep recovers the coordination-tax theory on its own grid at \kappa = 0.02. (a) Report the two flat maxima the run prints, then verify two cells of the parallel table by hand from the factory’s recipe — capability is S(n) times the topology gain, and tokens are \lfloor 200\,(1 + \kappa n(n-1)/2) \times \mathrm{cost} \rfloor: the peer cell at n = 2 (gain 1.03, cost 0.95) and the star cell at n = 8 (gain 0.97, cost 1.05). (b) The comparison the run prints is honest by construction — the theory is evaluated over the same four sizes the sweep can see; show what that honesty cannot rule out by recomputing the parallel regime at \kappa = 0.005: find the grid’s argmax and the true maximiser over n = 1, \dots, 100 with their speedups, say what compare_to_theory("parallel", kappa=0.005) reports, and state the instrument-design rule for any sweep whose maximum lands on the edge of its grid. (c) The failure census prints only sizes 1 and 8; from the factory’s schedule — \lfloor 6/n \rfloor competence errors and n - 1 join errors per run — derive the unprinted rows at sizes 2 and 4 with their bucket detail, locate the team size at which join errors first equal and then overtake competence errors, and say which empirical finding of Section 27.1 the schedule is a deliberate cartoon of. (d) In two sentences: what does a hermetic lab that recovers its own theory establish, and at which seam (frontier/runners.py) does the same instrument stop being scripted and start being evidence?
Exercise 5. In the scripted sweep, topology is a uniform factor that — as frontier/scaling_lab/sweep.py’s own comments admit — cannot move a column’s argmax, so every wiring peaks at the same team size, and the claim Section 27.2 most wants tested, that a topology is a placement of coordination cost, is designed out. Put it back: keep p = 0.9, set \kappa = 0.03, price the wiring per size as c_{\mathrm{star}}(n) = 2\kappa(n-1) (a report up and a digest down per worker), c_{\mathrm{chain}}(n) = \kappa(n-1) (one hand-off per link), c_{\mathrm{blackboard}}(n) = \kappa n (one post per member), and c_{\mathrm{peer}}(n) = \kappa n(n-1)/2 (every pair talks), with c(1) = 0 in every wiring, and evaluate S(n) = 1/\bigl((1-p) + p/n + c(n)\bigr). The single-wiring maximisers are already in hand: Chapter 1’s Exercise 4 found the fully connected mesh’s cube-root optimum n^{*} \approx (p/\kappa)^{1/3} and its Exercise 5 the star’s square-root one, so reuse those closed forms and spend this exercise on the placement. (a) Compute the four speedup columns on the lab’s grid, each wiring’s grid flat maximum, and its true maximiser over n = 1, \dots, 100, reporting the exact ties your search must break (the chain and the blackboard each tie two adjacent sizes) and what the lab’s _flat_argmax convention — the smallest size achieving the maximum — says about them. (b) Exhibit the two facts the uniform stand-in could not show: the wirings now peak at different sizes — compare the peer mesh’s true peak with the chain’s — and the mesh alone crosses below the single-agent baseline on the grid; find the crossing size and verify it against the inequality \kappa n(n-1)/2 > p(1 - 1/n), which is Figure 27.2’s caption made computable. (c) Show that the star and the mesh levy identical taxes exactly at n = 4 — solve 2\kappa(n-1) = \kappa n(n-1)/2 — so the running team of four pays the same token bill under either wiring; Exercise 6 reads the rest of the two invoices. (d) In a sentence or two, defend the repository’s choice to ship the argmax-invariant stand-in in a hermetic lab, and state what your per-size taxes now expose to refutation by a live sweep that the stand-in never risked.
Exercise 6. Every arrow in a topology diagram is a token bill, an attack path, and a contagion channel (Section 27.2); price the last two for the running team of four — orchestrator, coder, reviewer, tester — under the star (every channel meets the orchestrator), the chain (orchestrator–coder–reviewer–tester), and the peer mesh. A poisoned artefact enters one member, chosen uniformly at random, and spreads along shortest paths, surviving each hop independently with probability \beta — every relay is one more chance for the checks to catch it — so a colleague at distance d is corrupted with probability \beta^{d}. (a) Count each wiring’s channels; then, in exact fractions at \beta = 1/2, compute the expected number of corrupted colleagues for every origin and the uniform average per wiring — and set the result beside Exercise 5(c): at n = 4 the star and the mesh charge the same token tax, while the mesh carries twice the ingress channels and four-thirds the average contagion. (b) Recompute the three averages at \beta = 9/10 and rank the wirings; the chain now beats the star on the average while the star’s hub remains the worst single origin on the board — reconcile both facts with the section’s claim that the star bounds contagion, saying precisely what is bounded and what is merely relocated. (c) Grant the team one perfect verifier — a member that can be neither corrupted nor used as a relay — and place it optimally under each wiring: show that the expected corruption over the remaining origins falls to 0 in the star (verifier at the hub), 1/3 in the chain (verifier at an interior post), and 1 in the mesh, and state what the star’s concentration buys that the mesh’s egalitarian wiring cannot at any placement. (d) One sentence each: why the shortest-path model flatters the mesh, and which correlated-failure channel appears in no wiring diagram and survives every rewiring — the one whose coefficient Exercise 3 taught you to estimate.
Exercise 7. Section 27.3 compresses cumulative culture into an engineering formula — connectivity plus verification plus memory: the wire, the test, and the ledger — and warns that fashion is what the loop produces without the test. Build the smallest loop that can ratchet. The team’s asset ledger holds a working practice of quality z, initially 0; each run proposes one variant whose quality change \delta is a draw from the standard normal distribution; a verification gate pronounces the variant better or worse, wrongly with probability e; the ledger adopts exactly what the gate approves; you may use \mathbb{E}[\max(\delta, 0)] = 1/\sqrt{2\pi} \approx 0.399. (a) Show that the expected quality gained per run is (1 - 2e)/\sqrt{2\pi}, and evaluate it at e = 0, e = 0.2, and e = 1/2, reading the last value as the section’s fashion line — retention without selection accumulates nothing, however perfect the ledger. (b) Confirm all three by a seeded simulation: 400 runs, mean final z over 200 seeds, set against theory. (c) Add mortality: before each run, with probability \lambda, the ledger is wiped — the provider swapped the substrate, and the practices did not survive. Argue from the age of the most recent wipe that the long-run expected quality is (1 - 2e)/(\lambda \sqrt{2\pi}), and confirm by simulation at e = 0.1, \lambda = 1/50. (d) Read the result as the section does: identify the wire, the test, and the ledger in the formula; show that K independently tested proposals per run multiply the drift by K — Henrich’s larger-and-better-connected-populations engine — but only through the factor (1 - 2e), so that connectivity without verification multiplies nothing; and name the entry of Table 27.3 of which this toy is the smallest instance.
Exercise 8 (lab). Break a wise crowd, then repair it, on a live model. Fix twenty verification items whose ground truth you control — ten short functions from the running team’s repository as they stand, and ten with one planted bug each — and one model at fixed settings; the juror’s question is always “Does this function contain a bug? Answer BUG or CLEAN, with one sentence of evidence.” Convene five verdicts per item under four procurement regimes: (A) one call, five samples, the full file as context — the photocopy; (B) one call, the model prompted to role-play five named specialists and return five verdicts — the costumes; (C) five separate calls, same evidence and settings as (A); (D) five separate calls, each given a different evidence window — the function alone; with its tests; with its docstring and specification; with its call sites; with the file’s recent history. For each regime tabulate the mean individual accuracy \hat{p}, the majority accuracy, the mean pairwise disagreement rate \bar{d}, and the correlation estimate \hat{\rho} = 1 - \bar{d}/\bigl(2\hat{p}(1-\hat{p})\bigr) of Exercise 3. Then rule: does the predicted ordering \hat{\rho}_{\mathrm{A}} \approx \hat{\rho}_{\mathrm{B}} > \hat{\rho}_{\mathrm{C}} > \hat{\rho}_{\mathrm{D}} appear; in which regimes does the majority beat the mean juror by more than the estimator’s noise; and what, measurably, did the costumes of (B) buy over the photocopy of (A)? Record the model identifier and the date beside the table, note that at twenty items \hat{\rho} is a coarse instrument that can wobble past its design value and below zero, and remember that the durable finding is the gap between the regimes — the pattern, not any absolute rate.
Exercise 9. The final exercise is the delegation Section 27.4 promised: write a research proposal, four pages at most, for one numbered entry of the ledger of Table 27.2 through Table 27.7. It must contain: (i) the problem restated as a falsifiable claim or a computable question, sharpened at least one turn beyond the table’s wording; (ii) the two parent literatures — the section’s advice is to hunt at the seams, and a problem with two parents inherits two literatures’ tools — each with one named result the proposal will stand on and a sentence on why its tools transfer; (iii) the first experiment or the first theorem: an experiment designed to Chapter 24’s discipline, with the strong baseline named, the comparison paired, and the claim it would license stated as the canonical triple — better at this, at this price, not elsewhere — or a theorem with its statement, the sharpest counterexample not yet ruled out, and the toy case to be proved first; (iv) the kill condition — the result that would end the direction rather than adjust it — together with a perishability audit separating what in the proposal is durable from what is tied to today’s substrate; and (v) a closing paragraph answering the section’s two selection criteria: the absence you have personally paid for, or the seam whose two parent literatures you can actually read.