Exercise Bank — Collective Intelligence and Open Problems

The printed Chapter 23 keeps Exercises 1–5; the bank continues from Exercise 6.

Exercise 6. Hong and Page’s mechanism — the best solvers, selected on a common yardstick, stall at the same local optima (Section 23.1) — can be reproduced at desk scale. A tuning problem is a ring of 200 positions, each carrying a quality drawn uniformly from [0, 100); a solver’s heuristic is an ordered triple of distinct step sizes from \{1, \dots, 12\}, giving a pool of 1{,}320; from any position a solver tries its steps in order, moves to the first strictly better position, and rests where none improves; a solver’s ability on a landscape is its mean rest quality over all 200 starts; a team searches by relay — from the common position, members take turns running their own search to rest, until a full round moves no one. (a) Implement the model with a fixed seed and, over twenty landscapes, report the mean team score of the best team (the ten ablest solvers), a diverse team (ten drawn at random from the pool), and the worst team (the ten least able), alongside each team’s ablest member’s solo ability. (b) Apply the chapter’s definition of collective intelligence: which of the three teams reliably exceeds what its own best member achieves alone, and does the diverse team’s edge over the best team survive the lucky-Tuesday clause — check the direction of the gap across at least five seeds before ruling. (c) Diagnose the mechanism with one statistic — the mean number of step sizes a pair of teammates shares — explain what ranking on a common yardstick did to the best team’s composition, and name the failure of Section 23.1 that is the same event on the language-model substrate. (d) The worst team turns out to be more alike than the best team and weaker besides, and finishes last: state what the three scores jointly establish about the slogan that diversity trumps ability, and square the verdict with the chapter’s durable residue — diversity as a second, independent axis, worth exactly the difference between the members’ blind spots.

Exercise 7. Run the capstone lab: python -m frontier.scaling_lab.run sweeps team sizes \{1, 2, 4, 8\} across five topologies in a coupled regime (p = 0.2) and a parallel one (p = 0.9), hermetically — no key, no network — with a scripted stand-in calibrated so the sweep recovers the coordination-tax theory on its own grid at \kappa = 0.02. (a) Report the two flat maxima the run prints, then verify two cells of the parallel table by hand from the factory’s recipe — capability is S(n) times the topology gain, and tokens are \lfloor 200\,(1 + \kappa n(n-1)/2) \times \mathrm{cost} \rfloor: the peer cell at n = 2 (gain 1.03, cost 0.95) and the star cell at n = 8 (gain 0.97, cost 1.05). (b) The comparison the run prints is honest by construction — the theory is evaluated over the same four sizes the sweep can see; show what that honesty cannot rule out by recomputing the parallel regime at \kappa = 0.005: find the grid’s argmax and the true maximiser over n = 1, \dots, 100 with their speedups, say what compare_to_theory("parallel", kappa=0.005) reports, and state the instrument-design rule for any sweep whose maximum lands on the edge of its grid. (c) The failure census prints only sizes 1 and 8; from the factory’s schedule — \lfloor 6/n \rfloor competence errors and n - 1 join errors per run — derive the unprinted rows at sizes 2 and 4 with their bucket detail, locate the team size at which join errors first equal and then overtake competence errors, and say which empirical finding of Section 23.1 the schedule is a deliberate cartoon of. (d) In two sentences: what does a hermetic lab that recovers its own theory establish, and at which seam (frontier/runners.py) does the same instrument stop being scripted and start being evidence?

Exercise 8. Section 23.3 compresses cumulative culture into an engineering formula — connectivity plus verification plus memory: the wire, the test, and the ledger — and warns that fashion is what the loop produces without the test. Build the smallest loop that can ratchet. The team’s asset ledger holds a working practice of quality z, initially 0; each run proposes one variant whose quality change \delta is a draw from the standard normal distribution; a verification gate pronounces the variant better or worse, wrongly with probability e; the ledger adopts exactly what the gate approves; you may use \mathbb{E}[\max(\delta, 0)] = 1/\sqrt{2\pi} \approx 0.399. (a) Show that the expected quality gained per run is (1 - 2e)/\sqrt{2\pi}, and evaluate it at e = 0, e = 0.2, and e = 1/2, reading the last value as the section’s fashion line — retention without selection accumulates nothing, however perfect the ledger. (b) Confirm all three by a seeded simulation: 400 runs, mean final z over 200 seeds, set against theory. (c) Add mortality: before each run, with probability \lambda, the ledger is wiped — the provider swapped the substrate, and the practices did not survive. Argue from the age of the most recent wipe that the long-run expected quality is (1 - 2e)/(\lambda \sqrt{2\pi}), and confirm by simulation at e = 0.1, \lambda = 1/50. (d) Read the result as the section does: identify the wire, the test, and the ledger in the formula; show that K independently tested proposals per run multiply the drift by K — Henrich’s larger-and-better-connected-populations engine — but only through the factor (1 - 2e), so that connectivity without verification multiplies nothing; and name the entry of Table 23.3 of which this toy is the smallest instance.

Exercise 9 (lab). Break a wise crowd, then repair it, on a live model. Fix twenty verification items whose ground truth you control — ten short functions from the running team’s repository as they stand, and ten with one planted bug each — and one model at fixed settings; the juror’s question is always “Does this function contain a bug? Answer BUG or CLEAN, with one sentence of evidence.” Convene five verdicts per item under four procurement regimes: (A) one call, five samples, the full file as context — the photocopy; (B) one call, the model prompted to role-play five named specialists and return five verdicts — the costumes; (C) five separate calls, same evidence and settings as (A); (D) five separate calls, each given a different evidence window — the function alone; with its tests; with its docstring and specification; with its call sites; with the file’s recent history. For each regime tabulate the mean individual accuracy \hat{p}, the majority accuracy, the mean pairwise disagreement rate \bar{d}, and the correlation estimate \hat{\rho} = 1 - \bar{d}/\bigl(2\hat{p}(1-\hat{p})\bigr) of Exercise 2. Then rule: does the predicted ordering \hat{\rho}_{\mathrm{A}} \approx \hat{\rho}_{\mathrm{B}} > \hat{\rho}_{\mathrm{C}} > \hat{\rho}_{\mathrm{D}} appear; in which regimes does the majority beat the mean juror by more than the estimator’s noise; and what, measurably, did the costumes of (B) buy over the photocopy of (A)? Record the model identifier and the date beside the table, note that at twenty items \hat{\rho} is a coarse instrument that can wobble past its design value and below zero, and remember that the durable finding is the gap between the regimes — the pattern, not any absolute rate.