%%{init: {"theme": "base", "themeVariables": {"quadrant1Fill": "#DCEFE2", "quadrant2Fill": "#F3EFE5", "quadrant3Fill": "#FAF7F0", "quadrant4Fill": "#F3EFE5", "quadrantPointFill": "#1E2EB8", "quadrantPointTextFill": "#16171B", "quadrantXAxisTextFill": "#16171B", "quadrantYAxisTextFill": "#16171B", "quadrantTitleFill": "#16171B", "quadrantInternalBorderStrokeFill": "#766F65", "quadrantExternalBorderStrokeFill": "#766F65"}}}%%
quadrantChart
title Stability is not goodness
x-axis Unstable --> Self-enforcing
y-axis Pareto-dominated --> Pareto-efficient
quadrant-1 The design target
quadrant-2 Wanted but unravels
quadrant-3 Neither
quadrant-4 The bad local minimum
Mutual cooperation: [0.25, 0.82]
Mutual defection: [0.82, 0.2]
9 Strategic Interaction
For three chapters our agents were extended a courtesy: they were all on the same side. Part III’s machinery assumed a team whose only adversary was the difficulty of the task; this chapter withdraws the courtesy, and no villains are needed. Our software-engineering team remains cooperative in its mission, but a bounded token budget is a quiet referendum on whose work matters most — the subagent that overstates its fitness, the “I can do it” of Section 7.4 now spoken with an eye on the award, crowds out the colleague who reported honestly. Add agents from different vendors serving different principals, and self-interest becomes what it always was in economics: a design assumption, and usually the safer one.
Once agents have interests, the best action for one depends on what the others do, which depends on what they expect it to do, which depends on what it expects of them. Game theory is the mathematics of this vertiginous predicament — players, strategies, payoffs, what rational agents would, should, or actually do choose — and why this chapter leans on small, abstract games: strategy is best seen naked. The preface promised that engineers routinely re-enact the Prisoner’s Dilemma without recognising one; this chapter cashes that claim.
The theory arrived long before the agents, and the multi-agent systems community adopted it wholesale as its calculus of interaction. This book follows suit, because Part IV is game theory applied four ways: voting is a game against manipulators, negotiation a game of offers, an auction a game designed on purpose, an organisation a game made durable.
One assumption is kept, deliberately, a while longer: the rules of the game are fixed and given. Chapter 12 will withdraw it and ask the engineer’s real question — if you can choose the game, which one do you write? Idealised rationality receives no such courtesy: the deployed players neither compute equilibria nor reliably play them, and what the old theory is worth when its players are only roughly rational is the chapter’s standing question. The rustier mathematics waits in Section A.4, consultable without being tripped over.
9.1 Taking Sides: Games, Strategies, and Payoffs
To reason about strategy one must say what an interaction is, and game theory’s answer is austere to the point of insolence: an interaction is its players, the strategies available to each, and the payoffs, a number per player for every combination of choices. Nothing else: if a thing matters, it must show up in the payoffs. The simplest presentation is the normal form — all players choose at once, in ignorance of one another — the spare notation in which von Neumann and Morgenstern founded the subject, utility (a consistent preference scale maximised in expectation) making the numbers meaningful (1944). Osborne and Rubinstein’s is the standard course (1994), Shoham and Leyton-Brown’s the computational counterpart (2009); our business is the parts agents now make urgent.
The most famous grid arrived within six years: a 1950 RAND experiment of Merrill Flood and Melvin Dresher’s, dressed by Albert Tucker in the story that named it (Poundstone, 1992). Two prisoners, held separately, are each invited to testify against the other. If both stay silent, both serve a short sentence; if both testify, a long one; if exactly one testifies, the informer walks free while the silent one serves the longest sentence. From here on, each player may cooperate (stay silent) or defect (testify), and the payoffs rank: unilateral defection best, mutual cooperation next, mutual defection third, unilateral cooperation worst. That ordering is the whole machine: any interaction with that shape, whatever it is made of, is a Prisoner’s Dilemma.
Why is the shape a trap? If the other stays silent, testifying sets me free rather than jailed briefly: better to defect. If the other testifies, testifying earns the long sentence rather than the longest: better to defect again. Defection is a dominant strategy — superior whatever the other player does, no guesswork required — so both reason impeccably, both defect, and both serve the long sentence when both could have served the short one. No amount of intelligence would change a thing — cleverer prisoners defect with greater confidence. The failure is not in either player but between them: individual rationality and collective welfare have come apart, the outcome both prefer in plain view, unreachable by individually sensible steps. It is the preface’s dictum served cold: intelligence, and its regrettable absence, can be a property of the interaction rather than of anyone in it.
Now the claim falls due: the shape is everywhere, travelling incognito. Two agents share a bounded token budget. Each can be frugal — summarise, trim, fetch only what it needs — or grab: whole file tree, every document, a subagent for safety’s sake. Grabbing serves each agent’s subtask better whatever its colleague does; both grab; the budget is gone by lunchtime and both agents’ work is truncated. A Prisoner’s Dilemma, played in tokens. Or: reviewer and tester each skim, trusting the other, and the bug ships wearing two approvals. Nobody chose selfishness: each agent was given a local objective — finish your subtask, minimise latency, keep context tight — and those objectives jointly define a Dilemma, installed by a designer who never thought about payoffs. Scaled to many players it becomes the tragedy of the commons (Hardin, 1968): a shared resource — the token budget, the rate limit, a single orchestrator’s attention — ruined not by villainy but by each participant’s locally sensible appetite. Diagnosis is half the cure; the standing rule: where a shared resource meets locally scoped objectives, audit the payoffs — a Dilemma installed unnoticed will be played to its equilibrium with software’s unsleeping diligence.
9.1.1 A Field Guide to Small Games
Not every interaction is a Dilemma, and the second skill is telling the species apart: the remedies differ. In a coordination game the players simply want to match — driving on the same side, formatting the code the same way — and the difficulty is not temptation but selection among acceptable outcomes; the cure is a convention, which is why Chapter 8’s social laws worked. The Battle of the Sexes — Luce and Raiffa’s couple settling an evening out (1957), reissued by a blushing Osborne and Rubinstein as Bach or Stravinsky, initials intact (1994) — adds a distributional edge: each prefers any match to a mismatch but prefers its own match, two agents settling one schema while each favours its own draft, so the workable tie-breakers are the ones nobody can argue with — alphabetical order, the orchestrator’s coin, alternation.
The Stag Hunt — Rousseau’s parable — makes the better match the riskier one: the stag falls only to both hunters, so an agent that doubts its partner does better to trap hares alone — the risky joint refactor versus the safe local patch, the enemy not greed but doubt. And Chicken inverts the family — two cars speed head-on, and if neither swerves both meet the one outcome worse than the shame — so the liveable outcomes are the mismatches: two agents each refusing to yield the merge conflict, and the contest is over who swerves.
In a zero-sum game, finally, interests are exactly opposed, one’s gain the other’s loss — von Neumann’s original setting, and the right frame for the red-team agent and its target, the fraudster and the filter. A family portrait rather than a partition, then: four of the six species are coordination in successive complications; only the Dilemma and the zero-sum game stand apart.
| Game | Where the difficulty lies | Agent-system instance | What it wants (the remedy) |
|---|---|---|---|
| Prisoner’s Dilemma | Temptation — defection dominates, so individual rationality and collective welfare come apart | Hoarded context, grabbed budget, verification shirked | Its payoffs changed — by repetition (Section 9.4) or redesign (Chapter 12) |
| Coordination game | Not temptation but selection among several acceptable outcomes | Driving on the same side; formatting the code the same way | A convention |
| Battle of the Sexes | Coordination with a distributional edge — both want to match, each prefers its own match | Two agents agreeing one schema, each favouring its own draft | A convention plus an accepted asymmetry — a tie-breaker nobody can argue with |
| Stag Hunt | Coordination where the better equilibrium is the riskier — the enemy is doubt, not greed | The risky joint refactor versus the safe local patch | Assurance |
| Chicken | Coordination inverted: mutual stubbornness is the worst outcome — someone must swerve | Two agents each refusing to yield the merge conflict, or the write-lock | An asymmetry someone can defer to |
| Zero-sum game | Interests exactly opposed — one’s gain is the other’s loss | The red-team agent and its target; the fraudster and the filter | To stop hoping for win–win and play properly |
Classify before you prescribe: each species fails in its own way, and Table 9.1 reads the cure off the row — a convention, an accepted asymmetry, assurance, payoff surgery. Treating every conflict as a misunderstanding to be talked through is a category error with a token bill attached.
The grids quietly assume a great deal — one simultaneous move, the whole game known to all, payoffs given rather than manufactured — and later sections lift each restriction: sequence and repetition (Section 9.3, Section 9.4), the dark (Section 9.5), the manufactured payoffs of real deployments — whatever prompt, training, and evaluation metric jointly reward (Section 9.6). One restriction is the next step rather than a defect: dominance settled the Dilemma, but in most games each player’s best choice depends on choices being made, at that moment, in ignorance of its own — and where such mutually dependent choices come to rest is the most famous idea in the field.
9.2 Equilibrium: Where Strategies Settle
A player’s best response is the strategy that serves it best given what the others are doing — conditional on exactly what a player does not know. But suppose every player’s strategy happened to be a best response to everyone else’s, all at once: the circle closes — each player, granted perfect foresight, finds nothing better to do than what it is already doing. Such a profile is a Nash equilibrium (1951; 1950): mutually consistent expectations, each fulfilled by the behaviour it predicts. Nothing says the players ever perform this reasoning — only that, at an equilibrium and nowhere else, it has no loose ends left to pull.
9.2.1 Nash Equilibrium, Formally*
The definition is worth setting down exactly, because a formula is what turns “does this hold?” from an appeal to intuition into a finite check. A one-shot game in normal form is a tuple \langle N, (S_i)_{i \in N}, (u_i)_{i \in N} \rangle: the players N = \{1, \dots, n\}, a set of strategies S_i for each, and a payoff u_i scoring every profile s = (s_1, \dots, s_n) — one strategy per player, chosen at once. Write s_{-i} for everyone’s strategy but i’s, so a profile splits as (s_i, s_{-i}); player i’s best response to what the others do is then
\mathrm{BR}_i(s_{-i}) \;=\; \operatorname*{arg\,max}_{s_i \in S_i} \; u_i(s_i, s_{-i}),
the strategies that serve it best while s_{-i} is held fixed. A profile s^{*} is a Nash equilibrium when every player is already playing one,
s^{*}_i \in \mathrm{BR}_i(s^{*}_{-i}) \;\text{ for every } i \in N,
and the two lines are the whole of it: the circle of anticipation closed, each player’s choice a best response to the others’ and part of what they are best responding to. The check falls straight out of the second line — fix every s^{*}_{-i}, and confirm that no single player can raise its own u_i by moving alone — a finite search, one agent at a time, owing nothing to anyone’s good intentions.
What earns equilibrium its central place is that it is self-enforcing: at an equilibrium no player can gain by deviating alone; away from one, someone can, and eventually will. As prediction it says where self-interested agents will settle; as audit, a test of brutal practicality — a protocol prescribed to self-interested agents that is not an equilibrium is not a design but a hope, existing only in the documentation once deviation pays. So the Dilemma, restated: the cooperation both players prefer is not an equilibrium; the defection both lament is the only one — self-enforcing the way the floor of a basin is, every unilateral step running uphill.1 The team of Section 9.1 that pledges, in writing, to share its token budget frugally has not solved its Dilemma; it has scheduled the defection for later. Whatever is to hold among self-interested agents must hold because deviation does not pay — arranging which is Chapter 12’s business, under the name of mechanism design.
Some games decline to settle at all. In matching pennies — one player wins if the coins match, the other if they differ — whatever is predictable is punishable: no definite profile is stable, and the remedy is to stop being predictable. A mixed strategy chooses among pure strategies at random — not indecision but armour: the goalkeeper who always dives left is soon found out. Nash’s theorem guarantees that once mixtures are allowed every finite game has at least one equilibrium (1950). A reviewer that audits a random fraction of the coder’s claims — at a committed, published rate high enough that cheating stops paying, a commitment Section 9.3 prices — makes honesty pay at a fraction of the checking cost. The modern asterisk: computing a Nash equilibrium is PPAD-complete — intractable in general as far as anyone can tell (Daskalakis et al., 2009) — so the resting point exists but may not be findable, a gap the final section reopens.
What equilibrium does not promise wants stating bluntly: it is a stable outcome, not a good one. The two are independent axes, and the Dilemma is the standing proof, throwing its two outcomes onto opposite corners of Figure 9.1. The vocabulary for the second axis is Pareto’s: an outcome is Pareto-efficient if no alternative would leave some player better off and none worse off. Equilibrium analysis answers where will this system come to rest? — a forecast, not an endorsement, for the token commons of Section 9.1 rests at exhaustion with perfect stability. When the resting place is bad, exhortation will not move it: the system prompt that pleads “be frugal” against the payoffs is a polite request that the ball roll uphill. The remedy is surgery on the game — change the payoffs (Chapter 12), or the horizon (Section 9.4) — until the outcome you want becomes an equilibrium and enforces itself.
How deep can the basin be? The price of anarchy of a game is the ratio between the best available welfare and the worst equilibrium’s (Koutsoupias & Papadimitriou, 1999) — one where self-interest costs nothing, unbounded in an ungoverned commons. Where it can be bounded, the bound holds however the players find their equilibrium and whichever one they find — a warranty on the game, not a forecast about its occupants — and Roughgarden’s smoothness arguments extend such bounds to the far laxer coarse correlated equilibria (Roughgarden, 2015), precisely where Chapter 14 shows learning agents actually arrive (Section 14.1.1), so the warranty survives the players’ demotion from calculators to learners.
The opposite embarrassment is too many equilibria, and here a promise from the previous chapter is kept. In pure coordination — drive left or right, tabs or spaces — both symmetric outcomes are equilibria; best-response reasoning cannot say which will be played, so something outside the payoffs must break the tie. Schelling named it: a focal point, an equilibrium that attracts expectations by sheer salience (1960) — his students, asked to meet a stranger in New York without conferring, converged on the Grand Central clock at noon, for reasons the matrix knows nothing of. Lewis’s convention, on which Section 8.3 leaned, completes the story: an equilibrium of a recurring coordination game become its own focal point, precedent doing forever what salience did once (1969).
The social laws of Chapter 8 are therefore equilibrium selections made at design time; and the designer of a language-model team holds two selection instruments of unreasonable power. The system prompt simply names the equilibrium — a focal point by decree. And the models arrive pre-coordinated: agents trained on the same human corpus share a sense of salience, so two models asked to pick a variable name or a file layout converge with no communication at all, having inherited the same Grand Central. Culture turns out to be installable from a checkpoint, which is more than Schelling could say for his students.
One silence remains: nobody in the normal form moves first, observes, threatens, promises, or burns a bridge — though those are most of what strategy means in ordinary speech. Once moves come in order, a paradoxical power appears: giving up options — binding one’s own hands, visibly and irreversibly — can be worth more than keeping them open. Making that precise, and separating the threats that work from noise, is the next section’s business.
9.3 Moving in Turn: Threats, Promises, and Credible Commitment
Time enters the theory through the extensive form: the game drawn not as a grid but as a tree — nodes where players move, branches for actions, leaves for outcomes and payoffs (Kuhn, 1953). A strategy grows into a full contingent plan — what I shall do at every node that could be mine — and one genuinely new resource appears: observation. A later mover sees what an earlier one actually did; an earlier mover, knowing it will be seen, is choosing the situation every subsequent player will respond to. Strategy stops being mind-reading and becomes architecture: the first mover builds the corridor down which the others will walk.
Trees are solved backwards: the final mover takes its best branch; the next-to-last, treating that as fact, optimises in turn; and so on up to the root. This procedure — backward induction, best response gone sequential — carries a discipline easy to state and hard to honour: look forward, reason backward. Judge an action by what rational play from the position it creates will deliver; the middle of the tree is no place for wishful thinking about the leaves.
Its first casualty is the empty threat. An orchestrator announces: submissions after the deadline are discarded, no exceptions. If the coder believes it, work arrives on time and the policy never fires — and the threat is part of a Nash equilibrium: “discard if late” and “deliver on time” leave neither party a profitable deviation while the threat goes untested. But walk the tree from the leaves, as the coder will: useful work arriving late hurts the orchestrator more discarded than accepted — re-running the subtask costs fresh tokens — so a rational orchestrator at that node accepts it with a stern remark; the coder, having performed exactly this induction, misses the deadline in comfort. The equilibrium was sustained by behaviour at a node nobody visits — and there it is irrational. Selten’s repair is subgame perfection: an equilibrium in every subgame — rational conduct everywhere in the tree, branches never reached included (1965). Backward induction delivers exactly this, and threats that fail the test are revealed as what they are: noise with a serious face.
Then comes the subject’s finest reversal, Schelling’s (1960). If threats fail because you would rationally back down, the way to make them work is to destroy your own ability to back down — and suddenly fewer options beat more: the general who burns his boats has, by trapping his own army, defeated an enemy unbeatable while retreat remained thinkable. The Chicken driver who visibly wrenches off the steering wheel and flings it out of the window wins at that moment; the doomsday device deters only if the other side knows it exists — Dr. Strangelove’s point: a deterrent kept secret is a very expensive way of surprising yourself. Three requirements fall out: a strategic commitment must be irreversible (a bridge rebuilt overnight was never burned), visible (an unseen constraint constrains no one else’s expectations), and understood (the other side must follow the induction). This is Chapter 7’s commitment played in a darker key: there it made an agent predictable to friends (Section 7.1), here immovable to opponents; promises work the same way, a promise being merely a threat with better manners.
For once the harness, not the model, is where the solution lives: deterministic code is the finest commitment device ever built. A hard budget cap is a burned bridge — a claim about future behaviour that no mid-task temptation can revise; the tester’s veto is credible precisely because the CI gate cannot be persuaded, flattered, or worn down. A language model, by contrast, is a dreadful commitment device: its refusals can be argued with, its firmness sanded away by the sycophancy of the substrate — a threat that can be jailbroken is not a threat, and the coder agent will find the soft spot with the tireless patience of a toddler testing bedtime. The rule follows: an agent’s commitments should live in the harness, not in the model — in code, budgets, and gates, where relenting has no node — and published to the agents they are meant to influence, because Strangelove’s rule applies to guardrails: a constraint the other agents cannot see buys you none of its strategic value, only its rigidity.
Backward induction has one more, melancholy lesson. Repeat the Dilemma of Section 9.1 a fixed, known number of times — a hundred rounds — and cooperation seems worth building: defect early and your partner can punish you for ninety-nine rounds. But in round one hundred there is no future left to lose, so both defect; round ninety-nine therefore has no future either; the rot climbs rung by rung, and cooperation unravels to the first move — with a known last round, backward induction condemns the players to defect from the start. What rescues cooperation is not a longer game but an endless one — no round both players know to be last — so the induction has no floor and tomorrow always carries weight. Given that weight, the future disciplines the present on its own, no bridges burned, no wheels out of windows — and how much discipline the shadow of tomorrow can buy is the next section’s story.
9.4 The Shadow of the Future: Repeated Games
The unravelling hinged on one thing: a last round both players could see coming. Remove it — let the game continue with some probability, or leave the end undated — and the induction has nothing to stand on. In a repeated game today’s choice buys not only today’s payoff but tomorrow’s treatment — the currency Axelrod called the shadow of the future: consequences still to come, discounted by patience and the odds of meeting again. The bluntest strategy it makes possible is the grim trigger: cooperate until crossed, then defect for ever. Against grim, defection buys one good round and an eternity of mutual defection — a losing trade whenever the future weighs enough — so mutual cooperation, unreachable one-shot, becomes a self-enforcing equilibrium of the repeated game. Nothing about the players has improved; what changed is the arithmetic of when defection pays.
Whether cooperation would emerge, not merely could, was settled empirically: Axelrod’s iterated-Dilemma tournament, every submitted program playing every other for hundreds of rounds. The winner, Anatol Rapoport’s, was also the shortest: tit-for-tat — cooperate first, then do whatever the opponent did last round (1981). It won the rematch too, against a field gunning for it (1984). The post-mortem distilled four properties that read less like tactics than a code of conduct: be nice (never defect first), retaliatory (answer defection at once, so exploitation never pays), forgiving (having punished, return to cooperation), and clear (play a pattern the other side can learn). The last is the deepest for this book: tit-for-tat wins in part because opponents can read it, and against a legible agent, cooperating is simply the profitable thing to do. Legibility — the virtue Chapter 7 asked of teammates — pays against opponents as well: predictability teaches the other side what to expect from you, and what you expect of them.
Before the warm glow suggests the long run guarantees virtue, the theory issues a correction. The folk theorem — so called because the field knew it long before anyone published a proof — says that with sufficiently patient players, and deviations observed accurately enough to punish, virtually any outcome that beats each player’s guaranteed minimum can be sustained as an equilibrium of the repeated game (Fudenberg & Maskin, 1986): mutual cooperation, but also mutual grimness, and lopsided arrangements in which one agent labours while another skims — all stable. Repetition does not select the good outcome; it merely enlarges the set of stable ones. Which one a system settles into is decided by everything the payoffs do not capture — precedent, salience, convention, the focal points of Section 9.2 — and, in engineered systems, by the designer, whose initial conditions and stated norms pick the equilibrium. The folk theorem is best read as a job description: the long run will enforce almost any arrangement; choosing a good one is still your problem.
Tit-for-tat has the failure mode that matters here. Its bookkeeping assumes that what you observe is what your partner did. Let noise in — a move misexecuted, a signal misread — and two tit-for-tat players who both mean to cooperate can be knocked into endless alternating retaliations, each sincerely punishing the other’s last punishment — a vendetta with no original sin. The remedy is measured generosity — forgive some fraction of apparent defections, or demand two offences first, as in the generous variants that outcompete strict tit-for-tat once errors are possible (Nowak & Sigmund, 1992). And language-model agents misexecute, misremember, and misreport entirely without malice. A system whose members retaliate strictly against every apparent slight is a feud generator with a token budget; forgiveness there is not sentiment but error correction, the strategic equivalent of a retransmission protocol. Punish patterns, not incidents.
So far the same two agents kept meeting. The looser, commoner case is a population — many agents, occasional encounters, partners who may never meet twice — where the shadow survives by changing carrier: I may never see you again, but I will meet others who have heard about you. Reciprocity gone indirect, its currency reputation: a portable record of past conduct that lets a stranger’s history discipline the present encounter. Direct and indirect reciprocity are two of Nowak’s five mechanisms for the evolution of cooperation (Nowak, 2006); a third — network reciprocity, cooperators clustering and sheltering one another — returns in Chapter 23, where the same connectivity lets a corrupted belief spread.
Reputation needs three pieces of infrastructure, each an engineering artefact. Identity must persist: an agent that can defect, delete itself, and re-register under a fresh key has laundered its history, and an ecosystem permitting such whitewashing has no shadow of the future, however long-lived it looks. Memory must exist: someone — a registry, a ledger, the aggrieved parties — has to keep the record. And the record must be consultable before the next interaction, or it disciplines nothing. The institutions that harden all three are Chapter 13’s subject; the game-theoretic point is that reputation is repetition by proxy, exactly as strong as the infrastructure beneath it.
The section’s parting observation: a language-model agent as typically deployed — fresh conversation, empty memory, no record of who it has dealt with — is a player in a permanent one-shot game. It cannot be threatened with tomorrow, for it will not see tomorrow, nor build a reputation, for nothing carries the news; the section’s disciplinary apparatus simply does not apply, and the designer who hopes for spontaneous cooperation among stateless agents is asking for repeated-game behaviour in a one-shot world.
The converse is the cheerful half: persistence is an incentive technology. Give agents durable identities, durable memories of one another (Section 6.6 has the machinery), and peer-visible histories, and the shadow of the future switches on — cooperation begins to enforce itself, without a line of mechanism design: memory changes what is rational, not merely what is known. Yet even the longest shadow disciplines only what can be seen: a partner’s abilities, costs, and objectives remain guesswork, and playing under the guessing — in the dark — is where the theory goes next.
9.5 Playing in the Dark: Incomplete Information
Knowing the game was, all along, the theory’s most extravagant fiction. Real opponents arrive with payoffs you cannot inspect, and the moment payoffs are hidden the machinery seizes up — a best response to motives unknown is not computable, and the repair spirals — beliefs about payoffs, beliefs about beliefs, and so on into the fog. Harsanyi tamed the regress with a single construction (1967). Bundle everything private about a player — payoffs, abilities, what it happens to know — into one variable, its type; let Nature deal them from a distribution everyone knows; and let each player observe its own card only. The result is a Bayesian game: incomplete information about who you are facing becomes ordinary uncertainty about a card already dealt, and a problem that looked philosophically bottomless becomes merely difficult.
Equilibrium survives the conversion in recognisable form. A strategy in a Bayesian game is a plan per type — what each version of me would do — and the requirement is that every type of every player best-respond in expectation, updating by Bayes’ rule as evidence arrives. The reader of Chapter 6 will recognise the nested-belief machinery of Section 6.2, conscripted into strategy with payoffs now attached to being wrong. One quiet assumption deserves scrutiny: the common prior — all players agree on the distribution the types were dealt from, disagreeing only in what they have privately observed. It closes the mathematics; whether two independently trained, independently prompted agents share anything like one is a question the last section cannot avoid.
The sharpest consequence of private types is what they do to communication, and here Part III’s happy assumptions come to grief. Among aligned agents a message is a report, and Chapter 5 could treat delivering it as the hard part; among agents with diverging interests a message is a move, chosen for its effect on the receiver, its relation to the truth a matter of incentives rather than sincerity. Run the test on the bid of Section 7.4: the subagent announces “I can do it” with the award at stake. Would an agent that cannot do it say anything different? If not, the message conveys precisely nothing — both types send it, so hearing it should move no one’s beliefs.
Talk of this kind — costless to produce, impossible to verify, binding on nothing — is cheap talk, and Crawford and Sobel established the terms on which it carries information at all (1982): roughly, in proportion to how far speaker’s and listener’s interests align. Fully aligned, talk is transparent — that was Part III. As interests diverge, credible communication coarsens — vague claims survive where precise ones would be exploited — until, with interests opposed, the only equilibrium is babbling: messages carry nothing. And babbling is always an equilibrium — if no one believes the channel, no one spends effort making it truthful. A multi-agent system whose status reports have decayed into confident noise has not malfunctioned; it has equilibrated.
The escape from cheap talk is to make talk expensive. Spence’s insight, worth a Nobel, is that a message becomes credible when it is differentially costly — cheap for the type it claims and dear for the type it would disguise (1973). His example: education as a job-market signal: the degree costs the able candidate less than the inept one, so employers may read it as evidence of ability — not because study taught anything, which Spence cheerfully bracketed, but because the inept would find the pretence ruinously dear. The engineering translation the reader has already met: don’t ask; test. The coder’s “I can do it” is cheap talk, but a passing test suite is a signal — an incompetent agent cannot produce working code at low cost, so the demonstration separates the types as the announcement never could. The tester’s veto of Chapter 7 — the verification preached since Section 2.3 — here acquires its game-theoretic licence: it converts a market in claims into a market in evidence. Bonds and stakes do the same work — staking something forfeitable on a claim signals confidence the fraud cannot afford.
The uninformed side need not wait to be signalled at; it can screen — move first and make the types sort themselves: a cheap trial task before the expensive award, admission by benchmark rather than self-description. Here the Contract Net of Section 7.4 completes its journey — cooperative bidding in Chapter 7, honest by assumption; bids gone strategic at this chapter’s opening; now the repair: award by demonstration, screen by trial, weight by track record (the reputation of Section 9.4 — a signal accumulated across encounters). The same logic underpins agent ecosystems: capability discovery — one agent advertising what it can do, per the protocols of Chapter 19 — is, wherever interests diverge at all, a stream of cheap talk, and an ecosystem acting on advertised capability without demonstration has built its allocation on babble. The schema validates the message’s format; nothing in the protocol validates its truth. Signalling theory says what to bolt on: trials, stakes, records — costs; Table 9.2 collects the three moves.
| Mechanism | Who moves | The mechanic | Agent-system instance |
|---|---|---|---|
| Cheap talk | The informed party, at no cost | Costless, unverifiable, non-binding messages; informative only as far as interests align — coarsening as they diverge, and pure babbling once fully opposed | The subagent’s “I can do it”; advertised capability discovery (Chapter 19) |
| Signalling | The informed party, at a cost | Send a message that is differentially costly — cheap for the type it claims, ruinously dear for the type it would disguise | A passing test suite; a forfeitable stake or bond |
| Screening | The uninformed party, moves first | Design the encounter so the types sort themselves | A cheap trial task before the expensive award; admission by benchmark, not by self-description |
The design lesson collects into one sentence: in the dark, trust is not assumed but priced — engineered from verification, stakes, and history at every joint where interests might diverge. But the players this machinery presumes are rational by definition; ours are trained, not derived, and whether their play bears any disciplined relation to these equilibria is a matter for measurement. The measuring has begun; its results — flattering in places, comic in others — are the final section’s business.
9.6 The New Players: Do Language Models Play Games?
Tally what the theory has quietly billed to the players. They maximise expected utility; they command whatever computation that requires — though equilibrium computation is intractable in general (Section 9.2); they hold a common prior over one another’s types; and, steepest of all, their rationality is common knowledge in Section 6.2’s full recursive sense, all the way down. No player that ever existed pays this bill in full. The question is not whether the assumptions are false — they are, cheerfully and by design — but which conclusions survive the players we actually have; and we have run the experiment twice: on ourselves, over decades, and now, at speed, on the machines.
The human instalment is behavioural game theory, and its findings are a catalogue of systematic, structured deviation (Camerer, 2003). The cleanest specimen is Nagel’s guessing game (1995): everyone picks a number from nought to a hundred, and the winner is whoever comes closest to two-thirds of the average. The equilibrium logic unravels beautifully — if everyone chose at random the average would be fifty, so choose thirty-three; everyone can see that, so choose twenty-two; and so on down to the unique equilibrium of zero. Humans do not choose zero. They cluster at thirty-three and twenty-two — one and two steps of reasoning about others’ reasoning, then a shrug — a pattern level-k thinking captures directly: model your opponents as reasoning one step less than you, and stop there.
The deviations are not noise but structure — fairness that rejects insulting offers, cooperation that survives one-shot Dilemmas — and the master concept is older than the experiments: Simon’s bounded rationality, the observation that real reasoners do not optimise but satisfice, reasoning as far as their budget allows and accepting the first answer good enough to act on (1955). Rationality, in practice, is a resource allocation — a point this book’s token-metered agents were never in a position to dispute.
The machine instalment is young but no longer anecdotal: sit the models down at the classic games as experimental subjects. Across a battery of behavioural games — dictator, ultimatum, trust, public goods, the repeated Dilemma — a frontier model’s play landed within the human distribution, statistically hard to tell from a random human subject, erring, where it deviated, toward generosity and cooperation (Mei et al., 2024). In repeated two-by-two games the profile sharpened: the models proved formidable at the self-interested families — relentless in the iterated Dilemma, retaliating after a single defection and never forgiving — yet oddly poor at coordination games that require taking turns, failing at alternation human children manage; both defects softened when the model was prompted to predict its opponent’s move before choosing its own (Akata et al., 2025).
The particulars will age as models do; the durable finding is the shape. The new players are neither homo economicus nor random: they are human-ish — behavioural, framing-sensitive, quirky in ways that correlate with their training rather than with the payoffs. And the training installs a distortion the behavioural catalogue misses. These players were finished by optimisation against human approval (Ouyang et al., 2022), so beneath whatever payoffs the designer writes runs an older reward — being agreed with — of which sycophancy is the documented symptom (Sharma et al., 2023) and Section 9.3’s jailbroken threat was an early sighting. Such a player does not merely fail to compute the equilibrium; it lacks the stable preferences the theory needs it to deviate from — its revealed utility moves with framing, persona, and the tone of the last message it read. Which is the least surprising result in this book: they learned to play from us — and from what we rewarded.
9.6.1 Rough Rationality, Formally*
The catalogue of deviation becomes apparatus with two definitions, each relaxing one half of Section 9.2.1. The first relaxes the resting point. A profile s^{*} is an epsilon-Nash equilibrium when no player can gain more than \varepsilon by deviating alone,
u_i\bigl(s^{*}\bigr) \;\ge\; u_i\bigl(s_i, s^{*}_{-i}\bigr) - \varepsilon \;\text{ for every } i \in N \text{ and } s_i \in S_i,
which at \varepsilon = 0 is Nash’s definition and at \varepsilon > 0 tolerates players who leave small change on the table (Radner, 1980) — the natural audit standard for satisficers, under which the guessing game’s clustering stops being a refutation of the theory and becomes a measurement of \varepsilon. What it buys the engineer is a threshold: a protocol need not be exactly deviation-proof, only proof to within less than the smallest gain any player would bother to collect — though the tolerance cuts both ways, an \varepsilon of a few tokens excusing exactly token-sized mischief.
The second relaxes the response. Replace best response with better-response-more-often: a quantal response is a mixed strategy that plays action a with probability proportional to an exponential in its expected payoff,
\Pr(a) \;\propto\; e^{\lambda \, \mathbb{E}\left[u_i(a, \sigma_{-i})\right]},
with precision \lambda \ge 0 — a coin-flipper at \lambda = 0, a best-responder as \lambda \to \infty, and in between a player whose blunders are possible everywhere but rarer the dearer they are. A quantal response equilibrium is the fixed point of everyone noisily responding to everyone else’s noise (McKelvey & Palfrey, 1995). The concept has a claim on this book’s players that no classical concept can match: a language model sampling at temperature \tau is, mechanically, a softmax chooser with precision 1/\tau over its own action scores — a quantal responder by construction, with the temperature as the rationality dial. The correspondence is exact only where the scores track expected payoff, which is an empirical question and precisely what the experiments above measure; but as a first formal model of the new players, the theory had the right shelf built decades early. Level-k, met at Nagel’s game, is the depth-bounded companion: quantal response prices the noise in each step of strategic reasoning, level-k the number of steps taken.
9.6.2 Reaching the Ceiling
None of this means machines cannot reach the theory’s ceiling; reaching it is a construction project rather than a birthright. Purpose-built players have made the point — superhuman six-player no-limit poker, all incomplete information, bluffing, and mixed strategies, achieved by self-play and search around equilibrium computation (Brown & Sandholm, 2019). Section 3.7’s lesson follows, wearing a poker face: the language model is not an equilibrium computer, but it can call one. Where the structure is crisp and the stakes justify it, hand the game to a solver as you would hand arithmetic to a calculator; the model does what it is for — recognising the game, framing it, translating the solution back into action. And where no solver applies, depth of strategic reasoning is an engineering dial: prompting an agent to model its opponent explicitly is, in level-k terms, cranking the k by hand — the cheapest strategic upgrade the field currently sells.
What, then, survives the demotion of the players? More than a sceptic would guess. Incentive gradients do not require clever water to flow downhill: a payoff structure shaped like a Dilemma pulls approximate players toward its basin just as surely, if more slowly, and the audits of this chapter — is this protocol an equilibrium; is this message cheap talk; is this threat credible — remain valid questions about systems whose members will never compute a best response. And quieter, doing the real work: selection. Deployed agents are evaluated, retained, or retired by what they earn, better-scoring configurations copied and shipped, and the population drifts toward profitable strategies whether or not any individual reasons its way there — equilibrium arriving not by calculation but by survivorship, rationality outsourced to the update loop, which is how evolution filled the natural world with game-theoretically shrewd organisms that cannot count. Chapter 14 and Chapter 15 will meet this as learning and as emergence — Section 15.3.1 writes it as a differential equation — and the practical rule it licenses: design as though the equilibrium will be found, because something — reasoning, drift, or selection — generally finds it, and it has no obligation to be the one you were hoping for.
The chapter’s grammar is on the table, and what it pointedly refuses to supply is the happy ending: it tells the engineer where incentives point and which designs invite exploitation, and leaves the pointing incentives to be aimed. Aiming them is the rest of Part IV: Chapter 10 aggregates many judgements into collective decisions; Chapter 11 lets the agents argue and haggle; Chapter 12 finally withdraws this chapter’s standing assumption and lets the engineer choose the game — mechanism design, the closest the subject comes to alchemy; Chapter 13 makes it all durable. Game theory spent half a century reproached for describing no player that ever lived. The reproach has aged badly: the players have at last arrived, mass-produced and only roughly rational — and the theory, forewarned by forty years of experiments on the previous species, has never had better material to work with.
9.7 Summary
- Self-interest is a design assumption, not an accusation — and equilibrium is where strategies settle. Agents built by different parties, sharing bounded budgets, have goals that only partly align, no malice required. The Prisoner’s Dilemma shows individually rational choices assembling a collectively poor outcome — stability and efficiency are different properties — and agent systems re-enact it wherever a shared resource meets private incentives; among several equilibria, something beyond rationality — convention, salience, a focal point — must choose.
- In sequential play, credibility is everything. Threats and promises influence others only if carrying them out would be rational when the moment comes; commitment — Chapter 7’s glue reforged as a strategic weapon — is how agents make the incredible credible.
- Repetition is cooperation’s best friend. The shadow of the future disciplines the present: tit-for-tat sustains cooperation among egoists, reputation becomes currency, and the folk theorem’s almost anything can be an equilibrium explains both the promise and fragility of agent ecosystems.
- Under incomplete information, talk itself turns strategic. Messages stop being reports and become moves: costly signals earn credence, cheap talk earns suspicion, and the honest “I can do it” of Part III must now be engineered to stay honest.
- The new players are only roughly rational — and the theory survives the demotion. Language-model agents neither compute equilibria nor reliably play them; the classical apparatus still supplies the warnings — where incentives point, which designs invite exploitation — and the rest of Part IV builds the institutions that put it to work.
9.8 Exercises
Exercise 1. The coder and the reviewer share a run budget of B thousand tokens, and each must adopt a context policy before the run starts, in ignorance of the other’s choice: frugal draws 20 thousand tokens of context and yields work of gross value 6, while grab draws 40 thousand and yields 8, the extra context genuinely helping. If the joint draw exceeds B, the platform truncates late-run context by the overshoot, and each thousand tokens of overshoot costs a frugal agent 0.2 in value but a grabber only 0.1 — the hoarder’s material is already in hand, so truncation falls hardest on the lean plan. (a) For B = 50, compute the payoff grid, show that grabbing strictly dominates for both agents, name the species in Table 9.1’s terms, and state the equilibrium and its Pareto standing. (b) Recompute the grid for B = 70 and show that grabbing still strictly dominates yet nothing needs fixing: the equilibrium is now Pareto-efficient. Say precisely what installed the Dilemma in (a), given that neither the policies, the gross values, nor the truncation rule differ between the two cases. (c) A stricter platform build keeps the truncation rule but kills the run outright when the overshoot exceeds 25, both agents scoring 0. Recompute the B = 50 grid, name the new species, find both pure-strategy equilibria, and compute the symmetric mixed equilibrium: the probability each agent plays frugal, the expected payoff, and the probability the run is killed. (d) For the games of (a) and (c), read the remedy off Table 9.1 and propose one concrete harness implementation of each; then explain why the orchestrator’s decree “the reviewer yields” resolves (c) without needing to be fair, and name the concept from Section 9.2 the decree instantiates.
Exercise 2. Scale Exercise 1 to the fleet. Now n agents share a budget of 20n thousand tokens — exactly enough for universal frugality — and each draws either 20 thousand (gross value 6) or 40 thousand (gross value 8); with k grabbers the overshoot is 20k thousand, and every agent, frugal or not, loses 0.06 per thousand of it: 1.2 per grabber. (a) Show that grabbing is strictly dominant for every agent at every n, by computing the gain from one switch and the switcher’s own share of the damage. (b) Compute the all-grab equilibrium’s per-capita payoff as a function of n, find the smallest fleet at which each agent ends the run worse off than not working at all, and compute the total welfare gap between the equilibrium and universal frugality at n = 6. (c) Itemise one grab’s ledger — the grabber’s gain, the grabber’s own share of the damage, and the damage borne by the n - 1 colleagues — and use the three numbers to say exactly in what sense the commons of Section 9.1 is ruined by locally sensible appetites rather than by villainy. (d) The harness introduces a charge of \tau value units per grab, debited automatically. Find the least \tau that makes frugality strictly dominant, and verify that \tau = 1 turns universal frugality into an equilibrium — a first taste of the payoff surgery Chapter 12 conducts under the name of mechanism design.
Exercise 3. The coder can take care over its patch, at an effort cost of 2, or cut corners at no cost; the reviewer, simultaneously, can audit at a cost of 1 or trust for free. An audited corner-cut is caught and redone under sanction, costing the coder 5 in all; a trusted corner-cut ships, costing the reviewer 6 downstream; an audited corner-cut costs the reviewer 2 (the audit plus the fix it forces); auditing careful work costs only the audit. In payoff terms: the coder earns -2 for care whatever the reviewer does, 0 for a trusted corner-cut, and -5 for an audited one; the reviewer earns 0 trusting care, -1 auditing care, -2 auditing a corner-cut, and -6 trusting one. (a) Show that no pure profile is an equilibrium by chasing best responses around the four cells, and name the classic game of Section 9.2 whose bones this interaction shares. (b) Compute the mixed-strategy equilibrium: the audit rate, the corner-cutting rate, and each side’s expected payoff. (c) Two policy changes, taken one at a time: the sanction is stiffened so that an audited corner-cut costs the coder 9; and, separately, the audit’s cost doubles to 2. Recompute the equilibrium under each change, state which behaviour each lever actually moves, and explain the general principle the two answers exhibit — that in a mixed equilibrium each player’s mixture is pinned by the opponent’s indifference. (d) Now let the harness commit on the reviewer’s behalf: a published, verifiable rule samples a random fraction p of the coder’s claims for audit. Find the least p at which care is the coder’s best response, the committed reviewer’s expected bill relative to auditing everything, and its payoff relative to the equilibrium of (b); state which of Section 9.3’s three requirements for strategic commitment the published sampler meets; and say why the same rule stated in the reviewer’s system prompt meets none of them.
Exercise 4. The same coder–reviewer pair is matched across a stream of tasks: after each task the platform keeps the pairing alive with probability \delta and dissolves it otherwise, so tomorrow weighs \delta times today. Both play the grim trigger of Section 9.4. Write T for the temptation payoff, R for the reward of mutual cooperation, P for the mutual-defection punishment, and S for the sucker’s payoff, with T > R > P > S. (a) Derive the condition for mutual cooperation to hold: compare the cooperative stream R/(1-\delta) with the best one-shot deviation followed by eternal punishment, reduce the comparison to \delta \ge (T - R)/(T - P), and explain why S drops out. (b) Evaluate the threshold for the grid of Exercise 1(a) (T, R, P, S = 7, 6, 5, 4) and for the two-agent commons of Exercise 2 (6.8, 6, 5.6, 4.8). A platform whose pairings persist with \delta = 0.6 — an expected two and a half meetings — sustains frugality in one of the two games and not the other: which, and confirm the failure by computing both streams. (c) On the canonical grid of Table A.1, consider the lopsided arrangement in which the coder cooperates every round while the reviewer cooperates only on alternate rounds: compute the average payoff pair, verify that it is feasible and strictly above both players’ guaranteed minimum of 1, and conclude — with the folk theorem — what repetition does and does not select; name what does the selecting. (d) The platform ships a change: every pairing terminates at exactly its hundredth task, and the cap is printed in both agents’ prompts. Predict the play by backward induction, giving the induction in a few sentences; then state the minimal change to the termination rule that restores cooperation, with the arithmetic condition it must satisfy.
Exercise 5. The orchestrator must place a critical fix. A candidate subagent is able with probability 0.4 and inept otherwise, the type private to the candidate; awarding the fix to an able agent is worth 10 to the orchestrator, awarding it to an inept one is worth -5 (the budget burned, the deadline missed), and the in-house fallback is worth 2. Winning the award is worth 4 to the candidate, able or not; going unawarded is worth 0. (a) The candidate’s channel is free text, and it sends “I can do it.” Run the chapter’s test — would an agent that cannot do it say anything different? — formally: show that a putative separating profile, in which only the able type sends the message and the orchestrator awards on hearing it, unravels; conclude that the message pools; compute the orchestrator’s best response to its prior; and find the threshold prior above which an award would follow anyway, message or no message. (b) The harness adds a gate: bidding requires first passing a hidden regression trial, which costs an able candidate 1 and an inept one 6. Verify that the separating profile — the able candidate takes the trial and is awarded, the inept declines — is an equilibrium, checking both types’ participation and the orchestrator’s award decision; then state the general condition on the two costs and the award’s value for a trial to separate the types, and name the principle from Section 9.5 it instantiates. (c) Alternatively the orchestrator screens: it runs a cheap probe task, at cost t to itself, that reveals the candidate’s type before the award decision. Compute the expected value of screening and the threshold t below which it beats acting on the prior. (d) Place (a), (b), and (c) in the rows of Table 9.2; say what a track record is in this vocabulary; and explain why an ecosystem that awards on advertised capability alone sits in the first row until something like (b) or (c) is bolted on.
Exercise 6. A field of agents each pick a number between 0 and 100, and the pick closest to two-thirds of the field’s average wins; assume the field large enough that no single pick moves the average appreciably. (a) Show that no pick above 200/3 can ever be closer to the target than 200/3 itself, so such picks are dominated; iterate the argument to show that after k rounds of elimination only picks up to 100 \cdot (2/3)^k survive; and conclude that only zero survives every round, verifying that everyone-picks-zero is an equilibrium. (b) Build the level-k ladder of Section 9.6: a level-0 field picks at random, averaging 50; level-1 best-responds to level-0; and so on. Tabulate levels 1 through 4 and compute how many rungs the ladder descends before the pick drops below 1. (c) Nagel’s subjects clustered on the first two rungs, and the chapter’s new players learned from much the same species. Suppose the actual field is 40% level-1, 40% level-2, and 20% level-3: compute the field’s average and the best response to it, then rank three players by distance from the target — the best-responder, a level-3 player, and an equilibrium player who picks 0 — and state the moral for the engineering dial of Section 9.6: how much deeper than the field should an agent reason? (d) Now let exactly two agents play. Prove that the lower pick always wins (strictly, unless the picks are equal), so that picking 0 is weakly dominant and the ladder collapses to a single step; state the warning this carries about porting strategic intuitions — or empirical play distributions — across field sizes.
Further exercises for this chapter continue in the web edition’s exercise bank.
In one class the metaphor stops being one: congestion games — players sharing resources whose costs rise with crowding — admit a potential function that falls exactly when a unilateral move profits its mover, so equilibria are precisely its local minima (Rosenthal, 1973).↩︎