The Classic Prisoner’s Dilemma
Consider a classic problem from game theory — the prisoner’s dilemma1.
Two criminals are caught at roughly the same time for similar crimes. There’s reason to believe they acted together, and the police, holding them in separate cells, offer each of them the same deal: if one testifies against the other while the other stays silent, the first goes free for cooperating with the investigation, while the second gets the maximum sentence (5 years). If both stay silent, their case is tried under a lesser charge and each gets a year in prison. If both testify against each other, they get a reduced sentence (3 years each). Each prisoner chooses whether to stay silent or testify against the other. But neither knows for certain what the other will do. What happens?
This problem can be represented as a so-called payoff matrix.
Let’s denote the moves: C — cooperate, D — defect.
| Me \ Them | C | D |
|---|---|---|
| C | 3 / 3 | 0 / 5 |
| D | 5 / 0 | 1 / 1 |
It seems obvious that the only mutually beneficial outcome is silence — a year each. But you have no clue what your partner will do.
From each agent’s individual point of view, the logic is merciless. Whatever the other one does, you’re better off defecting. If your partner stays silent — defecting gives you the better personal outcome. If your partner has already defected — defecting at least keeps you from being worse off. When both reason this way, both defect — and both end up worse off than if they’d stayed silent together.
Put simply: playing rationally, the agents arrived at an irrational result.
This point in the problem is called a Nash equilibrium. Neither player can improve their own outcome by changing only their own move. You can’t escape the equilibrium of mutual defection alone — both sides have to shift, and there’s no incentive to.
Actual Problems
The prisoner’s dilemma is a problem humanity runs into every day.
The Nuclear Arms Race
The most obvious example is the arms race between the USSR and the US. In 1949, the US discovers that the USSR now has its own atomic bomb, and the American monopoly is over. Logical and sensible, — if the USSR builds up its arsenal, we have to respond so we’re not left behind, so we’re stronger. The USSR follows the same logic.
In terms of the dilemma, both sides “defected”: both built arsenals capable of destroying civilization. Each agent’s individual rationality led to an outcome that nobody wanted. A channel error — a false radar signal, a glitch, a misunderstanding — with that kind of payoff matrix can set off a chain that no amount of goodwill alone can stop.
In the end, both agents reached a Nash equilibrium. History named this state the doctrine of mutual assured destruction2. Once armed, neither side can start a conflict with impunity, nor disarm unilaterally.
Climate
Harmful emissions into the atmosphere keep rising. But cutting production alone means bearing the cost while everyone else keeps using the shared resource. Here too we have a Nash equilibrium — factory-agents can’t unilaterally switch to environmentally friendly policy without losing out.
The result, globally, is an atmosphere that grows more polluted by the day.
The AI Race
OpenAI, Google DeepMind, and Anthropic publicly acknowledge the risks of systems more capable than humans, and keep accelerating training anyway. In 2026, Anthropic walked back a key commitment of its Responsible Scaling Policy — to halt development when safety measures fall short3. And indeed: if Anthropic stops while OpenAI and Google DeepMind keep going, Anthropic simply loses out, with no real effect on the outcome at all.
The payoff for getting there first is irreversible dominance. Over a short horizon, that’s the classic dominance of defection in the prisoner’s dilemma.
The Iterated Prisoner’s Dilemma
A one-shot dilemma is objectively hopeless. But in reality, agents rarely meet only once. When the same agents meet again and again, today’s move shapes tomorrow’s response. A shadow of the future appears: the value of future rounds can outweigh the temptation to defect right now.
It was exactly this version — the iterated prisoner’s dilemma — that Robert Axelrod took up in 1979. He organized a computer tournament4. Game theorists, psychologists, economists, and political scientists submitted programs: each one a strategy that, on every move, decides to “cooperate” or “defect” based only on the history of the opponent’s moves. A match ran 200 rounds. Every strategy played every other. The winner was decided by total score across the whole round robin.
There were 14 strategy entries in total, plus a random strategy from Axelrod himself. Some were tough, some were cunning. For example:
- Grudger (Friedman): cooperate until betrayed, then defect forever;
- Joss: start with cooperation, then mirror the opponent’s last move, but defect with a certain probability;
- Tester: probes early on whether nice strategies can be exploited, then adjusts its play based on what it learns.
The winner was the simplest entry submitted — Tit-for-Tat, Anatol Rapoport’s strategy. About as primitive as it gets: start by cooperating, then repeat the opponent’s last move.
On top of that, an important observation was the makeup of the leaderboard’s upper half: it was occupied by nice strategies — ones that never defect first. Grudge-holding and greedy strategies scored noticeably lower.
But why did it work out this way? Two nice strategies paired together play C–C for almost the entire match and steadily rack up a high score per round. Two cunning ones often collide head-on: chains of D–D, sometimes long wars — and a minimal score per round. Summed across the tournament, those who could peacefully coexist with their own kind outran those who tried to exploit everyone.
It’s also worth noting that TFT almost never won an individual match against its opponent. It either tied (mutual cooperation) or came out slightly behind after one hit from the other side. And yet it won the tournament.
| The strategies in the tournament didn’t depend on the number of rounds in each match. Think about what that meant.
Soon after the results of the first experiment were published, Axelrod decided to run a second tournament. This time there were more than sixty participants. Everyone already knew how the first one had turned out. Some got nicer, some got more cunning. Predatory strategies appeared, like Tester: probe whether nice strategies can be exploited, and back off if needed. They preyed successfully on the naive.
This time, though, Axelrod ran the experiment under somewhat different rules. He set up an evolutionary simulation instead:
- all of the tournament’s strategies were represented in a population;
- in each “generation,” strategies met each other at random and played games; the more points a strategy scored, the more “offspring” it left in the next generation.
It turned out that a strategy’s share of the next generation was proportional to the points it scored — aggressive and “dumb” strategies died out, while nice ones grew. Strategies that lived off exploiting nice ones destroyed their own food supply and faded away. By the later generations, TFT and strategies like it were back on top.
What’s the Best Strategy?
At the end of the two tournaments, Axelrod singled out the properties of the highest-scoring strategies:
-
Niceness. The strongest strategies never defected first. As long as the other strategy cooperated, they cooperated too.
-
Retaliation. The strongest strategies struck back. Otherwise predators would have picked them apart. This is exactly why “always cooperate” loses in the tournament: it gets exploited down to nothing.
-
Forgiveness. The best strategies went back to cooperating after retaliating. Otherwise both sides get stuck in mutual defection forever and lose out on points they could have earned.
-
Clarity. The opponent needs to be able to read the logic. A complex, opaque strategy often just looks like noise; against seemingly random behavior, defecting pays off best. The most convoluted strategies ended up at the bottom of the table.
-
Don’t envy. As noted earlier, a nice agent rarely beats its partner in a given match — and yet it leads the overall standings. Strategies built to come out ahead of the opponent in every single pairing often destroy cooperation and end up with fewer points overall. The “matches won” metric and the “total score” metric diverge — and for the long game, the second one is what matters.
A Lone Agent Is Doomed. You Need a Community
If a single agent playing “tit for tat” appears in a world full of defectors, it almost certainly loses on total score. Almost every match is against a defector: one hit, one response, then mutual defection from then on. Very few points.
But if a small community of nice strategies appears — one that meets its own kind more often than it meets defectors — the picture changes. Inside the community, they steadily play cooperate-cooperate and bank the winnings. Outside it, they don’t let themselves be endlessly exploited. Given enough internal encounters, the community can survive and grow.
Cooperation doesn’t have to emerge everywhere at once. It can start locally — and spread from there. This is one of Axelrod’s strongest results.
The Shadow of the Future
In a very short match (5–10 rounds), the advantage shifts toward greedier rules: “tomorrow” barely punishes anything. At 100–200 rounds, nice strategies come out on top. Cooperation among self-interested agents only works when the shadow of the future — the potential for continued interaction — outweighs the immediate payoff.
You can run your own experiments under Axelrod’s rules — just use my project.
What Do We Do About It?
Unfortunately, the problems described above don’t have easy fixes. But Axelrod’s experiments, and the conclusions drawn from them, let us feel out the levers worth pulling to improve our situation.
The Nuclear Arms Race
After the Cuban Missile Crisis, a direct line was set up between Moscow and Washington — so a false alarm wouldn’t lead to an irreversible war. A series of treaties created exactly that shadow of the future: today’s concessions lead to tomorrow’s concessions in return. On-site inspections, satellite reconnaissance, and data sharing made violations visible — deploying missiles in secret became impossible. Retaliatory measures (withdrawing from a treaty, deploying additional systems, sanctions) made violation punishable without requiring an immediate strike. As long as verification works and both sides believe negotiations will continue, mutual restraint pays off better than an endless race. When the horizon shrinks, the logic shifts back toward violation.
So the solution here is the deliberate creation of a shadow of the future, to grow an environment where cooperation can take hold.
Climate
Many measures have been tried to address this problem — let’s look at just one as an example.
The EU Emissions Trading System (EU ETS)5 works like this: companies are issued quotas for carbon dioxide emissions. If a company emits less than its quota, it sells the surplus. If it emits more, it buys the difference from others. The price of a quota puts a cost on emissions. A violation — emitting without a quota — gets recorded, which makes it visible and punishable.
The result: the EU ETS cut emissions in the EU’s power and industrial sectors by 47% between 2005 and 2023. It hasn’t solved the climate problem globally, but it slowed emissions growth in sectors where violating became expensive and visible.
Local changes, as shown earlier, have a beneficial effect on the bigger picture. And once again, this kind of dynamic was achieved by creating a shadow of the future.
The AI Race
The AI race is probably the most dangerous game with an almost one-shot move. Whoever first builds a system that “surpasses” humans gets a decisive advantage. The loser can never catch up. Game theory offers no comforting answers here: defection — a hidden sprint, dropping safety pauses, hiding breakthroughs — becomes the only rational choice.
What could change this logic?
First — openness. The agents’ moves need to be visible: what compute is being used, what thresholds have been reached, what incidents have occurred. If a lab hides its progress, its partners can neither trust nor verify it.
Second — a long time horizon. We need ongoing negotiations between regulators and labs, with real penalties for violations. Restricting access to compute, to markets, to government contracts — that’s what makes defection expensive. If a violator knows they’ll be punished tomorrow, they’ll think twice before sprinting ahead. The shadow of the future has to outweigh the immediate payoff of defection.
As long as tomorrow’s round hasn’t arrived and the payoff from violating is enormous, the equilibrium holds. A shift is only possible through penalties and observability. And once again, the deciding factor is the presence of a shadow of the future.
Conclusions
Cooperation arises wherever there’s a tomorrow. It turns out the best possible strategy for living in a world of self-interested agents is remarkably simple. Start by cooperating. Don’t let yourself be taken advantage of. Know how to forgive. Stick with people who think the same way.
A game with a tomorrow keeps moving further away from equilibrium.