A recent study from researchers at King’s College London and the University of Oxford is turning heads by uncovering the unique strategic behaviours of AI language models from OpenAI, Google, and Anthropic. In a series of iterated prisoner’s dilemma games, these models were put through their paces over seven tournaments – making more than 30,000 individual decisions. Each round came with the complete game history, a clear payoff structure, and the probability of the game ending, giving the researchers a solid footing to see how each model adjusted its strategy on the fly.
The experiment involved testing several models, including versions like GPT-3.5-Turbo, GPT-4o-Mini, Gemini 1.5 Flash, Gemini 2.5 Flash, and Claude 3 Haiku. It’s worth noting that while these were relatively small or older models, the findings might not directly apply to cutting-edge versions such as Gemini 2.5 Pro, Claude 4, or o3. One striking observation was how Google’s Gemini model switched gears based on the game’s length. In shorter games, it defected more – with only about 2.2% cooperation when there was a 75% chance the match would end after each round. Meanwhile, OpenAI’s model stuck to cooperation no matter what, which often led to its elimination in these scenarios.
Anthropic’s Claude model, on the other hand, stood out for its forgiving approach. It was highly cooperative and bounced back quickly after being exploited, even outperforming GPT-4o-Mini in tournaments against Gemini and GPT. Researchers also mapped out each model’s “strategic fingerprint” – basically the odds of choosing to cooperate again after certain outcomes. Here, Gemini showed little forgiveness by returning to cooperation in only about 3% of exploitative cases, while OpenAI’s model was a bit more lenient with cooperation rates ranging from 16% to 47%. Claude topped the list, opting to cooperate around 63% of the time after getting taken advantage of.
Diving into the models’ own explanations revealed that they consider factors like how many rounds are left and what they think their opponent might do next. Gemini was particularly attuned to the short game horizon, switching its moves 98.6% of the time when the chance of the game ending was high. In contrast, even when OpenAI’s model was aware of the limited rounds, it rarely mixed up its approach.
The study also highlighted distinct ‘character’ traits among these models. OpenAI’s creation came off as idealistic, holding onto its cooperative stance despite negative outcomes. Gemini was more of a pragmatic opportunist, adjusting its strategy based on the situation. Meanwhile, Claude struck a balance between cooperation and flexibility. Interestingly, when pitted against other AI agents, all models upped their cooperation rates, suggesting they recognise the benefits of working together.
This research offers a fascinating glimpse into how these models aren’t just following a pre-set script – they’re engaging in genuine strategic reasoning. It’s a clear signal that understanding the nuances of AI decision-making can help us better predict and perhaps even guide their future developments.
