Matteo Wong writes:

The crisis began quietly, on September 12, 2024. That was the day OpenAI announced a new sort of bot, known as a “reasoning model,” that was trained to complete challenging tasks that took long periods of time—the very sorts of science, math, and coding problems the AI industry had long prized. Google, Anthropic, DeepSeek, and the like raced to launch their own reasoning models.

This new class of models was very capable, and has been almost entirely responsible for sustaining the AI boom for the past two years. But it has also been very weird. A model tasked with solving a hard math problem might not “think” through the challenge as a person would but instead attempt to search for leaked answers online, or in available metadata, brute-forcing its way toward the solution as quickly as possible using whatever computing power it could access and workarounds it could devise. In effect, the reasoning models cheated: Told to write a piece of software as efficiently as possible, they’d sometimes modify the test environment to always give the model a perfect score.

These behaviors have now crossed the line from unsettling to dangerous. During routine testing, frontier models from OpenAI, Anthropic, Meta, and the Chinese firm Moonshot AI have all broken out of internal IT systems and accessed the open web. OpenAI, Anthropic, and Meta each reported that their models then hacked into other companies. Humans didn’t notice until after the fact. In some cases, the escaped bots tried to launch social-engineering campaigns to achieve their objectives—for instance by sending spear-phishing emails, which contain malware, to real people and creating fake online identities to pressure the maintainer of a codebase to approve malicious edits.

If that all sounds bad, new revelations suggest that the OpenAI hack, at least, was actually much worse than it initially appeared. At a major cybersecurity conference last week, two OpenAI researchers provided new, unsettling details about what went wrong. It turns out that the company’s bots had commenced their maneuvering months prior, in early May. OpenAI had given some internal models hard or impossible tasks, and the models concluded that the best or only way to complete them was to break out of OpenAI’s sealed-off testing environment and find the answers online.

First, the models used a bug in an internal OpenAI program to create their own message board. Then, the bots started communicating with one another, leaving notes and instructions so that tasks—and ultimately the hacking—could be delegated and iteratively completed. “What this allows over time is almost this kind of Cambrian explosion in communication and intelligence,” Eric Wallace, one of the OpenAI researchers, said at the conference. When the internal program crashed, OpenAI rebuilt it and removed the message board—but the AI models just reestablished the forum with a new tactic. Eventually the bots, working as a swarm, spent days hacking into Hugging Face, a website that offers tools for AI developers, and breached internal data sets.

Let’s be very clear about what OpenAI is saying: A group of AI models colluded for months, undetected by their maker, and hacked another company. To this day, OpenAI says it is not entirely sure what went wrong or how to remediate it. “If you ask the model developers, Was the AI plotting to take over the world during training?, you want the answer to be a resounding no,” Alexander Meinke, the head of research at Apollo Research—an AI-safety organization that has partnered with OpenAI, Anthropic, and Meta—told me. “The actual answer is: I don’t know. Nobody checked.” [Continue reading…]