Karen Hao writes:

Last week, it seemed that OpenAI—the secretive firm behind ChatGPT—had been broken open. The company’s board had suddenly fired CEO Sam Altman, hundreds of employees revolted in protest, Altman was reinstated, and the media dissected the story from every possible angle. Yet the reporting belied the fact that our view into the most crucial part of the company is still so fundamentally limited: We don’t really know how OpenAI develops its technology, nor do we understand exactly how Altman has directed work on future, more powerful generations.

This was made acutely apparent last Wednesday, when Reuters and The Information reported that, prior to Altman’s firing, several staff researchers had raised concerns about a supposedly dangerous breakthrough. At issue was an algorithm called Q* (pronounced “Q-star”), which has allegedly been shown to solve certain grade-school-level math problems that it hasn’t seen before. Although this may sound unimpressive, some researchers within the company reportedly believed that this could be an early sign of the algorithm improving its ability to reason—in other words, using logic to solve novel problems.

Math is often used as a benchmark for this skill; it’s easy for researchers to define a novel problem, and arriving at a solution should in theory require a grasp of abstract concepts as well as step-by-step planning. Reasoning in this way is considered one of the key missing ingredients for smarter, more general-purpose AI systems, or what OpenAI calls “artificial general intelligence.” In the company’s telling, such a theoretical system would be better than humans at most tasks and could lead to existential catastrophe if not properly controlled.

An OpenAI spokesperson didn’t comment on Q* but told me that the researchers’ concerns did not precipitate the board’s actions. Two people familiar with the project, who asked to remain anonymous for fear of repercussions, confirmed to me that OpenAI has indeed been working on the algorithm and has applied it to math problems. But contrary to the worries of some of their colleagues, they expressed skepticism that this could have been considered a breakthrough awesome enough to provoke existential dread. Their doubt highlights one thing that has long been true in AI research: AI advances tend to be highly subjective the moment they happen. It takes a long time for consensus to form about whether a particular algorithm or piece of research was in fact a breakthrough, as more researchers build upon and bear out how replicable, effective, and broadly applicable the idea is.

Take the transformer algorithm, which underpins large language models and ChatGPT. When Google researchers developed the algorithm, in 2017, it was viewed as an important development, but few people predicted that it would become so foundational and consequential to generative AI today. Only once OpenAI supercharged the algorithm with huge amounts of data and computational resources did the rest of the industry follow, using it to push the bounds of image, text, and now even video generation.

In AI research—and, really, in all of science—the rise and fall of ideas is not based on pure meritocracy. Usually, the scientists and companies with the most resources and the biggest loudspeakers exert the greatest influence. Consensus forms around these entities, which effectively means that they determine the direction of AI development. Within the AI industry, power is already consolidated in just a few companies—Meta, Google, OpenAI, Microsoft, and Anthropic. This imperfect process of consensus-building is the best we have, but it is becoming even more limited because the research, once largely performed in the open, now happens in secrecy. [Continue reading…]