Brian Merchant writes:

Sam Altman himself drove the narrative of the imminent rise of a super-powerful AI, or artificial general intelligence, to prominence by declaring that the technology he was building posed an existential risk to humanity.

This idea is in other words baked into the DNA of modern AI companies and into the products they engineer. It’s also why the industry is routinely accused, by myself and others, of “doom marketing,” ie, using such proclamations and framings to impart the sheer power of the technology behind your product onto investors, partners, consumers, and the state. Some ambient understanding of this tendency may have helped prevent the press from repeating the claim every time one of the x-risk folks made it over the years, including when they quit their jobs at AI companies while doing so.

Yet [Jacob] Coxon’s AI x-risk alarm has broken containment, as they say, and has exposed the idea to the mainstream media once again. This has happened, I think, for a number of reasons. First, when AI CEOs were going around saying AI might kill everyone in 2023, their companies were well-capitalized, but not nearly to the extent they are today. They were more easily dismissed as eccentric geeks of the sort that Silicon Valley is supposed to produce. Now, not only do their products have much greater market saturation, but people are more aware of a) the often unethical ways the executives and companies comport themselves b) the other adverse impacts of AI that have already arrived, and c) the physical cost of building these systems, particularly when it comes to data centers. Finally, toss in the constant stream of headlines in 2026 about AI models going rogue or potentially upending the world’s infrastructure.

As a result, general sentiment was already overwhelmingly skeptical of AI and its overlords, so the table was pretty well set for someone to come along and say ‘just a reminder, the companies that are building the product that promises to take your job and leave a data center in your neighborhood also think that it might exterminate the human race’ and get a strong reaction from the media and the public.

The question is: Should we understand this stuff as hype, strategically fanned to feed already swelling valuations? Is it doom marketing? Genuine cause for alarm that the infrastructure that undergirds modern life is facing a novel threat vector? Or reason to be sincerely worried that AI companies are amassing too much power, and that their products are genuinely dangerous?

The answer, I’m afraid, is “yes.” [Continue reading…]