I don’t plan to make a habit of critiquing specific pieces or writers, but Rogé Karma’s latest piece for The Atlantic is so maddening that I’m making an exception.
The title and subtitle give a fair sense of Karma’s thesis:
Karma’s argument can be summarized as follows:
AI poses catastrophic risks.
Slowing or stopping data centers from being built in particular communities may address “local environmental concerns” but would not address the catastrophic risks.
Therefore, anti-data-center efforts are a “distraction.” People should instead build a global movement for AI nonproliferation, akin to the nuclear nonproliferation movement.
But (1) is a dubious statement and (2) ignores the very-much-not-dubious concerns driving grassroots opposition to data centers so (3) ends up making Karma sound like a condescending, out-of-touch elitist who has spent lots of time absorbing AI hype but little (if any) trying to understand and empathize with the concerns underpinning the anti-data-center movement.
To keep this piece reasonably short, I’ll focus on a few particular lines from Karma’s piece that stood out to me when I read them.
The first half of the piece is an attempt to convince people that AI poses apocalyptic-level risks. That starts with the article’s opening:
America’s biggest AI companies have spent the past few years churning out advanced new models as fast as they possibly can. Now the industry is screaming out with a nearly unified voice: Please, for the love of God, make us slow down.
That is a fair, if hyperbolic, characterization of some recent statements by AI companies’ executives. But it is foolish to take those statements at face value. The track record of Big Tech and venture-capital-backed companies is not that of people who care about the public interest or laws designed to protect it. Their record is instead one of particularly aggressive profit-maximizing corporations who show, at best, callous and reckless disregard for the harmful effects of their products, if not a conscious desire to inflict those harms, and view the law as something to be, in order of preference, exploited, ignored, or attacked.1
A more plausible interpretation of the recent “slow us down” statements would be: “We really need to attract money from investors and lenders, and, right now, rising public anxiety about AI means that an effective way of doing that is to make our models sound super powerful and scary. Please, for the love of God, signal-boost our scare mongering so that we can get more money.” Karma duly obliged.
A few lines later, Karma decided to engage in some hype-fueled AI scare mongering of his own:
Plenty of others in the industry have warned that we’ve already crossed a threshold from which there is no turning back. They worry that the most dystopian fears of an AI future—chatbot-generated bioweapons, major hacking attacks on banks or governments, rogue AI agents wreaking havoc on society—could soon become regular occurrences.
Leaving aside for the moment what exactly a “chatbot-generated bioweapon” is…
…actually no, let’s not leave that aside. Karma did not provide a link to explain what he meant by “chatbot-generated bioweapon,” so it’s unclear what exactly he was going for other than “I want to make AI risks sound as scary as possible, and I need to come up with something scarier-sounding than automated cyberattacks and more concrete than ‘rogue AI agents wreaking havoc on society.’” “Chatbot-generated-bioweapon” certainly sounds concrete and scary, but the hard part about building bioweapons has never been that no one knows how to formulate or even produce one—dark web marketplaces to obtain the components for bioweapons have been around for years.2 The hard part is getting the money, materials, equipment, and array of trained experts necessary to actually produce them in the physical world.
That points to the deeper flaw in Karma’s scare-mongering: all of the harms he mentioned are not really risks associated with AI, but with human actors. LLMs do not think, understand, or have intentions. They do not, on their own, decide to send phishing emails, launch cyberattacks (much less bioweapons), or go “rogue.” It’s true that the cybersecurity landscape in the coming years will be hellish because LLMs are great at helping humans launch powerful cyberattacks but not so great at helping them write secure code. But like other software used in cyberattacks, LLMs are only capable of doing dangerous things if there are malicious, or at least grossly negligent, humans that direct it to do those dangerous things and/or fail to put reasonable safeguards in place to stop them.
That is what what happened with recent incidents (including the UK AI Security Institute incident that Karma mentioned) where bots3 supposedly “broke containment” and hacked third parties. Despite the credulous media coverage that anthropomorphized the bots, the failures in each instance were thoroughly human. In each case, humans directed the bots to find and exploit security vulnerabilities while providing them an environment that contained security vulnerabilities. In the OpenAI “hack” of Hugging Face that grabbed so many media headlines, the researchers also did a blatantly awful job of monitoring the bots. Indeed, the weakness of the safeguards makes it more than plausible that Anthropic and OpenAI were low-key hoping the bots would break “containment” so they could get the resulting media attention from a hack. Which, of course, they did.4
That leads us back to why Karma wrote this piece in the first place. His thesis is that countries need to pass “an AI non-proliferation treaty” to address these catastrophic risks and that merely focusing on restricting data center construction is thus a (to use the word from his headline) “distraction.”
He elaborates on this point further down in the piece:
Unfortunately, the main form of AI-related legislation being proposed at the moment would not get the job done. State-level data-center moratoriums, such as those enacted recently in New York and Texas, might slow down AI companies a little bit by disrupting their construction plans, but they will eventually find other places to build. Even if Congress decided to pass a federal ban, AI companies could just set up data centers in other countries.
The point slapped Karma in the face and he still missed it.
Data center protestors are not stupid. Of course they realize that if they stop data centers from being built where they live, the AI companies will just build the data centers elsewhere. But making them find other places to build is the whole reason for the grassroots opposition to data center construction. The people showing up to their local council meetings to protest data centers are not there because they want to disrupt the coming of the robot apocalypse. They are trying to prevent water-guzzling, cost-hiking, grid-straining, noisy, polluting facilities from being built in their proverbial and often literal backyards.
A Gallup poll from this spring asked Americans whether they would support or oppose the construction of a data center in their area. 61% of respondents said they would oppose it. Those 61% were then asked to provide some of the reasons for their opposition. The most-mentioned reasons related to effects on resources (50%), quality-of-life concerns (22%), effect on costs (20%), and pollution (16%). Only 4% mentioned “Scared of AI/Don’t trust it/AI will take over,” which would seem to be the appropriate category for “robot apocalypse.”
At the very end of the piece, Karma seemed to acknowledge that more immediate concerns were the animating force behind data center development. But rather than empathizing with those concerns, he demeaned them:
Meanwhile, according to the latest polling, 75 percent of Americans, including the majority of Republicans, oppose data centers being built near them. More than 500 localities across the country have either restricted or banned them. Could this energy be channeled into a more sweeping—and more effective—framework for AI regulation? So far, the data-center backlash has mostly focused on local environmental concerns, not great-power diplomacy. But then again, that’s exactly how the anti-nuclear movement began.
First, the anti-nuclear movement did not start over “local environmental concerns.” Concern about nuclear technology started after many people saw the most powerful weapons in history kill over 150,000 people at the end of World War II. The first significant anti-nuclear protests occurred after 23 Japanese fishermen suffered radiation poisoning after a U.S. nuclear test. The first anti-nuclear organizations and truly mass protests were pacifistic in nature, focused on nuclear war and weapons testing, not “local environmental concerns.” The latter only became part of the milieu later, after civilian nuclear power plants started popping up.
More importantly, however, the condescension in this final passage is unbearable. “Local environmental concerns” makes it sound protesters are worried about whether data centers will block their view of a local creek rather than cost of living, noise and air pollution, and access to essential resources. The overall tenor of the last paragraph, and really the piece as a whole, is: “It’s too bad all these peasants are so focused on trivial concerns like the quality of life in their community. But maybe their energy can be useful if it can be ‘channeled’ into a crusade to stop chatbot-generated bioweapons.”
The fact that Karma views the data center fight as a distraction from stopping chatbot-generated bioweapons, rather than the other way around, is a sad illustration of just how out-of-touch the AI-pilled intelligentsia has become.
Matt Scherer is a fellow at the Open Markets Institute, where his research and advocacy focus on developing policy responses to the eventual bursting of the AI bubble. His Hard Reset pieces focus on highlighting the risks posed by the AI bubble and pushing back against the hype that is inflating it. The opinions expressed here are solely his own.
Karma also pointed to an open letter signed by a thousand-plus employees of AI companies. I am somewhat more reluctant to attribute ill intent to those employees, but it’s worth pointing out that the employees of AI startups, even moreso than the employees of other companies, have a vested interest in boosting narratives that help their companies financially because equity is the key part of most startups’ pay packages. To quote Hubbard’s Corollary to Hanlon’s Razor, “Never attribute to malice or stupidity that which can be explained by moderately rational individuals following incentives in a complex system of interactions.”
I try to use “bot” rather than “agent” to refer to these systems because I think “agent” unduly anthropomorphizes them.
In a tacit admission that phishing emails and cybersecurity tests gone wrong are not horsemen of an AI apocalypse, Karma also says:
Companies are entrusting more and more model development to their existing AI, creating a feedback loop that they expect to accelerate progress even further. If that happens, many experts believe, then the kind of leaps that currently occur every several months—such as the one that produced Mythos, the Anthropic model with superhacking skills—could begin happening every few weeks or days.
Karma does his best to make it sound like this is a new fear, but it’s really just the latest iteration of “the singularity is near” prophecies that very-online AI doomers have trotted out for years. For critiques of such magical thinking about AI, see this piece by Margaret Mitchell, this one by Adam Becker (from The Atlantic, no less), and, for more extended treatments, chapter 3 of Becker’s More Everything Forever and chapter 16 of Mitchell’s Artificial Intelligence: A Guide for Thinking Humans.
An example I'd like to borrow is the perchlorate contamination of groundwater issue, common during the late cold war era. On one hand the nation justifies these occurrences as national defense necessity; on the other hand whatever justifications that be, there is no justification for poisoning people's food and water and subjecting them to diseases.
Thus I say geopolitical and community concerns are two aspects of national AI mobilization, not two levels. The word "level" would invite the slippery thinking that higher level concerns ought to overrule lower level concerns. It is not necessarily true in every case, and not true in the case of today's AI construction spree. That slippery thinking was the norm, at the height of the cold war. This is why these lessons become the mandatory knowledge of university students, so that we know not to repeat these mistakes.
On the first point, it make sense, and in general I get the problem with thinking AI is magical, but I don't see what opposition one can have generally to the proposed non-proliferation framework? What is your general position about that?
"But making them find other places to build is the whole reason for the grassroots opposition to data center construction." But where are those other places? Climate apocalyse is a real possibility, even a reality now, even if superintelligence apocalypse is not, and data centers are going to accelerate it, or make things worse with respect to the environment.
So to base everything on what 'the people want' and what their reasons for opposing AI centres are, doesn't seem like a great idea to me. My guess is that not many in Wisconsin will be protesting against data centres being built in other parts of the world, where the people have lesser political agency, and those are going to mess the whole world up anyway, it seems like..
Non-proliferation of data centres that essentially no people in the world are asking for, or need, is a good idea maybe? Or are the climate concerns around data centres also unfounded?