Thank God it’s Sam Altman, not Dario Amodei, who’s reportedly joining a state dinner next week as part of Chinese President Xi Jinping’s summit with President Donald Trump.
To be fair: I wouldn’t invite any of our AI overlords to a high-stakes meeting with the leader of the world’s other superpower. I’m just realistic about how Trump collects oligarchs like they’re Pokémon cards. If Big Tech must be at a dinner discussion centered on growing concerns about AI killing everyone, I’d rather it feature Altman, the people-pleaser who’s famously “unconstrained by truth,” over Amodei, an apparently aspiring diplomat who just chortled to CBS News that he has “a hawkish position on China.”
Four days after former Anthropic researcher Jacob Coxon’s viral posts about how AI development is dangerously accelerating, Amodei released a manifesto called “We Must Pace the Frontier.” Previous Amodei manifestos have served as a pacifier for angsty AI followers; they’re well-read, but mostly among an insular crowd. My sense is this latest manifesto garnered a much larger audience, and for good reason: even passive observers of the AI industry are worried about what they’re reading and hearing. Amodei has fashioned himself a proponent of AI regulation and safety, and if you don’t squint very hard, he’s certainly better on those topics than his rivals.
First, the good news: “We Must Pace the Frontier” seems to have spurred dialogue among Amodei, Altman, Elon Musk, and Demis Hassabis. These four men do not share an affinity for one another; I wish they didn’t have unlimited power, but they do, so I’m glad Amodei is encouraging them to coordinate on matters of life and death. (Mark Zuckerberg is also trying to enter the fray, but I am not sure Meta’s frontier models matter enough.)
I’m content with another proposal in Amodei’s manifesto: having embedded, independent evaluators assess safety practices and emerging models. It’s the sort of thing that Congress should require, but Congress does not pass laws anymore. I’ll take what I can get, which is to say these evaluators will almost certainly run up against institutional roadblocks that we may or may not hear about in a reasonable amount of time.
Where Amodei loses me—where he always loses me—is his saber-rattling with China. The founder of a tech company should not be making diplomatic recommendations on behalf of the United States. That’s doubly true when the recommendations inflame tensions with China, the country that the United States most needs to work with on matters of AI safety and catastrophic risk. As Amodei sees it, any pacing slowdowns among American AI companies should be “limited by the lead” that those companies have “over authoritarian regimes, chiefly the Chinese Communist Party.”
According to Amodei, “If we slow down by more than this amount, then (unpaced) CCP-associated projects will pull ahead, creating significant national security risk.” Normally, the phrase “national security risk” is bandied about without offering specifics. To Amodei’s credit, he does say what he means a few sentences later: “The CCP-associated projects will run the alignment risks that US companies are carefully preventing, and even if they avoid those risks, they will be in a position to militarily dominate democracies (for example with AI-driven drones).”
Amodei is implying, without evidence, that people in China are unconcerned about AI alignment risks (losing control of an AI system, basically). In an ensuing CBS News interview, he was only willing to grant that America and China are probably capable of signing a treaty that bans the use of AI in the production of bioterrorism weapons. Somehow, his bare-bones concession was loaded: “Even the Chinese don’t want that,” he said of bioterrorism. What do you mean “even,” Dario?
Amodei additionally makes the dubious claim that American companies are “carefully preventing” misalignment, which is patently untrue. Researchers like Coxon would not be speaking out otherwise. OpenAI literally just admitted to six more examples of “concerning” behavior from its AI models. It also released a framework about if, and when, it will disclose future incidents of AI misalignment. There are a whole bunch of disturbingly convenient carve-outs in the framework.
As for the threat of “militarily dominating” democracies: Amodei comes across like he’s begging for an AI Cold War. China certainly has the capacity to misuse AI to nefarious ends. Already, there’s ample reporting that the CCP is taking advantage of AI tools to further clamp down on freedom of expression within the country. China’s surveillance network has reportedly been overhauled over the last few years to integrate AI and give local police forces access to invasive “predictive policing” technologies. I have no doubt the Chinese military has integrated AI in alarming ways, too.
But there’s only one country that’s currently relying on AI for military dominance, and it ain’t China. In fact, even while haggling with the Pentagon over a few red line use cases, Amodei and Anthropic remain enthusiastic backers of the U.S. military. Amodei has no qualms about teaming up with Palantir and the Pentagon to help the Trump Administration wage a costly and unwinnable war with Iran; in June, Amodei said he has no idea whether Claude played a role in the indiscriminate killing of 120 Iranian school children. His “red line” is whether an AI model “decides” to do a drone strike, rather than a human pushing the button after receiving intel from AI tools. It’s a weak distinction that hardly makes a difference.
To maintain America’s AI “lead” over China, Amodei recommends not selling AI chips or semiconductor manufacturing equipment to China. He’s calling for a crackdown on “unauthorized distillation by companies in authoritarian countries,” and to “strengthen security at the AI companies.”
Distillation isn’t a national security threat as it’s described by Amodei—it’s a commercial concern. You know who else uses distillation techniques? Musk’s xAI. He said as much on the stand, during his trial against Altman and OpenAI; there’s also the uncomfortable reality that frontier labs have scraped the internet without our collective permission, which makes any distillation whining pretty rich. In its own scary-sounding report about “China-Based AI Companies Targeting US AI Models,” America’s Cybersecurity and Infrastructure Security Agency (CISA) concedes that distillation is “a valid training method.” American tech companies just don’t like how distillation is a corner-cutting method “to acquire capabilities from competitors in less time and with less cost than developing them legitimately,” as CISA notes in its report.
Later in his manifesto, Amodei acknowledges that some people might conclude his hardline positions will make it “more difficult to cooperate with China,” but he believes the “opposite is true: these measures increase the leverage held by democracies and make an agreement more likely in the future.”
Gotcha, thanks Dario. Now over to China to hear how your proposed measures are being received:
“Narratives of threat, confrontation, and malicious competition serve only to disrupt the process of global AI governance and are not in anyone’s interest,” China’s foreign ministry spokesperson Guo Jiakun said of Amodei’s meddling. “All parties should work together to advance AI in a manner that is open, inclusive, beneficial to all, and oriented toward the good.”
And here’s another response, an opinion piece from the state-run Global Times:
“This ‘silent AI Cold War’ is hypocritical and short-sighted… Amodei’s attempt to discuss global AI security while simultaneously seeking to ‘contain China’ was not only a grossly flawed assessment but also a delusion fundamentally impossible to realize. As one of the largest and most open suppliers in the global AI ecosystem, China’s AI capabilities have long ceased to be a variable that can be ‘excluded.’”
It’s eminently reasonable to question China’s strategic messaging and motives here, too. Collaboration on AI safety and regulation will help China catch up to the capabilities of top frontier models in the U.S. But between the two sides, which one sounds more reasonable right now? Amodei can’t even get his story straight. In his manifesto and during his CBS News interview, he brought up highly fraught nuclear disarmament negotiations between the U.S. and Soviet Union as a historical precedent for AI talks. Here’s what happened next.
CBS News reporter: “Is that the way we should be thinking about AI? Using the Cold War as a model?”
Amodei: “I didn’t say we should use the Cold War as a model.”
CBS News reporter: “Well, those negotiations happened during the Cold War.”
Amodei: “I think we should approach China. Again, all of this is above my pay grade.”
CBS News reporter: “But you’re advising the president.”
Amodei: “I mean, if anyone asks me what I think they should do, I’m happy to say.”
Dario Amodei is correct that suggestions about “approaching China” are above his pay grade. My two cents? Go attend another wellness retreat. Breathe deeply and consider that you are neither the U.S. Secretary of State nor the U.S. Ambassador to China. No one has ever elected you or nominated you to a diplomatic role.
Take it down a few notches. No more manifestos or geopolitics for a while. Make yourself semi-useful and clean up your own messes. Coordinate among other tech oligarchs. Prove you’re willing to self-regulate enough so that, at the very least, random researchers of yours don’t quit Anthropic in horror because they’re terrified your company is on the verge of destroying the world.
Here’s what else we’re reading this week:
Universal, which holds the international rights to Alex Gibney’s buzzy four-hour Elon Musk documentary, is reportedly close to chickening out and not releasing the film. “The company has not dated the movie, has not communicated with the filmmaking team about its plans, and generally appears to be rethinking the release of a project that it once showed high enthusiasm for but is now politically sensitive for parts of the Trump administration,” writes The Hollywood Reporter.
This would not affect the Musk documentary’s release in the United States, because Bleecker Street, an independent film company, has the domestic rights and is proceeding with a limited opening on October 9. But it’s an ominous sign for the film’s supposed early 2027 streaming debut on HBO—another entertainment service that’s being yanked around by Trump Administration allies.
Ex-Anthropic wunderkind Jacob Coxon went on Fox News and said he agrees with President Trump that “global warming doomsday scenarios were overplayed.” I remain unconvinced that he is some sort of AI soothsayer.
I really liked Hamilton Nolan’s “Unionize AI” take, which presents a significantly more realistic option for reining in OpenAI and Anthropic than having Amodei and Altman run the show.
As Nolan writes, “There is no reason why a group of several thousand smart, well-paid people cannot organize themselves into an ad hoc union to demand a slowdown on AI research in the name of safety. These people, after all, do the research.”
I think Ryan Broderick of *Garbage Day* summed up the situation rather well:
"If you're having trouble following this, basically imagine the entire AI industry is the *I Think You Should Leave* Hot Dog Car Sketch [https://www.youtube.com/watch?v=WLfAf8oHrMo]. Oh, but, also, I guess they think the hot dog car is god or could become god in the future. And that if they hadn't crashed the hot dog car into the store, China would have. And they would rather Americans be the ones to do it."
(https://www.garbageday.email/p/they-want-you-to-be-scared-of-ai-in-a-very-specific-way)
Never lose sight of the fact that Altman, Amodei, and all their "AI"-boosting brethren are boundlessly dishonest, greedy, and ridiculous.