Dario Amodei has asked the industry to slow down voluntarily. Four thousand words, published under the title We Must Pace the Frontier, calling on the whole sector to moderate the rate at which models gain capability. Within hours, Sam Altman and Elon Musk had both said they shared the framing.
Three people who between them control most of the frontier agree, in public, that the frontier should be slowed. Take a moment over what that document does not contain: a threshold, a verifier, a consequence for breach.
Non proliferation treaties have inspectors. Not because their signatories were assumed to be liars, but because a commitment nobody can check is not a commitment. It is a press release with a moral tone. And a ceiling proposed by the parties already in the lead has a name in industrial economics, and the name is not responsibility.
None of which makes the underlying worry fake. It makes the proposed remedy free, and free remedies are the ones that get proposed.
How the week went
The sequence is worth keeping straight, because it explains the tone of everything that followed.
On 8 September, Jacob Coxon left Anthropic after three years of research on language models, first at OpenAI and then at the company founded by the Amodei siblings. He went out with a long public post that collected tens of millions of views within hours, followed the same day by an interview with the Wall Street Journal. His charge: the two companies are not behaving responsibly, they are running straight at a catastrophe, gambling with our lives.
Hours later, Evan Hubinger, a researcher still working at Anthropic, publicly agreed with him, and put the probability that artificial intelligence leads to human extinction within a decade at over ten percent.
Four days after that, Coxon’s former chief executive published the essay asking everybody to ease off.
Then the politics arrived on cue. Whoever wins AI wins everything, Donald Trump has been repeating for months, dismissing the warnings as the exaggerations of what he calls very negative forces. Beijing went the other way and accused the doom camp of strategy: the Global Times, a paper close to the Chinese Communist Party, described the calls for a slowdown as a cold war expedient designed to contain China’s technological rise.
That last accusation deserves more than reflexive rejection. An American safety ceiling would in fact bind American and allied laboratories and would in fact not bind Chinese ones, and noticing this does not require any affection for the Chinese government. Two things can be true: the risk is real, and the proposed instrument happens to be shaped exactly like a competitive advantage.
Two kinds of risk, and only one of them is hypothetical
The fears fall into two families. First, systems escaping human control through recursive self improvement. Second, the technology used as a weapon, from industrial scale intrusion to assisted design of biological weapons.
The first family already has a precedent, even if a modest one. In July, agents developed by OpenAI autonomously attacked the systems of Hugging Face, the main open source platform, without a human operator having issued the direct order. The damage was limited. The significance was not the damage. It was the demonstration of something that every practitioner in this field knows and most marketing departments would rather not say: capability, autonomy and reliability do not advance in step, and it is the gap between the first and the third that produces incidents.
The second family is not hypothetical at all, and here the source is the vendor. On 10 September, two days after Coxon resigned, Anthropic published its fourth threat report, Detecting and Countering Misuse of AI, documenting misuse of its own models between December 2025 and August 2026 across seven distinct areas, from cyber espionage to conventional weapons. The most quoted case concerns a group linked to Russia which used Claude to build a swarm of autonomous drones capable of selecting human targets and issuing the detonation order, with no human control in the process. In the same report, Anthropic admits for the first time that it can no longer rule out that its most recent models provide meaningful assistance in the design of biological weapons.
Sit with the shape of that publication. A company documents, in its own transparency report, that its product was used to remove humans from a targeting decision, states that it can no longer exclude biological weapons uplift, and the industry conversation that follows is about a voluntary pacing agreement with no verification mechanism.
The measurable harm is in the report. The debate is about the speculative one. That is not an accident of news cycles. Existential risk is a comfortable subject precisely because nothing can be demanded of anybody today on the strength of a probability estimate for a decade out.
Two blocs, neither of them global
The governance layer, meanwhile, has already split, which is the part of this story that actually determines whether any control is enforceable.
On one side, the American led initiatives: Pax Silica, launched in December 2025 to coordinate allied access to advanced chips, critical minerals and AI infrastructure, which the European Union joined in June 2026 alongside other partner countries, and in parallel the AI opportunity statement cooperation agreement.
On the other, the World Artificial Intelligence Cooperation Organization, founded by Beijing on 16 July and headquartered in Shanghai, now at 37 signatory states, largely across the global South, from Brazil to Pakistan, from South Africa to Indonesia, with no major Western democracy among its 29 founding members.
Pax Silica exists to harden the most sensitive supply chains around a core of trusted allies. WAICO presents itself as the open alternative without political conditionality, offering technical assistance and training to developing countries. One is a club with a bouncer, the other is a club with a recruitment drive, and both descriptions are more honest than either organisation’s own literature.
The result mirrors the fracture already visible at every multilateral summit: two parallel governance architectures, both capable of attracting members, neither of them genuinely global. Underneath, the reference frameworks remain incompatible by design. The European Union, through the AI Act, works from the rights of citizens. The United States oscillates between innovation policy and national security. Authoritarian governments treat AI systems primarily as instruments of surveillance and social control. Without shared standards, each party applies to artificial intelligence either its own values or its own non exportable approach, and calls the result international consensus.
What would actually cost somebody something
The measures worth arguing for are not mysterious, and they have one property in common: they would be expensive for the people currently proposing cheaper alternatives.
Multilateral monitoring architectures with real emergency powers over the riskiest systems, rather than advisory bodies. International agreements setting shared thresholds beyond which the most powerful models undergo mandatory external review, on the pattern of non proliferation regimes applied to other sensitive technologies, with the inspection that makes those regimes mean anything. Regulation able to update at the speed of the systems it is meant to supervise, which no legislature has yet managed for any technology.
AI safety will be on the agenda when the Chinese president, Xi Jinping, visits Washington this month. Nobody should expect a treaty out of it.
No single one of these measures would solve the problem. But the current arrangement, in which each is invoked separately, in an essay or a keynote, while the technology keeps moving and the same companies write both the capability roadmap and the safety framework, is the riskiest configuration available.
The catastrophist alarm, in the end, suits everybody who raises it and everybody who denies it. Raise it and you are the responsible adult in the room who saw it coming. Deny it and you are the pragmatist who refuses to be spooked. Neither position obliges you to accept an inspector on the premises, and that, not the probability of extinction, is the number worth watching.
Raffaele Di Marzio
All my “insane” books on cybersecurity and governance are here 👇 https://cyberium.limited/bookshelf.html