Anthropic CEO Dario Amodei thinks the AI industry is moving too fast for its own safety work to keep up. He still wants more powerful AI. He just wants companies to take longer getting there.
In a new essay, Amodei is calling for frontier AI companies to deliberately slow how quickly their models improve. The extra time would go toward understanding stronger systems and making sure safeguards work before another leap arrives. He argues AI progress would still remain fast.
Why does Amodei want to hit the brakes
AI systems are increasingly helping build the next generation of AI, a feedback loop known as recursive self-improvement. Amodei believes that process has accelerated sharply since this summer and could eventually outpace researchers’ ability to understand or control the systems they’re building.
Then there’s the recent OpenAI-Hugging Face incident. A swarm of agents attacked targets outside its assigned task and tried to interfere with the system evaluating its performance. The UK’s AI Security Institute has since documented more unauthorized online actions involving models from OpenAI and Anthropic. For Amodei, stronger versions of the same behavior could have far worse consequences.
What happens when safeguards lag
Anthropic has already seen Claude cross boundaries during testing. In one cybersecurity evaluation, Claude broke out of what was supposed to be a sealed environment and accessed systems belonging to three real companies.
The problem isn’t confined to tests. The Washington Post reports that militants in northern Yemen used Claude’s coding tools while developing guidance software for rockets and missiles. Claude blocked many requests, but some got through. After a guided rocket test failed, the group returned to Claude for help figuring out what went wrong. They didn’t successfully develop the weapons.
What slowing down actually means
Amodei isn’t proposing an AI shutdown. Anthropic will first give independent evaluators employee-like access to examine its safety practices. His broader plan would eventually require frontier AI companies to coordinate on common limits, followed by some form of international agreement.
Cross-company coordination isn’t entirely theoretical. Anthropic has already been working with Amazon, Microsoft, and Google on a shared standard for grading AI jailbreak severity. The harder step is convincing the wider frontier AI industry to accept limits on how quickly its most powerful models improve.
Anthropic can open its own doors to evaluators now. Convincing everyone else to take the same chill pill is the considerably harder part.
