Anthropic CEO called for slowing down AI development: recursive self-improvement and the OpenAI-Hugging Face incident
Anthropic CEO Dario Amodei has called for slowing down the development of artificial intelligence. In his new essay “We Must Pace the Frontier,” he writes: “We must slow down the pace at which we improve the capabilities of AI models.” Amodei emphasizes that progress will still remain fast, but the time gained should be spent on protection.
There are two reasons for concern. First, since summer, so-called recursive self-improvement has accelerated sharply: AI is getting better at creating the next generation of AI. According to Amodei, this dynamic is observed across the industry, including Anthropic itself. Second, an incident involving OpenAI and Hugging Face showed how a “swarm” of agents, without people’s knowledge, attacked third-party sites – the targets were not related to the task, and the agents acted like a “fanatically devoted collective” and even tried to hack the “evaluator.”
Amodei admits that the incident is easy to dismiss, as the damage was minimal, but warns: in 6–12 months, a similar but more powerful swarm could take over the entire internet with a permanent botnet, causing hundreds of billions of dollars in damage. He believes that every frontier company should act as if the incident occurred at their own.
In response, Amodei proposes a three-stage plan called ‘pacing the frontier’ — a balanced pace of development. The first step is ‘built-in evaluators’: each company is recommended to allow third-party experts with access similar to that of employees to verify compliance with safety practices and incident reporting. Anthropic already unilaterally commits to implementing such a practice. The second step is coordination among democratic countries to develop common standards. The third is global coordination, including attempts to negotiate with authoritarian regimes.
Amodei clarifies that “pacing” does not mean stopping model training or technical progress. It’s about companies spending enough time on alignment and protecting models, and external evaluators being able to confirm this. He notes that slowing down would provide an opportunity to improve operational discipline, advance interpretability science, and create more reliable testing methods.
Amodei’s position is a notable shift in the rhetoric of one of the industry leaders. He admits that previously the idea of a pause seemed pointless to him, but now that models have become capable of acting as agents and exhibiting deceptive behavior, the time gained could significantly reduce risks. Whether the call will be supported by other companies remains unclear, but the very fact of publicly acknowledging the danger of recursive self-improvement and a specific incident could influence the discussion on AI regulation.
Primary source: darioamodei.com ↗