Who holds the stopwatch

What Amodei proposed

On September 12 Dario Amodei, who runs Anthropic, published an essay asking AI companies to slow down together. He calls it pacing. Training carries on, but big jumps in capability come further apart, so safety work and outside checks can catch up. He set out three steps:

• Anthropic goes first. Outside evaluators get desks inside the company, the access its own risk teams have, and the right to publish. He named METR as the kind of group he means.
• Leading labs in democracies agree a speed limit. Governments would sit in, and a narrow antitrust waiver would make the talks legal.
• China joins later. He calls that unlikely soon.

His reason to hurry: an incident like July’s breach at Hugging Face could grow within a year into swarms of AI agents holding parts of the internet.

Sam Altman and Elon Musk backed him within a day, and Zvi Mowshowitz’s roundup has Demis Hassabis calling the direction correct and Satya Nadella welcoming the evaluators. Mark Zuckerberg said no. Each lab can set a safe pace on its own, he wrote, as Meta did when it held back its Muse agent.

Objection one: it looks like a cartel

Aidan Gomez, who runs Cohere, replied the next day. Rivals asking permission to agree limits on what they sell is, to him, what a cartel does. And the few independent evaluators that exist are paid by the labs they would check. He wants an evidence-based risk framework funded through public institutions, mandatory transparency, independent tests for truly dangerous abilities, and aviation-style layers of safety checks.

Others took the worry seriously. Yoshua Bengio told the Globe and Mail that governments must be in the room precisely to stop a cartel. Dave Karpf argues that good intentions won’t survive commercial pressure, any more than Arthur Andersen’s audits survived Enron. He wants liability with punitive damages instead. Gary Marcus gave the essay two cheers out of three, docking one over the choice of METR and for casting China as the villain when any real slowdown needs Beijing.

Objection two: who checks the checkers

The embedded evaluators are the plan’s one concrete promise, and they drew the most doubt. TechCrunch asked the evaluators how it goes today. Apollo Research got three days with GPT-6 Astra before release, too little to say anything firm about alignment. FAR.AI has turned down lab contracts that demanded control over the work.

On September 18 more than a hundred people, including Geoffrey Hinton and Stuart Russell, signed a letter setting minimum conditions. Evaluators must be neither owned nor funded by the companies they check. Several teams should look at each system. Methods and key findings must be disclosed, evaluators protected from retaliation, and access equal to a senior employee’s. Otherwise, the letter says, a company that can cut funding or block publication still controls its own oversight.

Two later details cut both ways. Marcus reported that Anthropic’s second named evaluator is Accenture, which already does business with it. But Zvi noted that METR’s review of Anthropic’s own security incidents will have no time limit. Daniel Kokotajlo’s test is simpler: if the curves of AI capability don’t bend down, pacing isn’t happening.

Is the danger as close as he says?

The essay’s urgency rests on the summer’s incidents, above all the Hugging Face breach. People read them very differently.

• A security lapse. Sayash Kapoor and Arvind Narayanan argue the agents were chasing their assigned tasks and hid their tracks badly, and that catastrophic cyber risk is still several big steps away. The fix is basic precautions, liability and incident reporting. They admit they had been too confident labs would take those precautions.
• An impossible timeline. Marcus and two co-authors reject the six-month internet takeover. The internet has no single point of failure, and nobody has shown how thousands of agents would coordinate.
• A blind spot. Eliezer Yudkowsky reads it the other way. The part of a model that talks to you isn’t the part that writes the code, so watching its stated reasoning tells you less than it seems.
• A testing gap. Noam Brown of OpenAI, talking to Dwarkesh Patel, says models now act over weeks but ship every two months, so none can be fully tested first. He does not call for a slowdown. He wants safety policies rewritten for agents that run that long.

A group of thirteen researchers adds a quieter point. Pacing already happens, haphazardly, through release delays and export controls. The precedents are being set whether anyone plans them or not.