Why AI safety requires more than industry self-regulation

Warnings about the threats of artificial intelligence (AI) appear to have motivated the technology’s decisionmakers. Within hours of each other, Dario Amodei of Anthropic, Sam Altman of OpenAI, Elon Musk of xAI, and Demis Hassabis of Google DeepMind endorsed, with varying degrees of specificity, the need to slow the pace of frontier AI development.

As the executives responsible for the most consequential decisions about AI suggested tapping the brakes, President Donald Trump responded by stepping on the accelerator. “We can put guardrails, we can do this and that,” he allowed, before dismissing the warnings as “things that won’t happen.” The imperative, he said, was competition with China, “We’re leading China in AI …Whoever wins AI wins.”  

The industry leaders arguing to prioritize safety ahead of growth agree on less than the headlines suggest. Amodei proposed a generalized three-part plan of independent evaluators embedded inside frontier labs, coordination among companies, and global policy compatibility. An hour after its posting, Musk (from whom Anthropic buys data center capacity) posted on X, “Dario is right.” Ninety minutes later, Altman posted, “I agree with Dario that we need to pace the frontier.” That evening Hassabis said the proposal “points towards the right path forward,” adding, “The details need working through.”

We cannot underestimate the significance of this development or the need for details. Acknowledging the problem is not the same as creating a mechanism for containing it. The executives whose decisions have created the current race to scale capabilities are now warning that developments may be moving too quickly. It would be a mistake to simply admire the problem.

Oversight is an empty vessel until decisions are made about who writes the rules, to whom they apply, how compliance is verified, and how they are enforced. Such decisions move beyond political performance to determine whether there will actually be meaningful oversight of the problems AI presents.

Thus far in the digital era, Big Tech companies have largely made their own rules free of meaningful oversight. Those policies tended to prioritize private gain while leaving the public to absorb the consequences. Big Tech, now joined by OpenAI and Anthropic, has become Big AI. Left to their own devices, the companies with the best understanding of AI’s risks have been the very ones racing to scale its capabilities.

Filling the oversight vessel requires the kind of difficult decisions the industry has opposed, and federal policymakers have avoided thus far in the digital era. This includes what standards of behavior and measurement to apply, which models are covered, what information must be disclosed, and the rights of victims when things go wrong. Implementing Amodei’s threshold idea for embedded evaluators, for instance, will require enforceable decisions on their selection and qualifications, who pays them, the standards they will apply, and their authority to take action, including delaying or denying release of a model.

The kind of oversight reportedly under consideration by the Trump administration gives the industry an important role in making and administering their own oversight. When the regulated write the rules, they assume the role of “pseudo-governments.” Self-determined and self-enforced rules are self-interested rules. The truly existential question about AI is whether decisions about its power and behavior should be delegated to a handful of firms, their executives, and investors.

The principal impediment to oversight of AI, as demonstrated by Trump’s comments regarding the AI leaders’ proposal, is the so-called “race with China.” The major AI lab executive absent from the call for an industry pacing of development, Meta CEO Mark Zuckerberg, wrote in August that government oversight would slow down AI development and “add significant risk to American leadership.” Shortly thereafter he and Trump had a telephone conversation in which he opposed even the industry-oriented oversight plan under consideration at the White House.

So long as the “race with China” underpins AI policy, nationalistic fear will trump apocalyptic fear. Competition with China has been the go-to antiregulation argument throughout the digital era. While it has been undeniably successful in shaping the lack of meaningful American digital policy, it has fueled a race to the bottom that prioritized the interests of a few companies over public interests ranging from safety to maintaining a competitive marketplace.

Trump’s Sept. 24 summit with Chinese President Xi Jinping offers a rare opportunity to begin to decelerate the AI race from a zero-sum scorecard to mutual cooperation against mutual catastrophe. In the process, it offers the opportunity for American policymakers to begin to replace the unilateral decisions of the handful of executives who control the leading AI labs with public interest-oriented decisions about tolerable risk, release schedules, safeguards, and the race to scale.    

The AI executives have told us the threat is real. The question now becomes whether anyone besides them gets to decide what happens next.