Why Are Tech CEOs Sounding the Alarm on the AI They Created?

The executives building America’s most powerful AI systems are calling for the industry to slow the pace of capability gains. Anthropic CEO Dario Amodei argues that safeguards need time to catch up with what the technology can do. Sam Altman expressed agreement and committed OpenAI to giving independent evaluators deeper access to its systems. Elon Musk responded, “Dario is right.” Meanwhile, their companies continue developing and releasing more powerful models, leaving the rest of us scratching our heads about what slowing the pace actually means.

President Trump has pushed back against additional restrictions, arguing that existing laws can address harmful uses of AI. He subsequently announced a Super Intelligence Force to coordinate the federal government’s response. His stated priority is to keep the industry moving:

“We will not in any way hinder or stifle the Growth of this incredible Industry. Rather, we will cherish it, help it, and watch over it, as it grows!”

— Donald Trump

The dispute comes as businesses gain more ways to use AI without buying access from the largest American providers. DeepSeek and Alibaba offer open-weight models, meaning their trained parameters are available for download, although the training data and development process may remain private. A call to slow the industry becomes commercially convenient when competitors are giving customers more reasons to leave your platform.

OpenAI and Anthropic earn revenue by selling access to models they operate. A customer using a competing model independently has less reason to buy that access. If new safety requirements make independent operation substantially more expensive, some customers may decide that paying a large provider is the easier choice. That gives the providers a financial stake in the rules they advocate, even when their concerns about the technology are sincere.

Businesses considering downloadable models face significant costs. Large systems require expensive equipment and enough steady work to justify it. Smaller models handling narrow tasks can run on much less hardware, but someone still has to manage the software and deal with failures. For most businesses, paying a hosting provider is cheaper than running the model themselves. But even when local operation is too expensive, Chinese models accessed through an API can still cost significantly less than leading American providers charge.

The stronger reason to operate a model internally is control. Confidential files can stay inside the company’s network, and the business can retain a working version if the developer changes direction. Its technical team can also adapt the system without waiting for a commercial provider to support the work. Companies may reasonably pay more for that independence.

The concern about releasing powerful models is what happens when someone uses that independence to remove safeguards. Restrictions intended to prevent assistance with dangerous activities can be weakened through modification. If a model has capabilities that could help carry out a major cyberattack or assist biological-weapons development, an altered version could make that assistance easier to obtain. The severity of the risk depends on the capabilities of the particular model.

A hosted provider can suspend an account or update its service when it discovers abuse. Downloaded copies can keep running after the original developer identifies a dangerous weakness. Their owners may refuse an update, and altered versions can continue circulating beyond the developer’s reach. That loss of control helps explain why the debate extends beyond an individual business’s data security.

Some safeguards can also be defeated through prompts alone. A September 2025 evaluation by the U.S. Center for AI Standards and Innovation found the DeepSeek models it tested substantially more vulnerable to known jailbreaking techniques than the American reference models. Those techniques try to make a system answer requests it was trained to refuse. The finding supports scrutiny of those models; it does not establish that every downloadable system presents the same danger.

The law determines who carries responsibility. California’s SB 53 requires frontier developers to report critical safety incidents and imposes additional transparency requirements on large frontier developers, including publishing their approaches to managing catastrophic risk. Anthropic supported the legislation. An ordinary business does not inherit that entire framework simply because it runs a downloaded model.

Broader requirements could change that. A business required to commission expensive evaluations would have fewer users absorbing the expense than a large hosting provider. The effect would depend on whether the rules covered routine internal use or substantial modifications to powerful systems. American open-weight models, including OpenAI’s gpt-oss, Google’s Gemma and Meta’s Llama, could also be affected.

Lawmakers need to identify the activities creating the danger before assigning the bill. Every requirement that raises the cost of independent operation should have a specific safety benefit. Otherwise, businesses could find themselves paying the companies advocating regulation to handle the expense it creates.