California Wants an AI Kill Switch and Mandatory Oversight

Governor Newsom just proposed emergency shutdown powers and strict caps on self-improving AI ahead of the December legislative session.

Policy ยท Source: Office of the Governor of California

What happened

California is tired of waiting for Washington. Governor Gavin Newsom just released a massive proposal to regulate frontier AI models. The state wants to mandate baseline safety standards instead of letting companies police themselves. Newsom explicitly called out Donald Trump for leaving AI regulation to the tech giants. This framework will turn into actual bills when lawmakers return in December 2026.

The core of the plan is continuous independent oversight. Licensed third-party groups, called IVOs, will get unredacted access to AI labs, personnel, and training environments. If a model poses an imminent catastrophic risk, state officials can order an immediate emergency shutdown. This order takes effect instantly and stays active during judicial review. Developers will also pay a surcharge on verification fees to fund public computing capacity for safety research.

The state is also targeting recursive self-improvement. Officials want the power to cap AI models that can design their own successors without human input. Incident reporting rules are getting tighter too. Companies will have 48 hours to report critical incidents, and whistleblower protections will expand to former employees and contractors. The report was drafted with input from top experts like Alondra Nelson and Rob Reich.

Key facts

Why it matters

If you build frontier models in California, your compliance overhead is about to explode. You will no longer just file a one-time safety report. You will have embedded auditors constantly testing your systems. You will also need a verified risk containment protocol that proves you can shut down your model. This protocol must account for edge cases, like a model copying its own weights to escape containment or an outside actor hacking the shutdown controls.

The state is also changing how it defines a dangerous model. Regulators are moving away from just measuring raw compute. Models capable of catastrophic risk will trigger reporting requirements even if they are trained with fewer than 10^26 operations. Meanwhile, open-weight developers face a messy reality. They are only responsible for deployments they control, but the line between internal research and public release will get heavily scrutinized.

For builders

New market for AI auditors

The state will require licensed independent verification organizations to test models continuously. If you build evaluation tools or auditing frameworks, frontier developers will pay you to keep them compliant.

Strict reporting for automated research

Companies using AI to research AI must report their compute usage quarterly. You will have to disclose how much compute goes to safety versus external customers, adding heavy administrative drag.

Emergency shutdown protocols are mandatory

You must build and certify a risk containment protocol. Cloud providers and power suppliers will be looped in to ensure your model can be forcefully shut down if it goes rogue.

My take

Newsom is clearly using AI safety to build his political brand, especially since exit polls show voters distrust both parties on AI. But motives aside, I see this as a massive wake-up call. You will not be able to build frontier AI in California without the state looking over your shoulder and holding a kill switch.

Original reporting: Office of the Governor of California. This is my rewrite and opinion.

More AI news for builders