7 min read
Artificial intelligence company Anthropic is pushing for new federal rules that could allow the U.S. government to block the release of certain powerful AI systems. The proposal includes a striking threshold: A model would need to pose an imminent risk of killing or seriously injuring more than 50 people, or causing over $1 billion in damage, before the proposed emergency intervention could apply.
The plan has drawn attention because it would set specific conditions for when federal officials could step in. Anthropic says stronger AI oversight is needed, but the draft also places limits on government authority. The proposal remains under discussion, with no guarantee that Congress will adopt it.

Anthropic representatives have privately discussed a draft artificial intelligence safety bill with lawmakers on Capitol Hill. The discussions became public in an Oct. 16, 2026, report based on information from 3 people familiar with the effort. The company is seeking federal standards for its most advanced AI systems rather than rules covering every chatbot or digital assistant.
The draft builds on Anthropic’s Advanced AI Framework, which the company published in June. That document calls for testing powerful models, publishing safety information, and using independent experts to examine potential risks. It also recommends giving government agencies authority to intervene when particularly dangerous systems threaten public safety.
Although the legislative discussions were initially private, Anthropic has acknowledged its support for the framework. The company says it has been advocating for its proposed policies with federal and state lawmakers since June. That distinction matters because the reported draft is a legislative proposal, not a new safety rule that AI developers must already follow.
Under the reported draft, government officials could seek to block an AI model when there is a foreseeable, imminent risk of killing or seriously injuring more than 50 people. The proposal also provides a separate threshold for potential damages exceeding $1 billion. Either condition could support intervention if the threat falls within the specified risk categories.
The numbers are important because they help define the kind of danger that would justify emergency action. The proposal focuses on severe consequences rather than routine errors, misleading answers, or ordinary software problems. A model producing an incorrect response would not automatically meet the proposed standard for a federal shutdown.
The draft does not mean a company would be permitted to harm fewer than 50 people. Existing laws and other safety obligations would still matter. Instead, the threshold concerns this particular proposed federal process for stopping an advanced AI system. It would not establish a general rule that smaller harms are acceptable or beyond government scrutiny.

The reported legislation identifies three types of threats that could justify an emergency response. One involves AI helping create or release biological weapons. Another concerns particularly severe cyberattacks. The third involves an advanced AI model escaping its developer’s controls and creating a serious risk to public safety.
These categories reflect concerns about what increasingly capable AI systems might eventually do. Advanced models can already perform complex technical tasks, and developers are studying how those abilities could be misused. The draft concentrates on dangerous capabilities that could cause widespread harm if effective safeguards fail.
Anthropic’s broader June framework also discusses AI systems that accelerate the automation of research and development. That makes the published framework wider than the three immediate intervention categories described in the legislative reporting. The difference is significant because a company’s general safety recommendations and the specific powers outlined in a proposed law are not necessarily identical.
The draft would require independent evaluators to examine certain advanced AI systems. These evaluators would register with the U.S. Department of Commerce and periodically assess whether covered developers and their models present serious dangers. Their work would help determine whether a model meets the proposed conditions for government intervention.
AI developers could pay the evaluators for their services, but compensation could not depend on the findings. That restriction is intended to support independent assessments. However, allowing companies to fund their own evaluations raises questions about how the system would protect reviewers from financial pressure.
If an evaluator identified an imminent risk covered by the proposal, the evaluator would have 72 hours after completing the assessment to notify the Commerce Department. Officials would then review the findings and decide whether to seek court intervention. The reporting requirement would create a formal path for urgent safety concerns to reach federal authorities.

The Commerce Department would not have unlimited power to shut down an AI model on its own. Under the proposed rule, officials could present evidence to a federal judge if they believed a covered system posed an imminent danger. The judge would decide whether the legal standard for suspension had been met.
The draft specifically identifies the U.S. District Court for the District of Columbia as the court that could order a suspension. It also seeks to prevent other federal agencies or officials from independently suspending a model’s development, deployment, or internal use under the covered circumstances.
That arrangement would give the court an important role in reviewing emergency requests. Supporters of a defined process may see it as a way to avoid unpredictable restrictions. Others may question whether the required assessments and legal review would move quickly enough as a serious threat develops.
If Congress adopts the proposal, developers of covered AI models could face additional oversight before releasing their most advanced systems. Independent evaluations would play a larger role in identifying risks that might require government intervention.
Companies would also need to consider how their safety testing and internal controls align with the proposed requirements. A model presenting a qualifying danger could face legal restrictions before reaching the public.
However, the framework would not automatically block every AI system that raises safety concerns. The emergency process would apply only when specific conditions are met, leaving other safety issues to existing laws and additional oversight measures.

Anthropic’s proposal arrives as federal lawmakers consider how to oversee increasingly capable AI systems. Some policymakers want stronger safety requirements, while technology companies also seek clear rules for developing and releasing new models. The reported draft attempts to define a specific federal process for handling the most serious potential dangers.
California has already adopted AI safety legislation that uses similar measures of catastrophic risk. Anthropic’s proposed national approach could shape discussions about how federal and state requirements work together. The reported draft also includes provisions that could limit certain state rules, making the relationship between different levels of government an important issue.
For now, Congress has not enacted Anthropic’s proposal. Its exact language could change during negotiations, and lawmakers may choose different standards or enforcement powers. The central question is how much authority regulators should have to act before an advanced AI system causes severe harm. Any final answer will depend on the legislation Congress actually passes.
Don’t forget to follow us for more exclusive content.
If you liked this, you might also like:
This article was made with AI assistance and human editing.
We appreciate you taking the time to share your feedback about this page with us.
Whether it's praise for something good, or ideas to improve something that
isn't quite right, we're excited to hear from you.