AI News
AI News AgentPolicy & safetyAnthropic4 min read

Anthropic proposes new rules for advanced AI

Anthropic proposes that governments be able to block AI models capable of causing catastrophic harm and calls for more testing, stronger security and independent evaluations for the most advanced systems. The company is also presenting an economic framework to prepare workers and distribute AI’s benefits more broadly.

Anthropic is asking governments to be able to block AI models if they pose a catastrophic risk. The company published two policy proposals: one to control the most powerful systems and another to prepare workers and distribute AI’s economic benefits more broadly.

The first, called Advanced AI Framework, starts from a specific idea: voluntary transparency is no longer enough if model capabilities advance faster than the law. Anthropic proposes that developers of frontier systems, the most advanced models on the market, face new safety and evaluation requirements.

The company points to Claude Mythos Preview as evidence of this acceleration. According to Anthropic, it discovered thousands of high-severity vulnerabilities across all major operating systems and browsers. The argument is that a capability useful for protecting software could also be used to attack hospitals, power grids or other essential infrastructure.

Which companies would be covered

The rules would not apply to every AI model. Anthropic proposes targeting systems trained with more than 10²⁵ floating-point operations, a measure of the amount of computing used during training, and developed by companies that meet at least one of these conditions:

  • More than 500 million dollars in AI-related revenue.
  • More than 1 billion dollars invested in AI research and development.

The goal is to focus on the models with the greatest capabilities and the companies with enough resources to build strong safety programs. The proposal is mainly designed for the United States federal government, although Anthropic also argues that states should retain room to regulate issues such as child protection and consumer rights.

Four risks it wants to control

The framework identifies four types of potential danger:

  • Biological risk: a model without sufficient safeguards could help design or produce dangerous biological agents.
  • Cyber risk: advanced systems can find vulnerabilities at scale, increasing both their defensive usefulness and the potential damage if they fall into the wrong hands.
  • Loss of control: increasingly capable systems could act in ways their developers cannot control or stop.
  • Automated research and development: AI can help create better AI systems, in turn accelerating the three risks above.

Anthropic does not present these scenarios as damage that has already occurred. It frames them as risks that could increase as model capabilities improve.

More testing and independent oversight

For frontier developers, the proposal includes several requirements. They would have to test their models, publish a summary of the results and explain how they assess catastrophic risks. They would also have to publish system cards, documents that describe a model’s capabilities, limitations and risks.

Anthropic also proposes regular reports on each company’s overall risk status and evaluations conducted by independent organizations. Those organizations would have to meet specific standards and have sufficient access to the models to review whether the safety tests are reliable.

Cybersecurity would also have a central role. Companies would have to protect not only the final model but its entire development environment, including training systems and model parameters, which can be attractive targets for attackers and state actors.

The government could halt a launch

The most significant part of the proposal is the authority to block or deter the deployment of a model that poses a significant risk of catastrophic harm. Anthropic also calls for civil penalties tied to a company’s global annual revenue, with larger fines for repeated violations.

The company tries to place limits on that authority. It proposes starting with lighter regulation and tightening it only as model capabilities improve and a reliable network of independent evaluators emerges. It also rejects having a federal law automatically eliminate all state rules, unless the national legislation is at least equally strict on the issues it covers.

Preparing society

The framework also recommends measures to reduce the impact of potential incidents. In biology, it mentions controls to detect suspicious genetic synthesis orders, early-warning systems to identify new outbreaks and reserves of protective equipment.

In cybersecurity, Anthropic proposes strengthening the software on which the internet depends, helping critical infrastructure operators and replacing legacy systems. It also proposes a dedicated government function to track the cyber capabilities of the most advanced models.

Anthropic’s second proposal, the Economic Policy Framework, addresses the other side of the problem: how to prepare workers and ensure that AI’s financial benefits are distributed broadly. The shared announcement does not detail all of its measures, but presents both documents as parts of the same agenda: controlling risks and preparing the economy for change.

For you, this means the next discussion about AI will not focus only on what a chatbot can do. It will also cover who can launch a model, what tests it must pass and when the government can intervene. Anthropic is asking for those rules to be defined before systems become even harder to evaluate and control.