In short
In short: the big AI labs write their own danger thresholds. OpenAI's Preparedness Framework has two levels, High and Critical, and three tracked areas: biological and chemical capabilities, cybersecurity, and AI self-improvement. Anthropic has had its Responsible Scaling Policy (RSP) since September 2023, now at version 3.4. A safety case is the next step: a written, structured, evidence-based argument that the risk is under control, of the kind used in aviation and nuclear power.
Think of the lift in your building. Inside there's a plate with the maximum load and the date of the last inspection. Almost nobody reads it, but everyone counts on someone having checked. The labs' thresholds are the plate. The question is who does the checking.
How the threshold works
Under OpenAI's framework from April 2025, a model at High could amplify existing pathways to severe harm and must have sufficient safeguards before it is deployed. Critical could open new pathways that didn't exist before, and then safeguards are needed while the model is still being developed. On September 1, 2026 OpenAI named Astra its first model at Critical in cybersecurity, and slowed parts of the release while it strengthened its protections.
The weak spot is that both the plate and the inspection belong to the company. On September 6 OpenAI's chief scientist Jakub Pachocki wrote that such frameworks should become widely mandated safety bars, enforced by third-party auditors, government agencies or international bodies.
A threshold the company writes and checks itself is a promise, not yet a rule.
Outside checking already has an example from the inside. On October 15, 2026 Anthropic announced that six people from METR, a nonprofit that evaluates frontier AI models, work at the company with work hardware, a seat in its office and access to its internal systems, roughly what its own risk-assessment teams have. Anthropic cannot stop their evaluations from being published, but it can ask for redaction within predefined categories, and it decides what falls into them. When the first analysis comes out has not been said.
The safety case
On September 28, 2026 OpenAI proposed that before any big reinforcement learning run there should be written safety documentation, ideally at the level of the safety cases used in aviation and nuclear power. The company admits this is a goal it is building towards, and that it's harder for AI because each new level of capability brings new complexity. Among its proposals: every safety case gets a written dissent from another team, and each senior leader who signs off can veto the run.
Primary sources
OpenAI: the updated Preparedness Framework (April 2025)Anthropic: Responsible Scaling PolicyOpenAI: Path to AstraOpenAI: Towards safety cases for frontier AI trainingOpenAI: Jakub Pachocki, An Alien MindAnthropic - An update on independent evaluators, 15.10.2026The visual is generated code-art, no third-party images.