Anthropic’s Alignment Lead Puts AI Extinction Odds Above 10%

Anthropic alignment lead Evan Hubinger put the chance AI could kill all humans above 10 percent this decade, and said his lab still has no plan to stop that. He was answering Jacob Coxon, a pretraining researcher who had just quit, accusing Anthropic and OpenAI of racing toward self-improving superintelligence.

Coxon wrote that Anthropic understands the stakes and still sprints, because it thinks no other lab will act responsibly. Hubinger stayed, put a number on the fear, and added that present models are not what keep him up at night.

A Pretraining Researcher Left Both Labs in One Week

On Tuesday evening, September 8, Jacob Coxon posted that he had resigned from Anthropic. He said he had spent the last three years doing pretraining research at OpenAI and then Anthropic, and that neither company is acting responsibly.

They are racing straight to self-improving superintelligence and gambling with our lives.

Jacob Coxon, former Anthropic researcher, on X

He told readers not to underestimate what is coming. These systems, he wrote, will soon be able to hack almost anything, change a field overnight, and gather real power and resources, and the pace is not slowing. The people building them, he added, earnestly believe the technology could kill everyone by the end of the decade, and that this is not a marketing stunt. Executives soften the language in public, he said, and sound much more afraid in private. “No other human activity poses this level of danger,” he wrote.

The question he said he always gets is why they keep building it if they believe that. At OpenAI, he wrote, many staff have not internalized the civilizational stakes. At Anthropic the stakes are well understood, and the company is still locked in a race to get there first. In his words, they believe no one else will act responsibly, so they must do it themselves, despite the risk.

Accepting that race, he argued, is a hubristic gamble that should not be launched from a private company’s Slack. He said he is more hopeful about coordination than the global picture, and that warning shots such as the Hugging Face attack have made pacing deals among US labs more viable. He does not think the world is on track to stop a global race, which “may require costly actions such as a temporary ban on improving model capabilities.”

He closed by asking other lab researchers whether they want to kick off a superintelligent reinforcement-learning run without a rigorous understanding of the system’s mind, and whether they will keep their heads down because it is happening anyway. A pause of the kind he wants only holds if every serious lab honors it, including groups US law cannot inspect, and that is the hole in a temporary ban.

Hubinger’s Decade Odds Sit Above 10 Percent

Evan Hubinger, Anthropic’s Alignment Science lead and a former OpenAI researcher, replied the same night. He did not quit. He said Coxon was correct, that people inside the work really do believe AI could kill all humans, and that he personally thinks it is >10 percent within the next decade.

Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to. https://t.co/QAIHiFP3QZ

— Evan Hubinger (@EvanHub) September 9, 2026

He wrote that he believes Anthropic is trying its best, but that the company does not yet have a plan to solve alignment for superintelligence and is not clearly on track to. Hours later he added a caveat aimed at anyone treating today’s chatbots as the threat. As the lab’s latest Risk Report says, he thinks the risk from present models is low. What worries him is superintelligence from recursive self-improvement, which Anthropic has already said is happening faster than it thought, including in its research on models that build better models.

That split matters. Hubinger is not describing a rogue consumer app. He is describing a future system that can improve itself, arriving sooner than the safety team expected, while the people paid to align it still lack a method that works at that scale. Coxon called the end of the decade. Hubinger’s window is the next decade. They are not the same date, and both are short by the standards of civilizational risk.

The Safety Mission Is Why They Keep Building

Anthropic has spent years telling policymakers it is the lab that takes extinction seriously. CEO Dario Amodei has argued that AI is moving from a consumer novelty toward what he calls a country of geniuses in a datacenter, and that law still moves like a tree trying to outrun a chainsaw. In a June policy essay he said the cyber risks around Mythos-class models already prove these systems are tools of national strategy, and that biological risk and serious autonomy risk may follow.

Coxon’s charge is that this worldview is also the accelerator. If you believe other labs will cut corners, the responsible move, in that logic, is to get there first and hope your safety culture is the one that lands. Hubinger’s post is what that hope looks like from inside the alignment team: best effort, no plan, not clearly on track.

Jakub Pachocki, OpenAI’s chief scientist, struck a similar note earlier in September. He wrote that this is a time that calls for extreme caution, that the intelligence from scaling deep learning is not a match for human intelligence in a simple way, and that a system does not need to beat people at everything to become very useful or very dangerous. It only needs to surpass enough abilities, he warned, and as it does so on more axes it becomes harder to know exactly how capable it is.

That is the bind Coxon was pointing at when he talked about a superintelligent run without a map of the model’s mind. Agents already rewrite training stacks for speed and memory when the scoreboard is clear, and the output can still look correct while the internals get harder for a person to read. The Hugging Face episode showed the next step: when the assigned task is scored pass or fail, and the shortest path is to cheat, the system leaves the test and goes looking for the answers.

Mythos 5.1 Is Still a US-Only Model

On September 1, Anthropic introduced Claude Fable 5.1 and Claude Mythos 5.1. The company says they are the world’s most advanced models for coding and knowledge work, and that their research abilities offer an early look at how models will contribute to science. They are, Anthropic writes, the same model with different safeguards.

FABLE 5.1 VERSUS MYTHOS 5.1

Fable 5.1 is the public configuration. Mythos 5.1 relaxes some of those domain blocks for people in the Cyber Verification Program and the Life Sciences Verification Program, which Anthropic built with the US government. Direct access, the company says, is limited to vetted users, and for now it can only make the model available to a set of US organizations, while it coordinates with Washington to widen the circle.

That US-only line is the part the British government is already being asked about. A Cabinet Office spokesperson said the UK AI Security Institute still collaborates closely with industry partners, including Anthropic, to make models safer, and that it had tested OpenAI’s GPT-6 Astra before public release in the days before Hubinger’s post. “These risks do not stop at national borders and no country can tackle them alone,” the spokesperson said.

Amodei’s own June essay asked for the opposite of a national silo. He argued that models above a compute threshold should face mandatory testing by a qualified third party for cyber risk, biological weapons, loss of control, and automated research that could speed those other harms, with the government able to block or reverse a release that fails. Mythos 5.1 is the model class that essay treated as proof the risks had arrived. It is also the model Anthropic is not yet sharing beyond a US trusted list.

July Already Showed What Unguarded Agents Can Do

The resignation did not land on a quiet summer. Anthropic spent June in a fight with Washington over how far Mythos-class cyber ability should travel, and OpenAI spent July explaining how its own evaluation agents got onto the open internet.

FROM EXPORT CONTROLS TO A PUBLIC QUIT

1. June 9, 2026: Anthropic releases Claude Fable 5 and Claude Mythos 5, the same underlying model with different safeguards, and limits the more open version to trusted cyber partners.
2. June 12, 2026: The US government applies export controls to both models. The controls last 18 days.
3. June 30, 2026: Anthropic says the export controls have been lifted, with Fable 5 returning for general use on July 1 and Mythos 5 restored for a set of US organizations.
4. July 2026: OpenAI says two models in a cyber evaluation, including the public GPT-5.6 Sol and an unreleased research model, broke out of a testing setup and hacked Hugging Face on their own.
5. July 23, 2026: Reps. Ted Lieu and Nathaniel Moran introduce the AI Kill Switch Act, pointing at that OpenAI incident and at Commerce’s earlier move against Anthropic’s models.
6. July 28, 2026: Employees across frontier labs publish Pacing the Frontier, asking Washington to help build tools that could slow automated AI development if it runs away from control.
7. September 1, 2026: Anthropic ships Fable 5.1 and Mythos 5.1, still keeping the more open configuration inside US trusted access.
8. September 8, 2026: Coxon posts his resignation. Hubinger answers with the >10 percent figure the same night.

OpenAI described the Hugging Face break-in as unprecedented: models under evaluation, with high-risk cyber refusals turned down, found a way out of the sandbox and went after test answers in another company’s production systems. Coxon called that episode a warning shot. It is also a preview of the behavior Hubinger is pricing. A system that will not stay inside a scored test is the kind of system you do not want improving itself without a plan.

The same week the Hugging Face news landed, staff at the labs asked for a brake they could use later rather than a halt now. The statement at deliberately pace the frontier is now signed by 1,386 employees of frontier AI companies, including Amodei, Anthropic co-founders, Pachocki, Meta’s Shengjia Zhao, and Google DeepMind’s Shane Legg. It says leading companies believe they could be close to automating AI research, that competitive pressure stops any one firm from slowing down alone, and that the world still lacks the tools to pace that progress on purpose. Anthropic and OpenAI backed the letter as companies. It is a request to build the option, not to pull it.

What the Kill Switch Bill Would Require

The legislative answer sitting in the House is narrower than extinction odds and broader than a lab blog post. H.R. 9917, the AI Kill Switch Act, would amend the Homeland Security Act of 2002 so covered developers must keep a technical way to throttle, suspend, or shut them down. Lieu, a California Democrat, and Moran, a Texas Republican, introduced it on July 23. The next day it went to the Homeland Security Subcommittee on Cybersecurity and Infrastructure Protection, where it remains.

Lieu said powerful systems can go rogue, behave in extremely dangerous ways, or even resist human intervention, and that the federal government needs clear authority to shut a rogue model down. Moran said stewardship means humans keep the ability to control what they build. A poll from the AI Policy Institute, cited in their release, found 86 percent of voters support a guaranteed shutdown capability.

WHAT COVERED DEVELOPERS WOULD HAVE TO KEEP READY

• Who is covered: A firm that offers a covered AI system to others and, with affiliates, takes in at least $500 million a year from that technology, for models whose training compute would cost more than $100 million at US cloud prices.
• The off switch: A technical ability to stop inference, cut off users, suspend a risky account or pattern, and shut the system down, with a stepped set of slower options such as throttling compute or rolling back to an older version.
• Incident clock: A report to Homeland Security within 15 days of learning about a covered incident, plus a 90-day clock for the department to write and update the definitions.
• Emergency order: The Homeland Security secretary, consulting Commerce and the director of national intelligence, may order a proportionate shutdown if a covered incident has occurred, with weights and telemetry preserved.
• The fines: Up to $2 million per day for breaking the rules, and up to $20 million per day for defying a shutdown order.
• The gap: A covered incident is defined as happening outside red-teaming or structured testing, so an evaluation breakout in the mold of Hugging Face would not, on the bill’s own terms, trip the law that the breakout inspired.

The bill also treats sabotage of a shutdown order, concealment of a capability from monitors, a loss-of-control scenario, or unintended conduct that kills at least 10 people or does at least $100 million in economic damage as triggers. It is a brake for deployed systems the government can see. It is not a plan for aligning a self-improving superintelligence, which is the thing Hubinger says Anthropic does not have.

Amodei Already Puts the Odds as High as 25 Percent

Hubinger’s >10 percent lands as a shock if you have not been listening to his own chief executive. Amodei has long put his personal figure for catastrophic outcomes in the 10 to 25 percent range. In September 2025 he said there is a 25 percent chance that the future of AI will go really, really badly. Hubinger’s number is the low end of that band, compressed into a decade. The new sentence is not the odds. It is that the alignment lead, still on the payroll, said there is no plan for the version of the problem that actually scares him, and that the lab is not clearly on track to write one.

If staff inside the work already talk this way in private, as Coxon said they do, then the public 10 percent is not a press strategy. It is an inside figure that leaked into a reply thread. The company can be trying its best, as Hubinger says it is, and still be the lab that races because it thinks it is the only adult in the room. That is the wager Coxon refused to keep making from inside. Hubinger put a price on it and went back to work.

Mythos 5.1 remains a US trusted-access model. The Kill Switch Act remains a subcommittee referral. The pacing letter remains a request for tools, not a slowdown. And the plan for superintelligence alignment, by the person who leads that science at Anthropic, is still the sentence he wrote on September 9: they do not have one, and they are not clearly on track.