Anthropic Warns AI Could Escape Human Control by 2028

Key Takeaways

• Anthropic estimates 60% probability of recursive self-improvement by 2028
• 80% of code merged into Anthropic's production systems is now AI-authored
• Critics argue the warning is a regulatory capture strategy ahead of Anthropic's IPO

Anthropic, the AI safety company behind Claude, published a warning this week that has split Silicon Valley. The core message: AI systems may soon design their own successors, and humans might lose the ability to stop them.

Co-founder Jack Clark put a number on the risk. He estimates a 60% probability that recursive self-improvement will occur by the end of 2028. That's not a distant sci-fi scenario. It's three years away.

What Recursive Self-Improvement Means

The blog post, written by Clark and Marina Favaro, head of Anthropic's research institute, describes a specific threshold. Once AI systems can design their own successors, the pace of improvement could accelerate beyond human oversight. Each generation of AI builds a smarter next generation. Humans become spectators.

“Right now, it's like the AI industry has a gas pedal, but it doesn't have a brake pedal.”

— Jack Clark, Co-founder, Anthropic

The numbers inside Anthropic suggest this future is arriving fast. Engineers at the company now ship code 8x faster per quarter compared to the 2021-2025 average, thanks to AI-assisted development. More striking: 80% of code merged into Anthropic's production systems is now authored by their AI models.

AI is already the primary driver of AI development at Anthropic. The company is building systems that build themselves.

Political Reactions Split Along Predictable Lines

David Sacks, Donald Trump's former crypto and AI adviser, dismissed the warning as a bid for government intervention. His critique was blunt.

"You compare it to nukes, threaten half of white-collar jobs, warn recursive self-improvement could end humanity, then race ahead anyway. In other words, you want the government to save us from you," Sacks wrote on X.

Mitt Romney took the opposite view. The former Republican presidential nominee called AI safeguards "our highest and most urgent national priority," citing risks from AI weapons, pathogens, mass unemployment, surveillance, and extinction.

AI safety researcher Eliezer Yudkowsky responded to Clark's 60% estimate with characteristic directness: "Then we will all perish."

The IPO Timing Question

Gary Marcus, NYU professor emeritus and vocal AI critic, argues Anthropic doesn't actually want a pause. He sees a rhetorical strategy ahead of the company's expected IPO this year.

"Reading between the lines, they want it both ways. They don't actually want a pause, at least for now. Rather, they want to rush ahead, hinting at 'least cautious actors' for justification," Marcus wrote. "They want people to talk about an 'option' they don't actually plan to take."

This skepticism echoes across HackerNews, where commenters labeled the proposal "regulatory capture." The argument: calling for global coordination on AI development would freeze out smaller competitors and cement Anthropic's lead.

Not Everyone Is Dismissing the Warning

Andrew B. Hall, a Stanford professor of political economy who previously advised Meta, called Anthropic's proposal "no longer far-fetched." He pointed to recent developments: executive orders, OpenAI's proposal for enhanced model review, and Google DeepMind founder Demis Hassabis's stated preference for separating consumer release from pure research.

"If nothing else, it might be possible to agree to slow down consumer release of models separate from development," Hall wrote.

The distinction matters. A pause on deployment is different from a pause on research. One slows consumer adoption. The other stops the race entirely. The former might be achievable. The latter faces prisoner's dilemma dynamics.

The 80% Statistic Divides Developers

On Reddit's r/MachineLearning, the debate centered on what it means that 80% of Anthropic's production code is now AI-authored. Some see a productivity explosion. Others see a loss of human architectural understanding.

The concern: if humans stop writing the code, do they stop understanding what the code does? And if they stop understanding, can they maintain control?

Anthropic's own warning suggests the answer may be no. "If it were possible to effectively slow the development of this technology to give ourselves more time to deal with its immense implications, we think that would likely be a good thing," the company wrote.

The key word is "if." Anthropic isn't stopping. Neither is anyone else.

Frequently Asked Questions

What is recursive self-improvement in AI?

Recursive self-improvement is when AI systems become capable of designing their own successors, each generation creating a smarter next version without human intervention.

Why is Anthropic warning about AI if it keeps building AI?

Anthropic argues that unilateral pausing would only benefit less cautious competitors. The company says global coordination is needed, but continues development until such coordination exists.

When does Anthropic predict recursive self-improvement could happen?

Co-founder Jack Clark estimates a 60% probability that recursive self-improvement will occur by the end of 2028.

What percentage of Anthropic's code is written by AI?

According to internal data, 80% of code merged into Anthropic's production systems is now authored by their AI models.

Is Anthropic planning an IPO?

Multiple sources indicate Anthropic is expected to go public in 2025, which some critics argue influences the timing of their safety warnings.

Source: mint / Aman Gupta