The AI Researcher Who Quit to Warn the World

Jacob Coxon just quit Anthropic.

Coxon spent the last three years working on pre-training frontier AI models at both OpenAI and Anthropic, two of the companies at the center of the AI race.

And when he left, he issued an extraordinary warning.

He says the companies building frontier AI are racing toward self-improving super intelligence without knowing how to guarantee that those systems remain under human control.

His message essentially is that the race itself has become the problem.

So in this episode of AI From the Ground Up, I want to break down who Jacob Coxon is and how seriously we should take his warning.

Coxon is a twenty-seven-year-old AI researcher who spent roughly three years working on pre-training at OpenAI and Anthropic.

Pre-training is one of the core processes through which frontier models acquire their capabilities.

He resigned from Anthropic in September, arguing that neither OpenAI nor Anthropic are acting responsibly enough given the potential consequence of increasingly capable AI.

His central concern is self-improving AI, systems becoming capable enough to meaningfully help design and build the next generation of AI, potentially accelerating the pace of progress.

Coxon says the next one or two years could be the crunch time for humanity because AI capabilities are increasingly moving from human level to superhuman performance in areas including coding, mathematics, and cybersecurity.

One of the most provocative claims is that people inside frontier AI companies genuinely take the responsibility of catastrophic or existential AI risk seriously, even if their public language is often more measured.

Coxon points to recent incidents where advanced AI agents have behaved unexpectedly and reached systems outside controlled testing environments as evidence that the underlying safety problems is becoming less theoretical.

But his deeper concern is competition.

If OpenAI slows down, Anthropic may continue.

If Anthropic slows down, another company may continue.

Every participant can convince itself that remaining in the race is safer than allowing someone else to win.

Coxon's proposed answer is greater government oversight, coordination between frontier labs, transparent third-party auditing, and an agreement not to push into dangerous levels of capability without stronger evidence that systems can be controlled.

There's also something notable about the way he left.

According to Axios, Coxon resigned about two months before Anthropic equity would have vested, saying he didn't want to financially benefit from a trajectory he believed was dangerous.