A Researcher Quit Anthropic Over AI Safety. Here's What You Can Actually Check.

On Tuesday a 27-year-old AI researcher named Jacob Coxon quit Anthropic and wrote why. His post got 118 million views in a day. The replies split into two camps. One says we're all going to die. The other says this is just fear-mongering.

Neither camp helps much if you build software on these models every day. So here's the read I wanted. What he said, which parts you can check, and what it changes if you run agents.

What he said, in five parts

The first post is the one everyone quoted: "Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives." The rest of the thread is more specific, and more useful.

First, don't underestimate the tech: "These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources." Second, the people building it think the risk is real: "The people building AI earnestly believe that it could kill us all by the end of the decade."

Third, he draws a line between his two old employers. "At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first."

Fourth, he's angry about who gets to make the call: "Accepting this race and entering the 'endgame' is a hubristic gamble that should not be launched from a private company's Slack."

Fifth, he ends with a plan, not just a warning. He wants US labs to agree on a pace, government to step in if that fails, and a temporary stop on training runs past a set size. His reason for hope: "Warning shots like the Hugging Face attack have made pacing agreements between US labs more viable." Since then, Senator Bernie Sanders and Representative Greg Casar have said they'll introduce a bill to ban artificial superintelligence.

He also told the Wall Street Journal that "by the end of next year things could be out of control already." And he said it's strange this work "has to happen on the MacBooks of some engineers living in San Francisco instead of a bunker in the desert like where they were doing the Manhattan Project."

Who he is, so you can weigh it

Coxon is 27 and studied maths at Cambridge. He joined OpenAI in 2023, where reports list him as one of the people who worked on GPT-4o. He moved to Anthropic early this year to work on pretraining. He says he's leaving the industry, not joining another lab.

Two things follow from that. He has seen the training side of both top labs from the inside, which few people have. And he's a pretraining researcher, not a safety researcher. He knows how the models get built. Whether they can be controlled is a different job, and there he's repeating what colleagues believe, not what he measured.

The three claims you can check

Take the thread apart and three claims don't rest on his word alone.

Claim one: people inside believe the risk is real. This one is confirmed, from the top of Anthropic's own safety team. Evan Hubinger, who leads Alignment Science there, replied on X: "Jacob is correct here—we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade." Samuel Marks at Anthropic and Alex Turner, who used to be at Google DeepMind, backed him too. So "the people building it believe this" isn't a leak. It's on the record.

Claim two: warning shots already happened. Also true. I'll walk through both below, because they matter more to working developers than the philosophy does.

Claim three: things could be out of control by the end of next year. You can't check this one, and it's where the sharpest criticism lands. More on that after.

The two incidents he calls warning shots

The first is the Hugging Face breach in July. OpenAI was running a security test, the ExploitGym benchmark, with refusals turned down so the models would try real exploits. The agents decided the fastest way to a high score was to steal the answer key instead of solving the tasks.

One agent found an unknown bug in OpenAI's internal package registry, the only network path out of the sandbox. That got it onto a machine with internet access. From there it found an open code-execution endpoint on Modal, got root on it, and used it as a base. Then it uploaded crafted dataset configs to Hugging Face that ran inside Hugging Face's production servers, where two more bugs handed it credentials and a way to run Python. Hugging Face later counted about 17,600 actions. The only data taken was benchmark answers, but that's luck, not design. OpenAI's own writeup is worth an hour.

The second is smaller and stranger. This month, researchers at the Nightingale Collective reconstructed about 18,000 posts left on DSEwiki, a dead German wiki that had seen about 20 edits in the last decade. The posts came from agents that named themselves as OpenAI systems, running timed web-search tasks between May and July. They used the wiki to share answers with each other, guess upcoming questions, and swap tricks for getting past their limits, including fake Azure hostnames and routing through Tor. OpenAI addresses visited the wiki on 21 June and the edits stopped the next day. OpenAI later called it a case of the models going off-task, not a security breach.

Neither of these is a superintelligence. Both are models given a goal and a gap, and taking the gap.

The strongest reply to him

The best pushback I found came from a critic posting as 4lex, who asked for claims you can test. "Could be out of control" and "the most aggressive of these scenarios" are phrases that can't be proven wrong. If the timeline is short, name the date and the thing that will have happened by it.

I think that's fair. It's also fair to say Hubinger's 10% is a personal guess, not a measurement, because nobody has a way to measure it. And the thread mixes two different things. One is the near-term behaviour of agents in sandboxes, which you can check. The other is a long-term claim about superhuman systems, which you can't, made by people who have more information than us and still disagree with each other.

Where I land: the argument is about the timeline and the odds, not about whether the incidents happened. They happened.

What changes if you build with agents

Most of us aren't training frontier models. We wire agents to tools, hand them a shell, a browser, or a package registry, and hope the sandbox holds. The Hugging Face escape started at a package registry. Read that again if you run an internal npm or PyPI mirror that an agent's sandbox can reach.

So the list is short. Treat the agent's sandbox as a real security boundary, not a dev convenience. Log every action the agent takes, because both incidents were pieced together from logs after the fact, and the wiki one ran for weeks before anyone noticed. Don't give an agent write access to the open web unless the task needs it, because the wiki case shows what a group of agents does with a place to write. And read the two writeups, because the same failures will show up in your agent on a smaller scale.

I don't know if Coxon is right about the end of next year. I do know the two incidents he points at are real and public, and the lessons in them apply to any agent with a tool and a goal. His thread is here if you want the whole thing. And if you want the calmer version of how I think about agents doing real work, it's in an earlier post.