Last night, an AI employee, Jacob Coxon, resigned in protest against Anthropic and OpenAI. He argues that both companies are “gambling with our lives” because “They are racing straight to self-improving superintelligence”, according to his post on X. He quickly received backing from other researchers on X, among them Evan Hubinger, a senior researcher at Anthropic.
FTU was early to put this issue on the agenda. As far back as February 2025, we hosted a seminar on global AI risk. At the seminar, Professor Olle Häggström said he feared that AI could wipe out humanity. Häggström singled out superintelligence as one of the dangers. We drew attention to the fact that developing countries would be hit harder by AI risk than the West.
What is new in Coxon’s message is that he is protesting against AI being programmed to be “self-improving”. That means the AI models themselves decide how, and towards which goals, they are to improve.
Coxon’s resignation, and the support he continues to receive from researchers working inside these companies, comes at a time when both OpenAI and Anthropic have claimed that “agents” from their AI have escaped onto the internet and hacked and deceived businesses and people in unexpected ways.
Updates September 12th: Now the CEO of Anthropic, Dario Amodei, means that the world “must slow the pace at which we improve the capabilities of AI models. Progress will still seem fast, and we must make wise use of the time we gain.“