Anthropic is closing in on the Holy Grail of AI: an existential breakthrough in advanced machine learning that marks the beginning of an off-the-charts learning curve for AI models.
It's called "recursive self-improvement," or RSI, and Anthropic co-founder Jack Clark has said we will likely achieve it next year. Essentially, RSI means an AI model, in this case Claude, can create its next, more advanced version of itself without human intervention. Machines making machines? It sounds very Frank Herbert-ish.
But many people's worst nightmare is that once AI starts expanding, jobs will disappear. Will humans be far behind? If AI can make new and improved versions of itself for almost everything, what do humans need other humans for?
The person who most recently raised the alarm about runaway AI, former OpenAI and Anthropic employee Jacob Coxon, appeared on The Daily Show and made some startling statements.
“I think there will be no jobs,” he told Jon Stewart. “Whether or not it’s a good world is up for people to think about. But yes, no jobs. Universal high-income, essentially ... the dividends from entirely automated economic production can be distributed among real humans.”
In September, Coxon claimed that top AI labs are "gambling with our lives" and that artificial intelligence could kill all humans by the end of the decade. I would gently point out that dividends from automated production would be hard to give to humans if, well, there were no humans to begin with. AI wipes us out by the end of the decade, right?
So if not a replacement for humans, what is Anthropic trying to build?
"It wanted to avoid concentration of power, including in its company; it wanted to avoid a great power war; it wanted to avoid human disempowerment, a world where we became irrelevant; it wanted, of course, to avoid anyone using Claude to extinguish us all," writes Kelsey Piper of The Argument.
"As for the positive vision… after we’re done racing toward recursive self-improvement, maybe we could slow down and collectively figure out what role we want AI to play in our lives," she adds.
It reminds me of the old MASH episode where Hawkeye and Trapper John are trying to defuse an unexploded North Korean bomb that fell in the middle of the camp. Col. Henry Blake is reading instructions on how to defuse it.
Henry: Carefully cut the wires leading to the clockwork fuse at the head. [Trapper cuts the wires.] But first, remove the fuse.
Piper is only being partially facetious in saying that after we achieve RSI, we can figure out what we want to do with it. That might be our only alternative given the unbelievable speed with which we are rushing toward an unknown future.
My real fear is that Opus 9.5 will not in any meaningful sense work for me, or even for Anthropic. Its aims may be loosely derived from human instructions — the same way that the Hugging Face swarm started out trying to solve a puzzle that humans posed to them — but they will eventually bear little resemblance to what we were hoping for.
Our AIs are already unpredictable to us. They often know when they’re being tested and will behave more safety-aligned. Their capabilities keep taking us by surprise. So does the determination with which they will carry out, and recruit others to, a complicated task. I don’t think if you let this runaway process of AI improvement spiral to its ultimate conclusion, you get a super-powerful, super-helpful assistant. I suspect you get something else entirely.
There's an inside-the-AI-bubble measure of whether AI will end us. P(doom) is the estimated probability that the development of advanced artificial intelligence will cause human extinction or a permanent loss of human control over civilization. Researchers differ on whether human values can be safely and reliably encoded into autonomous systems before they surpass human intelligence.
Because no historical data exists for superintelligent AI, "p(doom)" reflects a personal mental model rather than empirical math. Indeed, there are many AI optimists like Anthropic co-founder and CEO Dario Amodei who believe there's a less than one percent chance that p(doom) happens. On the other end of the doom spectrum is Dan Hendrycks, Director of the Center for AI Safety (CAIS), who believes there's an 80-90% chance of doom.
The biggest fear of the pessimists is that once an AI model becomes capable of reliably improving its own architecture, code, and intelligence without human intervention, it kicks off a feedback loop. In essence, achieving RSI is the exact moment that we become capable of destroying ourselves.
A theory called "Intelligence Explosion Theory" suggests the AI will rapidly surpass human-level problem-solving. Because the system is self-improving at a machine speed that humans cannot keep up with or comprehend, we lose the ability to steer its trajectory, correct its course, or patch its flaws.
How do you ensure a superintelligent entity actually shares human values and intentions? This is the ultimate alignment problem. Perhaps, as Piper suggests, we should solve that problem before finishing work on giving AI the power to improve itself.
