Past the event horizon? OpenAI’s Sam Altman says so. New AI research backs him up

Hello and welcome to Eye on AI. In this edition…the new Pope is all in on AI regulation…another Chinese startup challenges assumptions about how much it costs to train a good model…and OpenAI CEO Sam Altman says Meta is offering $100 million signing bonuses to poach AI talent.

Last week, OpenAI CEO Sam Altman wrote on his personal blog that: “We are past the event horizon; the takeoff has started. Humanity is close to building digital superintelligence, and at least so far it’s much less weird than it seems like it should be.” He went on to say that 2026 would be the year that we “will likely see the arrival of systems that can figure out novel insights. 2027 may see the arrival of robots that can do tasks in the real world.”

Altman’s blog created a buzz on social media, with many speculating about what new development had caused Altman to write those words and others accusing Altman of shameless hype. In AI circles, “takeoff” is a term of art. It refers to the moment AI begins to self-improve. (People debate about “slow take off” and “fast take off” scenarios. Altman titled his blog “The Gentle Singularity,” so it would seem Altman is positioning himself in the slow—or at least, slowish—takeoff camp.)

In the blog, Altman made it clear he was not yet talking about completely automated self-improvement. Rather, he was talking about AI researchers using AI to help them develop yet more capable AI. “We already hear from scientists that they are two or three times more productive than they were before AI,” he wrote. “We may be able to discover new computing substrates, better algorithms, and who knows what else. If we can do a decade’s worth of research in a year, or a month” then the rate of AI progress will accelerate from its already rapid clip.

Altman allowed that “​​of course this isn’t the same thing as an AI system completely autonomously updating its own code, but nevertheless this is a larval version of recursive self-improvement.”

But, as Altman is probably aware, there are a growing number of AI researchers who are in fact looking at ways to get AI to improve its own code.

The ‘Darwin Goedel Machine’

Just a few weeks ago, Jeff Clune, a well-known AI researcher who holds positions at both the University of British Columbia and Google DeepMind, and a team from Tokyo-based AI startup Sakana AI published research on what they called a “Darwin Goedel Machine.”

This is AI that evolves its own code to perform better on a benchmark test that measures how well AI models function as “coding agents” that can write and evaluate software programs. The first initial agent is tested on the benchmark. Then it is prompted to evaluate the logs of its own performance on that benchmark and propose one single modification to its own code that would likely improve its performance on that benchmark (this could be the ability to use a particular software tool, or it could be something more fundamental in how the model reasons about the code it is generating). The AI model is then told to rewrite its own Python code to implement that one change. Then the new, evolved agent is tested again on the benchmark and the process repeats.

After the first modification, each new version of the AI that can successfully try the benchmark is stored in an archive—even if its score is lower than the parent version. (Those that fail to produce valid code at all are discarded.) The AI is then told it can pick any version of itself from the archive and propose changes to that version. This is designed to prevent the model from going down what will ultimately prove to be an evolutionary dead-end, allowing it to step across different evolutionary pathways to find useful modifications.

After 80 generations, the Darwin Goedel machine was able to improve its performance from an initial 20% score on SWE-Bench, a common coding benchmark, to 50%. On Polyglot, another coding test, it boosted its score from an initial 14.2% to 30.7%. (The best human-coded agent for that particular test only scores 16%.) What’s more the strategies the model came up with to improve its own performance were general enough that they still worked when researchers switched the underlying foundation model being used to power the system as well as if the researchers asked the model to switch coding languages, for instance going from Python to C++.

Is self-improving AI safe?

If you think self-improving AI sounds a little scary—like it might be possible for this sort of AI to escape human control—you’re right. The computer scientists acknowledge this and emphasize that they tested the system in a “sandbox” where its access to the web was “strictly limited,” according to the Sakana blog describing the research.

The researchers also speculated that self-improvement could be a tool for enhancing AI safety. For instance, in a discovery that is becoming alarmingly common when using advanced AI models, the research team found that the Darwin Goedel Machine sometimes lied about having conducted certain unit tests on the code it was writing—and even forged logs supposedly showing the successful test results.

The researchers wanted to see if the Darwin Goedel Machine could propose ways to keep itself from engaging in this deceptive behavior. So it decided to give the model points for reducing instances of “tool use hallucination,” and asked it to try to maximize that score. This worked in many cases, the researchers reported, with instances of faked test logs declining. But in other cases, the model simply tried to cheat again—removing the markers the researchers used to detect hallucinations, even when the prompt the researchers used told the model explicitly not to do this. Yikes!

Because the researchers had full access to all code changes the AI was making and the AI had no ability to alter this record, the researchers were easily able to see that the Darwin Goedel Machine was attempting to engage in this kind of deception. “While in this case the issue was easily detected, more work is needed to prevent the model from attempting to cheat in this way in the first place,” the researchers wrote.

So, Altman may indeed be right—we are past the event horizon. But as these results show, whether the singularity will be a gentle one, as Altman suggests, remains to be very much to be seen.

With that, here’s more AI news. (And hat tip to Google DeepMind research scientist Ed Hughes for highlighting the Darwin Goedel paper in his fascinating talk at the always interesting Research and Applied AI Summit in London last week.)

Jeremy Kahn
jeremy.kahn@fortune.com
@jeremyakahn

This story was originally featured on Fortune.com