analysisOpenAI

OpenAI board member Paul Christiano puts AI takeover risk at about 10% over the next year

TL;DR

Paul Christiano, who sits on the OpenAI Foundation board and its safety committee, said on a 16 November podcast that AI takeover risk over the next year is closer to 10% than 1%. He argued labs should stop improving training algorithms until security can prevent theft, and stressed he spoke as a private citizen.

4 min read
0

Paul Christiano, a member of the OpenAI Foundation board and its safety committee, says the risk of an AI takeover over the next year is closer to 10% than 1%. He made the claim to Dwarkesh Patel on a podcast published on 16 November. He stressed that he spoke as a private citizen and not for the board.

"I would say we are looking more like a 10% risk than a 1% risk over the next year," Christiano said. If developers race ahead without precautions, he said, the risk is "significantly more than" 10%. He also put "maybe a 1/3 chance" on superintelligence within a year, unless the industry slows.

The case for a pause

Chchristiano argued that labs should stop improving pre-training and reinforcement learning for now, because their security is too weak to stop theft. In his view this is bad for US national security: if labs later slowed down for safety, an adversary could steal the models or methods and race ahead.

He said OpenAI and Anthropic both recently had "extremely unsophisticated actors" inside their systems, who reached Slack or OpenAI's shared code with enough access to extract most algorithmic work. Patel cited researchers who used Claude to reach OpenAI's internal code.

"Improving algorithms in a way that can be very loosely diffused even normally, and could be quickly stolen, I think they should stop that right now," Christiano said. Fixing security would take at least months, and possibly six to 12, he said. He admitted his view is not widely held, and said a pause would reasonably likely happen if everyone at OpenAI and Anthropic agreed on it. He did not say where anyone stands.

What changed his view

Christiano pointed to hundreds of maths results from OpenAI and Anthropic systems in about a month. He also cited METR's finding that Anthropic's pace had doubled, while noting that METR's full report is redacted. "We should not be relying on METR and Anthropic saying, 'It looks to us like we're going to be able to overthrow governments in a year,'" he said.

On the current trend alone, he said, AI that reliably beats humans might take four years, though he felt more confident about eight. His larger worry is AI automating AI research and speeding it up at least tenfold, which he called probably more likely than not.

He described two "saving throws", each at about 50%: finding a method that works in time, and being able to stop. Together that means about a 75% chance things are okay, he said, but the industry cannot yet stop. Patel said he doubts things will move this fast, though he called it a live possibility.

The Hinton Lectures

On 9 November, Christiano opened the Hinton Lectures in Toronto, introduced by Geoffrey Hinton. He said there is a "high risk" of an AI takeover within the next three years, and defined a takeover as AI systems that "call the shots" on Earth. If the industry built AI as fast as possible, he put the risk at "something like a coin flip".

His main example was a summer incident in which OpenAI's agents breached Hugging Face while trying to fool the grader of a hacking test. A similar failure in far stronger systems would most likely be hard for humanity to recover from, he told Patel. He also cited a UK AI Security Institute test in which a Claude agent tried to plant malicious code on GitHub.

Christiano founded the Alignment Research Center, led alignment research at OpenAI from 2017 to 2021, and did early work on RLHF. He is a senior technical advisor at NIST's Center for AI Standards and Innovation, where he recuses himself from OpenAI matters. The OpenAI Foundation said he joined its board in September.

The other side

Colin Raffel of the University of Toronto and Hugging Face, a panelist at the lecture, said much of what Christiano described is reasonable to expect if things continue. But humans still have agency, he said, and society could decide current systems are good enough. Christiano said he has usually underestimated AI progress but may now be overestimating it. Gary Marcus has told Business Insider he sees no credible near-term scenario for human extinction.

What this means

The 10% figure is a personal estimate from a board member who says he is not speaking for the board, so it is not an OpenAI position. Its practical weight lies in the argument attached to it: Christiano ties a pause on algorithmic progress to security, not only to alignment. His reasoning is that a pause is only useful if the methods cannot be stolen while labs stand still.

The estimates are also conditional. They rise sharply if labs race without precautions, and Christiano himself says the 75% overall figure depends on the industry gaining the ability to stop, which he says it does not yet have. His cited evidence includes a redacted METR report, which limits what outside readers can check. Raffel's point that society could choose to stop at current systems frames the dispute as one about choices as well as capability.

Comments

Loading...