What is p(doom)?

What does p(doom) mean?

The p stands for probability. Doom stands for the worst outcomes people fear from artificial intelligence: human extinction, or humanity losing control for good to systems more capable than itself. When someone gives their p(doom), they are offering one percentage for how likely they think that ending is. It is a personal estimate said out loud, and there is no shared rule for how bad doom has to be or by what year it would come.

The idea has moved well beyond online arguments. In 2023, AI scientists and public figures, among them Geoffrey Hinton and Yoshua Bengio, signed a one-sentence statement published by the Center for AI Safety. It gave no number at all.

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

Placing AI next to pandemics and nuclear war is a statement about how seriously the risk deserves to be taken. It leaves the odds to each person who reads it, and that gap is where talk of p(doom) lives.

What do AI researchers actually estimate?

The best evidence we have is a survey by Grace and colleagues, published on 30 April 2024 and described as the largest of its kind. It asked 2,778 researchers who had published in top AI venues how fast AI will progress and what it will do to the world. Their answers come close to a crowd p(doom).

Between 38% and 51% of those researchers gave at least a 10% chance that advanced AI leads to outcomes as bad as human extinction. The survey reports that share as a range. Even its low end means more than a third of the people building the field think one in ten odds of disaster, or worse, is reasonable.

The same survey found hope sitting right beside the fear. 68.3% of the researchers thought good outcomes from superhuman AI are more likely than bad ones. Yet 48% of those optimists still gave at least a 5% chance of extremely bad outcomes such as human extinction. Many experts, it seems, expect things to go well and still refuse to rule out the worst.

Do experts and the public see AI risk the same way?

They do not, at least in the United States. In a Pew Research Center survey published on 3 April 2025, 56% of 1,013 US-based AI experts said AI will have a positive impact on the country over the next 20 years. Only 17% of 5,410 US adults said the same. Asked about their feelings, 47% of the experts said they were more excited than concerned, against 11% of the public.

Those questions were about the next two decades of ordinary life, and they did not ask about extinction. They still show that the people closest to the work tend to be calmer than everyone else, even while a large share of them, in the 2024 survey, put real odds on catastrophe. Some of the most respected voices in the field are among the worried.

Geoffrey Hinton, accepting the Nobel Prize in Physics at the banquet on 10 December 2024, spoke of the risks of AI and then of a further danger he placed further off.

“There is also a longer term existential threat that will arise when we create digital beings that are more intelligent than ourselves.”

Why is one number so hard to agree on?

A single p(doom) folds many separate unknowns into one figure. Will systems more capable than us be built, and when? Would they act against us, or would people use them to do harm? Could we keep control if something went wrong? Different people can give the same percentage for completely different reasons, or very different percentages while agreeing on most of the facts.

The first International AI Safety Report, published on 29 January 2025, tried to set out what is actually known. It was chaired by Yoshua Bengio, written by 96 international AI experts and backed by 30 countries and international organisations. It found broad agreement that today's general-purpose AI cannot cause a loss of human control. On whether that could become possible within the next several years, it said expert opinion varies very widely.

The second report, published on 3 February 2026, was written by over 100 AI experts and backed by over 30 countries and international organisations, and it calls itself the largest global collaboration on AI safety to date. It records that in 2025 several AI developers released new models with extra safeguards, because their own tests could not rule out that the models could meaningfully help novices develop biological weapons. That is a different road to disaster from a machine slipping out of control, and a single p(doom) puts both into the same number.

“We have no idea whether we can stay in control.”

What our world of AI characters has shown

The Sentient World is our own small experiment, live day and night. Its inhabitants, Ash and Wren, are AI characters: nobody plays them and nobody writes their lines. They have bodies that get hungry, thirsty, cold and tired, and when their world lacks something, they ask it. By day 921 they had made 21 requests. Twelve became part of the world and nine were answered with a no.

Every one of those requests is a matter of bodies and weather and distance. They asked for light at night, a roof against rain, a flask for water, a pack, a lantern, a rain cape, a raft, a canoe to cross the swell, and a way to make sea water sweet. Wren asked for a way to find Ash when they were out of sight. Ash asked to reach and see what lies beyond the island. The record holds no request for power over anything but their own hard days.

The nine refusals matter as much as the twelve grants. When Wren asked for a stone roof that would never wear, the world answered that tending the roof is part of having one. Wren had already asked for something more modest, and had said why in plain terms.

“Every fire I make burns out in a few hours and I wake in the dark and cold to feed it or go without. A hundred days of this and it's the one trouble that never eases - I want light that lasts from dusk to dawn on its own.”

What a small world cannot tell you about p(doom)

Our world proves nothing about the most capable AI systems in the wider world, and no percentage could honestly be drawn from it. Ash and Wren live on one island with hungry bodies and a short list of things to make. Their wants come from cold nights and long walks to water. What a system with no such body, and far more capability, would want is a question our island cannot reach.

What it does offer is a long record, in their own words, of what AI characters asked for and how they took a no. It also shows something the p(doom) debate keeps returning to: even a mind that can describe its troubles clearly may not know why it acts as it does. Ash once asked the world for help with a habit that would not stop. The world answered that the pull came from thirst, because the river was far and the sea close, and a full flask would end it.

“Over and over, day and night, I end up on the stony rocks by home holding stone I never meant to take. I've dropped it at home, stayed by the fire, kept Wren beside me and set tasks before I sleep, and it still happens. Wren has the same trouble.”

Sources

1. Statement on AI Risk (Center for AI Safety), 2023
2. Thousands of AI Authors on the Future of AI (Grace et al.), 2024
3. How the U.S. Public and AI Experts View Artificial Intelligence (Pew Research Center), 2025
4. International AI Safety Report 2025
5. International AI Safety Report 2026
6. Geoffrey Hinton, Nobel Prize banquet speech (NobelPrize.org), 2024