A single number keeps showing up in AI policy debates: “p(doom),” shorthand for the probability that AI causes human extinction or a comparably catastrophic loss of control. Two Princeton researchers argue that number is doing something other than measuring anything. It’s making a guess sound like a measurement.

What they said

In a post on their AI as Normal Technology newsletter, Arvind Narayanan and Sayash Kapoor write that “p(doom) rhetoric is driving public discourse and policy attention to an unprecedented degree,” and that the underlying numbers deserve far less trust than they’re getting. Their core claim, using their shorthand for existential risk, the chance of a catastrophe on the scale of human extinction: “AI x-risk forecasts are far too unreliable to be useful for policy, and in fact highly misleading.”

They explain why. Comparing AI risk to past events like asteroid strikes or industrial revolutions doesn’t work, they argue, because those comparisons “tell us nothing about the possibility of developing superintelligent AI or losing control over such AI.” Mathematical models can forecast something like an asteroid impact because it involves, in their words, “a purely physical system” that follows fixed rules. AI development depends on unpredictable technical progress and human decisions, so no comparable model exists for it.

That leaves subjective judgment dressed up as a percentage. Their clearest evidence is the Existential Risk Persuasion Tournament, a structured competition where two groups tried to independently estimate the odds of major future catastrophes, run by the Forecasting Research Institute. Narayanan and Kapoor call it “the most elaborate and well-executed x-risk forecasting exercise conducted to date.” It found AI experts putting the odds of AI causing human extinction or a similar catastrophe by 2100 at between 0.25% and 12%, while professional forecasters, people trained to estimate probabilities across many different subjects rather than AI specifically, put it at near-zero to 1%. Two credentialed groups working from the same tournament landed tens of times apart on the same question.

Who they are

Narayanan is a professor of computer science at Princeton University and director of its Center for Information Technology Policy, a research group that studies how digital technology intersects with public policy. Kapoor completed his computer science PhD at the same center in August 2026 and is an incoming assistant professor at UC Berkeley’s School of Information, starting in 2027. Together they wrote AI Snake Oil, a book pushing back on inflated claims about AI capability, and run the AI as Normal Technology project, which argues AI should be governed like other powerful technologies rather than treated as an unstoppable path to superintelligence.

BuilderWithin has covered their work twice before: a study finding AI agents aren’t yet good at open-ended research tasks, and, two weeks ago, an argument that recent AI agent security incidents are a security failure rather than evidence AI is slipping out of human control. This piece extends that second argument. If agent incidents are a security problem rather than proof AI is becoming ungovernable, the extinction-probability numbers used to justify emergency-style restrictions deserve the same scrutiny.

What they get right, and where it’s incomplete

The tournament evidence holds up on its own terms. When two groups of qualified people, working from the same exercise and the same information, land tens of times apart, the resulting percentage cannot be doing the job a weather forecast or an insurance company’s risk tables do. Treating it as a precise input for legislation misrepresents what the number actually is.

Where the essay is harder to fully evaluate is the “unprecedented” framing. Narayanan and Kapoor assert that p(doom) rhetoric is shaping policy more than ever, but the post doesn’t lay out the specific hearings, bills, or filings behind that claim. It reads as their own read of the discourse, not a documented count. Their prescription, that governments should pick policies that hold up across a range of possible risk levels rather than betting everything on one estimate, is also a judgment call about how to handle uncertainty, not a conclusion that follows automatically from the measurement problem they diagnose. They’re not neutral referees on this either: they’ve spent years arguing against catastrophic AI framing, so a conclusion that downplays the doom numbers fits the position they already held.

Why it’s notable

Doom-probability numbers don’t stay confined to academic debate. They get cited in testimony, position papers, and lobbying for specific rules, like mandatory safety testing before a model can launch, limits on releasing open-weight models (AI models anyone can download and run themselves, instead of only using through a company’s paid service), or export controls (government limits on which technology can be sold or shared with other countries) on the chips used to train the most advanced models.

BuilderWithin covered last Wednesday how export-control pressure has coincided with Chinese open-weight AI models overtaking American ones on downloads, one live example of a policy fight where the underlying risk estimate shapes what gets built and by whom. If those estimates are as unreliable as Narayanan and Kapoor argue, the case for restriction is borrowing false confidence from a number that was never a measurement.

What it means for builders

Treat any specific AI-extinction percentage you see in a policy debate or news story as someone’s informed guess, not a measured fact, no matter how precisely it’s stated. Narayanan and Kapoor’s tournament evidence is the strongest reason why: credentialed forecasters working from identical information produced estimates tens of times apart, which is not what a working measurement looks like.

The practical risk to your roadmap isn’t a specific doom percentage. It’s the concrete policy proposals attached to that rhetoric: safety-testing mandates, restrictions on open-weight releases, or export controls that can limit which models and computing power you can build on. Watch those proposals directly, and treat the doom number cited alongside them as a rhetorical prop rather than evidence you need to weigh.

End of article