Matt Ridley: how seriously should we take AI doom warnings?

Might doomers be right for once?

[Welcome to the ~1,000 new rational optimists who’ve joined us in the last week! Our mission is to make the world more rationally optimistic for our kids.

As an ROS member, you’ll receive a column roughly once every other week from co-founder Matt Ridley, author of the bestselling Rational Optimist book. Today, he questions whether the AI doom narrative is exaggerated… as were all doom narratives before it.

You’ll also receive the weekly ROS Diary, written by Stephen McBride and Dan Steinhart, to brighten up your Sunday mornings and keep you abreast of all the exciting new companies solving some of humanity’s hardest problems.]

Imagine an artificial intelligence making copies of itself...the first replicator assembles a copy in one thousand seconds, the two replicators then build two more in the next thousand seconds, the four build another four, and the eight build another eight. At the end of ten hours, there are not thirty-six new replicators, but over 68 billion. In less than a day, they would weigh a ton; in less than two days, they would outweigh the Earth; in another four hours, they would exceed the mass of the Sun and all the planets combined.

This is the fate we face, according to many. Only notice that the paragraph above is in italics. That is because I did not write it.

It was written 40 years ago in a book published in 1986 and I have changed just one phrase, substituting “an artificial intelligence” for “a replicator floating in a bottle of chemicals”. The book was not about AI at all but nanotechnology.

Engines of Creation was written by Eric Drexler, and in it he coined the term “grey goo” for the fate of the world if self-replicating nanobots took off. Later he said he regretted the scare that his words sparked. Britain’s Royal Society, at the urging of the then Prince of Wales, wrote a lengthy report about the risk.

Yet nobody now talks about nanotechnology in such apocalyptic terms just as nobody says acid rain, ozone depletion, an imminent ice age or the Y2K computer bug are urgent existential threats. If you live as long as I have, you see many such scares come and go, and you get a tad cynical.

A Twitter account called Pessimists Archive, run by Louis Anslow, collects such cases. He recently pointed out that in 2026 “the New York Times cites seriously academics who say AI might end the world”, and in 1881 “the New York Times cites seriously academics who say telegraphy might end the world”.

The paper opined that meddling with the earth’s electric currents by building telegraph lines might freeze Africa and turn New York into a tropical jungle: “such are appalling results which, according to our esteemed scientific contemporary, are to follow the girdling of the earth with a continuous line of telegraph.”

At this point my more responsible friends remind me that the boy who cried wolf was eventually right and there was a wolf. Also, AI is not like the other scares, they insist: this time is different. If I reply “they said that last time, too,” it cuts little ice, and of course they have a point. At some point there may be a wolf.

But this is when I get even more exasperated. Because my world-weariness at promises of theoretical future calamity is matched by my bafflement at complacency about a current, urgent calamity. Why, when a wolf did suddenly descend on the fold, did the entire bien-pensant world dismiss it as an act of God?

I am referring to the COVID pandemic. In my view the evidence is now overwhelming that this began with a foolish experiment that went wrong in a laboratory in Wuhan—but even if you think that was only a small possibility, you should be really alarmed by the fact. If one virology experiment can kill 15 million people and do $45 trillion of damage to people’s livelihoods, then why have we largely ignored it while obsessing over some future potential threat from AI?

This point was brilliantly made recently by the celebrated venture capitalist Bill Gurley in a talk at the All-In summit in Los Angeles. “My p(2nd COVID) is way higher than my p(doom). And if their models are so ****ing smart, tell me what happened here.” To translate out of Silicon Valley-ese: we are facing a far greater risk of some virologist starting another pandemic than we face from AI. Yet nobody even wants to investigate, let alone prevent.

Gurley summarised in devastating detail how, when an accident happens, we usually leave no stone unturned to try find out whether it was somebody’s fault. When a building collapsed at Surfside in Florida, when two Boeing 737-Maxes crashed, when Hurricane Katrina overwhelmed New Orleans, when the Fukushima nuclear plant exploded, when the Challenger shuttle exploded, teams of investigators and journalists eventually found out that in all five cases these were man-made disasters, not natural accidents.

It took heroic determination to unearth the facts that, respectively, faulty design, faulty software, faulty levees, faulty flood protection and faulty cold resistance in the O-rings were the causes.

But in each of those cases the teams of relentless “searchers”, as Gurley calls them, had to fight like tigers to overcome powerful “blockers” from corporate and (yes) regulatory bureaucracies that did not want to find the truth. Yet those five events were pinpricks compared with COVID-19. Speaking of Richard Feynman’s battle to find out what happened to the Challenger, when he was told he should try being nicer, Gurley said: “If you have to get through niceties and seek truth when seven people died, how are you supposed to behave when 15 million people died?”

The blockers did an amazing job with COVID. They scared off almost all journalists, politicians, spies and lawyers by saying the lab leak was a debunked conspiracy theory, and it was left to a few self-funded amateurs to do most of the work finding out what happened in Wuhan.

Western authorities not only failed even to try searching; they did nothing to pressure Chinese scientists: no hint that collaboration should cease till they came clean about what happened, for example. There has not been a ghost of an attempt to improve the monitoring of what happens in virology labs in future. The opposite, in fact: the WHO was warning about the risks of lab leaks before the pandemic but since—barely a squeak.

Yet we are all supposed to be engaged, dawn till dusk, in an obsessive debate about the vague—rarely specified—way in which AI might doom us at some rarely specified point in the future. Ah, say my responsible friends, but it is precisely through virology that AI might harm us. Imagine how much worse it will be when AI starts designing virus genomes! Maybe, but would it not be a good start to pay a smidgen of attention to what human intelligence can do and is doing?

“A lot has been said about AI x bio trigger events that should lead to increased regulation or oversight,” writes my co-author, the virologist Alina Chan. “Yet, ~7 years since COVID emerged, the research pipeline that likely caused the pandemic is still legal to conduct outside of high containment without independent oversight.” To put it another way, we have just survived one shark attack and are swimming towards the shore in deep water, refusing to board life rafts, while discussing whether there might be a tiger where we land.