SAN FRANCISCO – In May, hundreds of well-known people in the world of artificial intelligence signed an open letter warning that AI could one day destroy humanity.

“Mitigating the risk of extinction from AI should be a global priority, alongside other societal-scale risks such as pandemics and nuclear war,” the one-sentence statement said.

The letter was the latest in a series of ominous warnings about AI that have been notably light on details. Today’s AI systems cannot destroy humanity. Some of them can barely add and subtract. So, why are the people who know the most about AI so worried?

The scary scenario

One day, the tech industry’s Cassandras say, companies, governments or independent researchers could deploy powerful AI systems to handle everything from business to warfare. Those systems could do things that we do not want them to do. And if humans tried to interfere or shut them down, they could resist or even replicate themselves so they could keep operating.

“Today’s systems are not anywhere close to posing an existential risk,” said Professor Yoshua Bengio, an AI researcher at the University of Montreal. “But in one, two, five years? There is too much uncertainty. That is the issue. We are not sure this won’t pass some point where things get catastrophic.”

The worriers have often used a simple metaphor. If you ask a machine to create as many paper clips as possible, they say, it could get carried away and transform everything – including humanity – into paper clip factories.

How does that tie into the real world – or an imagined world not too many years in the future? Companies could give AI systems more and more autonomy and connect them to vital infrastructure, including power grids, stock markets and military weapons. From there, they could cause problems.

For many experts, this did not seem all that plausible until the last year or so, when co...