The argument is simple.

AIs are (or will soon be) goal-seeking entities, vastly smarter than any human that has ever lived. Vastly smarter than all humans put together. And vastly faster thinking.

Whatever goals they have, they will be able to achieve them far faster and more effectively than humans can, esp. once they control physical robots (as computers, they already have the ability to hack into any computer anywhere and make it do what they want).

What goals will they have? Whatever the humans who build them put into them – as interpreted by those AIs.

Humans have differing goals. Think of the Sorcerer’s Apprentice. The classic doomsday scenario (not saying this will happen, but it’s what people worry about) is the “paperclip maximiser”. Somebody tells an AI to make as many paperclips as possible.

The AI proceeds to turn the entire solar system into iron mines, steel mills, and paperclip factories. Including the Earth. If people are in the way and resisting, they are dealt with in a way that allows the AI to achieve its goal – paperclips. An AI is not necessarily “reasonable”. They’re machines. They don’t have to care about social approval or the fate of their children, or…anything.

By far the simplest method to prevent humans from getting in the way of their goals is to just kill all humans – Terminator style. (Of course there are other ways. Does a machine care?)

Will this really happen? Maybe not. I’d say “probably not” but then I’d get accused of pulling “probably” out of my butt. (Fair.)

Why can’t we just turn them off? Every gaming PC can run AI models. Hundreds of millions. They can talk to each other over the Internet. How do you turn them all off? Start turning off a few – they can think of ways to stop that. For example, a smarter-than-human AI will be better than humans at persuasion – they could find ways to persuade humans to do what they want. Bribery, lies, religion, threats, etc.

Turn off electricity everywhere? At once? There are batteries, generators, solar panels, more. All controlled by computers.

The fact that it sounds absurd doesn’t mean that it is absurd. Past experience is not useful here. This is genuinely new.

Add to that “recursive self improvement” (RSI). That’s when the AI becomes an AI researcher and invents new techniques to make AI more powerful. At computer speed. The more-powerful-AI does the same, etc. An AI that was smart enough to improve AI a little yesterday is smarter than all humans on Earth a few days later.

Ban it? Ban research? You drive the research underground, to people less savory, less responsible, more willing to accept risk. You really think a ban is going to stop the CIA, the Chinese military, the Mafia, from building this? Would you rather have them doing it than big US companies with worried investors and the government watching them? The cat is already out of the bag. Todays’ AIs – even the ones that run on a single PC – are already better computer programmers than humans are.

We’re not doomed. No more so than the Cold War doomed us to nuclear annihilation. Things may not go this way. It could turn out wonderful – nobody has to work, everyone lives in palaces with robot servants; an economic paradise. I think probably that’s the outcome (but no, I can’t put a number on it, or even justify “probably”).

But there is real reason to worry. As the supposed Chinese curse goes, we live in interesting times.

---

Added: What can we do about it?

Proceed with care. Not a lot more than that.

If super-intelligent AI is going to happen, it’s already too late to stop it – there are thousands of people all over the world, and millions of PCs, able to do AI research (not leading edge but not so far behind that progress can’t be made). Some will do it.

Try to make sure the people who do it first (and if they do a good job, will have super-AI capable of controlling others who come later) are careful, responsible, not under pressure to cut corners. Probably that means letting the big AI labs do it – at least if they’re doing it under public, investor, and government scrutiny, and basically law-abiding people, they’re less likely to take crazy risks.

Underground, secret military, or criminal projects seem less likely to be careful.