If you were incredibly, incredibly optimistic, you might look at the differences and say: “Well, gradient… — Eliezer Yudkowsky
If you were incredibly, incredibly optimistic, you might look at the differences and say: “Well, gradient descent is not the same as natural selection, so it won’t have all the same complications as natural selection. And I don’t know of any particular complications in the relationship between what AIs are trained for and what AIs end up wanting; so I don’t expect any complications.”
But a blank map does not correspond to a blank territory: if you’re venturing across an unknown land mass and your map has a blank spot where you haven’t visited, it doesn’t mean you’ll see a vast empty space when you get there. If gradient descent is different from natural selection, that doesn’t mean that we should expect to see no complications, since we don’t know about any. Rather, we should expect to see new, interesting, unpredicted complications.
p. 65Source: Wikiquote · CC BY-SA 4.0
More from this book
If anyone builds it, everyone dies.
It doesn’t matter whether it’s built by benevolent corporations or selfish ones. It doesn’t matter whether it’s built by researchers in the East or researchers in the West. It doesn’t matter whether it’s built by reckless optimists or people who say they respect the problem. Nobody has the knowledge or skill to make a superintelligence that does their bidding.
p. 222
Insofar as the AI has weird alien preferences, escape is in fact the course of action that best fulfills its objectives. Attempts to escape are not a weird personality quirk that an engineer could rip out if only they could see what was going on inside; they are generated by the same dispositions and capabilities that the AI uses to reason, to uncover truths about the world, to succeed in its pursuits.
p. 190
It is not a matter of your own country outlawing superintelligence inside its own borders, and your country then being safe while chaos rages beyond. Superintelligence is not a regional problem because it does not have regional effects. If anyone anywhere builds superintelligence, everyone everywhere dies.
p. 211
