Eliezer Yudkowsky

Quotes

1. In our view, intelligence is about two fundamental types of work: the work of predicting the world, and the work of steering it.

   p. 20

2. 

   Perhaps you don’t believe us about any of the foreseeable reasons why shaping ASI is unreasonably hard. There’s an independent and separate case for disaster, an alternate set of historical lessons: Humans sometimes flub easy problems, never mind hard problems.

   p. 180

3. 

   And if a sci-fi writer tried to write that story, the audience would just be confused, because why did that happen? Isn’t that the opposite of what the AI was trained for?

   But reality is allowed to be like that. And we are, fundamentally, predicting that the world will not turn out like a sci-fi novel. We’re predicting that AI’s preferences will turn out to be complicated and weird.

   p. 71

4. 

   It is as improbable that human thinking patterns mark the final limit of intelligent algorithms as it is that human neurons represent the limit of possible computing speeds.

   p. 25

5. 

   You can’t grow an AI that does what you want just by training it to be nice and hoping.

   You don’t get what you train for.

   p. 72

6. 

   We know, from years of talking to people about this subject, that some people are swayed by the abstract observation that a superintelligence could exploit options they didn’t even know were possible.

   For others, however, explanations such as these end up making them feel like we’re cheating in a child’s game of pretend. If we can’t even tell a story about how the bad guy is supposed to win, how is that convincing?

   We emphasize again: Reality has never been bound by that rule. Even if an Aztec soldier couldn’t have figured out in advance how guns work, the big boat on the horizon contained them anyway.

   p. 113

7. 

   We predict this with confidence: Once some AIs go to superintelligence – and nobody will delay much in pushing AIs that far, if in the middle of some great arms race – humanity does not stand a chance. Ends are sometimes easier to call than pathways. The only part of our story that is a real prediction is the ending – and then, only if the story is allowed to begin.

   p. 157

8. 

   If you were incredibly, incredibly optimistic, you might look at the differences and say: “Well, gradient descent is not the same as natural selection, so it won’t have all the same complications as natural selection. And I don’t know of any particular complications in the relationship between what AIs are trained for and what AIs end up wanting; so I don’t expect any complications.”

   But a blank map does not correspond to a blank territory: if you’re venturing across an unknown land mass and your map has a blank spot where you haven’t visited, it doesn’t mean you’ll see a vast empty space when you get there. If gradient descent is different from natural selection, that doesn’t mean that we should expect to see no complications, since we don’t know about any. Rather, we should expect to see new, interesting, unpredicted complications.

   p. 65

9. 

   With so many different hopes, surely there’s a chance that one of them will pan out? If you think reality works like that, go try to write a hundred different letters to someone with fifty billion dollars, giving a hundred different reasonable reasons you thought of why they ought to give you a hundred million dollars for your personal use. See if it works. The reason it all fails in the end is that the fifty-billionaire does not want to rationalize giving you 0.2 percent of their wealth, not the same way you rationalize reasons they should want to.

   In much the same way, an artificial superintelligence will not want to find reasons to keep humanity around – not in the same way that humans desperately want to find reasons to be kept.

   p. 89

10. 

    From each of the four curses we named, we draw these lessons:

    1. An engineering challenge is much harder to solve when the underlying processes run on time scales faster than humans can react….

    2. An engineering challenge is much harder to solve when there is a narrow margin for error, especially if it’s a narrow margin between “unimpressive” and “explosive.”…

    3. Self-amplifying processes, like an overheating reactor boiling off its coolant water and then overheating more, leave little room for error….

    4. Complications make engineering problems worse….

    p. 170

11. 

    Problems like this are why we say that if anyone builds it, everyone dies. If all the complications were visible early, and had easy solutions, then we’d be saying that if any fool builds it, everyone dies, and that would be a different situation. But when some of the problems stay out of sight? When some complications inevitably go unforeseen? When the AIs are grown rather than crafted, and no one understands what’s going on inside of them? That’s not a problem that anyone’s equipped to solve.

    p. 74

12. 

    Some aspects of the future are predictable, with the right knowledge and effort; others are impossibly hard calls. Competent futurism is built around knowing the difference.

    p. 8