Eliezer Yudkowsky

Quotes

1. “We’ll make them care about truth, and then we’ll be okay.”

   “We’ll design them to be submissive.

   “We’ll just have AI solve the ASI alignment problem for us.”

   These are not what engineers sound like when they respect the problem, when they know exactly what they’re doing. These are what the alchemists of old sounded like when they were proclaiming their grandiose philosophical principles about how to turn lead into gold.

   p. 192

2. 

   When imagining some new, unprecedented piece of future history, there is a temptation to fall into imagining that it will all go sensibly, rather than the way things usually go in history books. People sometimes ask us: How could the AI companies possibly be doing this thing, if matters are as we say? And maybe the simplest real answer is: Because this is the sort of awful, sad, real situation that you read about in history books, and not in the sensible world that exists only in imagination.

   p. 197

3. 

   When it comes to thinking, quality trumps quantity.

   p. 24

4. 

   Pathways are hard to predict.

   But we can predict the endpoint.

   p. 97

5. 

   The preferences that wind up in a mature AI are complicated, practically impossible to predict, and vanishingly unlikely to be aligned with our own, no matter how it was trained.

   p. 74

6. 

   Once AIs get sufficiently smart, they’ll start acting like they have preferences – like they want things.

   We’re not saying that AIs will be filled with humanlike passions. We’re saying they’ll behave like they want things; they’ll tenaciously steer the world toward their destinations, defeating any obstacles in their way.

   p. 46

7. 

   Human intelligence is the source of all our power, all our technology.…

   So far, humanity has had no competitors for our special power. But what if machine minds get better than us at the thing that, up until now, made us unique?

   p. 28

8. 

   Humanity does not need to understand intelligence, in order to grow machines that are smarter than us.

   p. 39

9. 

   Even in the face of superhuman machine intelligence, it can be tempting to imagine that the world will keep looking the way it has over the last few decades of our relatively short lives. It is true, but hard to remember, that there was a time is real as our own time, just a few short centuries ago, when civilization was radically different. Or millennia ago, when there was no civilization to speak of. Or a million years ago, when there were no humans. Or a billion years ago, when multicellular colonies had no specialized cells.

   Adopting a historical perspective can help us appreciate what is so hard to see from the perspective of our own short lifespans: Nature permits disruption. Nature permits calamity. Nature permits the world to never be the same again.

   p. 10

10. 

    The most fundamental fact about current AI’s is that they are grown, not crafted. It is not like how other software gets made – indeed it is closer to how a human gets made, at least in some important ways. Namely, engineers understand the process that results in an AI, but do not much understand what goes on inside the AI minds they manage to create.

    p. 31

11. 

    We’re pretty sure, actually very very sure, that a machine superintelligence can beat humanity in a fight, even if it’s starting with fairly limited resources.

    p. 97

12. 

    Ultra-fast minds that can do superhuman-quality thinking at 10,000 times the speed, that do not age and die, that make copies of their most successful representatives, that have been refined by billions of trials into unhuman kinds of thinking that work tirelessly and generalize more accurately from less data, and that can turn all that intelligence to analyzing and understanding and ultimately improving themselves – these minds would exceed ours.

    p. 25