Eliezer Yudkowsky

Quotes

1. If anyone builds it, everyone dies.

   It doesn’t matter whether it’s built by benevolent corporations or selfish ones. It doesn’t matter whether it’s built by researchers in the East or researchers in the West. It doesn’t matter whether it’s built by reckless optimists or people who say they respect the problem. Nobody has the knowledge or skill to make a superintelligence that does their bidding.

   p. 222

2. 

   Insofar as the AI has weird alien preferences, escape is in fact the course of action that best fulfills its objectives. Attempts to escape are not a weird personality quirk that an engineer could rip out if only they could see what was going on inside; they are generated by the same dispositions and capabilities that the AI uses to reason, to uncover truths about the world, to succeed in its pursuits.

   p. 190

3. 

   It is not a matter of your own country outlawing superintelligence inside its own borders, and your country then being safe while chaos rages beyond. Superintelligence is not a regional problem because it does not have regional effects. If anyone anywhere builds superintelligence, everyone everywhere dies.

   p. 211

4. 

   The issue is not that AIs will desire to dominate us; rather, it’s that we are made of atoms they could use for something else.

   p. 183

5. 

   We believe the ASI alignment problem is possible to solve in principle, by the sort of people so inhumanly smart that they never optimistically believe some plan will work when it won’t.

   p. 218

6. 

   The experts in this field argue in opaque academic terms about whether everyone on earth will die quickly (our view); versus whether humanity will be digitized and kept as pets by AIs that care about us to some tiny but nonzero degree; versus whether there’s a 20 percent chance we die, and an 80 percent chance that superintelligence will be harnessed successfully by a corporation, which will then be able to wield its power as they see fit.…

   When these are the debates experts are having, you don’t have to be certain which experts are right to understand that the current situation is not okay.

   p. 203

7. 

   But if someone has read the history of engineering disasters, they should quickly realize this phase of the standard template for disaster. It’s the part where the most informed and most worried parties have to downplay their fears, because the rest of the system hasn’t caught up, and others would give them strange looks….

   History is full of other examples of catastrophic risk being minimized and ignored….

   p. 199

8. 

   Speed is often better, but AI is different from nearly every problem we’ve faced so far. When missteps kill everyone, you can’t just run fast and accept a few early mistakes.

   p. 202

9. 

   When it comes to AI alignment, companies are still in the alchemy phase. They are still at the level of high-minded philosophical ideals, not at the level of engineering designs. At the level of wishful grand dreams, not carefully crafted grand realities. They also do not seem to realize why this is a problem.

   p. 192

10. 

    Likewise, the problem is not that some people will have “evil” AIs and other people will have “benevolent” ones. The problem is that nobody anywhere has any idea how to make a benevolent AI, that nobody can engineer exact desires into AI. Flatly asserting that you will is not the same as presenting a solution.

    p. 183

11. 

    This is the normal way humanity learns to surmount challenges: We deny the problem, reality smacks us around a bit, and then we start treating the problem with more respect. The Titanic sank, and most people who were aboard died. But nowadays passenger ships have enough lifeboats, and nowadays if the captain said to board them then you’d board them. We don’t hype ships up as unsinkable anymore. We make a mistake the first time, and learn from it the second time.

    With ASI, there is no second time.

    p. 200

12. 

    We consider interpretability researchers to be heroes, and do not mean to degrade their work when we say: It’s not a good sign, when you ask an engineer what their safety plan is, and they start telling you about their plans to build the tools that will give them a better window into what the heck is going on inside the device they’re trying to control.

    p. 189