Eliezer Yudkowsky

Quotes

1. Computer security is a test of an engineer’s ability to nail down every single path the computer could take, in the face of adversaries who can search all possible ways to perturb the system. It is a famously losing battle – even though the engineers can fully control and craft their own computer’s code.

   We dub this central challenge the curse of edge cases: To be secure, a computer system must work in the face of cases that are outside of the normal and expected range, cases that occur on the edges of possibility.

   p. 174

2. 

   Many of these complications won’t show up in obvious, undeniable ways until after it’s too late for humans to do anything about them.

   p. 73

3. 

   The problem here is not that corporate executives might build AI servants and command them to do something monstrous. They’re not in control. It doesn’t matter whether they’re benevolent. Humanity is faced with an engineering challenge: How do we shape the preferences of AIs that we can’t understand? It doesn’t matter whether or not the engineers have an ethics team watching over their shoulder; the ethicists wouldn’t have any idea how to get an AI’s preferences to align with ours, either.

   p. 75

4. 

   Space probes. Nuclear reactors. Computer security. What do all these lessons add up to, and what can we learn from them about the difficulty of aligning an artificial superintelligence?

   An artificial superintelligence is like a space probe, in that we cannot test it in quite the same environment where it needs to work, and by default it is not retrievable or correctable once it rises high above us….

   An artificial superintelligence is like a nuclear reactor, in that its underlying reality involves immense, potentially self-amplifying forces, whose inner processes run faster than humans can react.

   An artificial superintelligence is like a computer security problem, in that every constraint an engineer tries to place upon the system might be bypassed by the intelligent forces that those constraints hinder.

   p. 175

5. 

   If any company or group, anywhere on the planet, builds an artificial superintelligence using anything remotely like current techniques, based on anything remotely like the present understanding of AI, then everyone, everywhere on Earth, will die.

   p. 7

6. 

   Most powerful artificial intelligences, created by any method remotely resembling the current methods, would not choose to build a future full of happy, free people. We aren’t saying this because we get a kick out of being bleak. It’s just that those powerful machine intelligences will not be born with preferences much like ours.

   p. 82

7. 

   A human brain can learn to navigate wider-ranging paths through a larger cross-section of reality than any other animal. That is our special power.

   p. 19

8. 

   If you were able to choose what an AI wants – the destinations toward which it steers – that might be good news for you. Or bad news, if you made a poor choice of destinations, or if some malicious person makes an AI that steers toward outcomes you dislike. But the problem facing humanity is not a problem of whether good people or bad people are in control of AI.

   No – we are facing an even harder problem: it’s much easier to grow artificial intelligence that steers somewhere then it is to grow AIs that steer exactly where you want.

   p. 54

9. 

   From these lessons in combination, we infer an additional lesson for engineers: If someone doesn’t know exactly what’s going on inside the complicated device subject to all these curses – speed, narrow margins, self-amplification, complications – then they should stop. They should shut it down immediately, at the moment the behavior looks strange; don’t wait until the behavior becomes visibly concerning.

   p. 171

10. 

    Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.

    p. 3

11. 

    History teaches that one kind of relatively easy call about the future involves realizing that something looks theoretically possible according to the laws of physics, and predicting that eventually someone will go do it. Heavier-than-air flight, weapons that release nuclear energy, rocket that go to the Moon with a person on board: these events were called in advance, and for the right reasons, despite pushback from skeptics who sagely observed that these things hadn’t yet happened and therefore probably never would.

    p. 8

12. 

    The real way a superintelligence wins a conflict is using methods you didn’t know were possible. And because we care about the truth more than about telling you things that are easy to swallow, that’s where will start.

    p. 98