Rationality- From AI to Zombies Read online

  Rationality

  From AI to Zombies

  Eliezer Yudkowsky

  Eliezer Yudkowsky is a Research Fellow at the Machine Intelligence Research Institute.

  Written by Eliezer Yudkowsky

  Published in 2015

  Machine Intelligence Research Institute

  Berkeley 94704

  United States of America

  intelligence.org

  Released under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported license.

  CC BY-NC-SA 3.0

  isbn-10: 1-939311-15-2

  isbn-13: 978-1-939311-15-3

  (epub)

  The Machine Intelligence Research Institute gratefully acknowledges the generous support of all those involved in the publication of this book, and the donors who support MIRI’s work in general.

  Cover by Tim Woolley.

  Contents

  Preface

  Biases: An Introduction

  * * *

  Book I: Map and Territory

  * * *

  A. Predictably Wrong

  1. What Do I Mean By “Rationality”?

  2. Feeling Rational

  3. Why Truth? And . . .

  4. . . . What’s a Bias, Again?

  5. Availability

  6. Burdensome Details

  7. Planning Fallacy

  8. Illusion of Transparency: Why No One Understands You

  9. Expecting Short Inferential Distances

  10. The Lens That Sees Its Own Flaws

  B. Fake Beliefs

  11. Making Beliefs Pay Rent (in Anticipated Experiences)

  12. A Fable of Science and Politics

  13. Belief in Belief

  14. Bayesian Judo

  15. Pretending to be Wise

  16. Religion’s Claim to be Non-Disprovable

  17. Professing and Cheering

  18. Belief as Attire

  19. Applause Lights

  C. Noticing Confusion

  20. Focus Your Uncertainty

  21. What Is Evidence?

  22. Scientific Evidence, Legal Evidence, Rational Evidence

  23. How Much Evidence Does It Take?

  24. Einstein’s Arrogance

  25. Occam’s Razor

  26. Your Strength as a Rationalist

  27. Absence of Evidence Is Evidence of Absence

  28. Conservation of Expected Evidence

  29. Hindsight Devalues Science

  D. Mysterious Answers

  30. Fake Explanations

  31. Guessing the Teacher’s Password

  32. Science as Attire

  33. Fake Causality

  34. Semantic Stopsigns

  35. Mysterious Answers to Mysterious Questions

  36. The Futility of Emergence

  37. Say Not “Complexity”

  38. Positive Bias: Look into the Dark

  39. Lawful Uncertainty

  40. My Wild and Reckless Youth

  41. Failing to Learn from History

  42. Making History Available

  43. Explain/Worship/Ignore?

  44. “Science” as Curiosity-Stopper

  45. Truly Part of You

  Interlude: The Simple Truth

  * * *

  Book II: How to Actually Change Your Mind

  * * *

  Rationality: An Introduction

  E. Overly Convenient Excuses

  46. The Proper Use of Humility

  47. The Third Alternative

  48. Lotteries: A Waste of Hope

  49. New Improved Lottery

  50. But There’s Still a Chance, Right?

  51. The Fallacy of Gray

  52. Absolute Authority

  53. How to Convince Me That 2 + 2 = 3

  54. Infinite Certainty

  55. 0 And 1 Are Not Probabilities

  56. Your Rationality Is My Business

  F. Politics and Rationality

  57. Politics is the Mind-Killer

  58. Policy Debates Should Not Appear One-Sided

  59. The Scales of Justice, the Notebook of Rationality

  60. Correspondence Bias

  61. Are Your Enemies Innately Evil?

  62. Reversed Stupidity Is Not Intelligence

  63. Argument Screens Off Authority

  64. Hug the Query

  65. Rationality and the English Language

  66. Human Evil and Muddled Thinking

  G. Against Rationalization

  67. Knowing About Biases Can Hurt People

  68. Update Yourself Incrementally

  69. One Argument Against An Army

  70. The Bottom Line

  71. What Evidence Filtered Evidence?

  72. Rationalization

  73. A Rational Argument

  74. Avoiding Your Belief’s Real Weak Points

  75. Motivated Stopping and Motivated Continuation

  76. Fake Justification

  77. Is That Your True Rejection?

  78. Entangled Truths, Contagious Lies

  79. Of Lies and Black Swan Blowups

  80. Dark Side Epistemology

  H. Against Doublethink

  81. Singlethink

  82. Doublethink (Choosing to be Biased)

  83. No, Really, I’ve Deceived Myself

  84. Belief in Self-Deception

  85. Moore’s Paradox

  86. Don’t Believe You’ll Self-Deceive

  I. Seeing with Fresh Eyes

  87. Anchoring and Adjustment

  88. Priming and Contamination

  89. Do We Believe Everything We’re Told?

  90. Cached Thoughts

  91. The “Outside the Box” Box

  92. Original Seeing

  93. Stranger than History

  94. The Logical Fallacy of Generalization from Fictional Evidence

  95. The Virtue of Narrowness

  96. How to Seem (and Be) Deep

  97. We Change Our Minds Less Often Than We Think

  98. Hold Off On Proposing Solutions

  99. The Genetic Fallacy

  J. Death Spirals

  100. The Affect Heuristic

  101. Evaluability (and Cheap Holiday Shopping)

  102. Unbounded Scales, Huge Jury Awards, and Futurism

  103. The Halo Effect

  104. Superhero Bias

  105. Mere Messiahs

  106. Affective Death Spirals

  107. Resist the Happy Death Spiral

  108. Uncritical Supercriticality

  109. Evaporative Cooling of Group Beliefs

  110. When None Dare Urge Restraint

  111. The Robbers Cave Experiment

  112. Every Cause Wants to Be a Cult

  113. Guardians of the Truth

  114. Guardians of the Gene Pool

  115. Guardians of Ayn Rand

  116. Two Cult Koans

  117. Asch’s Conformity Experiment

  118. On Expressing Your Concerns

  119. Lonely Dissent

  120. Cultish Countercultishness

  K. Letting Go

  121. The Importance of Saying “Oops”

  122. The Crackpot Offer

  123. Just Lose Hope Already

  124. The Proper Use of Doubt

  125. You Can Face Reality

  126. The Meditation on Curiosity

  127. No One Can Exempt You From Rationality’s Laws

  128. Leave a Line of Retreat

  129. Crisis of Faith

  130. The Ritual

  * * *

  Book III: The Machine in the Ghost

  * * *

  Minds: An Introduction

  Interlude: The Power of Intelligence

/>   L. The Simple Math of Evolution

  131. An Alien God

  132. The Wonder of Evolution

  133. Evolutions Are Stupid (But Work Anyway)

  134. No Evolutions for Corporations or Nanodevices

  135. Evolving to Extinction

  136. The Tragedy of Group Selectionism

  137. Fake Optimization Criteria

  138. Adaptation-Executers, Not Fitness-Maximizers

  139. Evolutionary Psychology

  140. An Especially Elegant Evolutionary Psychology Experiment

  141. Superstimuli and the Collapse of Western Civilization

  142. Thou Art Godshatter

  M. Fragile Purposes

  143. Belief in Intelligence

  144. Humans in Funny Suits

  145. Optimization and the Intelligence Explosion

  146. Ghosts in the Machine

  147. Artificial Addition

  148. Terminal Values and Instrumental Values

  149. Leaky Generalizations

  150. The Hidden Complexity of Wishes

  151. Anthropomorphic Optimism

  152. Lost Purposes

  N. A Human’s Guide to Words

  153. The Parable of the Dagger

  154. The Parable of Hemlock

  155. Words as Hidden Inferences

  156. Extensions and Intensions

  157. Similarity Clusters

  158. Typicality and Asymmetrical Similarity

  159. The Cluster Structure of Thingspace

  160. Disguised Queries

  161. Neural Categories

  162. How An Algorithm Feels From Inside

  163. Disputing Definitions

  164. Feel the Meaning

  165. The Argument from Common Usage

  166. Empty Labels

  167. Taboo Your Words

  168. Replace the Symbol with the Substance

  169. Fallacies of Compression

  170. Categorizing Has Consequences

  171. Sneaking in Connotations

  172. Arguing “By Definition”

  173. Where to Draw the Boundary?

  174. Entropy, and Short Codes

  175. Mutual Information, and Density in Thingspace

  176. Superexponential Conceptspace, and Simple Words

  177. Conditional Independence, and Naive Bayes

  178. Words as Mental Paintbrush Handles

  179. Variable Question Fallacies

  180. 37 Ways That Words Can Be Wrong

  Interlude: An Intuitive Explanation of Bayes’s Theorem

  * * *

  Book IV: Mere Reality

  * * *

  The World: An Introduction

  O. Lawful Truth

  181. Universal Fire

  182. Universal Law

  183. Is Reality Ugly?

  184. Beautiful Probability

  185. Outside the Laboratory

  186. The Second Law of Thermodynamics, and Engines of Cognition

  187. Perpetual Motion Beliefs

  188. Searching for Bayes-Structure

  P. Reductionism 101

  189. Dissolving the Question

  190. Wrong Questions

  191. Righting a Wrong Question

  192. Mind Projection Fallacy

  193. Probability is in the Mind

  194. The Quotation is Not the Referent

  195. Qualitatively Confused

  196. Think Like Reality

  197. Chaotic Inversion

  198. Reductionism

  199. Explaining vs. Explaining Away

  200. Fake Reductionism

  201. Savannah Poets

  Q. Joy in the Merely Real

  202. Joy in the Merely Real

  203. Joy in Discovery

  204. Bind Yourself to Reality

  205. If You Demand Magic, Magic Won’t Help

  206. Mundane Magic

  207. The Beauty of Settled Science

  208. Amazing Breakthrough Day: April 1st

  209. Is Humanism a Religion Substitute?

  210. Scarcity

  211. The Sacred Mundane

  212. To Spread Science, Keep It Secret

  213. Initiation Ceremony

  R. Physicalism 201

  214. Hand vs. Fingers

  215. Angry Atoms

  216. Heat vs. Motion

  217. Brain Breakthrough! It’s Made of Neurons!

  218. When Anthropomorphism Became Stupid

  219. A Priori

  220. Reductive Reference

  221. Zombies! Zombies?

  222. Zombie Responses

  223. The Generalized Anti-Zombie Principle

  224. GAZP vs. GLUT

  225. Belief in the Implied Invisible

  226. Zombies: The Movie

  227. Excluding the Supernatural

  228. Psychic Powers

  S. Quantum Physics and Many Worlds

  229. Quantum Explanations

  230. Configurations and Amplitude

  231. Joint Configurations

  232. Distinct Configurations

  233. Collapse Postulates

  234. Decoherence is Simple

  235. Decoherence is Falsifiable and Testable

  236. Privileging the Hypothesis

  237. Living in Many Worlds

  238. Quantum Non-Realism

  239. If Many-Worlds Had Come First

  240. Where Philosophy Meets Science

  241. Thou Art Physics

  242. Many Worlds, One Best Guess

  T. Science and Rationality

  243. The Failures of Eld Science

  244. The Dilemma: Science or Bayes?

  245. Science Doesn’t Trust Your Rationality

  246. When Science Can’t Help

  247. Science Isn’t Strict Enough

  248. Do Scientists Already Know This Stuff?

  249. No Safe Defense, Not Even Science

  250. Changing the Definition of Science

  251. Faster Than Science

  252. Einstein’s Speed

  253. That Alien Message

  254. My Childhood Role Model

  255. Einstein’s Superpowers

  256. Class Project

  Interlude: A Technical Explanation of Technical Explanation

  * * *

  Book V: Mere Goodness

  * * *

  Ends: An Introduction

  U. Fake Preferences

  257. Not for the Sake of Happiness (Alone)

  258. Fake Selfishness

  259. Fake Morality

  260. Fake Utility Functions

  261. Detached Lever Fallacy

  262. Dreams of AI Design

  263. The Design Space of Minds-in-General

  V. Value Theory

  264. Where Recursive Justification Hits Bottom

  265. My Kind of Reflection

  266. No Universally Compelling Arguments

  267. Created Already in Motion

  268. Sorting Pebbles into Correct Heaps

  269. 2-Place and 1-Place Words

  270. What Would You Do Without Morality?

  271. Changing Your Metaethics

  272. Could Anything Be Right?

  273. Morality as Fixed Computation

  274. Magical Categories

  275. The True Prisoner’s Dilemma

  276. Sympathetic Minds

  277. High Challenge

  278. Serious Stories

  279. Value is Fragile

  280. The Gift We Give to Tomorrow

  W. Quantified Humanism

  281. Scope Insensitivity

  282. One Life Against the World

  283. The Allais Paradox

  284. Zut Allais!

  285. Feeling Moral

  286. The “Intuitions” Behind “Utilitarianism”

  287. Ends Don’t Justify Means (Among Humans)

  288. Ethical Injunctions

  289. Something to Protect

  290. When (Not) to Use Probabilities

  291.
Newcomb’s Problem and Regret of Rationality

  Interlude: The Twelve Virtues of Rationality

  * * *

  Book VI: Becoming Stronger

  * * *

  Beginnings: An Introduction

  X. Yudkowsky’s Coming of Age

  292. My Childhood Death Spiral

  293. My Best and Worst Mistake

  294. Raised in Technophilia

  295. A Prodigy of Refutation

  296. The Sheer Folly of Callow Youth

  297. That Tiny Note of Discord

  298. Fighting a Rearguard Action Against the Truth

  299. My Naturalistic Awakening

  300. The Level Above Mine

  301. The Magnitude of His Own Folly

  302. Beyond the Reach of God

  303. My Bayesian Enlightenment

  Y. Challenging the Difficult

  304. Tsuyoku Naritai! (I Want to Become Stronger)

  305. Tsuyoku vs. the Egalitarian Instinct

  306. Trying to Try

  307. Use the Try Harder, Luke

  308. On Doing the Impossible

  309. Make an Extraordinary Effort

  310. Shut Up and Do the Impossible!

  311. Final Words

  Z. The Craft and the Community

  312. Raising the Sanity Waterline

  313. A Sense That More Is Possible

  314. Epistemic Viciousness

  315. Schools Proliferating Without Evidence

  316. Three Levels of Rationality Verification

  317. Why Our Kind Can’t Cooperate

  318. Tolerate Tolerance

  319. Your Price for Joining

  320. Can Humanism Match Religion’s Output?

  321. Church vs. Taskforce

  322. Rationality: Common Interest of Many Causes

  323. Helpless Individuals

  324. Money: The Unit of Caring

  325. Purchase Fuzzies and Utilons Separately

  326. Bystander Apathy

  327. Collective Apathy and the Internet

  328. Incremental Progress and the Valley