What could go wrong if we develop technology to significantly amplify the intelligence of human minds? Intelligence is tricky to understand and I get confused when comparing it to the related concepts of wisdom and rationality. I’d like to draw clear distinctions between them. In a nutshell, rationality is the tendency to apply the capacity of intelligence, whereas wisdom describes the embodied knowledge of human behavioral patterns, specifically in terms of failure modes.
The relationship between rationality and intelligence seems better understood. My favorite exposition is in the excellent What Intelligence Tests Miss (good summary on LW). Of course, LessWrong itself is partially devoted to understanding this distinction and CFAR was built to see if we can isolate and train rationality (as opposed to intelligence). Intelligence is typically viewed as the capacity to perform the relevant moves — explicit reasoning, analogical application of past experiences, and avoiding biased heuristics of thought — when presented with a well-formed problem. In practice, the hard part of taking advantage of intelligence is having the awareness that one is facing a situation where intelligence can be explicitly applied. Thus, one can perform well when formally posed a problem, such as on an IQ or SAT test, yet still behave foolishly in the real world where the problems are not clearly structured and labeled. A colloquialism which approximates this dynamic is the idea of “book” and “street” smarts. Thus, to be rational requires not only some capacity for intelligence but, more importantly, the habits of identifying when and where to apply it in the wild.
How does wisdom fit into this? Informally, wisdom refers to the ability to think and act with sound judgment and common sense, often developed through a diversity of life experiences. We tend to look to the aged members of society as a font of wisdom rather than those with merely a large raw capacity for reasoning (intelligence). This corresponds with the heuristic of listening to your elders even when it doesn’t always make sense. Wisdom is often associated with conservativism and functions as a regulatory mechanism for societal change. The young and clever upstart has the energy and open-mindedness to create new technology and push for change while the old and wise have seen similar attempts fail enough times to raise a note of caution. The intelligent (and rational) are not more careless than the wise but rather seem to have more blind spots — perhaps as a result of seeing fewer well-laid plans fail in unexpected ways. To anticipate failure — to predict the future — we rely on models. Ideally, we deduce from known laws — this is possible in the physical sciences. In messier and more complex systems, like human interactions, we are forced to primarily rely on experience from analogous situations (inductive and abductive reason). It is no surprise that the hardest failures to predict relate to how humans will act — politics, not rocket science.
Looking through the literature on measuring wisdom (1, 2, 3), one major commonality is the emphasis on modeling psychological dynamics: intrapersonal (knowing thyself) and interpersonal (making sense of interactions with, and between, other humans). Proficiency in these domains seems to only become possible through experience (specifically, exposure to extremes) interacting with other humans and introspecting, or reflecting, on experience. In contrast, a foundation in the physical sciences and mathematics seems to be learnable by interaction with text, thought, exercises, and experiments performable without significant interpersonal dynamics. In a sense, we can say that proficiency in the “hard” sciences is intelligence-constrained whereas proficiency in predicting and interacting with humans is constrained by a lack of diverse personal experience data and the ability to act upon heuristics extracted from it.
This can be understood as a difference in modelability — the extent to which we can formalize useful (predictive) models of the system. With mathematics and the physical sciences — at least when applied to sufficiently simplified slices of reality — we are able to constrain non-determinism into a probabilistic model with well-behaved errors. On the other hand, modeling humans presents us with an uncertainty of a kind that we struggle to reduce (see: the struggle of the social sciences to successfully science). Even residing in a deterministic universe amenable to reductionism, and being armed with excellent models of sub-atomic interactions, we are unable to build the machinery necessary to predict the behavior of human beings. There are too many moving parts for a supercomputer, let alone the highly-constrained working memory of a human brain, to make useful predictions by analyzing the interactions of the component parts. On the other hand, the human brain has evolved to be quite good at modeling itself and other humans — we are social animals, after all. We perform this feat by observing behavior and automatically chunking it into categories and schemas to be recalled in future situations that appear similar enough. Unfortunately, we have not yet found a shortcut for developing this repository of experiences and the corresponding heuristics derived from it. This is the hard-to-replicate thing we tend to call wisdom.
Also published on Medium.