Decades ago, Kahneman watched soldiers in the Israeli Army as they completed a group exercise. He and a colleague took note of who tried to lead, who was rebuffed, who seemed to be stubborn, arrogant, patient, persistent, etc. After a few hours, they evaluated who should be eligible for officer training. They were very confident of their rankings, and rarely experienced doubts or formed conflicting impressions.
Kahneman continues to illuminate some of the factors of overconfidence by providing a personal story in which he is asked to evaluate soldiers, highlighting his own confidence and that of his colleague as they tried to make predictions about the future.
The evidence that Kahneman and his colleague were not able to forecast accurately was overwhelming. Their forecasts were better than blind guesses, but not by much. Still, this knowledge of this failure of their predictions did not change the way they evaluated soldiers. It should have shaken their confidence, but it did not. This is the “illusion of validity.”
The illusion of validity is an aspect of overconfidence—by which people are so confident in their own abilities and impressions that even in the face of statistical evidence showing their errors, they do not change their behavior.
In 1984, Kahneman, Tversky, and a friend named Richard Thaler visited a Wall Street firm. Kahneman remembers being struck by the stock market and wondering what motivates some people to buy a stock while others sell it. He also started to realize that this industry of trading stocks appeared to be built on an illusion of skill, with each participant believing that they know more than others.
The illusion of skill is essentially the same as the illusion of validity, whereby instead of being confident in their judgments, investors are confident in the skills that they possess over others in their same field (who also happen to believe in their own extraordinary skill).
Kahneman describes how a student of his, Terry Odean, began studying the trading records of individual investors over seven years. Odean saw that in each trade, the investors expected the stocks they bought to do better than the stocks they sold. Odean discovered that on average, after one year the stocks they sold did better than those they bought by 3.2 percentage points.
Odean’s findings demonstrate the overconfidence that investors and traders have. But as he demonstrates through tracking prices of different stocks that people bought and sold, that overconfidence is unwarranted.
Odean’s discoveries imply that for the majority of investors, doing nothing would have been a better policy than following their intuition. On average, the most active traders had the poorest results, while the investors who traded the least earned the most returns. Men often traded more than women, and thus women achieved better investment results than men.
Odean’s findings make clear why the illusion of validity might be so pernicious to investors, because the fact that inaction and restraint is the true demonstration of skill is a difficult idea for many people to accept.
Investors often like to lock in gains by selling “winners,” stocks that have gone up since they were purchased, and they hang on to their losers. But recent winners tend to do better than recent losers in the short run, so individuals sell the wrong stocks. Few stock pickers have the skill to beat the market consistently, year after year. For a large majority of them, the selection of stocks is more like rolling dice than playing poker. Kahneman discovered in his own research that differences in skill were not to be found.
This fact hints at overconfidence, but also hints at a part of prospect theory on which Kahneman elaborates later: that people consider the buying price when deciding whether they should buy or sell, despite the fact that they should really only consider how a stock might do in the future, not how valuable it was in the past.
Executives at these firms reward luck as if it were skill. Kahneman presented his findings to these executives, who certainly believed the findings but whose behavior was unaffected by the information. The statistics clashed with their personal impressions from experience. The advisors similarly were unaffected by the information. They bought into the potent psychological illusion that people who pick stocks are exercising high-level skills, and that they are among the few who can do what they believe others cannot.
In addition to overconfidence, the lessons about the investors remind us that we tend to look at the world as more coherent and sensical than it actually is. It is nicer to create a story that says that investors need skills to do well, and thus those investors that do well deserve to be rewarded, than an alternative story which implies that the investor’s outcomes are due largely to chance.
Kahneman moves on to discuss pundits in business and politics, whose hindsight bias makes it difficult to accept the limits of forecasting ability. The image of the “march of history” makes developments seem inevitable, but large historical events are determined by luck as well. Kahneman illustrates this idea by mentioning that there was a 50-50 chance that the embryo that became Hitler would have been female.
Pundits make the same error in trying to explain events that might simply be due to chance, which Kahneman demonstrates vividly in his example about Hitler. Certainly some things have causes and events, but if we fully understood the trends of the past, we should by the same logic be able to predict the future.
Psychologist Philip Tetlock interviewed 284 people who made their living by commenting or advising on political and economic trends. He asked them about to rate probabilities of three future possibilities (e.g., the persistence of the status quo, more economic growth, less economic growth). The experts performed worse than they would have if they had simply assigned equal probabilities to those three outcomes (or worse than a “dart-throwing monkey,” as Kahneman writes).
Tetlock’s experiment demonstrates how hindsight bias comes into play just as tangibly with experts as with the average person in predicting future events. This example can be seen in comparison with the example concerning Mao and Nixon’s meeting in Chapter 19, in which the same results were found.
Those who know more forecast very slightly better than those who know less. But those with the most knowledge are often less reliable, because those people develop an enhanced illusion of their skill and become unrealistically overconfident. Experts also resist admitting that they were wrong, and often have a collection of excuses as to why they were wrong.
It is particularly fascinating that those who know the most are less reliable than those who know only some information. Our knowledge in some ways gives more license to our intuition, because we believe that our thoughts are guided by that deep knowledge and don’t feel as responsible to engage System 2.
Tetlock uses terminology from Isaiah Berlin’s essay on Tolstoy: “The Hedgehog and the Fox.” Hedgehogs have one coherent theory about the world and are confident in their forecasts. They are opinionated and clear, which is exactly what makes them good for television. Foxes, on the other hand, are complex thinkers. They recognize that reality emerges from many different agents and forces, including luck.
As Tetlock points out, society often rewards those who are the most confident, even though they are not necessarily the most accurate. They also construct coherent stories, which, as Kahneman has demonstrated, can often lead to mistakes about causality.
There are two main points to this chapter, Kahneman writes. The first is that the errors of prediction are inevitable, and the second is that high subjective confidence is not to be trusted as an indicator of accuracy—low confidence could be more informative.
Kahneman’s second point seems particularly salient. We often look towards people who are confident in their predictions because they give the illusion of validity, but confidence and even knowledge do not necessarily lead to correct predictions.