Selection Bias

Selection bias is an error that arises when the sample being analyzed fails to evenly represent the whole population. If certain cases systematically enter or drop out of the sample more often, the conclusion is distorted even if the calculation itself is accurate.

The key is 'who remained in the data.' What we see is only what remained after passing through some selection process, and forgetting the things that didn't pass and disappeared means judging the world with only half of it in view.

The survivorship bias covered earlier is also a type of selection bias — a special case where only 'what survived' remains in the sample and 'what vanished' drops out.

During WWII, the statistician Abraham Wald was asked to advise on where to add armor (protective plating) on bombers. The military examined the planes that returned from missions and wanted to reinforce the areas with the most bullet holes (mainly the wings and fuselage). It seems commonsensical to protect where they got hit the most.

But Wald thought the opposite. 'What we're seeing now is only the planes that came back. These planes came back alive despite being hit in those areas. The planes that were actually shot down and didn't return must have been hit elsewhere — namely the engines and cockpit, where there are almost no bullet holes.'

So Wald recommended reinforcing not the areas with many holes but rather the engines and cockpit with no holes. He saw through the fact that the data contained only 'planes that came back alive' — which is exactly selection bias.

The investing world is full of selection bias.

First, conclusions gathered only from success stories. Content like 'what people who got rich from stocks have in common' samples only the successful. The far greater number who invested the same way and failed never appear in the story. So the illusion that 'do this and it works' arises.

Second, cherry-picking favorable start dates and stocks. Showing only the start years or stocks that perform well in a backtest is a biased selection of the sample. For example, take a point right after a crash as your start, and any strategy looks good.

Third, fund performance statistics that leave out liquidated funds (fund survivorship bias) are ultimately selection bias too.

Good analysis tries to count not just what survived but also what vanished. Just as Wald imagined the 'planes that didn't return' in the bomber case, in investing you must also picture the 'stocks, funds, and strategies that failed and disappeared.'

The Return of Almost Everything tries not to cherry-pick and show only successful results. It shows the actual performance of a given asset invested in over a long, steady period — without hiding the maximum drawdown and loss periods of that process. Rather than slicing out only the dazzling success stretches, including the time you had to endure is the path to reducing selection bias.

Every time you look at numbers, ask as a habit: 'How was this sample chosen? What was left out here?' This one question filters out many illusions.