Statistical Bias4 分钟阅读

The Small-Sample Trap

If you flip a coin four times and it lands heads three times, is it a coin that favors heads? When the sample is small, it is easy to fall into this kind of illusion.

The Illusion of the "Law of Small Numbers"

In 1971, psychologists Amos Tversky and Daniel Kahneman published a paper titled "Belief in the Law of Small Numbers."

They showed that people tend to mistakenly believe that "even a small random sample sufficiently represents the population." In other words, they look at just a few pieces of data and hastily conclude that "this represents the whole."

It is named the "law of small numbers" because it arises from a misunderstanding of the actual formal statistical law, the "law of large numbers."

Source: Tversky & Kahneman (1971), Psychological Bulletin 76(2), 105-110.

Why Small Samples Are Dangerous

When a sample is small, variation due to chance is large, and extreme values easily appear.

If you flip a coin four times, getting heads three times (75%) is common. But if you flip it 4,000 times, the proportion of heads gets very close to 50%. That is because the larger the sample, the more the random fluctuations cancel out.

The problem is that people look at the "75%" from a small sample and generalize that "this coin favors heads." In reality it is nothing more than random fluctuation.

Contrast with the Law of Large Numbers

The law of large numbers is a formal statistical law stating that "as the sample grows, the sample mean converges to the true mean."

The law of small numbers is the mistaken expectation that this law "will apply equally to small samples." But small samples have no such stability.

So before you judge based on some result, you should first ask, "How many data points is this conclusion based on?" A small sample means the confidence interval around the conclusion is wide, and it is hard to separate luck from skill.

What to Watch for in Investing

The investing world is full of small-sample traps. A representative one is the judgment that "the last three years' returns were good, so this is a skillful fund." Three years is a short period in which luck plays a large role, so extremely good or bad performance appears often.

The same is true in personal investing. If you feel certain of your ability after a few winning trades, you may be mistaking chance produced by a small sample for skill.

That is why, when you look at performance, you should look over as long a period as possible and across multiple phases (bull, bear, and sideways markets) together. That is exactly why this site shows you long-term results rather than a short-window scorecard, together with the maximum drawdown and the loss period.

常见问题

Q. So how many years of performance do I need to judge skill?

There is no fixed answer, but a short window like three years is very heavily influenced by luck. The longer the period and the more market phases it spans (both bull and bear markets), the easier it is to separate luck from skill. What matters is the attitude that "the shorter the period, the more humbly you should draw conclusions."

Q. How is small-sample bias different from recency bias?

Small-sample bias is an illusion that arises "because there are too few data points," while recency bias is an illusion that arises "from placing too much weight on the most recent data." The two often appear together, such as accepting a few recent results (a small sample) as if they were the whole.