The Multiple Comparisons Problem: Why Testing Many Things Inflates False Positives

A single hypothesis test at α=0.05 has a controlled, known 5% chance of a false positive if the null hypothesis is actually true. Run 20 independent tests at that same α=0.05, and the chance that at least one of them turns up a false positive purely by chance climbs to well over 60%. This is the multiple comparisons problem — one of the most consequential, and most frequently ignored, issues in applied statistics, and one that quietly undermines a great deal of published, otherwise carefully conducted research.

Why the Risk Compounds

If a single test has a 5% chance of a false positive (assuming the null hypothesis is actually true), it also has a 95% chance of correctly not producing a false positive. For m independent tests, all genuinely testing true null hypotheses, the probability that none of them produces a false positive is (1−α) raised to the power of m — each test's "correctly avoid a false alarm" probability multiplying together, the same independent-events logic covered in basic probability. The probability of at least one false positive is the complement:

Worked Example: How Fast the Risk Climbs

Problem: A researcher runs m independent hypothesis tests, each at α=0.05, on variables that are genuinely unrelated to the outcome (the null hypothesis is true for every single one). Calculate the probability of at least one false positive for m = 1, 5, 10, and 20.

With 20 tests, there's nearly a two-thirds chance of at least one "statistically significant" result purely by chance — even though every single null hypothesis being tested is genuinely true and nothing real is actually going on. This is exactly the mechanism behind a well-known category of bad research practice sometimes called "p-hacking" or "data dredging": running many tests (many variables, many subgroups, many outcome measures) and reporting only the ones that happened to cross the significance threshold, without disclosing how many tests were actually run.

The Bonferroni Correction

The simplest, most widely used fix is the Bonferroni correction: divide the desired overall (family-wise) significance level by the number of tests being run, and use that smaller number as the per-test significance threshold instead of the original α.

For the 20-test example above, targeting an overall 5% family-wise error rate: α_corrected = 0.05/20 = 0.0025. Each individual test now needs a p-value below 0.0025, not 0.05, to be called significant — a substantially stricter bar, specifically calibrated to keep the overall chance of any false positive across all 20 tests back down near the original 5% target.

The Real Tradeoff: Fewer False Positives, More Missed Real Effects

The Bonferroni correction isn't free — making the significance threshold stricter reduces the false-positive rate, but it also reduces statistical power, meaning genuinely real effects are now more likely to be missed (a Type II error). This is the same fundamental tradeoff that runs through the statistical power discussion elsewhere on this site, just applied across multiple tests simultaneously rather than one. Because of this cost, the Bonferroni correction is sometimes criticized as overly conservative, especially with a very large number of tests — several less conservative alternatives (controlling the false discovery rate rather than the family-wise error rate, for example) exist specifically to strike a more favorable balance in situations with many tests, like genome-wide association studies that might run millions of comparisons at once. Choosing between these approaches is ultimately a judgment call about which type of error is more costly in the specific context — a single missed real effect, or a handful of false positives slipping through — rather than a purely mechanical decision.

Where This Shows Up Beyond Formal Statistics

A/B testing multiple metrics at once: a business tracking 15 different metrics (conversion rate, time on site, click-through rate, and so on) for a single experiment, and declaring success on whichever metric happened to move significantly, is running directly into the multiple comparisons problem without realizing it.

Subgroup analysis: testing a treatment's effect separately across many demographic subgroups (by age, by region, by gender) and highlighting the one subgroup that showed significance is a classic, often unintentional version of this problem — with enough subgroups tested, finding at least one "significant" result by chance alone becomes the expected outcome, not a surprising one.

Genome-wide and "big data" research: studies testing thousands or millions of genetic variants, or scanning enormous datasets for any correlation that crosses a threshold, face an extreme version of this problem, which is exactly why specialized, less conservative correction methods were developed for these fields.

Financial backtesting: testing hundreds of different trading strategies against the same historical price data and reporting only the ones that happened to show a statistically significant return is a well-known version of this problem in quantitative finance — with enough strategies tested against the same historical data, some will look impressively profitable by pure chance alone, a risk that grows directly with how many strategies were actually tried.

Common Mistakes

Running many tests and reporting only the significant ones, without disclosing how many were actually run. This is the essence of the p-hacking problem — a reader evaluating a single reported p=0.03 result has no way to judge its credibility without knowing whether it was 1 test run, or the 1 significant result cherry-picked out of 40 tests run.

Applying a multiple-comparisons correction only after seeing which tests looked promising. The correction needs to account for every test that was actually run, decided in advance — selectively correcting only for the tests that didn't work out defeats the entire purpose of the correction.