False Discovery Rate and the Benjamini–Hochberg Procedure

Run one hypothesis test at the 5% level and the chance of a false alarm, when nothing is going on, is 5%. Run ten and the chance of at least one false alarm climbs to about 40%. Run twenty thousand, as gene-expression studies routinely do, and false alarms are guaranteed. The multiple comparisons problem is usually handled by making each test stricter, the Bonferroni approach, but that protects against making any false discovery at the price of missing many real ones. The Benjamini–Hochberg procedure takes a different view: rather than forbidding all mistakes, it controls the proportion of mistakes among the findings you announce. That trade is often far more useful in practice.

Two Ways to Count Mistakes

The family-wise error rate (FWER) is the probability of making at least one false rejection across the whole set of tests. Bonferroni and Holm control it. The false discovery rate (FDR) is the expected fraction of rejected hypotheses that are actually false discoveries. If a procedure controls the FDR at 5% and announces 100 findings, on average no more than about 5 of them are duds.

The two suit different situations. If one false claim would be catastrophic, such as a confirmatory trial deciding whether a drug is safe, controlling the FWER is appropriate. If the goal is to screen many candidates and follow up on the promising ones, as in genomics, marketing experiments, or a large batch of A/B tests, a controlled share of false leads is an acceptable price for much greater power.

The Benjamini–Hochberg Procedure

Given m tests and a target FDR level q (commonly 0.05):

• Sort the p-values from smallest to largest: p₍₁₎ ≤ p₍₂₎ ≤ … ≤ p₍ₘ₎.
• For each rank i, compute the threshold (i ÷ m) × q.
• Find the largest rank k for which p₍ₖ₎ ≤ (k ÷ m) × q.
• Reject the hypotheses for ranks 1 through k, including any earlier ones whose own p-values did not clear their thresholds.

The thresholds start strict (q ÷ m for the smallest p-value, the same as Bonferroni) and relax steadily until the largest p-value is compared against q itself. This "step-up" design is what gives the method its power.

Worked Example

Problem: A team tests 10 website changes, and the resulting p-values, sorted, are 0.001, 0.008, 0.012, 0.018, 0.031, 0.038, 0.049, 0.20, 0.45, 0.72. Control the FDR at q = 0.05.

Rank i	p-value	Threshold (i ÷ 10) × 0.05	p ≤ threshold?
1	0.001	0.005	Yes
2	0.008	0.010	Yes
3	0.012	0.015	Yes
4	0.018	0.020	Yes
5	0.031	0.025	No
6	0.038	0.030	No
7	0.049	0.035	No
8	0.20	0.040	No
9	0.45	0.045	No
10	0.72	0.050	No

The largest rank whose p-value clears its threshold is k = 4 (0.018 ≤ 0.020). Everything from rank 1 to 4 is rejected: four changes are declared real. Ranks 5 to 7 have p-values below 0.05 but fail their stricter thresholds, so they are not declared.

Comparing the Three Approaches

Method	Rule	Rejections
No correction	p < 0.05	7
Benjamini–Hochberg (q = 0.05)	step-up rule above	4
Bonferroni (α = 0.05)	p < 0.05 ÷ 10 = 0.005	1

Uncorrected testing announces seven findings, but if all ten changes had no effect, the chance of at least one false positive is 1 − 0.95¹⁰ ≈ 0.40, and the seven-finding list may contain several duds. Bonferroni is safe but keeps only the single strongest result, discarding three others with p-values of 0.008, 0.012, and 0.018 that look genuinely promising. Holm's step-down method, a uniformly better version of Bonferroni, also rejects only one here. Benjamini–Hochberg lands in between, keeping four findings while guaranteeing that, on average, no more than 5% of the announced discoveries are false.

Adjusted p-values

Software usually reports BH-adjusted p-values (often called q-values), which can be compared directly with the chosen FDR level. Each is computed as (m ÷ i) × p₍ᵢ₎, then forced never to exceed the adjusted value of any higher-ranked test. For the data above:

Rank	1	2	3	4	5	6	7
Raw p	0.001	0.008	0.012	0.018	0.031	0.038	0.049
Adjusted p	0.010	0.040	0.040	0.045	0.062	0.063	0.070

For instance, rank 2 gives (10 ÷ 2) × 0.008 = 0.040, and rank 4 gives (10 ÷ 4) × 0.018 = 0.045. Ranks 1 to 4 have adjusted values at or below 0.05, matching the four rejections, while ranks 5 to 7 sit just above 0.05. Reading adjusted p-values is easier than repeating the step-up procedure, and the answer is the same.

What the Guarantee Does and Does Not Say

The Benjamini–Hochberg procedure guarantees that the expected proportion of false discoveries is at most q, provided the tests are independent or positively dependent. Three cautions follow:

• It is an average over many repetitions of the whole experiment. In any single analysis, the actual share of false discoveries could be higher or lower than 5%.
• It says nothing about any individual finding. A rejected hypothesis with an adjusted p-value of 0.045 is not "95% likely to be real."
• Its guarantee is proven for independent tests and for certain kinds of positive dependence. For arbitrary dependence, a more conservative variant, the Benjamini–Yekutieli procedure, is available.

Real-World Uses

Genomics: tens of thousands of genes are tested for differences between conditions, and researchers want a shortlist for follow-up experiments in which a known small fraction of leads may be false.

Experimentation platforms: companies running many concurrent experiments, or examining many metrics per experiment, use FDR control to keep the proportion of spurious "wins" in check without giving up nearly all power.

Exploratory research: scanning many variables for associations, an activity closely tied to the p-hacking risks discussed in the hypothesis testing framework.

Common Mistakes

Stopping at the first failure. The rule looks for the largest rank that passes, not the first that fails. If rank 5 fails but rank 6 passes, ranks 1 to 6 are all rejected.

Selecting the tests after seeing the data. The correction must count every test that was performed, including ones not reported. Choosing which p-values to submit after looking at them defeats the purpose.

Confusing FDR with FWER. An FDR of 5% does not mean a 5% chance of any error. With 100 findings, several may be false and that is within the guarantee.

Using it for a single confirmatory claim. When only one hypothesis matters and a false positive is costly, FDR control is the wrong tool. Follow-up planning with statistical power in mind is a better use of effort.

Is Benjamini–Hochberg more powerful than Bonferroni?

Yes, generally by a wide margin when there are many tests and a fair number of true effects. It controls a weaker error rate, the false discovery rate rather than the family-wise error rate, which is precisely why it can reject more hypotheses.

What q level should I choose?

There is no universal rule. 0.05 is conventional, but exploratory screens sometimes use 0.10 or higher when follow-up is cheap, while costly follow-up justifies a stricter level such as 0.01.

How does this relate to post-hoc tests after ANOVA?

Post-hoc procedures such as Tukey's HSD control the family-wise error rate across pairwise comparisons. FDR control is usually chosen instead when the number of comparisons is large and the goal is screening.

Can I apply the correction to confidence intervals?

FDR-style adjustments for intervals exist, but they are more delicate than for p-values. When intervals are the main output, simultaneous intervals based on Bonferroni or related methods are more commonly reported.