Effect Size and Cohen's d: Why Significance Isn't Enough

A study finds a "statistically significant" difference between two groups. Should you care? On its own, a p-value can't answer that — it only tells you whether an effect is probably real, not whether it's large enough to matter. Effect size fills that gap, giving a standardized measure of how big a difference actually is, independent of sample size.

Why P-Values Alone Aren't Enough

Effect size is the natural companion to every hypothesis test covered elsewhere on this site — t-tests, chi-square tests, and ANOVA all answer "is this difference probably real," while effect size answers the equally important "how big is it," and a complete report of any test result includes both.

A p-value is heavily influenced by sample size. With a large enough sample, even a genuinely trivial difference between two groups can produce a very small p-value — the sample is simply large enough to detect that tiny difference with confidence. This means "statistically significant" (p < 0.05) and "practically important" are genuinely separate questions, and conflating them is one of the most consequential misreadings in applied statistics. Effect size answers the second question directly.

Cohen's d: The Standard Effect Size for Two Means

Cohen's d expresses the difference between two group means in units of pooled standard deviation, making it comparable across studies that measure completely different things on different scales. A d of 1.0 means the two group means differ by a full standard deviation — a substantial, visually obvious difference; a d of 0.1 means they differ by only a tenth of a standard deviation — a difference that would be nearly invisible on a histogram of the two groups.

Worked Example

Problem: A new teaching method produces a class mean of 78 (SD 10), compared to the standard method's class mean of 74 (SD 10, same for both groups for simplicity). Calculate Cohen's d.

d = (78 − 74) / 10 = 4/10 = 0.4

By conventional benchmarks (below), this is a small-to-medium effect — a real, meaningful difference, but not an enormous one. Reporting this alongside the t-test's p-value gives a much fuller picture than the p-value alone: "statistically significant, moderate effect size" is a genuinely different and more complete finding than "statistically significant" on its own.

Conventional Benchmarks

• d ≈ 0.2 — small effect
• d ≈ 0.5 — medium effect
• d ≈ 0.8 — large effect

These benchmarks come from psychologist Jacob Cohen's own influential (if somewhat arbitrary, by his own admission) 1988 proposal, intended as rough, field-general orientation points rather than strict thresholds. What counts as a meaningful effect genuinely varies by field — a d of 0.2 in medicine might represent a clinically important, life-saving difference; the same d of 0.2 in an educational intervention might be considered too small to justify the cost of implementing it widely.

Why Large Samples Make Small Effects "Significant"

Consider two studies testing the same real underlying effect (a true d of 0.1, genuinely small). Study A samples 30 people per group; Study B samples 3,000 people per group. Study B is far more likely to produce a statistically significant p-value for that same small effect, purely because larger samples make hypothesis tests more sensitive to detecting any real difference, however small. This is exactly why very large studies (common in tech A/B testing, where millions of users can be included) routinely report statistically significant results for effects too small to matter commercially — and why sophisticated readers of research always check effect size, not just significance, especially for large-sample studies.

Real-World Uses

Meta-analysis: combining results across many studies requires converting each study's finding into a common effect size metric (often Cohen's d), since raw means and p-values from different studies with different scales and sample sizes aren't directly comparable.

Clinical trial design: a trial's required sample size is calculated based on the smallest effect size considered clinically meaningful — a trial without a specified target effect size, aiming only for "statistical significance," risks being underpowered for effects that matter or overpowered (wastefully large) for ones that don't.

Educational research: effect sizes are standard for comparing the impact of different teaching interventions, since they allow interventions studied across different schools, tests, and scales to be meaningfully ranked against each other.

Common Mistakes

Reporting only significance, never effect size. A p-value alone leaves the reader unable to judge whether a "significant" finding is practically meaningful — reputable journals and reporting standards increasingly require effect size alongside every significance test for exactly this reason.

Applying Cohen's benchmarks rigidly across every field. As covered above, Cohen himself cautioned these were rough, general-purpose reference points, not universal cutoffs. A "small" effect by these generic benchmarks can be enormously important in a high-stakes medical context, and a "large" effect can be unremarkable in a field where natural variability is very high.

Confusing a large effect size with statistical significance, or vice versa. These are genuinely independent properties of a result — a study can have a large effect size but fail to reach statistical significance (typically due to a small sample, giving low statistical power to detect even a real, substantial effect), or a tiny effect size that IS statistically significant (typically due to a very large sample). Both numbers are needed together to properly evaluate a finding.

Effect Size and Statistical Power

Effect size and statistical power are closely linked in study planning. Before collecting data, researchers specify the smallest effect size they consider worth detecting, then calculate the sample size needed to detect that effect with adequate power (conventionally 80%) — the same logic behind the sample size calculator elsewhere on this site, just applied to a hypothesis test's power rather than a confidence interval's margin of error. Skipping this step and simply collecting "as much data as convenient" risks an underpowered study (too small to reliably detect a real effect of practical interest) or an overpowered one (large enough to find statistically significant, but practically trivial, effects and waste resources doing so).

A Second Worked Example: Interpreting a Weak but Real Effect

Problem: A large study (n=50,000 per group) finds two website designs differ in average session length by 2 seconds, with SD of 60 seconds in both groups, and this difference is highly statistically significant (p < 0.001). Should the company redesign the site based on this result?

d = 2/60 ≈ 0.033 — a genuinely tiny effect, far below even the "small" benchmark of 0.2.

Despite the impressively small p-value, the effect itself is close to negligible in real terms — 2 seconds out of a typical 60-second session is barely noticeable to any individual user. Whether this justifies a redesign depends entirely on cost versus benefit at scale (across millions of users, small per-user effects can aggregate into meaningful business impact) — but the p-value alone, without the effect size, would have given a seriously misleading impression of how large this finding actually is.