Statistical Power: Why 'Not Significant' Doesn't Mean 'No Effect'

A study finds no significant difference between two treatments. Does that mean there's genuinely no effect — or did the study simply fail to detect a real one because it was too small? Statistical power answers exactly this question: it's the probability a test correctly detects a real effect, when one genuinely exists. A low-power study can miss real, important effects entirely, and understanding power is essential for designing a study that's actually capable of answering its own research question.

Power and Type II Error

Power is the natural companion to the significance level and effect size covered in hypothesis testing and effect size — where α controls the risk of a false positive, power controls the risk of missing a real finding, and the two are designed and reported together, not considered in isolation.

Recall from hypothesis testing that a Type II error (β) is failing to reject H₀ when it's actually false — missing a real effect. Power is simply 1 − β: the probability of correctly detecting a real effect when one exists. A power of 0.80 (a common conventional target) means an 80% chance of correctly detecting a real effect of the specified size, and correspondingly a 20% chance of missing it (a Type II error) even though it's genuinely there.

The Four Factors That Determine Power

• Sample size (n): larger samples increase power — more data gives a test a better chance of distinguishing a real effect from random noise.
• Effect size: larger true effects are easier to detect — a huge, obvious difference needs less data to detect reliably than a subtle, small one.
• Significance level (α): a stricter α (like 0.01 instead of 0.05) makes it harder to reject H₀ at all, which — holding everything else constant — reduces power, since fewer results clear the higher bar.
• Variability in the data: less noisy, more precise data increases power, since a real effect is easier to distinguish from a smaller amount of background variability.

These four factors are mathematically linked — a power analysis, run before data collection, typically fixes three of them (a target power like 0.80, a chosen α like 0.05, and the smallest effect size worth detecting) and solves for the fourth: the required sample size. This is exactly the same underlying logic as the sample size calculator's confidence-interval version, applied instead to a hypothesis test's power.

Worked Example (Conceptual)

Problem: A researcher wants to detect a Cohen's d of 0.5 (medium effect) between two groups, with 80% power at α = 0.05 (two-tailed). Standard power tables and software for this exact scenario indicate a required sample size of approximately 64 per group (128 total).

If the researcher instead only collects 20 per group, the study's power for detecting that same medium effect drops substantially — well below 80%, meaning a real medium-sized effect would likely go undetected, producing a "not significant" result that reflects the study's low power more than the true absence of an effect.

Post-Hoc (Retrospective) Power: A Common Misuse

Calculating power AFTER seeing a non-significant result — using the observed effect size from the very data that failed to reach significance — is a widely discouraged practice among statisticians. This "post-hoc power" is mathematically almost a direct restatement of the p-value itself, adding no genuinely new information, and using it to argue "the study just didn't have enough power" after the fact is generally considered statistically unsound. Power analysis is meant to inform study design BEFORE data collection, using an independently justified target effect size — not to explain away an unwelcome result afterward.

Real-World Uses

Clinical trial design: regulatory bodies typically require a documented power analysis before approving a trial, ensuring the planned sample size is genuinely capable of detecting a clinically meaningful effect if one exists.

Grant applications and research funding: funders often require a power analysis as part of a study proposal, to confirm the requested resources will produce a study capable of answering its stated research question.

A/B testing: the sample-size planning covered in A/B testing is a direct application of power analysis, ensuring a test runs long enough to reliably detect a meaningful conversion-rate difference if one truly exists.

Common Mistakes

Skipping power analysis entirely. Collecting "as much data as time and budget allow" without an explicit power calculation risks either an underpowered study wasting resources on a result too ambiguous to interpret, or an overpowered one that finds trivial effects significant, misdirecting attention toward practically unimportant findings.

Treating a single study's non-significant result as definitive. Especially with modest sample sizes, a non-significant finding is genuinely ambiguous evidence, not proof of no effect — this is exactly why systematic reviews and meta-analyses, which pool evidence across multiple studies, carry more weight than any single underpowered study's null result.

Using post-hoc power to defend a non-significant finding. As covered above, this is a widely discouraged practice precisely because it adds little genuine information beyond what the original p-value already conveyed, despite feeling intuitively like a useful diagnostic after the fact.

Power Analysis in Practice

Modern power analysis is almost always conducted with software (G*Power is a widely used free tool, and most statistical packages include power-calculation functions) rather than by hand, given how many interacting parameters (effect size, α, desired power, and the specific test being used) feed into the calculation. The manual conceptual example above illustrates the logic, but real power analyses for anything beyond a simple two-group comparison typically rely on these tools, and increasingly on simulation-based approaches for complex study designs where a closed-form power formula doesn't exist.

A Second Worked Example: The Cost of Underpowering

Problem: A pilot study finds a promising new drug shows improvement in 15 of 20 patients (75%) versus 10 of 20 on placebo (50%) — a seemingly large difference, but the study isn't statistically significant (p = 0.11 with this small sample). What should a researcher conclude?

With only 20 patients per group, this study has low power to detect even a meaningful 25-percentage-point difference in response rates — the observed effect looks genuinely promising, but the sample was too small to distinguish it confidently from chance. The correct conclusion isn't "the drug doesn't work" — it's "this study was underpowered to reach a confident conclusion, and a larger, properly powered follow-up trial is warranted" — exactly the kind of situation power analysis, done before the pilot, would have flagged as a risk in advance.