Simpson's Paradox: When Aggregated Data Reverses the Real Trend

Simpson's paradox is one of the most genuinely startling results in applied statistics: a trend that appears clearly in every subgroup of a dataset can completely reverse once those subgroups are combined. It's not a computational error or a trick — the arithmetic on both sides is entirely correct. It's a real structural feature of how aggregation can hide, or even invert, what's actually happening underneath.

A Real Example: Comparing Two Kidney Stone Treatments

This example is based on data from a well-known study comparing two treatments for kidney stones. Patients were split by how severe their case was — small stones versus large stones — since large stones are inherently harder to treat successfully regardless of which treatment is used.

Look closely at what's happening here. Within small stones, Treatment A succeeds more often (93.1% vs. 86.7%). Within large stones, Treatment A also succeeds more often (73.0% vs. 68.8%). Treatment A wins in every single subgroup. And yet, in the combined totals across both subgroups, Treatment B has the higher overall success rate (82.6% vs. 78.0%). If you only looked at the combined row, you'd conclude Treatment B is better — the exact opposite of what every subgroup actually shows.

Why This Happens: An Unbalanced Lurking Variable

The reversal isn't a coincidence or an error — it happens because of how patients were distributed across the two treatments. Look at the sample sizes: Treatment A was given to far more large-stone patients (263, the harder cases) than small-stone patients (87), while Treatment B was given mostly to small-stone patients (270, the easier cases) and relatively few large-stone patients (80).

Because large stones have a lower success rate no matter which treatment is used, and Treatment A was disproportionately used on the harder large-stone cases, Treatment A's combined average gets pulled down by that heavier weighting toward difficult cases — even though it was genuinely the better treatment within each difficulty level. Stone size here is a lurking variable (sometimes called a confounding variable): a factor that influences the outcome and is unevenly distributed between the groups being compared, distorting the combined comparison even though it says nothing about which treatment is actually better.

The General Mechanism

Simpson's paradox requires two conditions together: a lurking variable that genuinely affects the outcome (stone size affects success rate, regardless of treatment), and that lurking variable being unevenly distributed across the groups being compared (Treatment A saw far more of the harder cases than Treatment B did). Remove either condition and the paradox disappears — if stone size didn't affect success rates, or if both treatments had been given to the same proportion of small vs. large cases, the combined totals would tell the same story as the subgroups.

How to Check Whether the Paradox Might Be Hiding in Your Data

Whenever comparing two groups on some overall outcome, it's worth actively asking: is there a plausible lurking variable — something that affects the outcome and might not be evenly distributed between the groups? If so, the comparison should be broken down by that variable before trusting the combined number, exactly as the kidney stone data needed to be split by stone size to reveal what was actually happening. This is precisely the logic behind stratified analysis and randomized controlled trials in medical research — randomization specifically works to spread lurking variables evenly across treatment groups, which is one of the reasons well-designed randomized trials are trusted more than simple observational comparisons.

Other Well-Known Instances

The kidney stone example above is one of the most frequently cited illustrations because the reversal is so stark, but the same pattern shows up across many fields. Batting averages in baseball have famously produced Simpson's paradox cases, where one player has a higher average than another in both halves of a season individually, yet a lower average for the full season combined — driven by an uneven number of at-bats in each half. University admissions data has also produced well-known cases where an entire institution's admission rates by gender looked biased in one direction, while admission rates within each individual academic department told a different, sometimes opposite, story — driven by which departments each group tended to apply to in different numbers, a department-level lurking variable playing the same role stone size played above. In every case, the resolution is the same: the aggregated figure and the subgroup figures are both arithmetically correct, and the real work is figuring out which one actually answers the question being asked.

Connection to Correlation, Causation, and Confounding

Simpson's paradox is a close relative of the confounding problem covered in Pearson correlation's discussion of why correlation doesn't imply causation — both involve a third variable distorting what looks like a direct relationship between two others. The paradox is really just confounding taken to its most dramatic possible extreme: instead of merely weakening or exaggerating an apparent relationship, an unbalanced lurking variable can flip its direction entirely. Anyone drawing a causal conclusion from combined or aggregated data, without checking whether a relevant subgroup breakdown tells a different story, is exposed to exactly this risk.

Common Mistakes

Trusting the aggregated number without checking for meaningful subgroups. Whenever group sizes could plausibly differ in composition (different proportions of a relevant lurking variable), the combined figure alone isn't safe to interpret causally — always ask what the picture looks like broken down by any variable that might matter.

Assuming a subgroup breakdown is always the "more correct" view. This isn't universally true either — sometimes the aggregated figure genuinely is the right one to report, if the aggregated population is what actually matters for the decision being made (a hospital administrator choosing a policy for all patients, regardless of stone size, might legitimately care about the overall combined rate). The key isn't "subgroups are always right" — it's understanding which comparison actually answers the real question being asked, and being explicit about why.

Manufacturing suspicion of every aggregated statistic. Simpson's paradox is a real and important risk to check for, but it doesn't occur in every dataset — most aggregated comparisons are perfectly trustworthy. The lesson is to actively check when a lurking variable is plausible and the group sizes look unbalanced, not to distrust every combined number by default.