Simpson's Paradox: When Every Subgroup Disagrees With the Whole

Q: What is Simpson's paradox?

Simpson's paradox occurs when a statistical association observed within every subgroup of a population reverses direction when the subgroups are combined. It is not a logical contradiction: the aggregated and disaggregated numbers are both correct arithmetic. The paradox lies in the assumption that both can be read as the same causal claim, which they cannot.

Q: Should you always look at the subgroups rather than the aggregate?

No. Which analysis is correct depends on the causal structure, not on the data. If the grouping variable is a common cause of both the treatment and the outcome, you must condition on it. If the grouping variable is itself affected by the treatment — a mediator — conditioning on it blocks part of the causal effect and the aggregate is correct. Identical tables demand opposite answers under different causal stories.

Q: What is a confounder?

A confounder is a variable that causally influences both the treatment and the outcome, creating a non-causal path between them. In a causal diagram this is a common ancestor of treatment and outcome. Because the association it induces is not causal, it must be adjusted for — by stratification, matching, regression, or weighting — before the remaining association can be read as an effect.

Q: What does the do-operator mean?

P(Y | do(X = x)) denotes the distribution of Y in a hypothetical world where X is set to x by intervention, severing the arrows that normally determine X. This differs from the conditional P(Y | X = x), which describes passive observation and therefore includes information carried by the causes of X. Judea Pearl's do-calculus provides rules for deciding when the interventional quantity can be computed from purely observational data.

Q: Did Simpson's paradox really occur in the Berkeley admissions case?

Yes. In the 1973 graduate admissions data, the overall acceptance rate was about 44 percent for men and 35 percent for women, yet most individual departments admitted women at an equal or slightly higher rate than men. The explanation is that women applied disproportionately to departments with low acceptance rates for all applicants. The 1975 Science paper by Bickel, Hammel, and O'Connell remains the standard reference.

In 1973 the University of California, Berkeley, examined its graduate admissions for evidence of sex bias. The aggregate figures looked damning: about 44% of male applicants were admitted and about 35% of female applicants. The difference was far too large to attribute to chance across more than twelve thousand applications.

Then the statisticians disaggregated by department. In department after department, women were admitted at a rate equal to or slightly higher than men’s. Six of the largest departments showed either no bias or a small advantage for women. Every part of the data pointed one way; the whole pointed the other.

Nothing had been computed incorrectly. Both sets of numbers were exactly right. This is Simpson’s paradox, and resolving it requires something that no amount of additional data can supply.

The Arithmetic

The paradox is a fact about fractions, and a small example makes it transparent. Two treatments for kidney stones, split by stone size:

	Small stones	Large stones	Overall
Treatment A (open surgery)	81/87 = 93%	192/263 = 73%	273/350 = 78%
Treatment B (puncture)	234/270 = 87%	55/80 = 69%	289/350 = 83%

Treatment A wins on small stones. Treatment A wins on large stones. Treatment B wins overall. These are real figures, from a 1986 British Medical Journal study by Charig and colleagues.

The mechanism is unequal weighting. Treatment A was given mostly to large stones (263 of 350 cases), which are harder to treat; treatment B mostly to small stones (270 of 350), which are easier. Each treatment’s overall rate is a weighted average of its two subgroup rates, and the weights differ wildly between rows. Algebraically, there is no obstacle to

$\frac{a_1}{n_1} > \frac{b_1}{m_1}, \quad \frac{a_2}{n_2} > \frac{b_2}{m_2}, \quad \text{yet} \quad \frac{a_1 + a_2}{n_1 + n_2} < \frac{b_1 + b_2}{m_1 + m_2}.$

A geometric picture makes it intuitive. Represent each subgroup’s record as a vector $(n, a)$ — trials and successes — so that the success rate is the slope. Combining subgroups adds the vectors. Two vectors can each have smaller slope than their counterparts while their sum has larger slope, provided the lengths are arranged appropriately. Vector addition does not respect slope comparison.

Simpson described the phenomenon in 1951, though Karl Pearson noted it in 1899 and Udny Yule in 1903; it is sometimes called the Yule–Simpson effect. The reversal is not rare. Given random $2\times2\times2$ tables, it occurs in roughly 2% of cases — common enough to expect in any large observational dataset.

Which Number Is Right?

Here the discussion usually goes wrong. A frequent piece of advice is that the disaggregated analysis is always the correct one, since it “controls for” a variable. This is false, and the demonstration is that identical tables can require opposite answers.

Case 1: the grouping variable is a cause of treatment. In the kidney stone data, doctors chose open surgery for severe cases. Stone size influenced both which treatment a patient received and how likely they were to recover. Size is a confounder. The aggregate comparison is contaminated: treatment A looks worse partly because it was given to sicker patients. The correct answer is the subgroup analysis — treatment A is better.

Case 2: the grouping variable is an effect of treatment. Now imagine a drug for hypertension, and the third variable is post-treatment blood pressure. The drug works by lowering blood pressure. Splitting patients by their post-treatment blood pressure and comparing within strata removes precisely the mechanism you are trying to measure. Here the aggregate is correct and the subgroup analysis is the misleading one.

The two cases can produce numerically identical tables. No procedure operating on the numbers alone can distinguish them. The information needed — whether the third variable precedes or follows treatment in the causal order — is not in the data at all. It comes from knowledge of the subject matter.

This is the general lesson, and it applies well beyond Simpson’s paradox: statistical association is symmetric, causation is not, and no symmetric quantity can settle an asymmetric question without additional input.

Causal Diagrams

The formal framework that makes the distinction precise is Judea Pearl’s structural causal models, built on directed acyclic graphs.

A causal DAG has variables as nodes and a directed edge $X \to Y$ whenever $X$ is a direct cause of $Y$ . The graph encodes a factorisation of the joint distribution,

$P(x_1, \ldots, x_n) = \prod_{i} P\!\left(x_i \mid \mathrm{pa}(x_i)\right),$

where $\mathrm{pa}$ denotes the parents of a node. Three elementary configurations describe how association flows along a path:

Chain $X \to Z \to Y$ : $Z$ is a mediator. $X$ and $Y$ are associated; conditioning on $Z$ removes the association.
Fork $X \leftarrow Z \to Y$ : $Z$ is a confounder. $X$ and $Y$ are associated with no causal link between them; conditioning on $Z$ removes the spurious association.
Collider $X \to Z \leftarrow Y$ : $X$ and $Y$ are independent; conditioning on $Z$ creates an association between them.

The collider is the counterintuitive one, and it explains why “controlling for more variables” is not automatically safer. If two independent causes both produce an effect, learning the effect makes them dependent: if the alarm sounded and it was not a burglar, it was probably an earthquake. Conditioning on a collider — or on any descendant of one — manufactures correlation where none existed. This is the mechanism behind Berkson’s paradox in hospital data and behind selection bias generally.

Formally, a path is blocked by a conditioning set $S$ when it contains a chain or fork whose middle node is in $S$ , or a collider whose middle node and all its descendants are outside $S$ . Two variables are d-separated by $S$ when every path between them is blocked, which implies conditional independence given $S$ .

The Do-Operator and the Back-Door Criterion

Pearl’s central notational move separates seeing from doing. Write $P(Y \mid X = x)$ for the distribution of $Y$ among individuals observed to have $X = x$ , and

$P(Y \mid \mathrm{do}(X = x))$

for the distribution of $Y$ in a population where $X$ is set to $x$ by intervention. In the graph, the intervention deletes all arrows into $X$ — the value is imposed rather than generated — leaving a modified graph $G_{\bar{X}}$ . The two quantities coincide only when $X$ has no incoming arrows from anything relevant, which is exactly what randomisation achieves: the coin flip replaces every natural cause of treatment assignment.

The question of when an interventional quantity can be recovered from observational data is answered, in the most common case, by the back-door criterion. A set $S$ of variables satisfies it relative to $(X, Y)$ if

no node in $S$ is a descendant of $X$ , and
$S$ blocks every path between $X$ and $Y$ that begins with an arrow into $X$ .

When $S$ qualifies, the adjustment formula applies:

$P(Y = y \mid \mathrm{do}(X = x)) = \sum_{s} P(Y = y \mid X = x, S = s)\, P(S = s).$

Read it carefully: this is a weighted average of the subgroup rates, but weighted by the marginal distribution $P(S = s)$ in the whole population — not by the treatment-specific distribution that produced the aggregate table. That difference in weighting is exactly the arithmetic of Simpson’s paradox, now derived rather than guessed.

Apply it to the kidney stones. The DAG is $\text{Size} \to \text{Treatment}$ , $\text{Size} \to \text{Recovery}$ , $\text{Treatment} \to \text{Recovery}$ . Size satisfies the back-door criterion. The marginal distribution of stone size is $357/700 = 0.51$ small, $343/700 = 0.49$ large. So

$P(\text{recover} \mid \mathrm{do}(A)) = 0.93(0.51) + 0.73(0.49) = 0.83,$ $P(\text{recover} \mid \mathrm{do}(B)) = 0.87(0.51) + 0.69(0.49) = 0.78.$

Treatment A is better, by 5 percentage points. The naive aggregate said the opposite because it weighted each treatment by its own patient mix rather than by a common standard.

Now apply the criterion to the blood-pressure case. There, the third variable is a descendant of treatment, so it violates condition 1. The back-door criterion refuses to license adjustment, and the correct estimate is the unadjusted one. Same arithmetic, different graph, different answer — and the framework tells you which without ambiguity.

Berkeley Revisited

In the admissions data, the causal structure is $\text{Sex} \to \text{Department choice} \to \text{Admission}$ , together with $\text{Department} \to \text{Admission}$ reflecting differing selectivity. Department is a mediator, not a confounder — applicants chose departments after, and partly because of, their other characteristics.

This changes what the two analyses mean. The aggregate difference is the total effect of sex on admission, and it was real: women were admitted less often overall. The within-department comparison is the direct effect, holding department fixed, and it was essentially zero or slightly favourable to women. Both are legitimate quantities answering different questions.

Bickel, Hammel, and O’Connell drew the appropriate conclusion in their 1975 Science paper: the disparity arose not from departmental gatekeeping but from the distribution of applications across fields, with women applying in greater numbers to humanities and social science departments that were more competitive for everyone. Whether that pattern itself reflects earlier discrimination is a real question — but it is a question about the arrow $\text{Sex} \to \text{Department}$ , not about the admissions committees.

Decomposing a total effect into direct and indirect components is the subject of mediation analysis, and Pearl’s mediation formula makes it precise, along with the conditions (no unmeasured treatment-mediator confounding among them) under which the decomposition is identified.

Continuous Versions and Everyday Sightings

The paradox is not confined to contingency tables. Lord’s paradox is the continuous analogue: whether a group difference in change scores is analysed by comparing raw gains or by regressing follow-up on baseline can yield opposite conclusions, and again the resolution depends on the causal model rather than on the statistics.

Reversals of this kind appear routinely once you know to look.

US median wages rose slightly from 2000 to 2013 in aggregate while falling within every educational subgroup — the composition of the workforce shifted toward more-educated workers.
Batting averages: David Justice out-hit Derek Jeter in both 1995 and 1996 individually, yet Jeter had the higher combined average, because Jeter’s at-bats were concentrated in his stronger year.
Hospital mortality: referral centres often show worse raw survival than community hospitals, because they receive the sickest patients. Risk adjustment exists entirely to handle this.
Vaccine efficacy: in a population where vaccination rates are much higher in elderly groups, the raw count of severe outcomes among the vaccinated can exceed that among the unvaccinated even when the vaccine reduces risk substantially in every age stratum. Age-stratified rates are the meaningful quantity, and confusion over this point was widespread during the COVID-19 pandemic.

The last example shows the stakes. A reversal that a statistician recognises instantly as a weighting artefact can circulate publicly as evidence of the opposite conclusion.

The Broader Point

Simpson’s paradox is often filed as a curiosity about fractions. It is better understood as the clearest possible demonstration of a structural limit: a probability distribution does not determine a causal conclusion. Two different causal worlds can generate the same joint distribution over observed variables and yet demand opposite decisions.

What resolves the ambiguity is a model of the data-generating process — an assumption, stated explicitly, about what causes what. Randomised experiments make that model trivially true by construction. Observational studies must state it and defend it, and the graphical machinery exists so that the assumptions are visible and their consequences checkable rather than buried in a regression specification.

Pearl’s formulation, developed through the 1990s and 2000s and recognised with the 2011 Turing Award, gave the discipline a language for claims it had long wanted to make and had lacked the vocabulary to state precisely. The lasting message of the paradox is the same one: before asking which number is larger, ask which question the number answers.

Frequently asked

What is Simpson's paradox?

Simpson's paradox occurs when a statistical association observed within every subgroup of a population reverses direction when the subgroups are combined. It is not a logical contradiction: the aggregated and disaggregated numbers are both correct arithmetic. The paradox lies in the assumption that both can be read as the same causal claim, which they cannot.

Should you always look at the subgroups rather than the aggregate?

No. Which analysis is correct depends on the causal structure, not on the data. If the grouping variable is a common cause of both the treatment and the outcome, you must condition on it. If the grouping variable is itself affected by the treatment — a mediator — conditioning on it blocks part of the causal effect and the aggregate is correct. Identical tables demand opposite answers under different causal stories.

What is a confounder?

A confounder is a variable that causally influences both the treatment and the outcome, creating a non-causal path between them. In a causal diagram this is a common ancestor of treatment and outcome. Because the association it induces is not causal, it must be adjusted for — by stratification, matching, regression, or weighting — before the remaining association can be read as an effect.

What does the do-operator mean?

P(Y | do(X = x)) denotes the distribution of Y in a hypothetical world where X is set to x by intervention, severing the arrows that normally determine X. This differs from the conditional P(Y | X = x), which describes passive observation and therefore includes information carried by the causes of X. Judea Pearl's do-calculus provides rules for deciding when the interventional quantity can be computed from purely observational data.

Did Simpson's paradox really occur in the Berkeley admissions case?

Yes. In the 1973 graduate admissions data, the overall acceptance rate was about 44 percent for men and 35 percent for women, yet most individual departments admitted women at an equal or slightly higher rate than men. The explanation is that women applied disproportionately to departments with low acceptance rates for all applicants. The 1975 Science paper by Bickel, Hammel, and O'Connell remains the standard reference.

Simpson's Paradox: When Every Subgroup Disagrees With the Whole

The Arithmetic

Which Number Is Right?

Causal Diagrams

The Do-Operator and the Back-Door Criterion

Berkeley Revisited

Continuous Versions and Everyday Sightings

The Broader Point

Frequently asked

Continue reading

Zero-Knowledge Proofs: Convincing Without Revealing

Percolation Theory: Phase Transitions in Random Media

Optimal Transport: Moving Earth as Efficiently as Possible

Keep exploring