The surprising result
Imagine comparing the success rates of two training programs, A and B. Overall, 82 of 100 people in program A succeed, compared with 35 of 100 in program B. At first glance, A looks much more effective. But suppose the participants faced tasks of different difficulty. Among people doing easier tasks, A has a 90% success rate and B has a 95% success rate. Among those doing harder tasks, A has a 10% success rate and B has a 20% success rate. B has the higher rate in both subgroups, yet A has the higher rate overall.
This reversal is an example of Simpson’s paradox. It is not a mathematical error: each percentage uses a different set of people as its denominator. The overall rates combine the easier and harder tasks in different proportions, so they answer a different question from the subgroup rates.
Work through the denominators
Here are the counts behind the example. In the easier-task group, 81 of 90 participants in A succeed, while 19 of 20 in B succeed. In the harder-task group, 1 of 10 in A succeeds, while 16 of 80 in B succeed. Adding the successes gives A 82 out of 100 and B 35 out of 100. Within each difficulty level, however, B’s rate remains higher: 95% versus 90% for easier tasks, and 20% versus 10% for harder tasks.
The key is where the participants are concentrated. Ninety percent of A’s participants have easier tasks; only 20% of B’s do. Because success is much more common on easier tasks, A’s overall rate gets a large contribution from that subgroup. B’s overall rate gets most of its contribution from harder tasks. The pooled comparison therefore reflects both success rates and the mix of task difficulty.
One way to see the effect of that mix is to give the two difficulty levels equal weight for each program. A’s average of the two subgroup rates is 50%; B’s is 57.5%. This calculation answers a hypothetical question about an equal mix of easier and harder tasks. It does not replace the observed overall rates, which describe the participants who were actually in each program.
Choose the comparison that answers your question
Neither the overall figure nor the subgroup figures are automatically the “right” result. If you want to describe what happened among all participants as enrolled, the overall rates are relevant. If you want to compare programs for people facing similar task difficulty, the subgroup rates may be more informative. State which question you are answering before deciding which comparison to emphasize.
Even then, these numbers alone do not show that either program caused a better outcome. Participants may differ in other ways, and task difficulty may not have been measured consistently. A subgroup comparison can remove one obvious difference between groups without removing every possible explanation. The example is a reason to investigate the study design, not to declare a winner.
How to check for a reversal in real data
Start with counts, not percentages alone. For each group being compared, write down the number of successes and the total number of observations, both overall and within subgroups. Check that the subgroup counts add up to the overall counts. Then compare rates within each subgroup and look at how observations are distributed across those subgroups. Large differences in that distribution are a clue that pooling could change the picture.
Choose subgroups for a reason rather than searching through many splits until an interesting reversal appears. Ask whether a characteristic, such as task difficulty, is relevant to the outcome and measured in a comparable way across groups. Also check subgroup sizes: a rate based on a handful of cases can shift sharply when one outcome changes. If you report a pooled rate, explain what it combines. If you report an adjusted or reweighted comparison, explain the weights and the question they represent.
The lesson beyond the paradox
Simpson’s paradox is a reminder that an average can conceal important differences in who or what is being averaged. When a striking overall trend appears, ask whether the groups had comparable starting conditions and whether a meaningful split changes the result. When subgroup trends appear, ask whether those groups are large enough and relevant enough to support the interpretation.
The goal is not to distrust every summary number. It is to connect each number to its denominator and to the question it can answer. Showing both overall and subgroup results, with their counts and a clear explanation of the group mix, lets readers see the pattern without mistaking one comparison for another.
