Bias and Traps: Simpson’s Paradox, Missing Answers and Cause
See how correct numbers can still mislead: hidden groups that reverse a result, surveys that only hear from some people, and correlation mistaken for cause.
In this lesson you will
- Explain Simpson’s paradox with an example
- Explain non-response and survivorship bias
- Say why correlation is not cause
- Ask whether groups face the same conditions
About 30 minutes
Numbers can be right and still mislead
Omar proudly tells Leila that Sam is the better mechanic: “Sam fixes things first time far more often than Ola.” He has the numbers. But Ola is not convinced, because she knows she gets the harder jobs. Who is right? This lesson shows that an analysis can have correct arithmetic and still reach a false conclusion. It introduces three traps that catch beginners and experts alike: hidden groups, missing answers and confusing correlation with cause.
Trap one: Simpson’s paradox
Simpson’s paradox happens when a trend appears in every group of data, but disappears or reverses when the groups are combined. The reason is a hidden factor that decides both who ends up in each group and how well they do. Here the made-up figures give each mechanic a number of easy and hard jobs, and the share fixed first time.
import random
rows = {"Ola": {"easy": (9, 10), "hard": (30, 90)}, "Sam": {"easy": (80, 90), "hard": (3, 10)}}
for who, jobs in rows.items():
done = sum(a for a, b in jobs.values())
total = sum(b for a, b in jobs.values())
parts = ", ".join(k + " " + str(round(a / b * 100)) + "%" for k, (a, b) in jobs.items())
print(who + ": overall " + str(round(done / total * 100)) + "% | " + parts)
rng = random.Random(8)
happy = [rng.choice([1, 2, 3, 4, 5, 5, 4, 3]) for _ in range(200)]
answered = [x for x in happy if rng.random() < (0.15 * x)]
print("everyone, mean rating:", round(sum(happy) / len(happy), 2))
print("responders only:", round(sum(answered) / len(answered), 2))
print("response rate:", str(round(len(answered) / len(happy) * 100)) + " percent")
Ola: overall 39% | easy 90%, hard 33%
Sam: overall 83% | easy 89%, hard 30%
everyone, mean rating: 3.4
responders only: 3.97
response rate: 46 percent
Look at the first two lines. Overall, Sam looks far better: 83 percent against 39 percent. But compare them job type by job type. On easy jobs Ola gets 90 percent against Sam’s 89, and on hard jobs Ola gets 33 against 30. Ola is better in both groups. The explanation is the mix: Ola was given 90 hard jobs out of 100, and Sam only 10. Hard jobs fail more often for everyone, so a mix full of them drags Ola’s overall figure down. The hidden factor is the difficulty of the job.
The lesson is not “always split the data”. It is: before you compare groups, ask whether they face the same conditions. Which view to trust, the overall or the split, depends on the real-world story, and here the split is the fairer one.
Trap two: the people who answered
The second half of the output simulates a survey. Two hundred customers have a true satisfaction, with a mean of 3.4. But happier customers are more likely to answer: each person answers with a chance that rises with their score. The result is a survey of only the people who responded, with a mean of 3.97. The response rate is 46 percent. The survey looks cheerful and is quite wrong, because the unhappy customers stayed silent. This is called non-response bias: the people who respond differ from those who do not. A close cousin is survivorship bias, where you only see the things that survived, such as the shops still open, and so learn from a lucky sample.
In the workshop file, five visits have no rating. If those customers were more often unhappy, the mean rating of 3.88 flatters the workshop. You cannot know, but you can say so.
Trap three: correlation is not cause
Two measures that rise and fall together are correlated. It is tempting to say that one causes the other. In the workshop, longer repairs cost more, but spending more money does not make a repair longer. Both are driven by a third thing, the type of repair. Often a hidden third factor creates a link. And sometimes it is pure chance, especially with few data points. To show a cause you need more than a pattern, for example a controlled test or a good explanation backed by evidence.
Comparing two people or groups on one overall number without asking whether they had the same kind of work. Hidden mix differences can reverse the story.
The mechanics, the job counts and the survey are all made up. The survey is a simulation with a fixed seed so that the numbers repeat. It illustrates how bias works, and the size of the effect in real life depends on the real behaviour of people.
Practice for this lesson
Check what you learned
Five quick questions. Pick an answer and you will see straight away whether it is right and why.
What is Simpson’s paradox?
- A calculation error
- A missing value
- A chart type
- A trend in every group that reverses when the groups are combined
Ola was better on easy and hard jobs but worse overall.
What caused the reversal between Ola and Sam?
- Sam cheated
- Ola was given far more of the hard jobs
- The data was duplicated
- The jobs were sorted by date
Hard jobs fail more often for everyone.
What is non-response bias?
- Respondents lie on purpose
- Surveys are always too short
- Nobody answers
- People who respond differ from those who do not, so results skew
Happier customers answered more often in the simulation.
What does survivorship bias mean?
- You only see what survived, so the sample is lucky
- You count the same item twice
- You survey everyone
- You ignore dates
Shops still open are not a fair picture of all shops.
Why is correlation not proof of cause?
- Correlation is always false
- Cause is never real
- A hidden third factor or chance may create the link
- Numbers cannot be compared
Repair type drives both time and cost.
Summary
- A trend in every group can reverse when the groups are combined, because of a hidden factor.
- Surveys that only hear from some people give a skewed picture.
- Correlation can come from a third factor or from chance.
- Before comparing groups, ask whether they faced the same conditions.
Questions
Which view should I trust, the overall or the split?
It depends on the story behind the data. If a hidden factor decides who is in each group, the split is usually fairer. Explain your reasoning in the report.
How do I reduce non-response bias?
Make it easy to answer, follow up, and compare the people who answered with those who did not on whatever facts you have. Always report the response rate.
Can I ever prove cause?
With careful design, such as a controlled test with random groups, you can come close. From ordinary records, you can usually show only that things go together.
