Evaluate evidence and communicate limits
Spot selection bias, confounding and Simpson’s paradox, and report findings in proportion to the evidence.
- Recognize selection bias and confounding
- Explain Simpson’s paradox with real numbers
- Communicate what a finding does and doesn’t show
Before trusting a conclusion, ask how the data could mislead:
- Selection bias - the data systematically over- or under-represents some cases: surveys answered only by the motivated, reviews left mostly by the angry, planes studied only if they returned.
- Survivorship bias - a famous special case: during World War II, Abraham Wald pointed out that bullet holes on returning planes showed where a plane could be hit and survive; the armor belonged where the holes weren’t.
- Confounding - a third factor drives both the “cause” and the outcome.
- Measurement - the question wording, the device or the definition shapes the number.
Try it
Up to 15 XPWhat’s wrong with this conclusion?
Name the main problem with each claim.
“An online poll on our site finds 90% of people love our product.”
“People who take vitamins are healthier, so vitamins improve health.”
“Hospitals with more doctors have more deaths, so doctors are dangerous.”
“A randomized trial of 2,000 users found the new onboarding raised retention by 4 points (95% CI 2-6).”
“Students who use the tutoring center get lower grades, so tutoring hurts.”
Simpson’s paradox
A trend can reverse when you combine groups. Suppose Hospital A has better survival rates than Hospital B for mild cases and for severe cases - yet worse survival overall. How? A treats far more severe cases, and severe cases have lower survival everywhere. Severity is a confounder, and comparing the raw totals mixes it in.
A famous real example: in 1973, UC Berkeley’s graduate admissions looked biased against women overall, but within most departments women were admitted at similar or higher rates - women had applied more to the departments with the lowest admission rates.
a_overall = (95 + 300) / (100 + 400)
b_overall = (360 + 60) / (400 + 100)
print(f"A overall {a_overall:.0%}, B overall {b_overall:.0%}")A overall 79%, B overall 84%
Key takeaways
Ask how cases got into the data before trusting it.
Confounders can create, hide or reverse relationships - including Simpson’s paradox.
Communicate scope, uncertainty and limitations along with the finding.
Lesson quiz
5 questions · pass with 4 correct · up to 50 XP
Passing this quiz completes the lesson and keeps your streak going. Questions you miss come back in review sessions later.
Practice: write Python
Write Python in the editor and run it against sample inputs. Python runs locally in your browser using a WebAssembly runtime.
Report a proportion
Read two integers on one line: the number of positive responses and the total responses. Print the positive-response proportion as a percentage rounded to the nearest whole percent, followed by %.
- 68 of 100
- One of three
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
Uncover Simpson’s paradox
Read a CSV with hospital,severity,survived,patients. Print each hospital’s overall survival rate as overall H: R% (alphabetically), then each severity’s rates as SEVERITY H: R% (severities and hospitals alphabetically), all to 1 decimal. Compare the two views.
- Two hospitals
Python runs in a sandboxed browser worker with a 60 second time limit. Its runtime loads from the Pyodide CDN; your code stays in this browser.
Questions about this lesson
Stuck? Ask. Figured something out? Share it. Explaining is one of the best ways to learn.
Loading posts…