Run 20 jelly bean tests on pure noise. Watch false positives appear at α=0.05. Apply corrections to control them.
Sign in to track progress
Jelly Bean Study
A researcher tests whether eating jelly beans of each of 20 colors is associated with acne. Each test compares the mean acne-severity score of a jelly-bean group against a control group (a two-sample test of means), but there is no real effect. All differences are pure noise.
Number of jelly bean colors tested (M)
Expected false positives under H₀:
M × α = 20 × 0.05 = 1.0
Even with no real effect, we expect ~1.0 tests to appear significant by chance.
Test Results
Multiple Comparison Corrections
Bonferroni Correction
Divide the significance threshold by the number of tests. Each test must now meet a much stricter standard to be called significant. Controls the family-wise error rate (FWER): the probability of making even one false positive.
α_adjusted = α / M = 0.05 / 20 = 0.0025
Caveat
Most conservative: very low false positive rate, but high false negative rate. Can miss true effects when M is large.
Run simulation first to apply corrections.
Assumption Checklist
0/4 checked4 assumptions remaining. Expand each to learn more.
Family-Wise Error Rate
FWER is the probability of making at least one false positive across all tests. Bonferroni and Holm control FWER at α.
False Discovery Rate
FDR is the expected proportion of false positives among all significant results. BH controls FDR at α (more lenient).
p-Hacking
Running many tests without correction and reporting only significant ones inflates the false positive rate. Pre-registration helps.