Run a simulated experiment with two groups. Watch the p-value shift as you change effect size and sample size. Use permutation tests to build genuine intuition for what p-values really measure, and what they don't.
Experiment Design
Medium effect: moderate separation
Small sample: limited power
Standard threshold: 5% false-positive rate
Test Type
Tests for a difference in either direction (p × 2)
Group Data
Showing preview: run the experiment to generate real data.
Test Type
Two-tailed: Are the groups different in either direction? Divides α between both tails. Use this when you have no prior directional hypothesis.
Null Hypothesis Distribution
Run an experiment to see the t-distribution.
t-distribution (df = 58) | α = 0.05 | Two-tailed
Permutation Test
Randomly shuffle group labels and recompute the t-statistic. How often does chance produce a result as extreme as observed?
Run an experiment first.
What p-value means
P(data this extreme | H₀ true). It is NOT the probability the null is true, nor the probability your result is due to chance alone.
Significance ≠ importance
With large samples, tiny and practically irrelevant differences become highly significant. Always consider effect size alongside p-values.
Permutation tests
Shuffling group labels repeatedly shows the null distribution empirically. No distributional assumptions required.