Walk through every stage of a rigorous A/B test: from power-based sample-size calculation to randomization, live data collection, and statistical analysis with a full hypothesis test.
Sign in to track progress
Define your null and alternative hypotheses before running the experiment.
Tests for any difference in either direction. More conservative: α is split across both tails, so z_α/2 = 1.960 at your α = 0.05
Choose the KPI you will measure and set the current baseline rate.
Set significance level (α), desired power (1−β), and minimum detectable effect.
n = [zα·√(2p̄(1−p̄)) + zβ·√(p₁(1−p₁) + p₂(1−p₂))]² / MDE²
p₁ = 0.050, p₂ = p₁ + MDE = 0.070, p̄ = 0.060
= [1.960×0.3359 + 0.842×0.3356]² / 0.000400
Per group (control & treatment)
2,213
Total: 4,426 participants
α
0.05
Power
80%
MDE
2.0%
Fill in both H₀ and H₁ to continue
Type I Error (α)
Rejecting a true null, a false positive. You choose α before the test.
Type II Error (β)
Failing to reject a false null, a false negative. β = 1 − Power.
MDE
Minimum Detectable Effect: the smallest difference worth detecting. Drives required sample size.
Peeking Problem
Checking results before full sample is collected inflates false positive rate. Stick to your plan.