Design A/B tests. Run 1000 experiments. Watch the confusion matrix tally true/false positives. See how power, Type I, and Type II error emerge from the math in real time.
Experiment Design
Medium effect: Medium real effect. Standard sample sizes can detect this with decent power.
Data Groups
Preview (deterministic)Confusion Matrix
0 totalEffect Exists
(H₁ true)
No Effect
(H₀ true)
Test
Rejects H₀
True Positive
Correct rejection
0
0% of all experiments
False Positive
Type I Error
0
0% of all experiments
Fails to
Reject H₀
False Negative
Type II Error
0
0% of all experiments
True Negative
Correct retention
0
0% of all experiments
Type I Rate
—
FP / (FP+TN)
Type II Rate
—
FN / (FN+TP)
Power (1−β)
—
TP / (TP+FN)
Running Summary
Current Parameters
Obs. Type I
—
Obs. Type II
—
Obs. Power
—
Insight
Run more experiments to see patterns emerge. Try at least 10 to get a sense of the randomness in hypothesis testing.
Run 100+ experiments and explore 3+ scenarios to complete this guide