A/B Test Design - Sample Size, MDE & Duration

Description
A/B test design: sample size formula, MDE, duration rules, no-peeking sequential testing checklist.

---

Half of all A/B tests fail because they run too short. 'Run for a week or until significant' biases toward false positives and false negatives. SAMPLE SIZE FORMULA: n = (Z_a + Z_b)^2 x (p1(1-p1) + p2(1-p2)) / (p2 - p1)^2 where Z_a=1.96 (95% confidence), Z_b=0.84 (80% power). MDE: The smallest effect detectable with 80% power. Smaller MDE = larger sample. A B2B SaaS with 500 visitors/month cannot detect a 1% lift. DURATION RULES: - Minimum: 1 full business cycle (1 week B2C, 2 weeks B2B) - Rule of 7: each variant needs at least 7 conversions - No peeking: each peek inflates false positive rate PRACTICAL CHECKLIST: Sample size calculated, MDE worth shipping, guardrails defined, 7+ conversions per variant, decision rule p<0.05 AND lift > MDE. OUTPUT: Computed sample size, MDE, minimum duration, completed checklist.

Copy Optimize