Bayesian A/B Test Analysis — Posterior Distribution Comparison

Description
Bayesian A/B analysis: priors, posteriors, win probability, practical significance, expected loss.

---

Frequentist p-values tell you whether a difference exists, not which variant is better or by how much. Bayesian analysis gives the probability B beats A by a practically significant margin. 1. PRIOR: Beta(1,1) uniform or informed Beta(α, β) from historical data. State source explicitly. 2. POSTERIOR: Beta(α+conversions, β+non-conversions) per variant. Example: A: 100/2000 → Beta(101,1901). B: 120/2000 → Beta(121,1881). 3. WIN PROBABILITY P(B > A): Sample both posteriors, count B > A. <0.90 = insufficient evidence. 4. PRACTICAL SIGNIFICANCE: Define δ (minimum detectable effect): checkout change 1%, button color 5%, copy 3%. Compute P(B > A + δ). Only declare winner if >0.95. 5. EXPECTED LOSS (PEAK): What if B is worse but you ship it? Expected loss = probability B is worse × cost of being wrong. Ship only if expected gain > expected loss. 6. SAMPLE SIZE (if no winner): n = (Z_α + Z_β)² × (σ_A² + σ_B²) / δ². Z_α=1.96 (95%), Z_β=0.84 (80% power). OUTPUT: Posterior parameters, win probability, practical significance, expected loss, sample size if needed.

Copy Optimize