Running an A/B test is easy. Running one that produces a trustworthy, actionable answer — instead of a number you can spin however you like — is the actual skill.
1Hypothesis Before Tool
Every test should start as a written hypothesis: 'We believe [change] will cause [metric] to move because [reasoning]. We'll know we're right if [threshold].' Skipping this step is how teams end up testing button colors for months without ever validating anything that matters to the business.
2The Peeking Problem
Checking results daily and stopping the moment you see significance inflates your false-positive rate dramatically — a phenomenon statisticians call 'peeking.' Decide your sample size and run duration before the test starts, and don't call the result until both are met, no matter how tempting the early trend looks.
3Step-by-Step Breakdown
Introduction. A/B testing splits users into two groups — a control seeing the current experience, a variant seeing a change — and measures which one performs better on a defined metric. It's how you replace 'I think this button should be green' with an actual answer.
Picking a Single Success Metric. Before launching, commit to ONE primary metric — signup rate, checkout conversion, 7-day retention. If you track ten metrics and cherry-pick whichever one moved, you're not experimenting, you're rationalizing. Secondary metrics are fine as guardrails, not as the verdict.
Statistical Significance and Sample Size. A result isn't real until it clears a significance threshold (commonly p < 0.05) with a large enough sample. Stopping a test early because the variant is 'winning' on day two is the single most common way teams ship changes that do nothing — or actively hurt the product.
Knowledge Check. Your A/B test hits statistical significance after only 6 hours, showing a 40% lift. What should you do next?
- →Ship the variant to 100% of users immediately
- →Let it run the full planned duration to account for novelty effects and day-of-week variation before deciding
Summary. A/B testing isn't a rubber stamp for ideas you already like — it's a tool for finding out when you're wrong. The discipline is in defining the metric and sample size upfront, then honoring the result even when it's not the one you wanted.
