Don't guess. Test.
1Designing the Test
Define your primary metric before you start. Are you testing for clicks, signups, or revenue? If you look at 50 metrics, you'll eventually find one that went up by chance. Stick to your primary goal.
2Sample Size & Duration
Run your test for at least one full business cycle (usually 7 days) to account for weekday vs weekend behavior. Use a sample size calculator to know when you have enough data to stop.
3The Ethics of Testing
Never test things that trick or harm the user (dark patterns). Experimentation should be used to improve the user experience, not to manipulate people into doing things they don't want to do.
4Step-by-Step Breakdown
A/B Testing (split testing) is comparing two versions of a web page or app feature to see which one performs better based on a specific metric.
Every experiment starts with a hypothesis: 'If we [change X], then [metric Y will increase] because [reason Z]'.
Statistical Significance is key. You need enough data (sample size) to be sure that the result wasn't just a coincidence.
In an A/B test, what is the 'Control' group?
- →The group that sees the new experimental feature
- →The group that sees the existing, unchanged version of the product
- →The group of engineers who run the test
- →The group of users who paid the most
Why is it important to test only ONE variable at a time in a standard A/B test?
- →Because testing more is too expensive
- →To isolate the cause of the result. If you change the color AND the text, you won't know which one caused the increase in clicks
- →Because the server can only handle one change
- →To make the data analysis easier for the PM
Level Up 🚀
Advanced cheat sheets, SEO tricks, and interview prep for this topic.
Browser Support
Fully supported.
Fully supported.
Fully supported.
Fully supported.
Accessibility (A11y)
1Inclusive Sample Design
An experiment's results are only valid for the population it was tested on. If your test cohort systematically excludes users on screen readers, older devices, or slow connections, the 'winning' variant may perform worse for those users even though the aggregate metric went up.
// Segment results, don't just trust the aggregate
const liftByCohort = groupBy(results, 'accessibilityMode');
// Check assistive-tech users didn't regress before shippingSEO Implications
- 1
Test Duration vs Crawl Stability
Running an A/B test on indexable pages (e.g., swapping headline copy or layout) can create inconsistent content for search crawlers hitting different variants. Use cloaking-safe testing tools and run tests only as long as needed to reach significance, then converge on a single canonical version.
Best Practices
Pre-Register Your Primary Metric
Write down the single metric that decides success before the test starts. Deciding after the fact which of 20 metrics 'moved' is p-hacking, not experimentation.
Track Guardrail Metrics
Monitor secondary guardrail metrics (page load time, error rate) alongside your primary metric so a win on conversion doesn't hide a regression elsewhere.
Frequent Bugs
Peeking at results daily and stopping the test the moment it looks significant, instead of waiting for the pre-calculated sample size.
Commit to a minimum runtime and sample size before launch, and only make a call once that threshold is met — early peeking inflates the false-positive rate.
Real-World Examples
Checkout Button Color Test
An e-commerce team suspects a green 'Buy Now' button will outperform the existing blue one. They run a 50/50 split test over two full weeks.
// Simplified experiment config
{
name: 'checkout-button-color',
variants: ['control-blue', 'treatment-green'],
primaryMetric: 'checkout_conversion_rate',
minSampleSize: 12000,
guardrails: ['page_load_ms', 'cart_abandonment_rate']
}