A/B Testing: The Statistics Behind Product Decisions

A/B testing is hypothesis testing's most widely deployed real-world application — the engine behind product decisions at nearly every major tech company. Show one group of users version A of a webpage, another group version B, measure which performs better on some metric, and use statistics to decide whether the observed difference is real or just noise. The mechanics are exactly the hypothesis-testing framework covered elsewhere on this site, applied to a specific, extremely common business question.

The Basic Structure

Everything here rests on the same hypothesis testing framework covered elsewhere on this site: a null hypothesis, an alternative, a significance level chosen in advance, a test statistic, and a decision rule. What makes A/B testing distinct isn't the statistics — it's the specific, practical discipline of pre-registering a sample size and resisting the temptation to peek, both covered in detail below.

Visitors are randomly assigned to see either the control (A, typically the existing version) or the variant (B, the new version being tested). Random assignment is what makes the comparison valid — it's the mechanism that ensures, on average, the two groups differ only in which version they saw, not in any other systematic way that could confound the result.

The hypotheses map directly onto the standard framework: H₀ is "no difference in the metric between A and B"; H₁ is "there is a difference." A significance level (conventionally α = 0.05) and a test — usually a two-proportion z-test for conversion rate, or a two-sample t-test for a continuous metric like order value — produce a p-value, and the decision follows the same rule as any hypothesis test: reject H₀ if p < α.

Worked Example

Problem: Version A (control) converts 420 of 8,000 visitors (5.25%). Version B (new checkout flow) converts 480 of 8,000 visitors (6.0%). Is this a statistically significant improvement at α = 0.05?

Pooled proportion: p̄ = (420+480)/(8000+8000) = 900/16000 = 0.05625

Standard error: SE = √[p̄(1−p̄)(1/n₁+1/n₂)] = √[0.05625(0.94375)(1/8000+1/8000)] ≈ √[0.05308 × 0.00025] ≈ 0.00364

Z-statistic: z = (0.06 − 0.0525) / 0.00364 ≈ 0.0075 / 0.00364 ≈ 2.06

P-value (two-tailed): p ≈ 0.039

Decision: p (0.039) < α (0.05) → reject H₀ — the improvement is statistically significant at the 5% level.

Sample Size: Calculated Before the Test Starts, Not After

A properly run A/B test calculates its required sample size before launching, based on the baseline conversion rate, the smallest effect size worth detecting, the desired significance level, and the desired statistical power (commonly 80%). Skipping this step and simply running the test "until it feels done" is one of the most common and consequential A/B testing mistakes — it invites exactly the peeking problem covered next.

Statistical Significance vs. Practical Significance

With enough traffic, even a tiny, commercially meaningless difference can become statistically significant — a large sample is very good at detecting small real effects with confidence, but "detectable" and "worth acting on" are different questions. A 0.02 percentage-point conversion lift with p = 0.01 might be real, but if it doesn't clear the cost of implementing and maintaining the change, it isn't necessarily worth shipping. This is exactly why reporting the effect size (the actual magnitude of the lift, with a confidence interval) alongside the p-value matters — the p-value alone answers "is this probably real," not "does this matter."

Common Metrics and Their Matching Test

• Conversion rate (binary: converted or didn't) — a two-proportion z-test, as in the worked example above.
• Average order value, time on page (continuous) — a two-sample t-test.
• Click-through rate across several page variants — a chi-square test of independence, or ANOVA for continuous metrics across more than two variants.

Threats to a Valid A/B Test

Novelty effects: a new design can temporarily perform better (or worse) simply because it's different and draws attention, independent of any genuine underlying improvement — effects that often fade after the initial period, which is why some tests run longer than the minimum required sample size alone would suggest, to let novelty wear off.

Simpson's paradox: an A/B test can show one result overall while showing the opposite result within every individual subgroup, if the subgroups are unevenly distributed between the two test groups — a genuine, well-documented statistical trap that argues for checking randomization balance across key segments before trusting an aggregate result.

Multiple testing: running many simultaneous A/B tests, or testing many different metrics within one test, without correcting for the number of comparisons inflates the overall false-positive rate across the whole testing program, exactly as running many separate hypothesis tests does anywhere else in statistics.

Real-World Uses Beyond Websites

Email marketing: testing subject lines, send times, or call-to-action wording against open and click-through rates uses exactly the same two-proportion framework as the checkout example above.

Product features: gradually rolling out a new feature to a random subset of users while measuring engagement, retention, or revenue against the group still on the old version is a large-scale, longer-running version of the same A/B testing logic.

Pricing: testing different price points against conversion rate and total revenue (not conversion rate alone, since a lower price can convert better while generating less total revenue) is a common, higher-stakes application where the effect-size-versus-significance distinction matters enormously.

Public health and policy: randomized rollouts of an intervention (a new public health messaging campaign, a policy pilot program) to randomly selected regions or groups apply the identical statistical logic outside a commercial context.

Bayesian A/B Testing: An Alternative Approach

Everything covered above follows the frequentist hypothesis-testing framework. A growing alternative uses Bayesian methods instead — continuously updating a probability that "B is better than A" as data arrives, rather than computing a single p-value against a fixed significance threshold. This approach handles the peeking problem more gracefully (results can be checked continuously without the same false-positive inflation) and produces a more directly interpretable output ("there's an 87% probability B is better," rather than a p-value), which is part of why several major A/B testing platforms have adopted Bayesian methods as an alternative or default over the traditional frequentist approach.

A Worked Example of the Peeking Problem

Simulated experiments make the peeking problem concrete: running a test with truly no underlying difference between A and B, but checking the p-value every day and stopping the moment it dips below 0.05, produces a false "significant" result far more often than 5% of the time — in some realistic simulations, the effective false-positive rate under continuous peeking climbs well above 20%, four times the intended rate. This is not a hypothetical edge case; it's a well-documented, measurable consequence of not committing to a pre-calculated sample size and single analysis point in advance.