A/B Test Calculator

Find out whether the difference between two versions of a page, email, or ad is statistically significant or just chance.

Your experiment

Version A (control)

Visitors

Conversions

Version B (variant)

Visitors

Conversions

Alternative hypothesis

Significance level (α)

The confidence interval uses 1 − α.

Rate A

Rate B

6.5%

Relative lift

+30%

Statistically significant at α = 0.05: z = 2.232, p = 0.0256. The difference is unlikely to be explained by chance alone.

Pooled rate p̂ = (120 + 156) ÷ (2400 + 2400) = 0.0575

z = (p̂_B − p̂_A) ÷ √[p̂(1 − p̂)(1/n_A + 1/n_B)] = 0.015 ÷ 0.00672 = 2.232

Difference (B − A): 1.5%; 95% CI: 0.184% to 2.816%

How to use this tool

Enter the number of visitors and conversions for version A (the control) and version B (the variant).
Choose a two-sided test, or a one-sided test only if you decided the direction before running the experiment.
Read the conversion rates, the relative lift, the p-value, and the confidence interval for the difference.
Check the verdict, and read the note if any expected counts are small.

Formula Reference

Conversion rate

p̂ = conversions ÷ visitors

Pooled rate

p̂ = (x_A + x_B) ÷ (n_A + n_B)

z statistic

z = (p̂_B − p̂_A) ÷ √[p̂(1 − p̂)(1/n_A + 1/n_B)]

CI for the difference

(p̂_B − p̂_A) ± z* × √[p̂_A(1−p̂_A)/n_A + p̂_B(1−p̂_B)/n_B]

ArticleA/B Testing Explained ArticleStatistical Power ArticleThe Multiple Comparisons Problem CalcSample Size Calculator CalcChi-Square Test Calculator GlossaryGlossary: A/B Test

Frequently Asked Questions

What does statistically significant mean here?+

It means that if the two versions truly converted equally well, a difference at least this large would occur by chance less than α of the time, usually 5%. It says nothing about whether the difference is big enough to matter for the business.

Can I stop the test as soon as it becomes significant?+

Not safely. Checking the p-value repeatedly and stopping at the first significant result inflates the false-positive rate well above 5%. Decide the sample size in advance and analyse once, or use a method designed for sequential testing.

Why is my result not significant even though B looks better?+

With small samples, random variation is large, so an apparent lift can easily be chance. Look at the confidence interval: if it stretches from negative to positive, the data cannot tell whether B is better or worse.

Is this the same as a chi-square test?+

For a 2 × 2 table, the two-sided z test for two proportions gives the same p-value as a chi-square test of independence without continuity correction: the chi-square statistic equals z squared.

How to use this tool

Enter the number of visitors and conversions for version A (the control) and version B (the variant).

Choose a two-sided test, or a one-sided test only if you decided the direction before running the experiment.

Read the conversion rates, the relative lift, the p-value, and the confidence interval for the difference.

Check the verdict, and read the note if any expected counts are small.