A/B Test Significance Calculator

Enter visitors and conversions for two variants to get the conversion rates, relative uplift, z-score, p-value and whether the difference is statistically significant.

Confidence that B differs from A95.5%
Rate A
3%
Rate B
3.45%
Relative uplift (B vs A)
14.93%
p-value (two-sided)
0.045
z-score
2.0051

Results update as you type. Values are kept in the URL, so you can share or bookmark this exact calculation.

Formula

p̂ = (conversions A + conversions B) ÷ (visitors A + visitors B)
SE = √( p̂(1−p̂) · (1/nA + 1/nB) )
z = (rate B − rate A) ÷ SE
p-value = 2 · (1 − Φ(|z|))   ·   Confidence = 1 − p-value

How to read A/B Test Significance

This is the two-proportion z-test used by most A/B testing tools. The p-value is the probability of seeing a difference at least this large if the two variants actually converted at the same rate. Confidence is simply one minus that. The conventional threshold is 95% confidence (p < 0.05); above it the result is called statistically significant.

Significance is not the same as importance. A 0.2% relative uplift can be significant with millions of visitors and still not be worth shipping; a 20% uplift can miss significance with too little data and still be real. Read the confidence together with the uplift and the sample size you planned before the test.

Frequently asked questions

The result is 93% — is that good enough?
Below the 95% convention, so by the usual standard you have not shown a difference. Rather than lowering the bar, check whether the test reached the sample size you planned; if not, keep running it to that sample and read the result once.
Why does the calculator say significant but my testing tool does not?
Tools differ in method: some use a one-sided test, some use Bayesian estimates, some apply sequential-testing corrections for peeking. This calculator is a plain two-sided z-test on the final numbers, which is the most conservative simple approach.
Can I use this for revenue per visitor?
No — this test is for rates (converted or not). Revenue per visitor is a continuous metric with a skewed distribution and needs a t-test or a bootstrap. Use it for conversion, click-through, sign-up and similar binary outcomes.
Sample Size CalculatorWork out how many visitors each variant of an A/B test needs to detect a given lift in conversion rate at your chosen confidence and power — and how long that takes at your traffic.Conversion Rate CalculatorCalculate conversion rate from conversions and visitors, sessions or clicks — and see how many visitors each conversion currently costs you.CTR CalculatorCalculate click-through rate (CTR) from clicks and impressions for ads, search results, emails or any placement that is shown and clicked.