A/B Testing Statistical Significance Calculator
Compare two conversion rates and estimate whether the difference is statistically significant at a chosen confidence threshold.
Overview
Use this A/B testing statistical significance calculator to compare control and variation conversion rates. Enter the visitor and conversion totals for both versions, then choose a critical z-score to assess whether the observed difference reaches your selected threshold.
How it works
The calculator first finds each version's conversion rate and the difference between them. It then applies a pooled two-proportion z-test, which estimates how large the observed difference is relative to expected sampling variation. The absolute z-score is divided by your critical z-score to create the significance ratio. A ratio of 1.00 or more means the test statistic reaches that selected two-sided threshold. This result should be considered alongside sample quality, test duration, tracking accuracy, and the practical value of the lift.
How to use this calculator
- 1Enter the total number of eligible visitors in the control group.
- 2Enter the number of control visitors who converted.
- 3Enter the visitor and conversion totals for the variation.
- 4Set the critical z-score for your chosen confidence threshold, such as 1.96 for 95%.
- 5Review the significance ratio, z-score, and conversion lift.
Example Calculation
Control visitors
5000
Control conversions
250
Variation visitors
5000
Variation conversions
300
Critical z-score
1.96
Significance ratio
1.12 ×
The control conversion rate is 5.00% and the variation rate is 6.00%, an absolute lift of 1.00 percentage point. The z-score is about 2.19, producing a significance ratio of about 1.12×, which reaches the 95% two-sided threshold.
Frequently asked questions
What is statistical significance in A/B testing?
It indicates whether an observed difference is large relative to the random variation expected from the sample sizes. It does not indicate how valuable or permanent the difference is.
What critical z-score should I use for a 95% confidence level?
For a two-sided test, 1.96 is the commonly used critical z-score for a 95% confidence threshold. Common alternatives are 1.64 for 90% and 2.58 for 99%.
How do I interpret the significance ratio?
A ratio of 1.00× or above means the z-score meets or exceeds your selected critical z-score. A ratio below 1.00× does not reach that chosen threshold.
Why can a large conversion lift still be insignificant?
A large apparent lift can occur in a small sample where random variation is high. More observations generally reduce uncertainty and make it easier to distinguish a real difference from noise.
Does statistical significance mean the variation is better for my business?
Not necessarily. Also consider the size of the lift, revenue or user impact, implementation costs, guardrail metrics, and whether the result is likely to apply to your wider audience.
Can I use this calculator for metrics other than conversions?
Yes, when the metric is a binary outcome per observation, such as signup completed, purchase made, or button clicked. It is not designed for averages such as order value or time on page.
Explore Related Calculators
Assumptions and warnings
Assumptions
- This calculator uses a two-proportion z-test with a pooled conversion-rate estimate.
- Each visitor or observation is assumed to be counted once and assigned independently to one version.
- The control and variation are assumed to measure the same conversion event over comparable time periods.
- A critical z-score of 1.96 represents a commonly used 95% two-sided confidence threshold.
- Results are statistical estimates and do not measure business value, practical importance, or implementation quality.
Warnings
- Statistical significance does not prove that the variation caused the observed difference or that the result will persist.
- Avoid making decisions from very small samples, incomplete tracking, repeated peeking at results, or tests with major audience differences.