A/B Testing Statistical Significance (Monthly) Calculator
Compare monthly A/B test conversion results to estimate the conversion-rate lift, z-score, and whether the result clears your selected significance threshold.
Overview
Use this monthly A/B testing statistical significance calculator to compare two variants using their visitor and conversion totals. It estimates each conversion rate, the relative lift for variant B, and a two-proportion z-score against your chosen confidence threshold.
How it works
The calculator first divides conversions by visitors to find the conversion rate for each variant. It then calculates the difference between those rates and estimates the expected sampling variation using the combined, or pooled, conversion rate. Dividing the absolute rate difference by that standard error gives the z-score. If the z-score is at least as large as your selected threshold, the difference is statistically significant under a two-sided test. For example, the usual 95% two-sided threshold is 1.96.
How to use this calculator
- 1Enter the monthly visitor total for variant A.
- 2Enter the number of conversions recorded for variant A.
- 3Enter the matching visitor and conversion totals for variant B.
- 4Choose a two-sided confidence threshold, such as 95%.
- 5Review the z-score and threshold margin to assess the result.
Example Calculation
Variant A monthly visitors
10000
Variant A conversions
500
Variant B monthly visitors
10000
Variant B conversions
575
Significance threshold
1.96
Statistical significance z-score
2.35
Variant A converts at 5.00% and variant B converts at 5.75%. That is a 15.00% relative lift for B, with an approximate z-score of 2.35, which is above the 95% threshold of 1.96.
Frequently asked questions
What z-score is statistically significant for an A/B test?
For a two-sided test, a z-score of 1.645 corresponds to 90% confidence, 1.96 to 95% confidence, and 2.576 to 99% confidence. The z-score should meet or exceed your chosen threshold.
Does a statistically significant result mean variant B is better?
It indicates that the measured conversion-rate difference is unlikely to be due to random sampling variation under the test assumptions. You should also consider the direction and size of the lift, data quality, and practical impact.
Why does this calculator use a two-sided test?
A two-sided test checks for a meaningful difference in either direction, whether B performs higher or lower than A. It is a common default when a test could reveal either outcome.
Can I use this calculator for low conversion rates?
It can be useful when each variant has enough visitors and conversions for the normal approximation to be reasonable. Very small samples or very few conversions may need a more suitable exact statistical method.
Can I combine several months of A/B test data?
You can add counts across periods only when the test setup, audience, conversion definition, and variant experience remained consistent. Do not simply average monthly conversion percentages; use the combined visitor and conversion counts.
Why is the monthly visitor count important?
Larger visitor samples generally reduce uncertainty around conversion rates. A small conversion-rate difference may be significant with a large sample but inconclusive with a small sample.
Explore Related Calculators
Assumptions and warnings
Assumptions
- This calculator uses a two-sided two-proportion z-test for the conversion rates in variants A and B.
- Each visitor is counted once per variant and either converts or does not convert for the selected event.
- Visitors and conversions are assumed to come from the same monthly test period and comparable traffic sources.
- A positive threshold margin indicates that the observed difference meets the selected confidence threshold.
- Results are statistical estimates and do not measure commercial value, implementation risk, or long-term performance.
Warnings
- Statistical significance does not necessarily mean the result is practically important or financially valuable.
- Check that conversions do not exceed visitors and that test groups were assigned and measured consistently.