A/B Testing Statistical Significance Formula
Learn how a pooled two-proportion z-test estimates whether two A/B test conversion rates differ at a selected threshold.
This calculator compares the conversion rates of a control and variation group. It measures the observed rate difference against expected sampling variation, then compares the resulting z-score with your chosen critical z-score.
- 100% Free
- No Sign-Up Required
- Private & Secure
- Mobile Friendly
A/B Test Significance Ratio
Where:
First calculate how far apart the two conversion rates are. Then divide that difference by the estimated random variation and your selected cutoff. A ratio of 1.00 or higher reaches the selected threshold.
Variables Explained
| Variable | What It Means | Unit |
|---|---|---|
| p_control - Control conversion proportion | Control conversions divided by control visitors, expressed as a decimal proportion. | number |
| p_variation - Variation conversion proportion | Variation conversions divided by variation visitors, expressed as a decimal proportion. | number |
| c_control - Control conversions | Number of eligible control observations that completed the target action. | number |
| n_control - Control visitors | Number of eligible observations assigned to the control group. | number |
| c_variation - Variation conversions | Number of eligible variation observations that completed the target action. | number |
| n_variation - Variation visitors | Number of eligible observations assigned to the variation group. | number |
| p_pooled - Pooled conversion proportion | Combined conversion proportion across both groups used to estimate the standard error. | number |
| standardError - Standard error | Estimated sampling variation in the difference between the two conversion proportions. | number |
| zScore - Absolute z-score | Size of the observed conversion difference relative to its estimated standard error. | number |
| criticalZScore - Critical z-score | Selected two-sided significance cutoff, such as 1.96 for a commonly used 95% threshold. | number |
Step-by-Step Calculation
Calculate the control conversion rate
Divide control conversions by control visitors to obtain the control rate as a percentage.
controlConversionRate = (controlConversions / controlVisitors) * 100
Calculate the variation conversion rate
Divide variation conversions by variation visitors to obtain the variation rate as a percentage.
variationConversionRate = (variationConversions / variationVisitors) * 100
Find conversion lift
The absolute lift is the difference between rates in percentage points. Relative lift compares that difference with the control rate.
absoluteLift = variationConversionRate - controlConversionRate
Calculate the pooled conversion proportion
The pooled rate combines the results from both groups for the two-proportion z-test estimate.
pooledConversionRate = (controlConversions + variationConversions) / (controlVisitors + variationVisitors)
Calculate the standard error
The standard error estimates how much the observed difference may vary from sampling randomness under the pooled-rate model.
standardError = sqrt(max(pooledConversionRate * (1 - pooledConversionRate) * ((1 / controlVisitors) + (1 / variationVisitors)), 0.000000000001))
Calculate the z-score
Divide the absolute difference in decimal conversion proportions by the standard error.
zScore = abs((variationConversionRate - controlConversionRate) / 100) / standardError
Compare with the selected threshold
A ratio of 1.00 or above means the observed z-score meets or exceeds the selected critical z-score.
significanceRatio = zScore / criticalZScore
Worked example: 5% control rate versus 6% variation rate
Control conversion rate
(250 / 5,000) × 100
5.00%
Variation conversion rate
(300 / 5,000) × 100
6.00%
Absolute and relative lift
6.00% − 5.00%; (1.00% / 5.00%) × 100
1.00 percentage point; 20.0% relative lift
Pooled conversion proportion
(250 + 300) / (5,000 + 5,000)
0.055
Standard error
sqrt(0.055 × 0.945 × ((1 / 5,000) + (1 / 5,000)))
0.00456
Z-score
abs(0.0600 − 0.0500) / 0.00456
2.19
Significance ratio
2.19 / 1.96
1.12×
Final Result
The variation's 6.00% conversion rate has a 1.00 percentage point lift over control. Its z-score is about 2.19 and its significance ratio is about 1.12×, so it reaches the selected 95% two-sided threshold.
Assumptions
- ✓The calculation uses a pooled two-proportion z-test for a binary outcome, such as converted or did not convert.
- ✓Each eligible observation is counted once and is independently assigned to either control or variation.
- ✓Both groups measure the same conversion event during comparable periods and under comparable tracking conditions.
- ✓The selected critical z-score represents a two-sided threshold; 1.96 is commonly used for 95%, 1.64 for 90%, and 2.58 for 99%.
- ✓The result is interpreted as a fixed-horizon calculation rather than repeated checking and stopping based on interim results.
Limitations
- !A threshold result does not establish that the variation caused the difference if assignment, tracking, or audience quality is flawed.
- !Statistical significance does not measure practical importance, revenue impact, user experience, or implementation cost.
- !The calculator is for binary conversion metrics, not continuous measures such as average order value, average time, or revenue per visitor.
- !Very low conversion counts can make normal z-test approximations less reliable.
- !Multiple variations, multiple metrics, segmentation, or repeated peeking can increase the chance of misleading findings unless the testing plan accounts for them.
Common Mistakes to Avoid
Entering conversions that exceed the number of visitors in a group.
Comparing groups that use different conversion definitions, attribution windows, or eligibility rules.
Treating a positive relative lift as meaningful without checking the absolute percentage-point lift.
Using 1.96 as though it guarantees a result will repeat exactly in future tests.
Stopping a test immediately after the ratio crosses 1.00× during frequent checks.
Using this binary-outcome method for metrics that are averages rather than conversion events.
Related Formulas
Frequently Asked Questions
What is the formula for A/B test statistical significance?
This calculator uses an absolute two-proportion z-score: the absolute difference between the two decimal conversion rates divided by the pooled standard error. The significance ratio is that z-score divided by the selected critical z-score.
How is the pooled standard error calculated?
It is sqrt(p_pooled × (1 − p_pooled) × ((1 / n_control) + (1 / n_variation))), where p_pooled is the combined conversion proportion from both groups.
What does a significance ratio of 1.00× mean?
It means the calculated z-score exactly equals your selected critical z-score. Values above 1.00× exceed that selected threshold; lower values do not.
Why does the formula use an absolute z-score?
The absolute value tests whether the rates differ in either direction. The direction and size of the change are shown separately by the conversion lift.
What z-score corresponds to a 95% two-sided threshold?
A critical z-score of 1.96 is commonly used for a 95% two-sided threshold.
Can a small lift be statistically significant?
Yes. With sufficiently large and well-measured samples, even a small conversion-rate difference can exceed the chosen statistical threshold.
Ready to calculate your result?
Use the calculator to get instant results with your own inputs.