Monthly A/B Test Statistical Significance Formula
Learn how monthly A/B test conversion rates, relative lift, pooled standard error, and z-scores are calculated.
This calculator uses a two-proportion z-test to estimate whether the conversion-rate difference between two monthly A/B test variants is larger than expected from sampling variation alone. The result helps you compare the observed z-score with a chosen two-sided significance threshold.
- 100% Free
- No Sign-Up Required
- Private & Secure
- Mobile Friendly
Absolute z-score
Where:
First calculate each variant's conversion rate. Then divide the absolute difference between the rates by the estimated random variation between them. A higher z-score means the observed difference is less consistent with random sampling variation.
Variables Explained
| Variable | What It Means | Unit |
|---|---|---|
| pA - Variant A conversion rate | The proportion of Variant A visitors who completed the conversion event: conversionsA divided by visitorsA. | percent |
| pB - Variant B conversion rate | The proportion of Variant B visitors who completed the same conversion event. | percent |
| p - Pooled conversion rate | The combined conversion rate across both variants, used to estimate standard error under the no-difference assumption. | percent |
| nA - Variant A visitors | Number of eligible monthly visitors assigned to Variant A. | visitors |
| nB - Variant B visitors | Number of eligible monthly visitors assigned to Variant B. | visitors |
| z - Absolute z-score | The size of the conversion-rate difference relative to its estimated standard error. | N/A |
| confidenceThreshold - Selected significance threshold | The two-sided z-score threshold selected for 90%, 95%, or 99% confidence. | N/A |
Step-by-Step Calculation
Calculate Variant A's conversion rate
Divide Variant A conversions by its monthly visitor total.
conversionRateA = conversionsA / visitorsA
Calculate Variant B's conversion rate
Divide Variant B conversions by its matching monthly visitor total.
conversionRateB = conversionsB / visitorsB
Find the conversion-rate difference
This is the absolute percentage-point difference before percentage formatting.
conversionDifference = conversionRateB - conversionRateA
Calculate the pooled rate
Combine both variants' conversions and visitors to estimate the shared rate used by the test.
pooledRate = (conversionsA + conversionsB) / (visitorsA + visitorsB)
Estimate standard error
The standard error estimates the expected sampling variation between the two observed conversion rates.
standardError = sqrt(pooledRate * (1 - pooledRate) * ((1 / visitorsA) + (1 / visitorsB)))
Calculate z-score and threshold margin
A non-negative margin means the z-score meets or exceeds the selected two-sided threshold.
zScore = abs(conversionDifference) / standardError; significanceMargin = zScore - confidenceThreshold
95% confidence monthly conversion test
Calculate Variant A rate
500 / 10,000
5.00%
Calculate Variant B rate
575 / 10,000
5.75%
Find rate difference
5.75% - 5.00%
0.75 percentage points
Calculate relative lift
0.75% / 5.00% × 100
15.00%
Estimate standard error
sqrt(0.05375 × 0.94625 × (1/10,000 + 1/10,000))
0.00319
Calculate z-score
0.0075 / 0.00319
2.35
Final Result
Variant B shows a 15.00% relative lift and a z-score of about 2.35. It clears the 95% two-sided threshold by about 0.39.
Assumptions
- ✓The calculation uses a two-sided two-proportion z-test.
- ✓Each visitor is counted once in the assigned variant and has either converted or not converted for the selected event.
- ✓Both variants use the same conversion definition and are measured over comparable monthly periods.
- ✓Traffic assignment and measurement are sufficiently independent for the normal approximation to be reasonable.
- ✓The selected threshold is interpreted as a statistical comparison rule, not a measure of commercial value.
Limitations
- !Very small samples or very few conversions can make the normal approximation less reliable.
- !A significant z-score does not show whether the observed lift is large enough to matter operationally or commercially.
- !The calculation does not detect tracking problems, audience imbalance, bot traffic, or changes made during the test.
- !Repeatedly checking a test or testing many metrics can increase the chance of misleading findings.
- !The z-score alone does not provide a confidence interval, statistical power estimate, or expected long-term effect.
Common Mistakes to Avoid
Entering conversions that exceed visitors.
Comparing variants with different conversion events or inconsistent attribution windows.
Treating percentage-point difference and relative lift as the same measure.
Averaging monthly conversion percentages instead of combining the underlying visitor and conversion counts.
Calling a result meaningful solely because it clears a statistical threshold.
Using a one-sided interpretation for this calculator's two-sided threshold.
Related Formulas
Frequently Asked Questions
How is an A/B test z-score calculated?
The calculator divides the absolute conversion-rate difference by the pooled standard error for two proportions.
What z-score is significant at 95% confidence?
For the calculator's two-sided test, the usual 95% threshold is 1.96. The z-score must be at least 1.96 to clear that threshold.
What is the pooled conversion rate in a two-proportion z-test?
It is total conversions across A and B divided by total visitors across A and B. It is used when estimating standard error under the assumption of no underlying rate difference.
How is relative conversion lift calculated?
Relative lift equals the conversion-rate difference divided by Variant A's conversion rate, multiplied by 100.
Why does a larger visitor count affect the z-score?
Larger samples generally reduce the estimated standard error, so a given rate difference can produce a larger z-score.
Ready to calculate your result?
Use the calculator to get instant results with your own inputs.