A/B Testing Statistical Significance (Monthly) Calculator Examples
Worked monthly A/B testing examples showing conversion rates, relative lift, z-scores, and threshold comparisons.
These examples show how different monthly traffic volumes and conversion-rate differences can produce significant or inconclusive A/B test results. Each uses the calculator's two-sided two-proportion z-test approach.
Small lift with high monthly traffic
An ecommerce team compares a current checkout against a simplified checkout over one month.
Input Summary
Variant A visitors
50,000
Variant A conversions
2,500
Variant B visitors
50,000
Variant B conversions
2,650
Threshold
95% (z = 1.96)
Calculation Breakdown
- 1Variant A conversion rate2,500 / 50,0005.00%
- 2Variant B conversion rate2,650 / 50,0005.30%
- 3Relative lift(5.30% - 5.00%) / 5.00% × 1006.00%
- 4Z-score0.0030 / sqrt(0.0515 × 0.9485 × (1/50,000 + 1/50,000))3.05
Result Summary
Z-score
3.05
A/B Testing Statistical Significance (Monthly) Calculator
The z-score of about 3.05 exceeds 1.96, so the observed difference clears the selected 95% threshold.
Large observed lift with limited traffic
A new landing page receives limited traffic during its first month.
Input Summary
Variant A visitors
1,000
Variant A conversions
40
Variant B visitors
1,000
Variant B conversions
50
Threshold
95% (z = 1.96)
Calculation Breakdown
- 1Variant A conversion rate40 / 1,0004.00%
- 2Variant B conversion rate50 / 1,0005.00%
- 3Relative lift(5.00% - 4.00%) / 4.00% × 10025.00%
- 4Z-score0.0100 / sqrt(0.045 × 0.955 × (1/1,000 + 1/1,000))1.52
Result Summary
Z-score
1.52
A/B Testing Statistical Significance (Monthly) Calculator
The z-score of about 1.52 does not reach the 95% threshold of 1.96.
Variant B performs worse
A subscription business tests a revised renewal message across comparable monthly audiences.
Input Summary
Variant A visitors
20,000
Variant A conversions
1,200
Variant B visitors
20,000
Variant B conversions
1,080
Threshold
95% (z = 1.96)
Calculation Breakdown
- 1Variant A conversion rate1,200 / 20,0006.00%
- 2Variant B conversion rate1,080 / 20,0005.40%
- 3Relative lift(5.40% - 6.00%) / 6.00% × 100-10.00%
- 4Absolute z-scoreabs(-0.0060) / sqrt(0.057 × 0.943 × (1/20,000 + 1/20,000))2.58
Result Summary
Absolute z-score
2.58
A/B Testing Statistical Significance (Monthly) Calculator
The z-score of about 2.58 exceeds the 95% threshold, while the negative lift indicates that Variant B performed worse.
How to Read Your Results
Compare the z-score with the selected two-sided threshold: 1.645 for 90%, 1.96 for 95%, and 2.576 for 99%.
A positive threshold margin means the z-score is at or above the selected threshold.
Read relative lift alongside the conversion rates; a high relative percentage can still be a small absolute rate difference.
A negative relative lift means Variant B converted at a lower observed rate than Variant A.
Use the result as an estimate based on the supplied monthly counts, not as proof of future performance.
Assumptions & Important Notes
- Each example uses one conversion opportunity per visitor and a consistent conversion definition across variants.
- The visitor totals and conversion totals are from comparable monthly test periods.
- The examples use a two-sided two-proportion z-test with pooled standard error.
- Results are rounded for display, so manually recalculated figures may vary slightly.
Related Examples
Frequently Asked Questions
Can a 25% lift fail an A/B significance test?
Yes. Relative lift can look large when the baseline is low, while a small sample may still leave too much uncertainty for the result to clear the chosen threshold.
Can a negative lift be statistically significant?
Yes. In a two-sided test, a sufficiently large negative difference can be significant, indicating that B performed lower than A.
Should I use percentage points or relative lift to compare variants?
Use both. Percentage points show the absolute change, while relative lift shows the change relative to Variant A's baseline.
Why do rounded example z-scores differ slightly from a calculator?
The calculator uses full precision internally, while examples display rounded conversion rates and intermediate values.
Ready to calculate your own result?
Use the live calculator with your own inputs, timing, and preferences.