A/B Testing Statistical Significance (Annual) Calculator Examples
Worked annual A/B testing examples showing conversion rates, uplift, z-scores and threshold progress.
These examples show how different annual traffic levels and conversion-rate differences can change the observed uplift and z-score. They are illustrative estimates based on the calculator's two-proportion z-test method.
Large sample with a modest conversion improvement
Equal annual traffic of 50,000 visitors per variant.
Input Summary
Variant A
50,000 visitors; 2,500 conversions
Variant B
50,000 visitors; 2,750 conversions
Threshold
1.96
Calculation Breakdown
- 1Conversion rates2,500 / 50,000 and 2,750 / 50,0005.00% and 5.50%
- 2Relative uplift(5.50% - 5.00%) / 5.00%10.00%
- 3Z-scoreTwo-proportion z-test using pooled rate of 5.25%About 5.01
- 4Threshold progress5.01 / 1.96About 256%
Result Summary
Threshold progress
About 256%
A/B Testing Statistical Significance (Annual) Calculator
Variant B shows a 10.00% relative uplift and meets the 1.96 threshold in this example.
Small annual sample with a visible uplift
A small business compares two signup forms.
Input Summary
Variant A
1,000 visitors; 50 conversions
Variant B
1,000 visitors; 60 conversions
Threshold
1.96
Calculation Breakdown
- 1Conversion rates50 / 1,000 and 60 / 1,0005.00% and 6.00%
- 2Relative uplift(6.00% - 5.00%) / 5.00%20.00%
- 3Z-scoreTwo-proportion z-test using pooled rate of 5.50%About 1.01
- 4Threshold progress1.01 / 1.96About 52%
Result Summary
Threshold progress
About 52%
A/B Testing Statistical Significance (Annual) Calculator
Variant B has a 20.00% observed uplift, but the z-score is about 1.01.
Unequal traffic allocation with a negative uplift
A checkout experiment has uneven annual allocation.
Input Summary
Variant A
20,000 visitors; 1,200 conversions
Variant B
30,000 visitors; 1,650 conversions
Threshold
1.96
Calculation Breakdown
- 1Conversion rates1,200 / 20,000 and 1,650 / 30,0006.00% and 5.50%
- 2Relative uplift(5.50% - 6.00%) / 6.00%-8.33%
- 3Z-scoreTwo-proportion z-test using pooled rate of 5.70%About -2.34
- 4Threshold progressabs(-2.34) / 1.96About 119%
Result Summary
Threshold progress
About 119%
A/B Testing Statistical Significance (Annual) Calculator
Variant B shows an observed 8.33% relative decline and the absolute z-score is about 2.34.
How to Read Your Results
A positive relative uplift means Variant B converted at a higher observed rate than Variant A; a negative value means it converted at a lower rate.
Absolute conversion-rate difference is best read in percentage points, while uplift is relative to Variant A's rate.
Compare the absolute z-score with the threshold you selected, not just the direction of the z-score.
Threshold progress of 100% means the absolute z-score equals the threshold; a higher value exceeds it.
Review the size of the rate difference alongside the z-score, since a small effect can be significant with large traffic.
Assumptions & Important Notes
- Examples use independent visitor groups and one binary conversion opportunity per visitor.
- The same conversion event and measurement rules apply to both variants.
- Z-scores are rounded for display and may vary slightly with exact precision.
- The selected 1.96 threshold is a common two-sided benchmark, not a universal requirement.
Related Examples
Frequently Asked Questions
Can a 20% uplift fail an A/B significance threshold?
Yes. When visitor counts are small, sampling variation can be large enough that even a sizeable observed uplift does not meet the chosen threshold.
Can a small conversion difference be significant?
Yes. Large samples can make the standard error small, allowing a small percentage-point difference to produce a high z-score.
Do Variant A and B need equal traffic?
No. The calculator accepts unequal visitor counts, although heavily uneven allocation can reduce the precision available from the smaller group.
Why can the z-score be negative?
It is negative when Variant B's rate is below Variant A's rate. For a two-sided threshold comparison, use the absolute value.
Ready to calculate your own result?
Use the live calculator with your own inputs, timing, and preferences.