A/B Testing Statistical Significance Calculator Examples
See worked per-user A/B test examples for significant uplifts, inconclusive results, and negative conversion changes.
These examples show how user counts, conversion counts, and the selected confidence threshold affect the observed uplift, z-score, and evidence ratio. They use binary per-user conversion data and a two-sided test.
Large-Sample Signup Test With a Positive Result
A product team compares signup completion between an existing page and a redesigned page at a 95% confidence threshold.
Input Summary
Variant A users
10,000
Variant A conversions
500
Variant B users
10,000
Variant B conversions
600
Threshold
95% confidence
Calculation Breakdown
- 1Conversion rates500 / 10,000 and 600 / 10,0005.00% and 6.00%
- 2Observed uplift(0.0600 - 0.0500) / 0.0500 * 10020.0%
- 3Z-scoreabs(0.0600 - 0.0500) / 0.003223.10
- 4Evidence ratio3.10 / 1.961.58×
Result Summary
Evidence ratio
1.58×
A/B Testing Statistical Significance Calculator
Variant B converts at 6.00% versus 5.00% for Variant A and clears the 95% threshold.
Small-Traffic Test With an Inconclusive Uplift
A small website tests a new call to action using a 95% confidence threshold.
Input Summary
Variant A users
1,000
Variant A conversions
50
Variant B users
1,000
Variant B conversions
58
Threshold
95% confidence
Calculation Breakdown
- 1Conversion rates50 / 1,000 and 58 / 1,0005.00% and 5.80%
- 2Observed uplift0.0080 / 0.0500 * 10016.0%
- 3Z-scoreabs(0.0580 - 0.0500) / 0.010110.79
- 4Evidence ratio0.79 / 1.960.40×
Result Summary
Evidence ratio
0.40×
A/B Testing Statistical Significance Calculator
Variant B has a 16.0% observed relative uplift, but the evidence ratio is only about 0.40× at 95%.
Higher Baseline Conversion With a Clear Result
An ecommerce team evaluates a simplified checkout flow at a 95% confidence threshold.
Input Summary
Variant A users
20,000
Variant A conversions
2,000
Variant B users
20,000
Variant B conversions
2,200
Threshold
95% confidence
Calculation Breakdown
- 1Conversion rates2,000 / 20,000 and 2,200 / 20,00010.00% and 11.00%
- 2Observed uplift0.0100 / 0.1000 * 10010.0%
- 3Z-scoreabs(0.1100 - 0.1000) / 0.003073.26
- 4Evidence ratio3.26 / 1.961.66×
Result Summary
Evidence ratio
1.66×
A/B Testing Statistical Significance Calculator
Variant B has a 10.0% observed relative uplift and clears the 95% threshold.
Negative Conversion Change at a Stricter Threshold
A team checks whether a new onboarding flow changes activation at a 99% confidence threshold.
Input Summary
Variant A users
15,000
Variant A conversions
1,200
Variant B users
15,000
Variant B conversions
1,050
Threshold
99% confidence
Calculation Breakdown
- 1Conversion rates1,200 / 15,000 and 1,050 / 15,0008.00% and 7.00%
- 2Observed uplift(0.0700 - 0.0800) / 0.0800 * 100-12.5%
- 3Z-scoreabs(-0.0100) / 0.003043.29
- 4Evidence ratio3.29 / 2.5761.28×
Result Summary
Evidence ratio
1.28×
A/B Testing Statistical Significance Calculator
Variant B shows a 12.5% observed relative decline and clears the selected 99% threshold.
How to Read Your Results
Compare conversion rates in percentage points to understand the absolute size of the observed change.
Use relative uplift to describe the change relative to Variant A, especially when comparing tests with different baseline rates.
Read the z-score as the observed difference measured in estimated standard errors; higher absolute values indicate stronger evidence under the model.
An evidence ratio above 1.00 means the result clears the particular threshold selected in the calculator.
A result that clears a threshold still needs context such as metric quality, implementation effort, and possible effects on other outcomes.
Assumptions & Important Notes
- Every conversion is linked to a unique assigned user, rather than counting repeated events from the same person.
- Both variants have comparable measurement definitions and exposure periods.
- Examples use the calculator's unpooled two-proportion z-test approximation.
- Thresholds are two-sided and do not account for multiple comparisons or optional stopping.
Related Examples
Frequently Asked Questions
Why can a 16% uplift be inconclusive?
Relative uplift does not show how precisely the difference was measured. With few users, ordinary random variation can be larger than the observed rate gap.
Does a higher z-score always mean a larger business impact?
No. A higher z-score can result from more data as well as a larger effect. Business impact depends on the absolute change and relevant operational context.
Why does the negative example use an absolute z-score?
The z-score magnitude is used for a two-sided threshold. The negative rate difference and negative uplift still show that Variant B performed worse in the observed data.
Can I compare examples with different confidence thresholds directly?
Compare their z-scores directly, but remember that a stricter threshold has a larger critical z-score and therefore requires stronger evidence to clear it.
Ready to calculate your own result?
Use the live calculator with your own inputs, timing, and preferences.