A/B Testing Statistical Significance Formula
Learn how per-user conversion rates, uplift, standard error, z-score, and evidence ratio are calculated for an A/B test.
This calculator compares conversion outcomes from two independent user groups. It estimates whether the observed rate difference is large relative to expected random variation and compares that evidence with a selected two-sided threshold.
- 100% Free
- No Sign-Up Required
- Private & Secure
- Mobile Friendly
Evidence Ratio vs. Significance Threshold
Where:
First calculate each variant's conversion rate. Then divide the absolute gap between those rates by its estimated random variation and by the selected critical z-score. An evidence ratio above 1 means the observed z-score clears the chosen threshold.
Variables Explained
| Variable | What It Means | Unit |
|---|---|---|
| nA - Variant A users | The number of unique users assigned to Variant A. | number |
| cA - Variant A conversions | The number of Variant A users who completed the binary conversion event. | number |
| pA - Variant A conversion rate | Variant A conversions divided by Variant A users. | percent |
| nB - Variant B users | The number of unique users assigned to Variant B. | number |
| cB - Variant B conversions | The number of Variant B users who completed the same conversion event. | number |
| pB - Variant B conversion rate | Variant B conversions divided by Variant B users. | percent |
| zcritical - Critical z-score | The selected two-sided significance threshold, such as 1.96 for 95% confidence. | N/A |
Step-by-Step Calculation
Calculate Variant A's conversion rate
Divide the number of converting users in Variant A by the number of unique users assigned to it.
conversionRateA = conversionsA / usersA
Calculate Variant B's conversion rate
Use the same conversion definition and observation window as Variant A.
conversionRateB = conversionsB / usersB
Find the absolute rate difference
A positive value means Variant B converted at a higher observed rate; a negative value means it converted at a lower rate.
absoluteDifference = conversionRateB - conversionRateA
Calculate relative uplift
This expresses the rate change relative to Variant A. It is not defined when Variant A has a zero conversion rate.
relativeUplift = (absoluteDifference / conversionRateA) * 100
Estimate the unpooled standard error
The standard error estimates the expected random variation in the difference between two independent observed conversion rates.
standardError = sqrt((conversionRateA * (1 - conversionRateA) / usersA) + (conversionRateB * (1 - conversionRateB) / usersB))
Calculate the z-score
The z-score measures the observed difference in units of standard error.
zScore = abs(absoluteDifference) / standardError
Compare with the selected threshold
An evidence ratio above 1 means the observed z-score exceeds the selected two-sided threshold.
evidenceRatio = zScore / criticalZ
Worked Example: 5.00% vs. 6.00% Conversion
Calculate Variant A rate
500 / 10,000
0.0500 = 5.00%
Calculate Variant B rate
600 / 10,000
0.0600 = 6.00%
Calculate rate difference and uplift
0.0600 - 0.0500; then 0.0100 / 0.0500 * 100
1.00 percentage point; 20.0% uplift
Calculate standard error
sqrt((0.05 * 0.95 / 10,000) + (0.06 * 0.94 / 10,000))
0.00322
Calculate z-score
abs(0.0100) / 0.00322
3.10
Calculate evidence ratio
3.10 / 1.96
1.58×
Final Result
Variant B shows a 20.0% observed relative uplift. Its evidence ratio is about 1.58× at the 95% threshold, so the result clears that selected threshold under the calculator's assumptions.
Assumptions
- ✓Each assigned user is counted once and has one independent binary conversion outcome.
- ✓The variants use the same conversion definition, eligibility criteria, and observation window.
- ✓Variant assignments are independent enough for a two-proportion z-test approximation.
- ✓The chosen critical z-score represents a two-sided threshold.
- ✓The experiment was evaluated according to a plan rather than repeatedly stopped after favorable interim results.
Limitations
- !The result is an estimate from observed data, not a guarantee of future performance.
- !The calculator does not adjust for multiple metrics, multiple variants, segmentation, or repeated interim checks.
- !Very small samples or extremely rare conversion events may not be well represented by the normal approximation.
- !Statistical significance does not measure practical value, implementation effort, or possible side effects.
- !Relative uplift can be unstable or undefined when Variant A's conversion rate is near zero.
Common Mistakes to Avoid
Entering total conversion events rather than unique converting users for a per-user metric.
Using different conversion windows or event definitions for the two variants.
Treating a 1.00× evidence ratio as proof that a change will perform the same after rollout.
Confusing a relative uplift with a percentage-point difference.
Stopping or checking many tests without accounting for the increased false-positive risk.
Related Formulas
Frequently Asked Questions
How is A/B test statistical significance calculated for conversion rates?
This calculator uses an unpooled two-proportion z-test approximation. It calculates each per-user rate, estimates the standard error of their difference, and divides the absolute difference by that standard error.
What evidence ratio is statistically significant?
An evidence ratio greater than 1.00 clears the selected threshold. For example, with a 95% threshold, the observed z-score must be greater than 1.96.
Why does this calculator use an absolute z-score?
The selected thresholds are two-sided, so the test checks for a difference in either direction. The sign of the rate difference still shows which variant had the higher observed rate.
What is the difference between absolute difference and relative uplift?
The absolute difference is measured in percentage points, while relative uplift divides that difference by Variant A's conversion rate and expresses it as a percentage.
Can this formula be used for revenue per user?
No. This calculation is for a binary per-user outcome, such as converted or did not convert. Continuous metrics require a different statistical method.
Ready to calculate your result?
Use the calculator to get instant results with your own inputs.