A/B Testing Statistical Significance Formula
Learn how annual A/B test conversion uplift and the two-proportion z-score are calculated.
This calculator compares the annual conversion rates of Variant A and Variant B. It estimates the observed uplift and standardizes the difference with a two-proportion z-test, helping you compare the result with a chosen z-score threshold.
- 100% Free
- No Sign-Up Required
- Private & Secure
- Mobile Friendly
Two-Proportion Z-Score
Where:
Subtract Variant A's conversion rate from Variant B's rate, then divide that difference by the estimated sampling variation. The resulting z-score is compared with the selected threshold.
Variables Explained
| Variable | What It Means | Unit |
|---|---|---|
| pA - Variant A conversion rate | Annual conversions for Variant A divided by annual eligible visitors for Variant A. | percent |
| pB - Variant B conversion rate | Annual conversions for Variant B divided by annual eligible visitors for Variant B. | percent |
| p - Pooled conversion rate | Combined conversions divided by combined visitors across both variants. | percent |
| nA - Variant A visitors | Number of eligible annual visitors assigned to Variant A. | visitors |
| nB - Variant B visitors | Number of eligible annual visitors assigned to Variant B. | visitors |
| z - Z-score | Standardized difference between the two observed conversion rates. | N/A |
Step-by-Step Calculation
Calculate Variant A's conversion rate
Divide Variant A conversions by Variant A eligible visitors.
conversionRateA = annualConversionsA / annualVisitorsA
Calculate Variant B's conversion rate
Divide Variant B conversions by Variant B eligible visitors.
conversionRateB = annualConversionsB / annualVisitorsB
Find the absolute rate difference
This is the difference in conversion rates, expressed as a decimal or percentage points.
absoluteDifference = conversionRateB - conversionRateA
Calculate relative uplift
Relative uplift shows the difference as a proportion of Variant A's conversion rate.
relativeUplift = absoluteDifference / conversionRateA
Calculate the pooled rate
The pooled rate is used to estimate variation for the two-proportion z-test.
pooledConversionRate = (annualConversionsA + annualConversionsB) / (annualVisitorsA + annualVisitorsB)
Calculate standard error and z-score
The standard error measures expected sampling variation. Dividing the rate difference by it produces the z-score.
standardError = sqrt(pooledConversionRate * (1 - pooledConversionRate) * ((1 / annualVisitorsA) + (1 / annualVisitorsB))); zScore = absoluteDifference / standardError
Check the selected threshold
A result of 1 or more, or 100% or more when displayed as a percent, meets or exceeds the selected threshold.
thresholdProgress = abs(zScore) / criticalZScore
Example: Annual landing-page conversion test
Variant A rate
2,500 / 50,000
0.0500 or 5.00%
Variant B rate
2,750 / 50,000
0.0550 or 5.50%
Absolute difference
0.0550 - 0.0500
0.0050 or 0.50 percentage points
Relative uplift
0.0050 / 0.0500
0.1000 or 10.00%
Pooled rate
(2,500 + 2,750) / (50,000 + 50,000)
0.0525 or 5.25%
Z-score and threshold check
0.0050 / sqrt(0.0525 × 0.9475 × (1/50,000 + 1/50,000))
z ≈ 5.01; threshold progress ≈ 256%
Final Result
Variant B has a 10.00% relative uplift versus Variant A, with a z-score of about 5.01. It exceeds the selected 1.96 threshold under this test's assumptions.
Assumptions
- ✓The variants are independent groups and visitors are assigned without systematic bias.
- ✓Each visitor is counted once and has one opportunity to complete the same binary conversion event.
- ✓Both variants use the same conversion definition, audience eligibility criteria, measurement period, and tracking method.
- ✓The normal approximation used by a two-proportion z-test is suitable for the entered sample and conversion counts.
- ✓The selected z-score threshold represents the test's chosen decision rule.
Limitations
- !The calculation does not identify tracking errors, bot traffic, audience overlap, or assignment problems.
- !Statistical significance does not establish causation if the test design or measurement process is flawed.
- !A result meeting a threshold may still have a small practical or commercial effect.
- !Annual totals can hide changes over time, such as seasonality, campaigns, product changes, or site outages.
- !The calculator is for binary conversions, not continuous outcomes such as revenue per visitor.
Common Mistakes to Avoid
Entering conversions greater than visitors.
Comparing variants with different conversion definitions or attribution windows.
Treating a positive uplift as meaningful without considering its absolute size.
Using a z-score threshold as proof that the result will persist in future traffic.
Counting repeat visits as independent visitors when the experiment is intended to be user-based.
Related Formulas
Frequently Asked Questions
What is the formula for A/B test statistical significance?
This calculator uses a two-proportion z-score: the difference between conversion rates divided by the pooled standard error of that difference.
How is relative conversion uplift calculated?
Relative uplift equals the Variant B conversion rate minus the Variant A conversion rate, divided by the Variant A conversion rate.
What does a positive z-score mean?
A positive z-score means Variant B's observed conversion rate is higher than Variant A's. The absolute z-score is used for the threshold check.
What does a z-score of 1.96 mean?
For a standard two-sided normal approximation, 1.96 is commonly associated with a 95% confidence threshold.
Why does the calculator use a pooled conversion rate?
The pooled rate combines outcomes from both variants and is used by the standard two-proportion z-test when estimating variation under the no-difference assumption.
Ready to calculate your result?
Use the calculator to get instant results with your own inputs.