A/B Testing Statistical Significance (Per-User) Calculator
Compare two per-user conversion rates to estimate the observed uplift, z-score, and whether the result clears your chosen significance threshold.
Overview
Use this per-user A/B testing calculator to compare conversion outcomes for two independent variants. Enter each variant's unique users and conversions, then select a confidence threshold to review the conversion rates, relative uplift, z-score, and evidence ratio.
How it works
The calculator first divides conversions by users to find each variant's per-user conversion rate. It then calculates the difference between the rates and estimates its unpooled standard error using both sample sizes and observed rates. Dividing the absolute difference by that standard error produces a z-score. The evidence ratio compares that z-score with the critical z-score for your selected two-sided confidence threshold. An evidence ratio above 1 suggests the observed difference is statistically significant at that threshold under the stated assumptions.
How to use this calculator
- 1Enter the number of unique users exposed to Variant A.
- 2Enter the number of converting users in Variant A.
- 3Enter the same user and conversion counts for Variant B.
- 4Choose a two-sided confidence threshold.
- 5Review the evidence ratio; a value above 1 indicates the z-score clears the chosen threshold.
Example Calculation
Variant A users
10000
Variant A conversions
500
Variant B users
10000
Variant B conversions
600
Significance threshold
1.96
Evidence ratio vs. threshold
1.58 ×
Variant A converts at 5.00% and Variant B at 6.00%, a 20.0% relative uplift. The z-score is about 3.10, giving an evidence ratio of about 1.58 at the 95% confidence threshold.
Frequently asked questions
What does the evidence ratio mean?
It is the observed z-score divided by the critical z-score for your selected threshold. A value above 1 means the result clears that threshold; a value below 1 does not.
Is this a one-sided or two-sided A/B test?
The selected critical values are for two-sided tests. They assess whether the conversion rates differ in either direction.
Why does sample size matter for statistical significance?
Larger user samples generally reduce the standard error, making it easier to distinguish a real rate difference from ordinary random variation.
Can I use this calculator for revenue or average order value?
This calculator is designed for binary per-user outcomes, such as converted or did not convert. Continuous metrics such as revenue per user need a different test.
What if conversions are greater than users?
The data are not valid for a per-user binary conversion test. Check for duplicate events, mismatched reporting windows, or whether you should use a different metric.
Does statistical significance show business significance?
No. A statistically significant difference can still be too small to matter commercially. Consider the absolute change, expected value, implementation cost, and experiment quality as well.
Explore Related Calculators
Assumptions and warnings
Assumptions
- Each user is counted once in the assigned variant and contributes one independent binary conversion outcome.
- Both variants use the same conversion definition, observation window, and audience eligibility rules.
- The calculation uses an unpooled two-proportion z-test approximation.
- The selected threshold is two-sided, meaning it tests for either an increase or a decrease.
- Results are estimates based on observed experiment data and do not account for repeated interim checks or multiple comparisons.
Warnings
- This calculator provides a statistical estimate, not a guarantee that a variant will perform the same in future.
- Check that conversions do not exceed users in either variant before interpreting the result.
- Repeatedly checking a test and stopping when it appears significant can increase false-positive risk.