A/B Testing Statistical Significance (Monthly) Calculator FAQ
Answers to common questions about monthly A/B test conversion rates, z-scores, significance thresholds, and result interpretation.
This FAQ explains the inputs, statistical method, and practical interpretation of the monthly A/B Testing Statistical Significance Calculator. It is educational information and results should be treated as estimates.
Using the calculator
Questions about the data to enter and the outputs shown.
What information does the monthly A/B test calculator need?
Enter monthly visitors and conversions for Variant A and Variant B, then choose a two-sided significance threshold.
What counts as a visitor?
Use the number of eligible users who were exposed to each variant and could complete the selected conversion event.
What counts as a conversion?
A conversion is a visitor completing the same defined event, such as a purchase, signup, or renewal, for both variants.
Can conversions be greater than visitors?
No. This calculator assumes each visitor either converts once for the selected event or does not convert.
Z-scores and thresholds
Questions about statistical significance output.
What does the z-score mean in an A/B test?
It expresses the observed conversion-rate difference in units of estimated sampling variation. Larger absolute values indicate stronger evidence of a difference under the test assumptions.
What threshold should I select?
The calculator offers 1.645 for 90%, 1.96 for 95%, and 2.576 for 99% two-sided confidence. The appropriate standard depends on your measurement process and decision context.
What does a positive threshold margin mean?
It means the calculated z-score is greater than or equal to the selected threshold.
Does a result below the threshold prove there is no difference?
No. It means the supplied data does not clear that selected threshold using this method; the data may still be too limited to distinguish a difference reliably.
Formula and interpretation
Questions about the calculation method and rate changes.
Why does the calculator use a two-sided test?
A two-sided test checks for a difference in either direction, so it can identify whether B is higher or lower than A.
What is the difference between relative lift and percentage points?
Percentage points are the direct difference between conversion rates. Relative lift divides that difference by A's rate, expressing the change relative to the baseline.
Why is the pooled conversion rate used?
The pooled rate is used to estimate standard error for the standard two-proportion z-test under the no-difference assumption.
Does statistical significance mean Variant B is better?
Not necessarily. Check whether B's lift is positive, how large the absolute effect is, whether tracking is sound, and whether the outcome is practically useful.
Data quality and monthly reporting
Questions about combining periods and common sources of error.
Can I combine several months of results?
You can combine underlying visitors and conversions when the variant experience, audience, conversion definition, and measurement setup remained consistent across periods.
Should I average monthly conversion rates?
Do not use a simple average of monthly percentages. Combine the conversion counts and visitor counts, then calculate the overall rate.
Can I use the calculator for very low conversion counts?
Use caution. With very few conversions or very small groups, the normal approximation used by a z-test may be less suitable.
Does this calculator account for multiple tests or repeated checks?
No. Testing many variants or metrics, or repeatedly reviewing an in-progress test, can affect how a significance threshold should be interpreted.
What z-score is statistically significant at 95% confidence?
For this calculator's two-sided test, a z-score of 1.96 or higher clears the usual 95% confidence threshold.
Explore Related Questions
Ready to see what you can calculate?
Open the calculator and get personalized results in seconds.