A/B Testing Sample Size Calculator FAQ
Answers to common questions about A/B test sample size, minimum detectable effect, confidence, power, traffic requirements, and duration estimates.
Use these answers to understand what the calculator measures, how its inputs affect the result, and when a simple conversion-rate sample-size estimate may not fit your experiment.
General A/B Test Sample Size Questions
Core concepts for planning a two-variation conversion experiment.
What is sample size in A/B testing?
Sample size is the number of eligible visitors needed in each test variation to evaluate a planned conversion-rate difference with selected confidence and power.
Why do I need a sample size estimate before testing?
It helps determine whether available traffic can support the target decision and provides a traffic-based estimate of test length.
Is the result per variation or total traffic?
The primary result is visitors per variation. For a 50/50 A/B test, multiply it by two to get combined traffic.
What conversion outcomes can this calculator handle?
It is intended for binary outcomes, such as purchase versus no purchase, sign-up versus no sign-up, or click versus no click.
Inputs and Calculation Method
How baseline rate, uplift, confidence, and power affect the formula.
What is the minimum detectable effect in this calculator?
It is the smallest relative conversion-rate uplift the test is designed to detect. For example, a 20% uplift on a 10% baseline implies a 12% variant rate.
What is the difference between relative uplift and percentage points?
Relative uplift is measured against the baseline. Moving from 10% to 12% is a 20% relative uplift and a 2-percentage-point increase.
What confidence level should I choose?
A 95% confidence setting is a common planning choice. Higher confidence increases the required sample size.
What does statistical power mean?
Power is the modeled probability of detecting the target effect when it is truly present. Higher power requires more visitors.
Why does a smaller MDE increase sample size?
Smaller expected differences are more difficult to distinguish from random variation, so more observations are needed.
Accuracy and Test Duration
Important practical factors that can make actual test conditions differ from the estimate.
How accurate is an A/B test sample size calculator?
It is a planning estimate based on the supplied inputs and statistical assumptions. Actual traffic, conversion behavior, and analysis choices can change the outcome.
Why might my test run longer than the estimated days?
Eligible traffic may be lower than expected, allocation may not be exactly even, or the test may need to include normal weekly or seasonal cycles.
Can I stop the test early if the result looks significant?
Repeatedly checking and ending a test solely on interim significance can affect interpretation. Define a measurement approach before running the experiment.
Does this estimate account for multiple variants?
No. Tests with more than one variant generally need additional planning because traffic is split across more groups and comparisons.
Does daily traffic mean all website visitors?
No. Use visitors who are eligible for the experiment and can reasonably encounter the tested experience.
When to Use Another Method
Situations where a binary two-proportion calculation may not be sufficient.
Can I use this calculator for revenue per visitor?
Not directly. Revenue per visitor is a continuous and often skewed metric, so it usually requires a different variance-based method.
Can I use this calculator for average order value?
Not directly. Average order value is not a binary conversion metric and generally needs a calculation based on its distribution and variability.
Can I use it for an email A/B test?
It can be used for a binary email outcome, such as click or conversion, when recipients are independently assigned and the denominator is defined consistently.
What if the target variant conversion rate exceeds 100%?
Reduce the relative uplift or review the baseline input. A conversion probability cannot exceed 100%.
What is a minimum detectable effect?
The minimum detectable effect is the smallest relative conversion-rate uplift that the test is planned to identify with the selected confidence and power.
Explore Related Questions
Ready to see what you can calculate?
Open the calculator and get personalized results in seconds.