A/B Testing Sample Size Formula
Learn how an annual A/B testing sample size estimate is calculated from conversion rate, target lift, confidence, power, and traffic.
This calculation estimates the number of eligible visitors needed to detect a planned change in conversion rate between a control and a variant. It helps teams judge whether their annual traffic can support a test at the selected confidence and power settings.
- 100% Free
- No Sign-Up Required
- Private & Secure
- Mobile Friendly
Required Sample per Group
Where:
The formula estimates visitors needed in each group by comparing the baseline conversion rate with the expected variant rate. Smaller differences, higher confidence, and higher power all increase the sample required.
Variables Explained
| Variable | What It Means | Unit |
|---|---|---|
| p₁ - Baseline rate | Current control conversion rate expressed as a decimal. | percent |
| p₂ - Expected variant rate | Baseline rate after applying the planned relative lift. | percent |
| p̄ - Average conversion rate | Average of the baseline and expected variant rates. | percent |
| zα - Confidence z-score | Z-score representing the selected two-sided confidence level. | N/A |
| zβ - Power z-score | Z-score representing the selected statistical power. | N/A |
| n - Sample per group | Estimated eligible visitors required for each group under an equal split. | visitors |
| d - Rate difference | Absolute difference between the expected variant and baseline conversion rates. | percent |
Step-by-Step Calculation
Convert the baseline percentage to a decimal
A percentage such as 5% becomes 0.05 for use in the calculation.
baselineRate = baselineConversionRate / 100
Calculate the expected variant rate
The relative lift is applied to the baseline rate. A 10% lift on a 5% baseline gives 5.5%.
variantRate = baselineRate * (1 + minimumDetectableEffect / 100)
Find the average rate
The two-proportion approximation uses the average of the planned control and variant rates.
averageRate = (baselineRate + variantRate) / 2
Calculate the absolute rate difference
This is the conversion-rate gap the test is designed to detect.
rateDifference = variantRate - baselineRate
Estimate visitors required per group
Confidence, power, conversion-rate variability, and the target difference are combined to estimate the group sample.
samplePerGroup = pow((confidenceZScore * sqrt(2 * averageRate * (1 - averageRate)) + powerZScore * sqrt(baselineRate * (1 - baselineRate) + variantRate * (1 - variantRate))) / rateDifference, 2)
Calculate total sample and estimated duration
The per-group estimate is doubled for a 50/50 test and compared with annual eligible traffic.
totalRequiredSample = ceil(samplePerGroup * 2); estimatedTestMonths = totalRequiredSample / annualVisitors * 12
Example: 5% baseline rate and 10% relative lift
Convert the baseline rate
5 / 100
0.05
Calculate the expected variant rate
0.05 × (1 + 10 / 100)
0.055
Find the absolute difference
0.055 - 0.05
0.005
Estimate sample per group
[(1.96 × √(2 × 0.0525 × 0.9475)) + (0.84 × √(0.05 × 0.95 + 0.055 × 0.945))]² / 0.005²
31,197 visitors per group
Calculate total sample
ceil(31,197 × 2)
62,394 visitors
Estimate test duration
62,394 / 120,000 × 12
6.2 months
Final Result
The test requires approximately 62,394 eligible visitors in total, or about 31,197 per group, and would take about 6.2 months at 120,000 annual eligible visitors.
Assumptions
- ✓The calculation uses a normal approximation for two independent conversion rates.
- ✓The confidence z-score represents a two-sided significance threshold.
- ✓The core sample estimate assumes an even 50/50 allocation between control and variant.
- ✓Eligible visitors are independent and are counted once in the experiment.
- ✓Traffic volume and conversion behavior remain reasonably stable during the test period.
Limitations
- !The estimate does not account for multiple variants, multiple metrics, or repeated testing adjustments.
- !Seasonality, campaign changes, audience changes, and tracking problems can affect actual test duration and reliability.
- !Very low conversion rates or very small expected conversion counts may require a more specialized method.
- !Unequal traffic allocation is less efficient than an equal split, while the main formula is based on equal groups.
- !A planned sample size does not guarantee that the observed result will match the expected lift.
Common Mistakes to Avoid
Entering a percentage-point increase as a relative lift. For example, moving from 5% to 6% is a 20% relative lift, not 1%.
Using total site traffic instead of visitors who can actually enter the experiment.
Choosing a very small detectable lift without checking whether the available traffic can support it.
Treating a 50/50 sample estimate as exact when the actual allocation is uneven.
Ending a test early only because a result appears favorable before the planned sample is reached.
Related Formulas
Frequently Asked Questions
How is A/B test sample size calculated?
It is estimated from the baseline conversion rate, expected variant conversion rate, selected confidence z-score, power z-score, and the absolute difference between the rates.
Why does a smaller detectable lift need more visitors?
A smaller conversion-rate difference is harder to distinguish from random variation, so the test needs more observations at the same confidence and power.
What is the formula for a 10% conversion lift?
First calculate variantRate as baselineRate multiplied by 1.10. The detectable difference is then variantRate minus baselineRate and is used in the sample-size formula.
Does the formula use relative lift or percentage points?
The lift input is relative. For example, a 10% lift applied to a 5% baseline produces a 5.5% expected variant rate, a 0.5 percentage-point increase.
What z-score is commonly used for 95% confidence?
A z-score of 1.96 is commonly used for approximately 95% two-sided confidence.
What does 80% power mean in sample-size planning?
It means the chosen design is intended to have about an 80% chance of detecting the planned effect if that effect is truly present, subject to the calculation assumptions.
Ready to calculate your result?
Use the calculator to get instant results with your own inputs.