A/B Testing Service-Level Agreement Calculator FAQ
Answers to common questions about A/B test SLA completion, traffic allocation, sample targets, and test duration.
This FAQ explains what the calculator measures, how its estimates are produced, and what its results do and do not establish.
General calculator questions
Understand the purpose and scope of the SLA calculation.
What does this A/B testing SLA calculator measure?
It estimates progress against the minimum per-group sample and minimum live-duration requirements entered for an experiment.
What does SLA completion mean?
It is an operational progress measure. A score of 100% means both entered requirements have been met.
What is a required sample per variant?
It is the minimum number of eligible visitors required in each experiment group under the test plan or SLA.
Can I use the calculator before launching a test?
Yes. Enter expected daily eligible traffic and the planned allocation to estimate a target test duration.
Traffic and allocation
Questions about visitor counts and experiment group splits.
Why does traffic allocation affect duration?
It changes daily traffic to each group. The lower-traffic group determines when both groups can meet a shared sample target.
Is a 50/50 split required?
No. The calculator accepts any allocation from 1% to 99%, but an uneven split generally lengthens the time needed for the smaller group.
What counts as an eligible visitor?
An eligible visitor is someone who can enter the experiment under its targeting and exclusion rules.
Should I adjust for expected traffic loss?
If known exclusions, outages, or tracking losses materially reduce usable traffic, use a conservative eligible-traffic estimate.
Formula and results
How completion, shortfall, and duration are calculated.
How is overall SLA completion calculated?
The calculator takes the lower of sample-size completion and duration completion, with each capped at 100%.
How is sample shortfall calculated?
It subtracts the smaller estimated group size from the required per-group sample, with a minimum shortfall of zero.
How is recommended total test duration calculated?
It uses the longer of the minimum required duration and the estimated days for the smaller group to reach the required sample.
Why does the calculator use complete days?
The days-to-sample estimate rounds up because a partial day is not a complete calendar day of testing.
Accuracy and interpretation
Important limits of an SLA-based estimate.
Does 100% completion mean the result is statistically significant?
No. Completion only confirms the entered traffic and duration thresholds. Use the agreed statistical method separately.
Does the calculator determine which version wins?
No. It does not use conversions, revenue, confidence intervals, guardrails, or effect size.
Why might actual completion take longer than estimated?
Traffic may decline, allocation may differ, users may be excluded, tracking may fail, or the experiment may be paused.
Can a test run longer than the recommended duration?
Yes. The recommended duration is a traffic-and-duration estimate, not a stopping rule or conclusion about the experiment.
What does 100% A/B testing SLA completion mean?
It means the entered per-group sample target and minimum test duration have both been met.
Explore Related Questions
Ready to see what you can calculate?
Open the calculator and get personalized results in seconds.