A/B Testing Statistical Significance Calculator FAQ

It describes whether the observed difference is large relative to the random variation expected from the sample sizes under the test model. It does not measure business value or prove permanence.

It compares two binary conversion rates, such as purchases per visitor, signups per visitor, or clicks per eligible visitor.

The control is the baseline version used as the comparison point for the variation.

The variation is the alternative version whose conversion rate is compared with the control rate.

A ratio of 1.00× or higher means the z-score meets or exceeds the critical z-score you selected. A ratio below 1.00× does not reach that chosen threshold.

The absolute z-score shows how large the observed conversion-rate difference is relative to estimated sampling variation. Larger values indicate stronger evidence of a difference under the model.

Common two-sided values are 1.64 for 90%, 1.96 for 95%, and 2.58 for 99%. Use the cutoff that matches the testing threshold selected for your analysis.

It is the variation conversion rate minus the control conversion rate, expressed in percentage points.

It is the absolute lift divided by the control conversion rate, expressed as a percentage. It can appear large when the control baseline is small.

It uses a pooled two-proportion z-test to estimate a standard error and an absolute z-score for the difference between two conversion proportions.

The pooled rate is the combined rate across both groups. In this z-test approach, it is used to estimate sampling variation under the no-difference reference model.

No. For this calculator, conversions must be no greater than eligible visitors in the same group.

Yes, provided the denominator represents comparable eligible observations in both groups and each observation has a consistent opportunity to convert.

No. Revenue per visitor is generally a continuous or skewed metric, not a binary conversion outcome, and needs a method suited to that metric.

No. Biased assignment, faulty tracking, changing audiences, or other test issues can produce misleading results.

Small samples and low event counts create larger uncertainty. The apparent difference may be compatible with random variation.

Yes. Repeatedly looking for a threshold crossing and stopping early can increase false-positive risk unless the analysis plan accounts for sequential monitoring.

More comparable, correctly measured observations generally reduce statistical uncertainty, but they do not correct biased traffic, inconsistent tracking, or a poorly defined conversion event.

No. Use it as one estimate alongside experiment quality, practical lift, guardrail metrics, and the relevance of the tested audience.

Yes, if each eligible recipient is assigned to one version and the outcome is binary, such as opened or clicked.

Yes, when clicks are measured as a binary outcome per comparable impression, visitor, or eligible observation.

This calculator compares one control and one variation. Testing multiple variants requires additional planning because multiple comparisons can affect interpretation.

You can calculate a segment separately, but exploratory slicing across many segments can create misleading threshold results. Define important segments in advance where possible.