A/B Testing Statistical Significance Calculator FAQ

It compares two observed per-user binary conversion rates and reports the rate difference, relative uplift, z-score, and evidence ratio against a selected threshold.

It means a unique user either completed the defined conversion during the measurement window or did not. Each user should be counted once per assigned variant.

Binary user-level metrics such as signup completion, purchase conversion, activation, or click-through conversion can fit when users are counted uniquely.

Revenue per user, average order value, session counts, repeated event totals, and other continuous or non-binary metrics require a different approach.

Enter the number of unique eligible users assigned to each variant during the same analysis period.

Enter the number of those unique users who completed the same defined conversion event. Conversions cannot exceed users for this per-user calculation.

It is Variant B's conversion rate minus Variant A's conversion rate. It is best read in percentage points.

It is the absolute rate difference divided by Variant A's rate, expressed as a percentage. A negative value means Variant B's observed rate is lower.

It is the absolute observed rate difference divided by its estimated standard error. It indicates how large the observed gap is relative to estimated random variation.

It is the z-score divided by the selected critical z-score. A value above 1.00 means the result clears the selected two-sided threshold.

The calculator offers 90%, 95%, and 99% two-sided thresholds. A higher confidence threshold requires a larger z-score to clear it.

It uses two-sided critical z-scores. That means it tests for a conversion-rate difference in either direction.

No. It means the observed data do not clear the selected threshold under this calculation. The true difference may be smaller, absent, or not precisely estimated with the available data.

No. A two-sided significant result can indicate either an increase or a decrease. Also consider the sign and size of the observed change and the quality of the experiment.

Larger groups generally reduce the standard error, making a given conversion-rate difference easier to distinguish from ordinary random variation.

No. Repeatedly checking results and stopping when they look favorable can increase false-positive risk.

No. Testing many variants, segments, or metrics can require adjustments beyond this simple two-group calculation.

Yes. The formula uses each variant's own user count and conversion rate, although substantially uneven allocation can reduce precision compared with balanced groups at the same total traffic.

No. Statistical evidence and practical importance are different. The absolute change, likely value, costs, risks, and guardrail metrics may also matter.