A/B Testing Password Strength Monthly Impact Formula
Learn how to estimate expected monthly account compromises and the reduction associated with a lower weak-password rate.
This calculation estimates expected account compromises under two password-strength approaches. It converts weak-password rates and risk assumptions into a full-population monthly comparison so differently sized A/B test groups can be compared fairly.
- 100% Free
- No Sign-Up Required
- Private & Secure
- Mobile Friendly
Estimated Monthly Compromises Avoided
Where:
For each approach, estimate the compromise risk for weak-password and stronger-password accounts, then multiply the difference by the monthly account population.
Variables Explained
| Variable | What It Means | Unit |
|---|---|---|
| N - Monthly active accounts | The total active accounts included in a typical month. | accounts |
| wA - Weak-password rate in Variant A | The share of Variant A accounts that meet the selected weak-password definition. | percent |
| wB - Weak-password rate in Variant B | The share of Variant B accounts that meet the same weak-password definition. | percent |
| rw - Monthly compromise risk for weak passwords | Estimated monthly probability of compromise for an account with a weak password. | percent |
| rs - Monthly compromise risk for stronger passwords | Estimated monthly probability of compromise for an account that meets the strength standard. | percent |
| A - Variant A traffic share | The percentage of test accounts assigned to the existing password approach. | percent |
Step-by-Step Calculation
Split accounts between test variants
Calculate how many monthly active accounts were assigned to the current Variant A approach.
variantAAccounts = monthlyAccounts * (variantAShare / 100)
Calculate Variant B accounts
The remaining accounts are assigned to the Variant B test approach.
variantBAccounts = monthlyAccounts - variantAAccounts
Estimate weak-password accounts
Apply each variant's weak-password rate to its account group.
weakAccountsA = variantAAccounts * (weakPasswordRateA / 100); weakAccountsB = variantBAccounts * (weakPasswordRateB / 100)
Estimate stronger-password accounts
Accounts not classified as weak are treated as stronger-password accounts in this model.
strongAccountsA = variantAAccounts - weakAccountsA; strongAccountsB = variantBAccounts - weakAccountsB
Estimate compromises for each variant
Apply the relevant monthly compromise-risk assumption to each password-strength subgroup.
expectedCompromisesA = (weakAccountsA * (weakPasswordRisk / 100)) + (strongAccountsA * (strongPasswordRisk / 100)); expectedCompromisesB = (weakAccountsB * (weakPasswordRisk / 100)) + (strongAccountsB * (strongPasswordRisk / 100))
Standardize and compare the variants
Scale both observed variant results to the same full monthly population before finding the expected difference.
monthlyCompromisesAvoided = (expectedCompromisesA * (monthlyAccounts / variantAAccounts)) - (expectedCompromisesB * (monthlyAccounts / variantBAccounts))
Stricter password-strength approach for 10,000 accounts
Assign accounts
10,000 × 50% = 5,000 accounts in A; 10,000 − 5,000 = 5,000 accounts in B
5,000 accounts per variant
Find weak-password accounts
A: 5,000 × 30% = 1,500; B: 5,000 × 15% = 750
A: 1,500; B: 750
Find stronger-password accounts
A: 5,000 − 1,500 = 3,500; B: 5,000 − 750 = 4,250
A: 3,500; B: 4,250
Estimate Variant A compromises
(1,500 × 0.50%) + (3,500 × 0.05%) = 7.5 + 1.75
9.25 accounts/month
Estimate Variant B compromises
(750 × 0.50%) + (4,250 × 0.05%) = 3.75 + 2.125
5.875 accounts/month
Scale to all accounts and compare
A: 9.25 × 2 = 18.5; B: 5.875 × 2 = 11.75; 18.5 − 11.75 = 6.75
6.75 expected compromises avoided/month
Final Result
Variant B is estimated to avoid 6.8 compromised accounts per month, a 36.5% reduction versus Variant A under these assumptions.
Assumptions
- ✓Variant assignment is random, and both groups represent the same underlying account population.
- ✓One consistent definition of a weak password is used in both variants.
- ✓Weak-password and stronger-password risk assumptions apply consistently within each group for the month.
- ✓The standardized results represent what could occur if each approach were used across all included accounts.
- ✓Password strength is isolated from other security controls in the calculation.
Limitations
- !Expected compromises are modeled averages, not a prediction of the number of actual incidents in a future month.
- !Compromise risk can change with phishing campaigns, credential-stuffing activity, breach exposure, and attacker behavior.
- !The model does not separately measure multifactor authentication, password managers, rate limiting, recovery flows, or detection controls.
- !A lower weak-password rate may be associated with usability or completion changes that are outside this calculation.
- !Small test groups or short observation periods can produce unstable observed rates.
Common Mistakes to Avoid
Using different weak-password definitions for Variant A and Variant B.
Entering a probability as a decimal such as 0.005 when the field expects 0.5%.
Comparing raw compromise totals from unequal A/B groups instead of standardized totals.
Treating expected compromises such as 6.8 as a guaranteed count of incidents.
Assuming every account not marked weak has identical compromise risk.
Ignoring changes in account mix, such as new users being disproportionately assigned to one variant.
Related Formulas
Frequently Asked Questions
What is the formula for expected password-related compromises?
Expected compromises equal weak-password accounts multiplied by weak-password risk, plus stronger-password accounts multiplied by stronger-password risk.
Why does Variant A traffic share not appear in the simplified avoided-compromises formula?
After each variant is standardized to the same full account population, the test-group size cancels out. Traffic share still matters when calculating the observed group totals.
How is compromise reduction calculated?
It is estimated monthly compromises avoided divided by standardized expected compromises for Variant A, multiplied by 100.
What happens if Variant B has a higher weak-password rate?
The avoided-compromises result becomes negative, indicating that Variant B has a higher estimated compromise count under the supplied risk assumptions.
Can weak-password risk and stronger-password risk be the same?
Yes, but then changing the weak-password rate will not change the estimated compromise outcome in this model.
Ready to calculate your result?
Use the calculator to get instant results with your own inputs.