A/B Testing Token Usage Per-User Formula
Learn how to calculate token usage per user for two A/B test variants and the traffic-weighted average across the experiment.
This calculation estimates the model tokens consumed by a typical participating user in each variant, then blends those totals using the experiment traffic split. It is useful for comparing prompt or response designs before scaling an A/B test.
- 100% Free
- No Sign-Up Required
- Private & Secure
- Mobile Friendly
Weighted Average Tokens per Test User
Where:
Add input and output tokens for each variant's average request, multiply each by the requests per user and its traffic share, then add the two weighted amounts.
Variables Explained
| Variable | What It Means | Unit |
|---|---|---|
| requestsPerUser - Average requests per user | Average number of model requests made by a participating user during the measurement period. | number |
| aInputTokens - Variant A input tokens per request | Average prompt, system, context, and other tokens sent with one Variant A request. | tokens |
| aOutputTokens - Variant A output tokens per request | Average generated tokens returned by one Variant A request. | tokens |
| bInputTokens - Variant B input tokens per request | Average prompt, system, context, and other tokens sent with one Variant B request. | tokens |
| bOutputTokens - Variant B output tokens per request | Average generated tokens returned by one Variant B request. | tokens |
| variantAShare - Variant A traffic share | Percentage of participating users assigned to Variant A. Variant B receives the remaining share. | percent |
Step-by-Step Calculation
Find Variant B traffic share
Variant B receives all traffic not assigned to Variant A.
variantBShare = 100 - variantAShare
Calculate Variant A tokens per request
This combines the average input and generated output for one A request.
aTokensPerRequest = aInputTokens + aOutputTokens
Calculate Variant B tokens per request
This combines the average input and generated output for one B request.
bTokensPerRequest = bInputTokens + bOutputTokens
Calculate tokens per user for each variant
Each request total is multiplied by the average request frequency per user.
aTokensPerUser = aTokensPerRequest * requestsPerUser; bTokensPerUser = bTokensPerRequest * requestsPerUser
Calculate blended usage
The final result weights each variant's per-user usage by its share of test traffic.
weightedTokensPerUser = (aTokensPerUser * variantAShare / 100) + (bTokensPerUser * variantBShare / 100)
Compare A with B
A positive result means A uses more tokens per user than B; a negative result means it uses fewer.
tokenDifferencePercent = ((aTokensPerUser - bTokensPerUser) / bTokensPerUser) * 100
Example: Shorter Variant A Prompt vs Longer Variant B Prompt
Variant B traffic share
100% - 50%
50%
Variant A tokens per request
800 + 300
1,100 tokens
Variant B tokens per request
1,000 + 400
1,400 tokens
Variant A tokens per user
1,100 × 10
11,000 tokens
Variant B tokens per user
1,400 × 10
14,000 tokens
Weighted average tokens per user
(11,000 × 50%) + (14,000 × 50%)
12,500 tokens
A versus B difference
(11,000 - 14,000) / 14,000 × 100
-21.4%
Final Result
The experiment is estimated to use 12,500 tokens per participating user on average. Variant A uses 11,000 tokens per user and Variant B uses 14,000.
Assumptions
- ✓Input and output token figures are representative averages for requests in the selected measurement period.
- ✓Users assigned to each variant make the same average number of requests.
- ✓Traffic allocation is the only weighting factor used for the blended average.
- ✓Input and output tokens are counted together as usage volume, without applying provider pricing.
Limitations
- !Actual token use can vary substantially by conversation length, retrieval context, tools, and user behavior.
- !Averages can hide heavy users and unusually long requests, which may matter for capacity planning.
- !The calculation does not separate cached, reasoning, tool, or other provider-specific token categories.
- !This is a token-volume estimate and does not calculate API charges.
Common Mistakes to Avoid
Entering a total test request count instead of the average requests made by one user.
Using a maximum prompt length as though it were the observed average input-token count.
Forgetting that Variant B traffic share is the remainder after Variant A's share.
Comparing tokens per request when request frequency differs between real variant cohorts.
Treating the weighted average as the amount used by every individual user.
Related Formulas
Frequently Asked Questions
How do I calculate tokens per user in an A/B test?
Add input and output tokens per request for the variant, then multiply by the average requests per user.
How is average token usage across A/B test traffic calculated?
Multiply each variant's tokens per user by its traffic share as a decimal, then add the two values.
What does a negative A minus B token difference mean?
It means Variant A uses fewer tokens per user than Variant B.
Why use Variant B as the percentage comparison baseline?
The percentage formula divides the A-minus-B difference by Variant B tokens per user, so it describes A relative to B.
Does a 50/50 split always produce the midpoint of A and B usage?
Yes, when both variants use the same average requests per user, the blended result is the arithmetic midpoint of their per-user totals.
Ready to calculate your result?
Use the calculator to get instant results with your own inputs.