UtilibyWritten by the Utiliby editorial team · Last updated October 15, 2026 · Rules and formulas reviewed October 2026
What this free online A/B test calculator does
An A/B test shows two versions of a page, email, advert or button to different visitors and counts how many take the action you want. The hard part is deciding whether the difference you see is real or just luck. This free online A/B test calculator answers that. Enter visitors and conversions for version A and version B and it reports the conversion rate of each, the relative lift, the p-value, the z-score, a confidence interval for the gap, the chance that B beats A and a plain verdict.
It also includes a sample size calculator to plan a test before you start, a visual of both conversion rate ranges that updates as you type, and warnings about the most common mistakes: too little data, an uneven traffic split and stopping too early.
How to read your A/B test result
- Check the numbers. Visitors are people who saw each version. Conversions are people who took the action.
- Read the verdict. Green means a significant winner, amber means not enough evidence yet.
- Look at the lift range. The confidence interval shows how small or large the true gain could be. If it includes zero, the result is not conclusive.
- Check the chart. Curves that barely overlap mean strong evidence.
- Read the warnings. A traffic split far from 50/50 or fewer than 25 conversions means you should not trust the result yet.
- Decide. Weigh the gain against the cost of making the change.
A/B testing terms in plain language
| Term | Meaning |
|---|---|
| Conversion rate | Conversions divided by visitors for a version |
| Control (A) | The current version you compare against |
| Variant (B) | The new version you want to test |
| Relative lift | The change in rate as a share of the control rate |
| p-value | How surprising the gap would be if there were no real difference |
| Confidence level | One minus the false positive rate you accept, such as 95% |
| Confidence interval | The range of gaps consistent with the data |
| Power | The chance of detecting a real effect of a chosen size |
| Minimum detectable effect | The smallest lift worth finding |
Planning a test: sample size, duration and the minimum detectable effect
Most failed tests were doomed before they began, because they were too small to find anything. The planner uses the standard formula for comparing two proportions. Give it your current conversion rate and the smallest relative lift that would matter to the business, choose your confidence and power, and it returns the visitors needed in each version, the total and the number of days at your traffic level. Smaller lifts need far more traffic: halving the lift you want to detect roughly quadruples the sample. If the number is too high for your traffic, test a bigger change, test on a page with more traffic, or use a metric higher in the funnel.
Whatever the sample size says, run the test for at least one full week, ideally two, so that weekday and weekend behaviour are both included, and avoid holidays and sales that distort normal traffic.
Common A/B testing mistakes
- Peeking. Stopping when the result first looks significant inflates false positives. Decide the sample size in advance.
- Tiny samples. Small samples produce wild swings that do not repeat.
- Many variants and metrics. Every extra comparison raises the chance that something looks significant by luck.
- Uneven splits. Visitors should be split as planned. A mismatch signals a bug.
- Changing the test midway. Changing traffic or the page during a test invalidates it.
- Ignoring practical significance. A significant 0.1% gain may not be worth the effort.
- Counting the same visitor twice. Use unique visitors for both numbers.
What to test and how to link it to the rest of your marketing
The biggest wins usually come from testing the message, not the colour of a button. Good starting points are the headline, the offer, the call to action, the form length, the price presentation, images and social proof. Write stronger copy with the Headline Analyzer, then test it. Tag each variant with the UTM Link Builder so that your analytics can separate traffic. Measure the value of a lift with the Marketing ROI Calculator, and use the Engagement Rate Calculator for social tests.
Frequentist versus Bayesian results
The p-value and confidence interval come from the classic (frequentist) approach, and the chance that B beats A is a simple Bayesian estimate. They usually agree when sample sizes are reasonable. The p-value answers: how unusual is this data if nothing is going on? The Bayesian figure answers: given this data, how likely is B to be better? Many teams find the second easier to explain, but neither removes the need for a planned sample size and enough data.
Worked example: reading a real test
Suppose version A, the control, was shown to 5,000 visitors and 250 of them signed up, a rate of 5.00%. Version B was shown to another 5,000 visitors and 305 signed up, a rate of 6.10%. The absolute difference is 1.10 percentage points and the relative lift is 22%. Load this as an example in the tool and you will see a p-value of about 0.016, below the 0.05 threshold for 95% confidence, so B is a significant winner. The confidence interval for the difference runs from about 0.2 to 2.0 points, so the real gain could be small or large. A careful team would plan for the lower end of that range when estimating the value of rolling out B.
Now imagine the same rates with only 500 visitors each. The lift is the same, but the p-value is far above 0.05 and the interval includes zero. The tool says there is no significant difference yet, and the sample size planner shows how many visitors you would need. This is why rate alone is never enough: the amount of data decides how much you can trust it.
How to report an A/B test to your team
A good test report is short and honest. State the hypothesis, the dates, the number of visitors and conversions in each version, the lift with its confidence interval, the p-value and the decision. Mention anything unusual, such as a holiday, a traffic spike or a tracking change, and say what you will do next. Use the print option to save a one-page report with the chart. Keep a log of every test, including the ones that failed to find a winner, because a record of what did not work stops the same idea from being tested again and again.
Accuracy, limits and responsible use
The calculator uses a two-proportion z-test with a pooled standard error for the p-value and an unpooled interval for the difference, and results have been checked against reference statistical software. These methods assume large enough samples, independent visitors and random assignment. They do not apply to averages such as revenue per visitor, they do not correct for many comparisons and they cannot detect biased tracking. Treat the output as decision support, not proof. This free online A/B test calculator is for lawful, honest use only.
Frequently asked questions
How do I use this free A/B test calculator?
Enter the number of visitors and the number of conversions for version A and version B, pick a confidence level and read the result. The tool shows the winner, lift, p-value, confidence interval and a chart.
What is statistical significance in an A/B test?
A result is statistically significant when the p-value is below your threshold, for example 0.05 for 95% confidence. It means a gap this large would be unlikely if both versions truly performed the same.
What is a p-value?
The p-value is the probability of seeing a difference at least as large as yours if there were no real difference. A small p-value is evidence against no difference. It is not the probability that B is better.
What confidence level should I choose?
95% is the common default. Use 99% when a wrong decision is costly and 90% for low-risk tests where speed matters. Choose the level before you start, not after seeing the data.
What is the difference between one-sided and two-sided tests?
A two-sided test checks whether B is different from A in either direction. A one-sided test checks only whether B is better. Two-sided is safer and is the usual choice.
What is relative lift versus absolute difference?
Absolute difference is the gap in percentage points, such as 5.0% to 5.8% is 0.8 points. Relative lift divides that gap by the control rate, which is 16% in this example.
How many visitors do I need for an A/B test?
It depends on your current conversion rate and the smallest lift you care about. Use the Sample size planner. Smaller lifts and lower conversion rates need many more visitors.
How long should I run an A/B test?
Until you reach the planned sample size and for at least one full business cycle, usually one to two weeks, so weekday and weekend behaviour are both covered.
Can I stop the test as soon as it is significant?
No. Checking repeatedly and stopping at the first significant result greatly raises the false positive rate. Decide the sample size first and look at the final result.
What is statistical power?
Power is the chance that a test detects a real effect of the size you care about. 80% is the usual minimum. Low power is the main reason real improvements go undetected.
What does the chance that B beats A mean?
It is a Bayesian estimate that treats each conversion rate as uncertain and asks how likely B is higher. The tool uses a flat prior and a normal approximation, which is accurate with reasonable sample sizes.
What is a sample ratio mismatch?
It happens when the split of visitors differs from the planned split, for example 50/50 giving 5,000 against 4,500. It usually signals a tracking or targeting bug, so fix it before trusting results.
What if there are very few conversions?
With fewer than about 25 conversions per version, the normal-approximation test is unreliable. The tool warns you. Collect more data or use an exact test.
Can I test more than two versions?
This calculator compares two versions. For several variants, compare each against the control with a stricter threshold, such as a Bonferroni correction, to limit false positives. The planner accepts several versions to total the traffic.
Does this work for click-through rate, sign-ups or purchases?
Yes. Any yes-or-no outcome per visitor works: clicks, sign-ups, opt-ins, purchases or opens. It does not apply to revenue per visitor, which needs a different test.
Is the calculation accurate?
It uses a two-proportion z-test with a pooled standard error for the p-value, which matches standard statistical software, and an unpooled interval for the difference. We checked the results against reference values.
Is my data stored?
No. All calculations run in your browser and nothing is uploaded.
Sources and further reading
- Wikipedia, A/B testing.
- NIST, Engineering Statistics Handbook: Comparing two proportions.
- Kohavi, Tang and Xu, Trustworthy Online Controlled Experiments.
- Evan Miller, How not to run an A/B test.
- Penn State STAT 415, Comparing two population proportions.
- Fabijan et al., Diagnosing sample ratio mismatch in online controlled experiments.
Sources were consulted in October 2026. Standards and rules can change, so always check the official page for your situation.