Ad test significance calculator
Ad Test Significance Calculator
Check whether your A/B test result is statistically significant, and work out the sample size and test duration a new ad test needs from your budget and CPM. Works with impressions and conversions from Google Ads, Meta, TikTok or any ad platform.
Copy the numbers from your ads platform. Everything updates as you type.
Variant A · control
Variant B
Verdicts use a two-proportion test at 95% confidence and 80% power.
KEEP RUNNING
Promising, but not proven
B is converting 9.7% better than A, but at this volume a gap this size still appears by luck in roughly one test in 8. Calling it today is a weighted coin flip.
Keep running: roughly 222,703 more impressions per variant. Add how many days it has run and this becomes a date.
This exact test, rerun 1,000 times
Same volume, same gap. Each square is one rerun; the odds are computed, not sampled.
218
found the winner
781
read as a tie
1
picked the loser
Common Ad Testing Mistakes
The three patterns we find most often when auditing accounts that test creative.
Calling the test on a good day
Checking a running test daily is normal. Stopping it the first time the numbers look significant is where accounts go wrong: a test checked every day and stopped on its best reading will ship false winners several times more often than the confidence level suggests. Decide the sample size before launch, then judge the test once it gets there.
Daily read of the same test
One lucky day out of twelve. The greyed days never happened.
Testing a small improvement on a small budget
Detecting a 10% lift in conversion rate takes several times more traffic than most advertisers expect, and halving the test length roughly doubles how often a real winner reads as a draw. If the budget cannot fund the sample size, test bigger creative differences instead of subtle ones. Our PPC budget calculator helps set the spend the test needs.
the test you budgeted
the test a 10% lift needs
Same budget, one-fifth of the sample. The rest reads as a draw.
Ignoring the weekly cycle
Weekend buyers behave differently from Tuesday-morning researchers. A test that runs Monday to Thursday compares a different audience against itself and can flip when the weekend arrives. Run whole weeks even when the sample size is reached mid-week. This discipline is the core of how we run paid social creative testing for clients.
Two weeks of buyers
Teal days were tested. The weekend buyer never got a vote.
A/B Test Statistics Explained
The calculator runs a two-proportion significance test, the same maths behind Google Ads experiments and Meta A/B tests. These are the six terms that matter.
Statistical significance
A result is statistically significant when the gap between two ads is big enough, at the volume tested, that it would rarely appear by chance. It does not mean the gap is large or valuable, only that it is probably real.
Ad A
Ad B
Statistical significance: A result is statistically significant when the gap between two ads is big enough, at the volume tested, that it would rarely appear by chance. It does not mean the gap is large or valuable, only that it is probably real.
Confidence level: How sure you want to be before calling a winner. At 95% confidence, a gap this size would appear by luck in fewer than one test in twenty. Higher confidence needs more data.
Statistical power: The chance your test spots a winner that genuinely exists. At 80% power, one real winner in five still slips through as a draw. Power is what the green share of the rerun field shows.
Sample size: The number of impressions each variant needs before the test can give a reliable answer. It depends on your baseline rate and the smallest lift you care about, not on how the test is going.
Minimum detectable effect: The smallest improvement your test is set up to find. Chasing a smaller effect means a much bigger sample: detecting a 5% lift takes roughly four times the traffic of a 10% lift.
Peeking: Checking a running test repeatedly and stopping the moment it looks significant. It feels efficient and quietly multiplies false winners, which is why sample size gets decided before launch.
These six numbers drive both modes above. The calculator runs a two-proportion significance test, the same maths behind Google Ads experiments and Meta A/B tests.
Ad Test Significance FAQs
Enter the impressions and conversions (or clicks) for each variant in the calculator above. It runs a two-proportion significance test and gives a verdict: safe to call, keep running, or stop. At the default 95% confidence, "safe to call" means a gap this size would appear by chance in fewer than one test in twenty.
Until each variant reaches the sample size the effect needs, which depends on your baseline conversion rate or CTR, the smallest lift worth acting on, and your daily traffic. Plan mode converts your daily budget and CPM into a test length in days and pounds. Run whole weeks where possible, so day-of-week behaviour evens out.
As a rule of thumb, detecting a 10% lift on a 1% conversion rate needs a six-figure impression count per variant, and halving the detectable lift roughly quadruples the requirement. Plan mode calculates the exact number for your rates, and Judge mode shows how far a running test still has to go.
Sometimes. Dropping from 95% to 90% confidence shortens the test but doubles the chance of shipping a false winner. It is a reasonable trade for low-stakes creative rotations, and a poor one for landing page or offer changes that will carry real budget. The strictness setting in Plan mode prices this trade for your numbers.
Yes. The maths is platform-agnostic: any test that reports impressions and conversions or clicks per variant can be judged here, including Google Ads experiments, Meta A/B tests and TikTok split tests. The platforms’ own dashboards use similar tests but rarely show you the sample size you still need.
Confidence covers random noise, not everything else. Traffic mix can shift mid-test, one variant can win the morning auction and lose the evening one, and a real lift can be smaller than the one you planned for. Significance is the floor for a decision, not a guarantee of the future.
How to Run Ad Tests Properly
Four steps, four free tools. This calculator is the last one.
Set the daily spend the test needs before it starts.
Write and compare the ad variants side by side.
Get the creative built to spec for every placement.
Run the test to its sample size, then check it in the calculator above.
Rather have this run for you? We run structured creative tests on Meta and LinkedIn accounts, and ad experiments inside the wider paid search programme.
Paid social creative testingPaid search managementTesting the ads is half the job
A senior specialist reviews the account around your tests: wasted spend, conversion tracking, and whether the winners are actually being fed budget.