A/B Testing Statistics: How to Tell a Real Winner From a Lucky One

Most "winning" A/B tests are not wins. They are noise that got called early.

That is not cynicism, it is arithmetic. If you run a test with no real difference between the versions and check it every day, stopping the moment it looks significant, you will "find" a winner roughly one time in three, purely by chance. The statistics exist to stop you from doing that to yourself. You do not need a degree in them. You need four ideas.

Short answer: when is an A/B test result real?

A result is trustworthy when three things are true: you decided the sample size before you started, you let the test reach it without stopping early, and it cleared your significance threshold (usually 95%). Miss any one of those and the number on your dashboard is a guess wearing a lab coat.

Idea 1: Statistical significance is a measure of doubt, not of size

Statistical significance answers one narrow question: if the two versions were actually identical, how likely is it that I would see a gap this big by pure chance?

"95% significance" (a p-value below 0.05) means there is less than a 5% chance the difference is a fluke of random variation. That is it. It does not tell you the difference is large, or that it will hold up next quarter, or that it matters to the business. It only tells you the result is unlikely to be noise.

Which leads to the trap: a result can be statistically significant and commercially pointless (a real 0.1% lift), or genuinely important but not yet significant because you haven't collected enough data. Significance is about confidence, not about whether the win is worth having.

Idea 2: Sample size is decided before the test, not discovered during it

The single most common mistake is starting a test with no idea how much traffic it needs, then stopping whenever the result looks good. That is not testing. That is fishing.

How much you need depends on three inputs: your current conversion rate, the smallest improvement you would care about detecting (the minimum detectable effect), and how confident you want to be. Plug them into a calculator like Evan Miller's before you launch and you get a number: X visitors per version. That number is the finish line, and you commit to it up front.

A useful, slightly counterintuitive consequence: small improvements are expensive to detect. Detecting a 5% relative lift can take roughly fifteen times the traffic of detecting a 20% lift. This is the statistical case for testing big swings, the offer, the form, the whole page, rather than button colors. Not because small changes never help, but because you often can't afford to prove they did.

Idea 3: Peeking is how honest people crown fake winners

Here is the one that catches almost everyone. You set up a clean test. You check it every morning. On day four it hits 95% significance, so you call it and ship the winner.

That process is broken, even though every individual step felt responsible. Every time you look at an in-progress test and let yourself stop on significance, you take another roll of the dice. Check continuously and stop at the first green light, and your real false-positive rate climbs from the 5% you think you have to 20-30% or more (Georgiev, Statistical Methods in Online A/B Testing, 2019; Kohavi, Tang, and Xu, Trustworthy Online Controlled Experiments, 2020).

The fix is dull and it works: pick the sample size in advance, and do not act on the result until you reach it. If you genuinely need to monitor tests as they run, that is a real branch of statistics (sequential testing) with its own math, but the everyday version of the discipline is simply: no peeking-and-stopping.

Idea 4: Run in whole weeks, and watch for the losing majority

Two smaller things that quietly wreck results.

Build landing pages with AI in 60 seconds

Drag-and-drop editor, 166+ templates, A/B testing, and no traffic caps. Try Leadpages free for 7 days.

Start free trial

Whole weeks. Your Tuesday traffic and your Sunday traffic convert differently, and they come from a different mix of sources. Run a test for four and a half days and you have baked a day-of-week bias into the result. Always run in complete weeks so every version sees the same rhythm.

Expect to lose. At Microsoft, only about a third of experiments produce a positive result, and plenty are flat or negative (Kohavi and Thomke, Harvard Business Review, 2017). A testing program that reports an unbroken streak of winners is not lucky, it is measuring wrong, usually by peeking. Losing tests are not failures. They are the system working, telling you an idea was wrong before you shipped it to everyone.

The whole discipline, in one paragraph

Write the hypothesis first. Calculate the sample size before you launch. Run in whole weeks until you hit that number, without stopping early. Read significance as confidence-it-is-not-noise, not as proof-it-matters, and sanity-check the size of the win against whether it is worth shipping. Do that and you will ship fewer changes, but the ones you ship will actually hold.

This pairs with the two decisions on either side of it: which test method to use before you start, and what to test first once you're ready. And none of it works if the tracking underneath is wrong, so confirm that first: Landing Page Tracking Setup.

How Leadpages helps

Good statistics are worthless if you can't run the test in the first place. Leadpages lets you build a campaign page, duplicate a variant, split test it, and track the result without a developer ticket, so the slow part of testing is the traffic, not the setup.

Try it now. Free for 7 days, full access, and you're not charged until day 7.

Frequently asked questions

What does statistical significance mean in A/B testing? It measures how likely your result is to be a fluke of random chance. A 95% significance level (p-value below 0.05) means there is less than a 5% probability the difference between versions is noise. It indicates confidence that the effect is real, not that the effect is large or commercially important.

How much traffic do I need for an A/B test? It depends on your baseline conversion rate, the smallest improvement you want to detect, and your confidence level. Calculate it with a sample size tool before launching. As a rough anchor, detecting a 20% relative lift on a 3% page takes on the order of 14,000 visitors per version.

Why shouldn't I stop an A/B test as soon as it's significant? Because checking repeatedly and stopping at the first significant result (called peeking) inflates your false-positive rate well beyond the 5% you intend, often to 20-30%. Commit to a sample size in advance and wait until the test reaches it.

How long should an A/B test run? Until it reaches your pre-calculated sample size, and always in complete weeks so day-of-week differences in traffic and behavior don't skew the result. For most pages that means one to four weeks, not a few days.

Is a statistically significant result always worth shipping? No. Significance tells you a difference is probably real, not that it's big enough to matter. A tiny but significant lift may not justify the change, and a promising result that hasn't reached significance isn't yet trustworthy. Judge both the confidence and the size.