Statistical significance for marketers, in plain English

The idea in one paragraph

Conversion rates wobble from week to week even when nothing changes. If version A converted 2.0% of visitors and version B converted 2.6%, the question is whether that gap is a real effect or the normal wobble. A significance test answers a narrow version of that question: if B were truly no better than A, how often would we see a gap at least this big? That probability is the p-value. If it is small (by convention, below 0.05), the gap is called statistically significant. Everything else on this page follows from that definition.

The terms you will meet, translated

Worked example: the same uplift at three traffic levels

Suppose a new headline on a demo request page lifts the conversion rate from 2.0% to 2.6%, a 30% relative improvement. Here is what a standard test makes of exactly the same rates with different amounts of traffic, in a randomised A/B test with visitors split evenly:

The observed lift is identical in every row. What changes is how much noise surrounds it. At 1,000 visitors per version, the data is consistent with the headline hurting badly or helping enormously. At 10,000 per version, the whole range is positive, so you can say with reasonable confidence that the new headline is better, by somewhere between about 9% and 51%.

How the 4,000 row was calculated

We used the standard two-proportion z-test, which is what most A/B testing calculators use, and checked every figure in this guide with a short script. You can follow it with a calculator:

1. Pool the rates. 184 conversions out of 8,000 visitors in total is 2.3%.
2. Work out the standard error of the gap under "no difference". The square root of 0.023 × 0.977 × (1/4,000 + 1/4,000) is about 0.00335, or 0.335 percentage points.
3. Divide the observed gap by it. The gap is 0.6 points, so z is 0.6 ÷ 0.335, about 1.79.
4. Turn z into a p-value. For a two-sided test, z of 1.79 gives p of about 0.073. That is above 0.05, so the result is not significant at the 5% level.
5. Build the confidence interval. Using each version's own rate, the standard error is about 0.335 points, so the 95% interval is 0.6 ± 1.96 × 0.335, from about −0.06 to +1.26 points. Divided by the 2.0% baseline, that is roughly 3% worse to 63% better.

A p-value of 0.07 is not "nearly significant" in a way that justifies calling a winner, and it is not proof that nothing happened. The honest summary is the interval: the change could be anything from slightly harmful to very helpful, and you do not yet know which.

What significance does not tell you

In 2016 the American Statistical Association published a statement with six principles on p-values¹. Three of them matter most for marketers:

• "P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone."
• "Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold."
• "A p-value, or statistical significance, does not measure the size of an effect or the importance of a result."

In practice, these are the readings to avoid:

How much traffic you need

Before a test, the question to ask is not "will it be significant?" but "with my traffic, what is the smallest effect I could detect?" These are our own calculations for a two-sided test at the 5% level with 80% power, using the standard formula for two proportions. Evan Miller's free sample size calculator³ uses a slightly different formula and gives slightly lower figures; either is fine for planning.

Turn it round and the picture for a lean B2B team is clear. With a 2% baseline, 2,000 visitors per version can only reliably detect a relative lift of about 72%; 5,000 per version, about 43%; 10,000 per version, about 30%. Those are visitors to the page under test, not to the whole site. This is why our guide to CRO for low-traffic B2B websites recommends bolder changes, pooled templates and qualitative evidence over small tweaks tested one at a time.

Four ways to manufacture a false winner

1. Peeking and stopping early

If you check results repeatedly and stop the first time the test looks significant, the 5% false alarm rate no longer holds. In Evan Miller's worst-case illustration, checking after every observation, up to 150 observations, produced a false winner 26.1% of the time, "more than five times what you probably thought the significance level was"⁴. Fix the sample size in advance, or use a method designed for continuous monitoring, such as a sequential test.

2. Judging many metrics at once

Each metric tested at the 5% level has a 5% chance of a false alarm when nothing changed. Across 10 independent metrics, the chance that at least one looks significant by luck is 1 − 0.95¹⁰, about 40%. Pick one primary metric before the test and treat the rest as supporting evidence.

3. Slicing into segments afterwards

Mobile, desktop, paid, organic, new, returning: every slice is another chance for noise to look like a finding, and each slice has less traffic than the whole. A segment result is a hypothesis for the next test, not a conclusion.

4. Forgetting how rare real wins are

Ron Kohavi and colleagues reported that only one third of ideas tested at Microsoft improved the metrics they were designed to improve, and that in less well understood domains the figures are much worse⁵. When real effects are uncommon, a bigger share of your significant results are flukes. An illustration, with assumed numbers: run 100 tests where 10 changes genuinely have the effect you planned for and 90 do nothing. At 80% power, you detect 8 of the 10. Of the 90 that do nothing, about 4 or 5 will look significant by chance, roughly half of them in the positive direction. So of about 10 apparent winners, around 2 are false. The lower your hit rate, the worse that ratio gets.

A checklist before you call a winner

1. The hypothesis, primary metric, sample size and end date were written down before the test started.
2. The test ran to its planned end, with no early stop because it looked good.
3. Traffic split as intended. A noticeably uneven split points to a broken test.
4. Tracking worked throughout. Check the conversion event fired in both versions every day.
5. The test covered whole weeks, so weekday and weekend behaviour are both included.
6. You are reporting the confidence interval, not just "significant" or "not significant".
7. The lower end of the interval is still worth having, or you have said plainly that it might not be.
8. No major outside event, such as a campaign, price change or holiday, overlapped the window.

When you cannot reach significance

On a B2B site with a few thousand visits a month to a key page, many properly run tests will end without significance. That does not mean you should stop measuring. It means you should report the result as inconclusive with its range, make decisions on the merits of the change, and use methods that fit your traffic. Our guide to inconclusive website test results covers what to do next, and the guide to measuring website changes honestly sets out the four verdicts we use: improved, worse, inconclusive and insufficient traffic.

Where Artiweave fits

Artiweave does not declare winners the data cannot support. Each change it publishes is reported with the numbers behind it, and when significance is out of reach the verdict says so: "inconclusive" or "insufficient traffic".

Frequently asked questions

What does statistically significant mean in A/B testing?

It means the difference between the versions is larger than you would usually expect from chance if the change really did nothing. At the common 5% level, a gap that large would appear less than 5% of the time with no real difference.

Is a p-value the chance my change did not work?

No. A p-value is calculated assuming the change made no difference, and measures how surprising your data would be under that assumption. It is not the probability that the change worked or failed.

Do I have to use 95% confidence?

No. The 5% level is a convention. A lower bar, such as 90%, finds more real effects but also more false winners. Choose the level before the test, and report the confidence interval whatever you choose.

Does a significant result mean the uplift will last?

Not necessarily. The observed lift is a noisy estimate, and on small samples significant results tend to overstate the true effect. Plan around the lower end of the confidence interval.

How long should an A/B test run?

Until it reaches the sample size you calculated in advance, in whole weeks. Do not stop early because it looks significant, and do not extend it because it does not.

Sources

1. American Statistical Association: Statement on Statistical Significance and P-Values, press release with the six principles (7 March 2016; accessed November 2026)
2. Altman and Bland: Absence of evidence is not evidence of absence, BMJ 1995;311:485 (19 August 1995)
3. Evan Miller: Sample Size Calculator (accessed November 2026)
4. Evan Miller: How Not To Run an A/B Test (18 April 2010)
5. Kohavi, Deng, Frasca, Longbotham, Walker and Xu: Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained (KDD 2012), introduction

Report what the data supports, nothing more.

Artiweave publishes only the changes your team approves, with automatic rollback, and reports every outcome honestly, including "inconclusive" and "insufficient traffic". It is in private beta.

Join the waitlist