CRO for low-traffic B2B websites: what to do when A/B tests cannot reach significance

Why low traffic breaks A/B testing

An A/B test splits visitors at random between the current page (A) and a variant (B), then asks whether any difference in conversion is larger than chance would produce. Four numbers decide how many visitors that takes:

• Baseline conversion rate: the share of visitors who convert today. The lower it is, the fewer conversions each visitor contributes, and the more visitors you need.
• Significance level: usually 5%, meaning you accept a 5% chance of declaring a difference when there is none.
• Power: usually 80%, meaning that if the true effect is as large as you planned for, the test will detect it 80% of the time.
• Minimum detectable effect (MDE): the smallest true change you want the test to catch, often stated as a relative change, such as "a 20% improvement on a 2% baseline", which is 2% to 2.4%.

The critical relationship is between the MDE and the sample. In Kohavi, Henne and Sommerfield's practical guide to online experiments, the required sample is approximately n = (4rσ/Δ)², where Δ is the smallest difference to detect¹. Because Δ is squared in the denominator, halving the effect you want to detect roughly quadruples the visitors you need. Their own worked example needed over 1.6 million users (at 95% confidence and 90% power) to detect a 5% change in revenue per user¹.

How many visitors you actually need

The table shows visitors needed per variant for a two-variant test at 5% significance (two-sided) and 80% power. We calculated it with the standard formula for comparing two proportions; you can reproduce any cell with Evan Miller's sample size calculator using a relative MDE². Double each figure for the total across A and B.

Whole-site B2B conversion rates mostly sit between 1% and 3% (see our B2B conversion rate benchmarks), which puts most realistic tests in the top three rows.

A worked example

Your main solution page gets 3,000 visitors a month and converts 2% of them into demo requests: about 60 a month. You want to test a new headline and expect perhaps a 20% improvement, so 72 a month instead of 60.

• Detecting a 20% lift needs 21,109 visitors per variant, 42,218 in total: about 14 months.
• Detecting a 50% lift (60 to 90 a month) needs 7,652 in total: about 2.5 months.
• Detecting a doubling needs 2,282 in total: under a month.

Fourteen months is not a test: seasonality, campaigns and search updates will all change in that time. This page can only test changes expected to have a large effect, or should use a different method.

The traps that make small tests worse

• Peeking. Checking results daily and stopping when they look significant inflates false positives sharply. In Evan Miller's illustration, a test of a change with no real effect, checked after every visitor and stopped at the first significant result, produced a false winner 26.1% of the time instead of the intended 5%⁴.
• Too many variants. Each extra variant divides your traffic further and multiplies the chance of a false winner. On low traffic, test one variant against the original.
• Optimising a proxy. Clicks on a button convert far more often than demo requests, so they reach significance sooner, but more clicks do not guarantee more qualified leads. Use proxies as supporting evidence, not as the verdict.
• Calling a null result a loss. "No significant difference" on an underpowered test means "we could not tell", not "it made no difference".

What to do instead

1. Fix clear defects without testing

Some changes need no test because nobody would choose the broken version: a form that fails on mobile, validation that rejects legitimate company email addresses, a call to action that links to a 404, a page that is slow to show its main content on a phone. Find these by completing every key journey yourself on a phone and a laptop and by checking crawl reports. Fix them, confirm the fix works, and note the date.

2. Do qualitative research with a handful of buyers

Qualitative research tells you why buyers hesitate, and it needs very few people. Nielsen Norman Group's guidance is that a usability test with five users finds around 85% of the usability problems, and that three rounds of five, fixing between rounds, beats one round of fifteen³. The same guidance notes the figure assumes similar users; distinct groups (say, technical evaluators and finance approvers) need a few participants each³.

• Usability sessions: ask people who resemble your buyers to complete a real task, such as "find out whether this integrates with your CRM and roughly what it costs", while thinking aloud.
• Sales and customer interviews: ask what nearly stopped recent customers from getting in touch, and what prospects most often misunderstand.
• A one-question on-page survey: on pricing or the demo page, ask "What, if anything, is stopping you from getting in touch today?"

Qualitative findings do not prove a change will work. They make it much more likely that you are changing the right thing.

3. Make larger changes

The sample table shows why bold changes suit low traffic: a change that could plausibly double conversion needs a small fraction of the visitors a subtle one does. On a B2B site, that means rewriting a solution page around the buyer's problem, replacing a generic "Contact us" with a specific offer, adding pricing guidance, or removing half of a long form. Base them on your qualitative findings so they are bold and well founded.

4. Pool traffic where it is honest to do so

If ten service pages share the same template and call to action, a change to that shared element can be tested across all ten at once, which multiplies the sample. The same applies to a form used site-wide. Only pool pages whose visitors are genuinely similar, or the result will describe no page in particular.

5. Use sequential testing if you must monitor as you go

If you need to check a test as it runs, use a method designed for it rather than peeking at a fixed-sample test. Evan Miller's simple sequential test sets the stopping rules in advance, and can reduce the observations needed by 50% or more in some circumstances when there is a real effect, but reaches a conclusion more slowly than a fixed test when there is none⁵. It makes good use of limited traffic; it does not remove the need for traffic.

6. Measure before and after, with Search Console and analytics

For most changes on a low-traffic B2B site, a careful before-and-after comparison is the practical method. It is weaker than a randomised test, so the method has to do the work:

1. Pick one page and one change. Write down what you expect to move: click-through rate from search for a title change, demo requests for a page rewrite.
2. Choose control pages. Pick similar pages you will not change during the period. If the control pages move as much as the changed page, the cause is probably outside it.
3. Record the "before" period. Use at least 28 days, ideally longer. In Google Search Console's Performance report, filter to the page and note clicks, impressions, click-through rate and average position. Search Console data normally arrives within 2 to 3 days, and not every query is shown, to protect user privacy⁶.
4. Ship the change and note the date. Change nothing else on that page during the comparison.
5. Record an equal "after" period, aligned by weekday, starting a few days after the change so it has been recrawled.
6. Compare the change on the page with the change on the controls. For a title change, check that average position held steady, since click-through rate rises and falls with position.
7. Write the verdict in plain words: improved, worsened, inconclusive, or insufficient traffic, with the numbers and the caveats, such as a search algorithm update or a campaign during the period.

Search Console shows search performance, not conversions. For lead outcomes, use the same before-and-after structure in your analytics and CRM, and remember that lead quality can take a full sales cycle to show.

Which method for which page

Reporting results honestly

On a low-traffic site, most results are uncertain, and a team that reports every change as a win soon stops learning. Keep a simple log with the page, the change, the date, the method, the numbers, and one of four verdicts: improved, worsened, inconclusive, or insufficient traffic. Over a year, that log shows which kinds of change tend to help on your site.

For the wider process of choosing what to change, see our B2B CRO guide. For how to judge a single change in more depth, see how to measure the impact of a website change.

Where Artiweave fits

Artiweave is built for sites with this kind of traffic. It proposes specific changes from evidence, your team approves, edits or rejects each one, and approved changes are published with automatic rollback. Outcomes are reported with the strength of the evidence, so "inconclusive" and "insufficient traffic" appear as results rather than being dressed up as wins. It is in private beta.

Frequently asked questions

How much traffic do you need to A/B test?

It depends on your conversion rate and the smallest change you want to detect. At a 2% conversion rate, detecting a 20% relative improvement with 95% significance and 80% power needs about 21,000 visitors per variant, or 42,000 in total. Detecting a doubling needs about 1,100 per variant.

What is a minimum detectable effect?

The minimum detectable effect is the smallest true change a test is designed to detect reliably, usually 80% of the time. The required sample grows with the inverse square of that effect, so halving it roughly quadruples the visitors needed¹.

Is a before-and-after comparison valid?

It is weaker than a randomised test because seasonality, campaigns and search algorithm changes happen at the same time. It becomes more useful when you compare equal periods aligned by weekday, track similar pages you did not change as a control, change one thing at a time and report inconclusive results as inconclusive.

Should low-traffic sites use Bayesian A/B testing?

Bayesian methods report a probability instead of a pass or fail, which some teams find easier to act on. They do not create information that the traffic does not contain: with few conversions the result will still be uncertain, and it should be presented that way.

Sources

1. Kohavi, Henne and Sommerfield: Practical Guide to Controlled Experiments on the Web (KDD 2007), section 3.2
2. Evan Miller: Sample Size Calculator
3. Nielsen Norman Group: Why You Only Need to Test with 5 Users (Jakob Nielsen, March 2000)
4. Evan Miller: How Not To Run an A/B Test (April 2010)
5. Evan Miller: Simple Sequential A/B Testing (October 2015)
6. Google Search Console Help: About Search Console data

Improve a low-traffic website without pretending every change is a win.

Artiweave proposes changes your team approves, publishes them with automatic rollback, and reports outcomes honestly, including "inconclusive" and "insufficient traffic". It is in private beta.

Join the waitlist