Measuring website changes honestly: baselines, windows and inconclusive results

Why most reports on website changes cannot be trusted

Almost every website change gets reported as a success. Someone rewrites the services page, conversions are up the following month, and the change gets the credit. The trouble is that the same report would have been written whatever happened, because the comparison was chosen after the result was known. We call this success theatre: reporting that looks like evidence but could not have shown failure.

The common ways it happens:

• The window is picked afterwards. Comparing "the two weeks since launch" with "the month before" lets whoever writes the report choose the period that looks best.
• The season moved, not the page. Many B2B sites see traffic dip in August and over Christmas, then recover in September and January. A change made in late August will look like a success in October whatever it did.
• Several things changed at once. A new title, a paid campaign and a webinar in the same fortnight cannot be separated.
• The page was chosen because it had a bad month. Pages that performed unusually badly tend to drift back towards their normal level on their own. Fixing them and then measuring the recovery credits the fix with something that would have happened anyway.
• The tracking changed. A new consent banner, analytics tag or form tool changes what gets counted, so before and after are no longer measuring the same thing.
• The test was stopped when it looked good. Checking results repeatedly and stopping at the first significant-looking moment inflates false positives. As Evan Miller puts it, "Repeated significance testing always increases the rate of false positives"¹.

1. Record a baseline before you change anything

A baseline is the page's normal performance before the change, measured the same way you will measure it afterwards. Without one, there is nothing to compare against.

• Use whole weeks. Most B2B sites have a strong weekly pattern, with far less traffic at weekends. Comparing a period with five weekdays against one with four distorts the result.
• Use a period long enough to be stable. For most B2B pages, 8 to 12 weeks is a sensible starting point; a page with very little traffic needs longer.
• Use the same metric definition. If the outcome is enquiries, use the same form and the same analytics event before and after.
• Note anything unusual in the baseline. Campaigns, outages, a press mention or a known data problem in your tools. Google publishes a list of Search Console data anomalies; for example, it reports that a logging error affected impressions, CTR and average position (but not clicks) from 13 May 2025 to 27 April 2026².

2. Fix the measurement window in advance

The measurement window is the period after the change that you will compare with the baseline. Decide it before you make the change and write it down, so the result cannot be shaped by choosing a convenient end date.

• Match the baseline's length where you can, and keep whole weeks.
• Allow for delay. Search changes need Google to recrawl the page and rankings to settle. Google's own guidance says some changes take effect in a few days and others take several months³.
• Skip the launch period if the change is something visitors notice, such as a new layout, because the first days after launch are the least representative.
• Do not stop early. If you set eight weeks, read the result at eight weeks, not at the first moment it looks good.

3. Allow for seasonality and outside events

The biggest threat to a before and after comparison is that something other than your change moved. Check these before reading a result:

• The same period last year. Google's guidance on traffic drops suggests comparing the last three months year over year, so a recurring seasonal pattern stands out³. If last year's line moved the same way at the same time, the season is the likely cause.
• Search demand. Google Trends shows whether interest in your key queries rose or fell across the web⁴. Google's debugging guide recommends it for telling a site problem from a wider trend³.
• Google updates. The Google Search Status Dashboard lists ranking updates and incidents with their dates⁵.
• Your own activity. Campaigns, emails, events, PR, price changes, sales outreach and new content all bring visitors.
• The calendar. Bank holidays, school holidays, the August slowdown, the end of the financial year and the Christmas break.
• Unchanged pages. Similar pages you did not touch are a simple control. If they moved as much as the changed page, the change probably did not cause the movement.

4. Know the minimum data you need

The amount of data needed to detect a change depends on how common the outcome is and how large an effect you hope to see. Small effects on rare outcomes need a lot of data. The table below shows our own calculation, using the standard formula for comparing two proportions in a randomised A/B test at 95% confidence and 80% power. Evan Miller's free sample size calculator⁶ uses a slightly different formula and gives figures about 6% lower; either is fine for planning.

These are visitors to the page being tested, not to the whole site. A B2B services page with 2,000 visits a month and a 2% enquiry rate would need around 21 months of traffic to detect a 20% relative lift with a properly randomised test. A before and after comparison, which cannot rule out outside causes, needs more evidence still, not less.

5. Why low-traffic sites should not expect statistically proven uplifts

Put the numbers together and the conclusion is uncomfortable but important. A lean B2B team with 5,000 to 100,000 visits a month across the whole site will rarely have enough traffic on a single page to prove a modest change in conversion rate. That is not a flaw in the team or the tool; it is arithmetic.

What this means in practice:

• Judge changes on more frequent outcomes. Clicks from search, click-through rate and visits to the next page in the journey happen far more often than enquiries, so they can show a change sooner. They are signals, not proof of revenue.
• Look for large, plausible effects. A modest change will often be invisible in the noise. Fixing an unindexed page or a broken form can produce effects large enough to see.
• Lean on reasoning as well as numbers. A change that makes a page clearly answer a buyer's question is worth keeping even if the data cannot prove it helped, as long as it did not make things worse.
• Be suspicious of tidy uplift claims. A vendor or agency report showing a precise percentage lift from a small sample, with no confidence range, is very likely reporting noise.

Our guide to CRO for low-traffic B2B websites covers what to do when A/B tests cannot reach significance.

6. Treat "inconclusive" as a valid result

Every change should end with one of four outcomes, decided against the plan you wrote before the change:

Recording "inconclusive" honestly has real value. It stops the team building a strategy on noise, it keeps the record of what actually worked clean, and it builds trust with whoever reads the report, because the improvements you do report are believable.

Choosing a measurement method

From strongest to weakest:

1. A randomised A/B test. Visitors are split at random between versions at the same time, which removes seasonality and outside events. It needs the traffic shown above.
2. Before and after, with a control. Compare the changed page with similar unchanged pages over the same periods. If both moved, the change probably did not cause it.
3. Before and after, checked against last year. Use year-over-year data to see whether the seasonal pattern explains the movement.
4. A simple before and after. Weakest, because anything else that changed is counted as your effect. Use it only with the checks above, and say so in the result.

For most changes on a B2B site, method 2 or 3 is the realistic choice. The step-by-step guide to measuring the impact of a website change shows how to run one.

Keep a change log

A simple record for every change makes all of this possible. For each one, write down:

• The page, the change and the date it went live.
• The expected effect and the metric that should show it.
• The baseline period and figures.
• The measurement window, decided in advance.
• Anything else that happened during the window.
• The outcome: improved, worse, inconclusive or insufficient traffic.

Search Console also lets you annotate its Performance chart with a short note on a given date, visible to everyone with access to the property⁷. It is a useful companion to the log, not a replacement for it.

How Artiweave measures

Artiweave was built around this approach. Every change it proposes is approved, edited or rejected by a marketer before it is published, with automatic rollback. Outcomes are measured honestly and reported as they are, including "inconclusive" and "insufficient traffic". It does not claim uplifts the data cannot support. It is in private beta.

Frequently asked questions

How long should I wait before measuring a website change?

Decide the window before the change and wait for all of it. For search changes, allow time for Google to recrawl the page and for rankings to settle; Google says some changes take days and others months. Several weeks of whole-week data is a sensible minimum for most B2B pages.

What does an inconclusive result mean?

It means the data cannot tell whether the change helped: the movement was within normal variation, or something else happened at the same time. It is a valid and common result on low-traffic sites. Keep the change if it is sound on its merits, but do not report it as a win.

Can I A/B test a low-traffic B2B website?

You can run the test, but a page with a few thousand visits a month will rarely reach significance for a modest change in conversion rate. At a 2% conversion rate, detecting a 20% relative lift needs around 42,000 visitors to the page in total. Use more frequent metrics, look for large effects and record inconclusive results honestly.

How do I account for seasonality?

Compare against the same period last year, check Google Trends for changes in search demand, and compare with similar pages you did not change. If they all moved together, seasonality or an outside event is the likely cause.

Guides in this section

• How to measure the impact of a website change
• Inconclusive website test results: what they mean and what to do next

Related: Finding quick wins in Google Search Console and CRO for low-traffic B2B websites.

Sources

1. Evan Miller: How Not To Run an A/B Test (2010)
2. Search Console Help: Data anomalies in Search Console, entry dated 3 April 2026 (accessed November 2026)
3. Google Search Central: Debugging drops in Google Search traffic (accessed November 2026)
4. Google Trends
5. Google Search Status Dashboard
6. Evan Miller: Sample Size Calculator
7. Search Console Help: Search Console annotations (accessed November 2026)

Signals over success theatre.

Artiweave publishes only the website changes your team approves, and reports each outcome honestly, including when the data cannot tell. It is in private beta.

Join the waitlist