How to Measure the Impact of a Website Change, Step by Step

The method in one line each

1. State the change and the expected effect. 2. Pick one primary metric. 3. Check you have enough data. 4. Record a baseline. 5. Make one change and log it. 6. Fix the measurement window. 7. Compare like with like, ideally against a control. 8. Check for outside causes. 9. Call the result. 10. Record it and decide.

This guide is the practical companion to measuring website changes honestly, which explains the principles behind each step. It assumes a typical B2B website: a few thousand to a hundred thousand visits a month, Google Search Console, and an analytics tool such as Google Analytics.

Step 1: State the change and what you expect it to do

Write one sentence before you touch the page: what you are changing, on which page, and what you expect to happen. For example: "Rewrite the title and opening paragraph of the managed services page so it matches the queries it already appears for. Expect a higher click-through rate from search for those queries."

Writing this down first is what makes the measurement honest. It stops the goal shifting afterwards to whichever metric happened to rise.

Step 2: Choose one primary metric

Pick the metric closest to what the change is meant to affect, and that happens often enough to measure:

Type of change	Primary metric	Where to get it
Page title or meta description	CTR and clicks for the page's main queries	Search Console Performance report, filtered to the page
New or expanded content for search	Impressions and clicks for the target queries	Search Console
Indexing fix	Indexed status, then impressions	URL Inspection and the Page indexing report
Call to action, layout or copy for visitors	Clicks to the next step, or form submissions	Your analytics tool's key events
Form changes	Form submissions per visit to the form page	Analytics key events, cross-checked with your CRM

In Google Analytics 4, the actions that matter most to your business are configured as key events¹. Record one or two secondary metrics as a check that nothing got worse, but decide the result on the primary metric.

Step 3: Check you have enough data

Before investing in measurement, check whether the page can produce an answer at all. Count how many times the primary metric happened in a recent period of the same length as your planned window.

A useful rough rule for counts such as clicks or form fills: when nothing has changed, a count of n will commonly vary by about two times its square root from one period to the next. This is an approximation that assumes events happen independently, but it gives a quick sense of the noise:

Count in the period	Rough normal variation	As a share of the count
25	About ±10	About ±40%
100	About ±20	About ±20%
400	About ±40	About ±10%
2,500	About ±100	About ±4%

So if a page gets around 25 enquiries in the window, a change would need to move enquiries by far more than 40% before you could tell it apart from chance. If the expected effect is smaller than the normal variation, plan to judge the change on a more frequent metric (such as clicks to the next page), or accept up front that the likely outcome is "insufficient traffic". For proper A/B tests, the hub page has a sample size table, and Evan Miller's calculator gives figures for your own baseline².

Step 4: Record a baseline

Choose a recent period of whole weeks, typically 8 to 12 weeks, ending the day before the change.
Export the primary and secondary metrics for the page over that period. In Search Console, filter to the page with + Add filter, then Pages, and export the daily figures from the Dates tab³.
Export the same figures for two or three similar pages you will not change. These are your control pages.
Export the same period a year earlier, if you have the data, to see the seasonal pattern. Search Console's date filter covers up to 16 months⁴.
Note anything unusual in the baseline: campaigns, outages, press coverage, or known data problems. Google lists Search Console data anomalies on a help page; for example, a logging error affected impressions, CTR and average position from 13 May 2025 to 27 April 2026⁵.

Step 5: Make one change, and log it

Change one thing on the page. If you change the title, the layout and the form together, you cannot tell which one mattered.
Do not change the tracking. A new consent banner, tag or form tool in the same period changes what gets counted.
Log the date and time it went live. In Search Console, right-click the Performance chart on the date to add an annotation of up to 120 characters⁶. Keep your own change log too.
For search changes, confirm Google has seen it. URL Inspection shows when the page was last crawled, and you can request indexing for an important page⁷. Measure from the recrawl, not from the edit.

Step 6: Fix the measurement window, and wait for all of it

Set the window before the change, matching the baseline's length in whole weeks where possible. Google notes that some search changes take effect within days and others take several months⁴, so for search changes allow a settling period before the window starts. For visible changes such as a new layout, skip the first days after launch, when returning visitors react to novelty.

Do not read the result early and stop when it looks good. Checking repeatedly and stopping at a favourable moment inflates false positives⁸. If you must look, look, but make the decision only at the planned end.

Step 7: Compare like with like

At the end of the window, compare three things:

Before and after for the changed page.
Before and after for the control pages, over the same dates.
The same weeks last year, to see whether the season explains the movement.

The useful number is the difference between the changed page's movement and the control pages' movement. If the changed page rose 20% and the control pages rose 18%, the change is credited with about 2 points, not 20.

In Search Console, the Compare option in the date filter puts two periods side by side and adds a Difference column to the table³. You can have one comparison at a time.

Step 8: Check for outside causes

Before calling the result, run through this list for the baseline and the window:

Did search demand change? Check your key queries in Google Trends⁹.
Was there a Google ranking update? Check the Google Search Status Dashboard¹⁰.
Did you run campaigns, emails, events or PR?
Were there bank holidays, school holidays or the August or Christmas slowdown in one period but not the other?
Did anything else on the site change, such as navigation, other pages or the form?
Did the tracking or consent set-up change?
Are there known data anomalies for the period⁵?

If any of these affected one period and not the other, note it. It may be enough on its own to make the result inconclusive.

Step 9: Call the result

Call it	When
Improved	The primary metric moved in the expected direction by clearly more than normal variation, more than the control pages moved, and no outside cause explains it
Worse	The primary metric fell by clearly more than normal variation, beyond the control pages
Inconclusive	There was enough data to look, but the movement was within normal variation, matched the controls, or coincided with an outside event
Insufficient traffic	The page did not get enough visits or events in the window to say anything either way

"Inconclusive" and "insufficient traffic" are valid results, not failures. On a low-traffic B2B site they will be common, and reporting them honestly is what makes your "improved" results believable.

Step 10: Record it and decide

Improved: keep the change, and consider the same kind of change on similar pages, measured the same way.
Worse: roll back, then look for the reason before trying again.
Inconclusive or insufficient traffic: keep the change if it is sound on its merits (clearer, more accurate, more useful), but do not count it as a win or use it to justify similar changes.

Add the outcome to your change log with the baseline, the window and anything that happened along the way.

A worked example

This example is illustrative; the figures are invented to show the method, not taken from a real site.

A B2B marketer rewrites the title of a services page that appears for many queries but is rarely clicked. The plan says: primary metric CTR for the page, 8-week baseline, 2-week settling period after Google recrawls the page, 8-week window, two similar service pages as controls.

Baseline: 6,000 impressions, 90 clicks, CTR 1.5%.
Window: 5,400 impressions, 108 clicks, CTR 2.0%.
Control pages, same dates: CTR rose from 1.6% to 1.9%.

Clicks rose by 18, but the rough normal variation for 90 clicks is about ±19, so clicks alone show nothing. CTR rose by half a percentage point, but the unchanged pages rose by 0.3 points over the same weeks, so most of the movement was shared. The honest result is inconclusive. The new title is clearer and more accurate, so the marketer keeps it, records the outcome as inconclusive, and does not claim a win.

That is a good result. A less careful report would have said "CTR up 33%".

Common mistakes

Choosing the comparison period after seeing the data.
Comparing a period that includes a holiday with one that does not.
Reporting relative changes on tiny numbers ("enquiries doubled", from 3 to 6).
Crediting a change with a recovery that the control pages also showed.
Changing the tracking or consent set-up during the measurement.
Stopping the measurement at the first good-looking week.

For finding changes worth making in the first place, see Finding quick wins in Google Search Console. If your traffic is too low for A/B tests, see CRO for low-traffic B2B websites.

Frequently asked questions

How do you measure the impact of a website change?

Decide the expected effect and one primary metric, record a baseline of several whole weeks, make one change and log the date, wait for a window fixed in advance, then compare the changed page with similar unchanged pages and with the same period last year. Call the result improved, worse, inconclusive or insufficient traffic.

Is a before and after comparison reliable?

On its own, not very, because anything else that changed in the same period is counted as your effect. It becomes more reliable with control pages, a year-over-year check, a window fixed in advance and a check for outside events.

How much traffic do I need to measure a website change?

It depends on how often the outcome happens and how big the effect is. As a rough guide, a count of n varies by about two times its square root between periods, so a page with 25 enquiries in the window needs a very large change before it shows. Use a more frequent metric where you can.

What should I do with an inconclusive result?

Record it honestly. Keep the change if it is sound on its merits, but do not report it as a win or use it to justify rolling out similar changes elsewhere.

Sources

Know what your website changes actually did.

Artiweave proposes changes to your existing B2B website, publishes only what your team approves, with automatic rollback, and reports each outcome honestly, including "inconclusive" and "insufficient traffic". It is in private beta.

Join the waitlist