Before-and-after SEO testing with Search Console data

Why a plain before-and-after comparison misleads

Search traffic moves for many reasons that have nothing to do with your change. Google's guide to debugging traffic drops lists several of them: algorithmic updates, technical issues, security issues, site moves, and changes in search interest, including seasonality throughout the year². If you rewrite ten service page titles in early December, clicks will probably fall in the following weeks. That probably tells you more about the Christmas holidays than about your titles.

A before-and-after test can never be as clean as a randomised experiment. What it can do is make the most likely alternative explanations visible, and the most useful tool for that is a set of control pages. For the general method that applies to any website change, see our guide to measuring the impact of a website change. This page covers what is specific to search.

Step 1: choose the changed pages and the control pages

A control page is a page you deliberately leave unchanged for the whole test. It is exposed to the same seasonality, the same Google updates and the same industry news as the changed pages, so its movement is your best guess at what would have happened to the changed pages anyway.

What makes a good control group

• Same page type. Compare service pages with service pages and blog posts with blog posts. Different page types answer different kinds of query and react differently to updates.
• Similar history. Over the past few months, the control pages' clicks should have risen and fallen roughly in step with the pages you plan to change. Plot both groups' weekly clicks side by side; if the lines already diverge before the change, the comparison will not be fair.
• Enough traffic. A control group of three pages with five clicks a week each is mostly noise. Pool several pages so that each group has a steady weekly total.
• Truly untouched. No edits, redirects, new internal links or template changes to the control pages during the test. Agree this with whoever else edits the site.
• Not competing with the changed pages. If a changed page starts ranking for queries a control page used to win, the control falls because of your change, and the comparison overstates the effect. Avoid controls that target the same queries.

Decide the groups by a rule you write down, for example "all service pages; the five with odd-numbered IDs change, the rest are controls", rather than hand-picking the pages you expect to improve.

Building the groups in Search Console

In the Performance report you can filter by multiple pages, and regex filters let you include or exclude several URLs at once with the | operator; matching is partial and case-insensitive by default⁶. Save one filter for the changed group and one for the control group, and export each group's daily clicks for the full test period. Two details matter when you read the numbers: the report labels daily data in Pacific Time, and the table shows at most 1,000 rows, while rare anonymised queries are left out of the table but counted in the chart totals⁵. Work from page-level totals, not from summed query rows.

Step 2: fix the windows before you make the change

The windows are part of the plan, not something you choose after seeing the data. We suggest two reads.

Google itself says some changes take effect in a few days while others can take several months, and that you will likely want to wait a few weeks before analysing the result in Search Console². Three practical rules follow:

1. Start the clock when Google has seen the change. The URL Inspection tool shows a "Last crawl" date for each page⁸. Count the after window from the date the last changed page was recrawled, and leave out the days in between.
2. Use whole weeks. B2B search traffic is usually much lower at weekends. Google's own help suggests weekly or monthly granularity when comparing date ranges, to smooth out these effects⁶.
3. Do not move the end date. Extending a test until it shows what you hoped for turns noise into false winners. If you need longer, decide the new end date before you look again.

Step 3: account for seasonality with year-over-year data

Control pages handle most seasonality, because they share it. A year-over-year comparison is a second, independent check: it shows whether the same weeks last year moved the same way. Google's debugging guide recommends viewing the last 16 months of data to put a change in context, and comparing a period year over year to see whether a movement "happened exactly as it did the previous year"².

• Align by weekday, not by date. Compare Monday 2 November 2026 with Monday 3 November 2025, not with 2 November 2025, which was a Sunday.
• Watch moving holidays. Easter, and the UK bank holidays around it, fall in different weeks each year. A window that contains Easter in one year and not the other is not comparable without adjustment.
• Know your own calendar. Trade shows, financial year ends, product launches and big campaigns all create seasonal patterns of their own. Note last year's as well as this year's.
• Check wider interest. Google suggests checking your queries in Google Trends to see whether a drop affected only your site or the whole web².
• Mind the 16-month limit. The Performance report offers up to 16 months of data². Export the data you will need for future year-over-year checks before it ages out.

Step 4: check for Google updates

The Google Search Status Dashboard lists ranking updates and crawling, indexing and serving incidents, with the dates Google says each began and how long it ran⁴. Check it for every window in your test, before and after. These are the ranking entries it showed for 2026 when we checked on 3 November 2026:

For core updates, Google recommends confirming the update has finished rolling out, waiting at least a full week after it completes before analysing your site in Search Console, and then comparing with a week before the update started³. In practice:

• If you can, schedule the change so that neither window overlaps a core update rollout.
• If an update lands mid-test, record it. Control pages absorb some of its effect, but an update can treat page types differently, so a result that spans one is weaker evidence.
• If the changed and control groups react very differently in the days of an update, suspect the update before you credit or blame your change.

Search Console's annotations feature lets you mark dates on the Performance chart, up to 120 characters each and 200 per property, and annotations are shared with other users of the property⁷. Annotate the change date, the recrawl date and any update, so the next person to read the chart sees the context.

Step 5: check for known data problems

Google keeps a list of data anomalies in Search Console. Check it for every test. The most important recent entry is dated 3 April 2026. Google says a logging error prevented Search Console from accurately reporting impressions from 13 May 2025 until 27 April 2026, that the issue has been resolved, and that you may notice a decrease in impressions as a result. Only impressions and the related metrics, CTR and average position, were affected; clicks were not¹.

This has three practical consequences for a before-and-after test:

1. Make clicks your primary metric. They were not affected, and they are closer to what the business cares about anyway.
2. Do not compare impressions, CTR or average position across 27 April 2026. A before window inside the affected period and an after window outside it measure impressions differently.
3. Year-over-year impression comparisons stay distorted until the end of April 2027. Most periods between 13 May 2025 and 27 April 2027, compared with the same weeks a year earlier, puts an affected period against an unaffected one. Compare clicks year over year instead.

The same page lists smaller, shorter anomalies, for example two days missing from bulk data exports for some properties (28 February and 1 March 2026), which Google says will not be recovered¹. If you work from exported data, check whether either day falls inside your windows.

Step 6: read the result with difference-in-differences

Difference-in-differences sounds technical, but the idea is simple. You take two differences and subtract one from the other:

1. How much did the changed pages move, from before to after?
2. How much did the control pages move over the same period?
3. The difference between those two movements is your estimate of the change's effect.

The control pages' movement stands in for "what would have happened anyway". It works only if the two groups were moving roughly in step before the change, which is why Step 1 asks you to check their history.

A worked example (illustrative)

A plain before-and-after reading says the rewrite added 15%. But the controls rose 8% without any change, perhaps because of seasonal demand or a ranking update. If the changed pages had followed the controls, they would have reached about 1,296 clicks (1,200 multiplied by 1.08). They reached 1,380, so the estimated effect is about 84 extra clicks over the 8 weeks, or roughly 6.5% above expectation. Put simply: 15% minus 8% is about 7 percentage points, less than half the headline figure.

Even that estimate is uncertain. With pages this small, a 6 to 7% gap can easily come from normal week-to-week variation. Before calling it, compare the gap with how far apart the two groups drifted in comparable periods before the change. If gaps of that size were common, the honest verdict is inconclusive. Our guide to inconclusive website test results explains what to do next and how to report it.

Verdicts to agree in advance

SEO split testing, and why it needs large page sets

SEO split testing is the more rigorous cousin of the before-and-after test. Because Googlebot is a single crawler and sees only one version of a page, you cannot split visitors the way an ordinary A/B test does. Instead, you split pages. SearchPilot, a vendor of SEO split testing software, describes the method this way: pages that share a template are assigned to control or variant groups, only the variant pages receive the change, and a model built on historical traffic (usually about 100 days) forecasts what each group would have received without the change. Actual traffic is then compared with that forecast and with the control group, and significance is judged at the 95% level⁹. SearchPilot says it moved from the open source Causal Impact package to a neural network-based approach in 2019, and that today its model builds a forecast of how the variant pages would have performed if left alone, using the control pages' data as training data, and is trained on many random shuffles of the data to reduce noise from individual pages¹⁰.

The catch is scale. SearchPilot says sites need at least hundreds of pages on the same template, and that it generally works with sites that have at least 30,000 organic sessions a month to the group of pages being tested. It adds that some customers test sections with only a few thousand sessions a month, but those tests need much larger changes in traffic to reach significance⁹. The reason is the same as for any experiment: the forecast is only as precise as the volume and stability of the data behind it, and SearchPilot notes that larger page groups, with more traffic and more predictable performance, allow smaller uplifts to be detected¹⁰.

Which approach fits:

• A dedicated split-testing platform is the better fit for sites with hundreds or thousands of templated pages and substantial organic traffic: ecommerce category pages, marketplaces, large publishers, directories.
• Before-and-after tests with control pages fit most B2B marketing sites, which have tens of important pages rather than hundreds, and far less traffic per template. The evidence is weaker, so report it with more caution, but it is far better than an unadjusted comparison. For more on testing with little traffic, see CRO for low-traffic B2B websites.

A checklist for your next SEO change

1. Write the plan: changed pages, control pages, metric (clicks), windows, verdict rules.
2. Plot both groups' weekly clicks for the last few months and confirm they move in step.
3. Check the Search Status Dashboard for any update in progress, and avoid launching during one.
4. Publish the change, then confirm each page's recrawl date in URL Inspection.
5. Annotate the change and recrawl dates in Search Console.
6. At 28 days, take the first read; act only on clear harm.
7. At 8 to 12 weeks, take the final read with difference-in-differences.
8. Check the dashboard and the data anomalies page for both windows.
9. Check year-over-year clicks for the same weeks, aligned by weekday.
10. Record the verdict, with the numbers and any caveats, in your change log.

Where Artiweave fits

Artiweave reports each change it publishes with one of the four verdicts this kind of test supports, improved, worse, inconclusive or insufficient traffic, together with the numbers behind it. For the wider approach, see our guide to measuring website changes honestly.

Frequently asked questions

How long should a before-and-after SEO test run?

Take a first read 28 days after Google has recrawled the changed pages, compared with the 28 days before the change, and a final read after 8 to 12 weeks. Fix both dates before you make the change.

Should I measure clicks or impressions?

Use clicks as the primary metric. Impressions, CTR and average position are useful supporting evidence, but Google reports that a logging error affected impressions and those related metrics, though not clicks, from 13 May 2025 to 27 April 2026.

What is a control page in SEO testing?

A control page is a similar page that you deliberately leave unchanged during the test. Because it is exposed to the same seasonality and Google updates as the changed pages, its movement shows what would probably have happened to the changed pages anyway.

What if a Google update happens during my test?

Note it, and treat the result with caution. Control pages absorb some of the effect, but an update can affect page types differently. Google recommends waiting at least a full week after a core update completes before analysing a site in Search Console.

Can a small B2B site run SEO split tests?

Usually not in the statistical sense. SearchPilot says split testing needs at least hundreds of pages on the same template, and that it generally works with sites that have at least 30,000 organic sessions a month to the pages being tested. Smaller sites are better served by careful before-and-after tests with control pages.

Sources

1. Search Console Help: Data anomalies in Search Console, entries dated 3 April 2026 and 28 February to 1 March 2026 (accessed 3 November 2026)
2. Google Search Central: Debugging drops in Google Search traffic (accessed 3 November 2026)
3. Google Search Central: Google Search's core updates (accessed 3 November 2026)
4. Google Search Status Dashboard: Ranking incident history (accessed 3 November 2026)
5. Search Console Help: Performance report (Search results): Troubleshooting data discrepancies (accessed 3 November 2026)
6. Search Console Help: Performance report (Search results): Advanced filtering and comparison (accessed 3 November 2026)
7. Search Console Help: Search Console annotations (accessed 3 November 2026)
8. Search Console Help: URL Inspection tool (accessed 3 November 2026)
9. SearchPilot: What is SEO A/B testing? (first posted 23 December 2020, updated 28 October 2026; accessed 3 November 2026)
10. SearchPilot: How SearchPilot analyses test data (December 2024; accessed 3 November 2026)

Measure your SEO changes honestly.

Artiweave publishes only the changes your team approves, with automatic rollback, and reports every outcome honestly, including "inconclusive" and "insufficient traffic". It is in private beta.

Join the waitlist