The short answer
"Inconclusive" means the data cannot tell you whether the change helped, hurt or did nothing. It is not a failure and not proof of "no effect"; it is an honest statement of what the evidence supports. Check the test ran properly, decide whether to keep the change on its merits, learn what you can, and report it with the range of plausible effects rather than a single number.
What "inconclusive" actually means
Every website change should end with one of four verdicts, judged against a plan written before the change went live: improved, worse, inconclusive, or insufficient traffic. Our guide to measuring website changes honestly sets out all four. Two of them are easy to confuse:
| Verdict | What happened | What it tells you |
|---|---|---|
| Inconclusive | There was enough data to look, but the movement was within normal variation, matched the control pages, or coincided with an outside event | The range of plausible effects still includes a gain and a loss, which on low traffic can both be large |
| Insufficient traffic | The page did not get enough visits or conversions in the window to say anything either way | Almost nothing about the effect; the test could not have detected it |
The key idea is that "no significant difference" is not the same as "no difference". Statisticians Douglas Altman and Martin Bland made the point in the BMJ in 1995: calling such a result "negative" wrongly implies the study showed there is no difference, when usually all that has been shown is an absence of evidence of a difference2. The same applies to a landing page.
A worked example
Your demo request page gets 3,000 visitors a month and converted 60 of them (2.0%) in the month before you rewrote the headline. In the month after, it converted 66 (2.2%), an apparent 10% improvement.
A standard 95% confidence interval for the difference between the two rates runs from about 0.5 percentage points lower to about 0.9 points higher: in relative terms, from roughly 26% worse to 46% better. We calculated this with the usual normal approximation for two proportions. The data is consistent with the new headline helping a lot, doing nothing, or hurting. That is inconclusive, and the honest report says so.
Even 72 conversions (a 20% improvement) would not settle it: the interval would still run from about 17% worse to 57% better. To detect a 20% relative improvement on a 2% baseline reliably, at 5% significance and 80% power, you need about 21,000 visitors per variant3. Our guide to CRO for low-traffic B2B websites works through those numbers in detail.
Why inconclusive is a valid result
- Most changes do not move the metric much. Ron Kohavi and colleagues reported that only one third of ideas tested at Microsoft improved the metrics they were designed to improve, and that in less well understood domains the figures are worse1. If your log shows nothing but wins, it is more likely that the method is generous than that your team is unusually good.
- It protects the record. A log where every "improved" was earned is worth far more than one padded with lucky noise. Strategy built on noise fails quietly, months later.
- It is what the statistics support. The American Statistical Association's 2016 statement on p-values says that decisions should not be based only on whether a p-value passes a threshold, that proper inference requires full reporting and transparency, and that statistical significance does not measure the size or importance of an effect5. Reporting an inconclusive result with its range does exactly that.
- It still narrows things down. If the interval rules out a large loss, you know the change did not do serious harm. If it rules out a large gain, you know this lever is not the big one.
- It builds trust. Stakeholders who see "inconclusive" reported plainly will believe your "improved" when it comes.
Before you call it: five checks
Some inconclusive results are really broken tests. Rule these out first:
- Was the change live for the whole window? Check the publish date, caching, and that nothing rolled it back. Check it on mobile as well as desktop.
- Was the measurement working? Confirm the conversion event fired throughout both periods. A form plugin update or consent change can silently stop tracking.
- Did enough people see the change? A change below the fold, or on one step of a journey few people reach, is diluted by everyone who never saw it.
- Did the traffic split as planned? In an A/B test, if one variant received noticeably more visitors than intended, something is wrong with the assignment, and the result should not be trusted either way.
- Did anything else change? Campaigns, price changes, a search algorithm update, a trade show or a holiday can all move the numbers. Note them; if a big one overlaps the window, the verdict is inconclusive regardless of the numbers.
If a check fails, the result is not inconclusive, it is invalid. Fix the problem and run the measurement again.
What to do next
There is no single right response. Choose based on what the change was and what the range of effects rules out.
| Situation | Sensible next step |
|---|---|
| The change is better on its merits: clearer copy, a fixed bug, an accessibility improvement, a shorter form | Keep it. Record it as inconclusive, not as a win |
| The change adds cost or complexity: a new tool, a heavier page, more maintenance | Roll it back. Without evidence of benefit, the cost is not justified |
| The range includes a meaningful loss and the change was a matter of taste | Revert to the original; it has the longer track record |
| The idea came from strong research but the change was subtle | Make a bolder version of the same idea, which is easier to detect |
| Similar pages share the same template or element | Test the change across all of them at once to pool the traffic |
| The main conversion is too rare to read | Add a closer supporting metric, such as form starts, alongside it next time, and treat it as supporting evidence only |
| You do not know why visitors behaved as they did | Run a handful of usability sessions or ask sales what prospects say about the page |
What not to do
- Do not extend the test until it turns significant. Repeatedly checking and stopping at the first significant result inflates false positives sharply: in Evan Miller's worst-case illustration, checking after every observation produced a false winner 26.1% of the time instead of the intended 5%4. Extending is fine only if you decide the new end date in advance and accept whatever it shows.
- Do not go hunting through segments for a winner. Slice the data by device, source, country and new versus returning, and something will look significant by chance. A segment result is a hypothesis for the next test, not a finding.
- Do not switch the metric after the fact. If you planned to judge demo requests, do not report the change as a win because time on page rose.
- Do not call it a loss. "Inconclusive" does not mean the idea was bad. It means this measurement could not tell.
- Do not rerun the identical test and hope. Unless something about the first run was broken, the same test on the same traffic will most likely give the same answer.
How to report it honestly to stakeholders
Most of the pressure to dress up results comes from the reporting, not the analysis. A consistent format takes the awkwardness out of it.
A five-line format
- What we changed and why: one sentence, including the evidence behind the idea.
- What we measured: the primary metric, the period and the comparison.
- What we saw: the numbers before and after, and the plausible range of the effect.
- Verdict: improved, worse, inconclusive or insufficient traffic, with any outside events noted.
- Decision and next step: keep, revert or iterate, and why.
An example
Demo page headline, October
Change: rewrote the headline around the buyer's problem, after three sales calls where prospects said the old one was unclear.
Measured: demo requests from the page, 28 days before against 28 days after, weekday-aligned.
Seen: 60 to 66 requests on about 3,000 visits each period. The plausible effect ranges from about 26% worse to 46% better.
Verdict: inconclusive. No campaigns or price changes in the window.
Decision: keep the new headline, which is clearer on its merits and shows no sign of serious harm. Next, test a bolder change to the offer itself across all three demo pages.
Words to use, and words to avoid
| Say | Avoid |
|---|---|
| "The data cannot distinguish a gain from a loss" | "It made no difference" |
| "Requests rose from 60 to 66, within normal variation" | "Requests up 10%" |
| "We are keeping it because it is clearer, not because it is proven" | "The new version won" |
| "Mobile looked stronger; we will test that directly" | "It worked on mobile" |
| "Inconclusive" | "Failed test" |
Two habits help further. Agree the verdict categories with stakeholders before the first result, so "inconclusive" is an expected outcome rather than an excuse. And report results in a running log rather than one at a time, so a quarter's worth of honest verdicts shows which kinds of change tend to help on your site. For the full step-by-step method, see how to measure the impact of a website change.
Where Artiweave fits
Artiweave proposes specific website changes from evidence, and your team approves, edits or rejects each one before anything is published, with automatic rollback. Each outcome is reported as improved, worse, inconclusive or insufficient traffic, with the numbers behind it, so inconclusive results are recorded as results rather than hidden. It is in private beta.
Frequently asked questions
What does an inconclusive test result mean?
It means the data cannot distinguish between the change helping, hurting or doing nothing. There was enough data to look, but the movement was within normal variation, matched the control pages, or coincided with an outside event.
Does an inconclusive result mean the change had no effect?
No. Absence of evidence of a difference is not evidence of no difference. A small real effect, positive or negative, may be hidden by normal variation. The confidence interval shows how large the effect could plausibly be.
Should I keep a change if the test was inconclusive?
Keep it if it is better on its merits, such as clearer copy, a fixed bug or a shorter form, and the range does not include a serious loss. Roll it back if it adds cost or complexity without evidence of benefit.
Can I run the test for longer to get a result?
Only if you set the new end date in advance and accept whatever it shows. Extending a test until it happens to turn significant inflates the chance of a false winner.
How is inconclusive different from insufficient traffic?
Inconclusive means there was enough data to look but no clear movement. Insufficient traffic means the page did not get enough visits or conversions in the window for the test to detect anything at all.
Sources
- Kohavi, Deng, Frasca, Longbotham, Walker and Xu: Trustworthy Online Controlled Experiments: Five Puzzling Outcomes Explained (KDD 2012), introduction
- Altman and Bland: Absence of evidence is not evidence of absence, BMJ 1995;311:485 (19 August 1995)
- Evan Miller: Sample Size Calculator
- Evan Miller: How Not To Run an A/B Test (April 2010)
- American Statistical Association: Statement on Statistical Significance and P-Values (7 March 2016)
Report what your website changes actually did.
Artiweave publishes only the changes your team approves, with automatic rollback, and reports every outcome honestly, including "inconclusive" and "insufficient traffic". It is in private beta.