AI SEO Tool Case Study: Choosing the Right Software

TL;DR: Most trials of the best generative engine optimization tool prove nothing, because the team changes several things at once over thirty days and then attributes whatever happened to the software. A usable trial needs a baseline measured before the tool arrives, a holdout group that does not receive the treatment, decision criteria written down in advance, and an honest account of what else changed. That structure takes an extra day to set up and is the difference between a decision and a guess.

Key Takeaways

• Set the decision criteria before you start. Write down what result would make you buy, and what result would make you walk away. Criteria chosen after seeing the data are not criteria.

• Thirty days is too short for citation outcomes. Retrieval-grounded changes move in days, but citation rates need 45 to 90 days. Use a trial to test workflow fit, then measure outcomes over a quarter.

• Keep a holdout. A set of prompts or pages you deliberately do not touch is the only cheap way to tell the tool's effect from everything else happening that month.

• Name the confounds. Seasonality, a product launch, a competitor's campaign and an engine update can all move your numbers inside a trial window. List them in the write-up.

• A percentage with no denominator is not a result. Any figure you publish needs a sample, a period, a baseline and a method, or a reader who checks will discount everything else you say.

• Trial fit, not outcomes. In thirty days you can reliably learn whether the tool fits your workflow and produces actionable output. That is a legitimate and sufficient reason to buy.

Anchor targets assume your CMS generates slug IDs from heading text, which WordPress, Webflow and most headless setups do by default. Confirm on staging and adjust the hrefs if your platform slugifies differently.

Why most tool trials prove nothing

A typical GEO tool trial runs for thirty days, during which the team also rewrites pages, ships schema, starts outreach and publishes new content. At the end, citation rate has moved and nobody can say which change caused it. The trial produced a feeling rather than a finding, and the renewal decision gets made on enthusiasm.

Related reading: AI SEO Tool Reviews: Best Platforms for 2026

The four structural problems

Everything changes at once. Tool adoption arrives bundled with new attention, new processes and new work. The software is one of six variables, and it is the one that gets credited.

The window is shorter than the effect. Crawler fixes propagate within days, but citation rates typically move between days 45 and 90. A thirty-day trial ends before the outcome it is supposed to measure.

There is no counterfactual. Without a holdout, you cannot know what would have happened anyway. Seasonality, a competitor's campaign or an engine update can move your numbers independently of anything you did.

Criteria get chosen after the data. The team looks at the result and decides whether it is good. Any result can be made to look good this way, which is why the decision was effectively made before the trial started.

What a trial can and cannot establish

The first four are legitimate reasons to buy. The honest framing is that a trial tests fit and data quality, and the outcome question gets answered over the following quarter by the measurement system the tool provides.

Why this matters beyond your own decision

The same rigour applies to the results you publish. A case study claiming a large percentage increase with no sample, period, baseline or method is the kind of claim a sceptical reader checks and discounts. In a category where buyers are already wary of inflated numbers, a modest result documented properly is worth considerably more than a dramatic one that cannot be examined.

Framework 3 covers what a defensible write-up contains. If you intend to publish your results, read that before you start the trial, because two of the six required elements have to be captured before the tool arrives.

Framework 1: The Trial Protocol

The Trial Protocol is a six-part structure that turns a tool trial into an interpretable result. Each part takes minutes to set up and the whole thing costs about a day before the trial starts. Skipping it is why most evaluations end in a debate about impressions rather than a decision.

The six parts

1. A baseline measured before the tool arrives. Two weeks of manual sampling: twenty category prompts, five runs each, across three engines. This is the only period in which you know nothing has changed, and it cannot be reconstructed later.

2. Written decision criteria. Before the trial, state what would make you buy and what would make you decline. For example: the tool must match manual spot checks on 18 of 20 prompts, surface at least five actionable findings we did not already know, and fit into the weekly triage without adding a second meeting.

3. A holdout. Choose thirty prompts or ten pages that receive no treatment during the trial. They are your comparison group. Without them, every external factor gets credited to the tool.

4. One variable at a time, where possible. If you must do other work during the trial, document what and when. A trial that coincides with a site migration measures the migration.

5. A fixed measurement method. Same prompts, same number of runs, same engines, same days of the week. Changing any of these mid-trial starts a new baseline rather than extending the old one.

6. A pre-committed review date. Set the decision meeting before you start. Trials that drift end in automatic renewal, which is the vendor's preferred outcome and rarely yours.

The protocol as a table

Under four hours of setup, and it is the difference between a decision you can defend and a renewal you cannot explain.

On trial length

Thirty days tests fit. Ninety days tests outcomes. If a vendor offers only a fourteen-day trial, use it purely to test data accuracy and workflow fit, and treat the outcome question as unanswered going into the purchase. That is an acceptable position as long as you state it rather than pretending the trial proved more than it did.

The criterion most teams forget

Add one explicit line: what would make us decline. A trial with no failure condition is a procurement formality. Writing the condition down in advance is uncomfortable precisely because it makes the decision real, which is the point.

Framework 2: The Attribution Firewall

The Attribution Firewall is a method for separating what the tool caused from what merely happened at the same time. It exists because every trial window contains events you did not control, and the default behaviour is to credit the software for all of them. The firewall is three techniques and a disclosure list.

The four confound categories

The three techniques

Holdout prompts. Thirty tracked prompts that receive no treatment during the trial. If your treated prompts improve 12 points and your holdout improves 9, the tool's contribution is closer to 3 than 12. This is the single most informative thing you can do and it costs nothing.

Paired pages. When testing content changes, match each treated page with an untouched page of similar age, type and traffic. Compare the pair rather than the treated page against its own history.

Pre-registration. Write down your hypothesis and expected effect size before you look at any data. A result that matches a prediction made in advance is far stronger evidence than the same result explained afterwards, and the discipline exposes how often the explanation was retrofitted.

The disclosure list

Keep a running log during the trial of everything that could have affected the numbers. At the end, publish it alongside the result. A short list looks like this:

That table is not an admission of weakness. It is what makes the result credible, and a reader who sees it trusts the headline number considerably more than one presented without context.

The honest conclusion format

Most trials end with a mixed picture, and the useful write-up says so. Something like: citation rate on treated prompts rose 14 points against 6 on the holdout, suggesting a tool contribution of roughly 8 points, though a crawler fix in week one likely accounts for much of that and would have happened without the tool once we noticed it.

That sentence is more useful than any percentage, because it tells you what to expect next quarter when there is no crawler fix available.

Framework 3: The Case Study Standard

The Case Study Standard is the six elements a published result must contain to survive a sceptical reader. It applies to your own marketing as much as to the vendor claims you evaluate. A figure missing any of the six is a number rather than a finding, and in this category readers have learned to discount those automatically.

The six required elements

What this does to a typical claim

Take a claim of the form "traffic increased 312%". Run it through the standard and the questions are immediate. Traffic from what source? Over what period? From what starting number, since 312% from a very low base is a different story from 312% from a substantial one? Measured how? What else was happening? Across how many sites, or is this one case?

A claim that cannot answer those is unusable in two directions. Your readers discount it, and you cannot learn anything from it internally, because you do not know the conditions under which it would reproduce.

The same standard applied well

Here is the same shape of claim meeting the standard, written as an illustrative example:

Across one B2B SaaS site, cited prompt rate on 220 category prompts rose from 19% to 31% between January 15 and April 15, 2026, measured at five runs per prompt across five engines with the method held constant. A holdout of 30 untreated prompts rose from 18% to 22% over the same period, suggesting a treatment effect of roughly 8 points. A CDN rule blocking two AI crawlers was fixed in week one and likely accounts for a majority of the gain.

That paragraph is less dramatic than 312% and considerably more persuasive to anyone making a purchase decision. It also tells the reader what to expect, which is the thing a case study is actually for.

When you genuinely cannot meet the standard

Say so, and label the example as illustrative. A clearly marked hypothetical that demonstrates a method is honest and useful. A hypothetical presented as a customer result is neither, and the distinction is the entire difference between teaching and misleading.

Using the standard on vendors

When a vendor presents a case study, ask for the six elements. The useful signal is not whether they have all six, since few will. It is how they respond to the question. A vendor who says the baseline was low and the effect was partly a crawler fix is giving you information you can act on. A vendor who repeats the headline figure louder is telling you the number has no structure underneath it.

A worked evaluation, start to finish

Everything in this section is a constructed example. No company named here exists and no figure is a measured result. It demonstrates the method and what a completed evaluation looks like, which is the only thing a hypothetical can legitimately do.

The setup

A mid-sized B2B software company, 140 employees, average contract value around $26,000. Their SEO is healthy and their category is competitive. They suspect they are losing recommendation prompts to two rivals and want to evaluate a citation tracking platform.

Before the trial

During the trial

The disclosure log, kept weekly:

The result against the criteria

The write-up they produced

Over a 35-day evaluation, cited prompt rate on 20 treated category prompts rose from 19% to 28%, measured at five runs per prompt across three engines. A 30-prompt holdout rose from 18% to 23% over the same window, suggesting a treatment effect of roughly 4 points rather than 9. A WAF rule blocking PerplexityBot was identified in week one and fixed on day 4, which probably accounts for most of the treated gain. The tool met all three pre-set decision criteria. We are purchasing, primarily because the crawler finding alone justified the annual cost and because the weekly workflow fits.

Why this write-up is useful

It names a smaller effect than the headline number suggests, credits the holdout, discloses the confound, and states the real reason for the purchase. The crawler finding is the honest justification, and it is a better argument than a percentage, because it is specific and checkable.

It also tells the team what to expect next quarter. There is no second WAF rule to find, so the gains from here will come from slower work, and nobody will be surprised when the second quarter looks less dramatic than the first.

What they deliberately did not claim

No revenue figure, because the sales cycle is longer than the trial. No traffic claim, because AI referral attribution is unreliable. No statement that the tool caused the full nine-point move. Each omission is a place where a less careful write-up would have produced a number that could not survive being checked.

Step by step: running the evaluation

Nine steps across roughly seven weeks: two weeks of baseline before the trial, a 30-day trial, and a week to decide and write up. The first three steps happen before you have access to any software.

Step 1: Write the decision criteria. Three to five specific, checkable conditions, including at least one that would make you decline. Circulate them to whoever will be in the decision meeting so nobody can quietly revise them later.

Step 2: Measure the baseline manually. Twenty category prompts, five runs each, three engines, over two weeks. Log every answer and every cited URL. This is the only clean period you will have.

Step 3: Define the holdout. Thirty prompts that get no treatment. Write them down separately so nobody accidentally optimises for them during the trial.

Step 4: Start the trial and verify data accuracy first. In week one, spot-check twenty prompts manually against what the tool reports. Independent testers have found meaningful undercounting in some products, so this check matters and it is cheap. If the data does not match, nothing else about the evaluation is meaningful.

Step 5: Keep the disclosure log weekly. Five minutes. Every change you shipped, every competitor move you noticed, every external event. Memory will not reconstruct this at the end.

Step 6: Act on findings, but log what you acted on. The point of a trial is to test whether the tool produces actionable output, so act. Just record each action and its date so you can separate tool effect from action effect afterwards.

Step 7: Hold the method constant. Same prompts, same runs, same engines. Resist the temptation to add prompts mid-trial, which is the most common way a baseline gets destroyed.

Step 8: Score against the criteria on the pre-set date. Go through the written criteria one at a time, with the holdout comparison in front of you. Decide. If the criteria were met, buy; if not, decline, even if the experience was pleasant.

Step 9: Write it up to the Case Study Standard. Sample, period, baseline, method, confounds, denominator. Keep it internal or publish it, but write it either way, because it is what makes the next evaluation faster and the renewal decision defensible.

The step people skip

Step 4. Teams assume vendor data is accurate because it is presented in a dashboard, and dashboards are persuasive. Twenty manual spot-checks take an hour and occasionally reveal that the product is measuring something other than what you think.

If you only have two weeks

Run steps 1, 4 and 8. Criteria, data accuracy check, decision against criteria. You will learn whether the tool is trustworthy and whether it fits, and you will not learn anything about outcomes. State that limitation in the decision rather than papering over it.

What to measure, and what will mislead you

During a trial, measure four things and treat everything else as context. The shortlist is deliberately narrow because trial periods generate a lot of data and most of it moves for reasons unrelated to the software.

The four trial metrics

The first three are reliable in thirty days. The fourth is directional at best, because the window is shorter than the effect, and it should be reported with that caveat attached.

The numbers that will mislead you

The second row catches more teams than the others. A visibility score has no pre-trial value by definition, so it can only be compared against itself over 30 days, which is the weakest possible evidence and the most confidently presented.

On data accuracy specifically

Independent testing has found that some AI visibility products undercount brand mentions compared with manual checking, which is why step 4 of the implementation sequence exists. Run your twenty spot checks and record the match rate as a number. If a product reports you in 11 of 20 prompts and manual checking finds you in 16, that is not a small discrepancy, and every downstream figure inherits it.

What to report at the decision meeting

One page. The four trial metrics, the criteria you set in advance with met or not met beside each, the disclosure log, and a recommendation with its reasoning. Anyone in the room should be able to disagree with the recommendation using the same page, which is the test of whether the write-up is honest.

Seven ways trials produce misleading results

Each of these turns a trial into a formality. They are listed in roughly the order they occur during an evaluation.

1. No baseline. The trial starts the day the tool does, so the first measurement is already contaminated. Fix: two weeks of manual baseline before access begins. This cannot be reconstructed afterwards.

2. Criteria written after the data. The team sees the result and decides whether it is good, which means any outcome justifies the purchase. Fix: write three to five criteria in advance, including one failure condition, and circulate them.

3. No holdout. Every external change gets credited to the tool. Since trials tend to coincide with periods of focused effort, the tool gets credit for the effort. Fix: thirty untreated prompts, named before you start.

4. Trusting the dashboard. Vendor data is presented confidently and is sometimes wrong, and independent testing has found meaningful undercounting in parts of this category. Fix: manual spot-check twenty prompts in week one and record the match rate.

5. Changing the method mid-trial. Adding prompts, adding an engine or changing run frequency restarts the baseline without anyone noticing. Fix: freeze the method for the trial and note any change as the start of a new measurement period.

6. Measuring outcomes the window cannot contain. Citation rates move over 45 to 90 days and pipeline over a sales cycle. A thirty-day trial reporting either is reporting noise. Fix: measure fit and data quality in the trial, outcomes over the following quarter.

7. Publishing the headline without the structure. A percentage with no sample, period or baseline reads as marketing and gets discounted, including by the people you most want to persuade. Fix: the six elements of the Case Study Standard, every time.

The pattern

Six of the seven are decisions about structure made before the trial starts. That is the argument for spending a day on the protocol: by the time the trial is running, most of the ways it could mislead you have already been locked in, and no amount of careful analysis at the end recovers a missing baseline or an absent holdout.

Trial timeline and evaluation checklist

Seven weeks end to end. Two weeks of baseline before any tool access, thirty days of trial, and a week to decide and document. The first phase is the one teams skip and the one that determines whether anything that follows is interpretable.

Weeks 1 to 2: before the tool

Weeks 3 to 6: the trial

Week 7: decide and document

Honest expectation setting: a well-run trial usually produces a smaller measured effect than the raw numbers suggest, because the holdout absorbs part of it. That is the protocol working. A trial that shows a dramatic unambiguous effect over thirty days is more likely to have a measurement problem than a remarkable tool.

Evaluation checklist

• Decision criteria written before the trial, including a failure condition

• Criteria circulated to everyone who will be in the decision meeting

• Two-week manual baseline completed before any tool access

• Baseline method documented: prompts, runs, engines, days

• 30-prompt holdout defined and ring-fenced from treatment

• Data accuracy spot-checked on 20 prompts in week one, match rate recorded

• Disclosure log started in week one and maintained weekly

• Every action taken during the trial logged with its date

• Method held constant: no prompts, engines or run counts changed mid-trial

• Final measurement run identically on treated and holdout sets

• Treated versus holdout delta calculated, not just the treated change

• Decision made on the pre-committed date against the written criteria

• Write-up contains sample, period, baseline, method, confounds and denominator

• Any hypothetical example clearly labelled as illustrative

• No revenue or pipeline claim made from a 30-day window

• Reasoning recorded so the renewal decision can be revisited in a year

FAQs

How long should a GEO tool trial run?

Thirty days tests workflow fit and data accuracy, which are legitimate reasons to buy. Citation outcomes need 45 to 90 days, so they land outside any standard trial window. Use the trial for fit, then measure outcomes over the following quarter.

What should I measure during a trial?

Four things: whether the tool's data matches your manual spot checks, how many actionable findings it surfaces that you did not already know, how much weekly process overhead it adds, and the change in treated prompts compared against an untreated holdout.

Why do I need a holdout group?

Because trials coincide with periods of focused effort, competitor activity and engine updates. Without a set of prompts you deliberately leave alone, every external change gets credited to the software. The holdout is the cheapest way to estimate the tool's actual contribution.

Can I trust the numbers a GEO platform reports?

Verify them. Independent testing has found meaningful undercounting of brand mentions in parts of this category. Spot-check twenty prompts manually in week one and record the match rate. Every downstream figure inherits any error in that measurement.

What makes a case study credible?

Six elements: sample, period, baseline, method, disclosed confounds and a stated denominator. A percentage without those cannot be evaluated or reproduced. A modest result documented properly persuades a buying committee more than a dramatic number that cannot be checked.

How do I separate the tool's effect from everything else?

Use a holdout, pair treated pages with untouched comparables, and write your hypothesis down before looking at data. Then keep a weekly log of everything that changed, and publish it alongside your result rather than leaving it out.

Should I publish my own results?

Yes, if you can meet the Case Study Standard. A documented result with a modest effect and disclosed confounds is a genuine differentiator in a category full of unverifiable claims. If you cannot meet the standard, label the example as illustrative instead.

What if the trial result is ambiguous?

That is the normal outcome and it is still decidable. Fall back to the criteria you wrote in advance: data accuracy, actionable findings and process fit. Those three are reliably answerable in thirty days, and they are sufficient grounds for a decision either way.

Evaluating the best generative engine optimization tool for your team

The protocol in this article costs about a day of setup and converts a vendor evaluation from an impression into a decision. Write the criteria, measure a baseline before anything changes, hold back a group of prompts, log what else happened, and decide on a date you set in advance. None of it is difficult. It is simply the part that gets skipped when a trial starts with a demo and some enthusiasm.

The same discipline applies to the results you publish afterwards. In a category where buyers have learned to discount dramatic percentages, a documented modest result is a competitive advantage, and it is available to anyone willing to keep a baseline and a holdout.

When not to run a trial at all

• You have not diagnosed the problem. If you do not know whether you are invisible, uncited or unrecommended, a trial will not tell you. Run twenty prompts manually first and find out.

• You have no capacity to act on findings. A trial that surfaces fifteen fixes nobody will make proves only that the tool works. Fix the capacity question before the tooling question.

• The purchase is already decided. If budget is allocated and the decision is made, a trial is theatre. Skip it, buy the tool, and spend the time on a proper baseline so you can evaluate the renewal honestly in a year.

• Your crawler access is broken. Fix that first. It is usually a one-line change and it will dominate any trial result, making the tool look better than it is.

Start with the baseline

If you want a measured starting point before any trial begins, run a free audit with Blazly GEO. It reports citation rates across ChatGPT, Gemini, Perplexity, Claude and Grok, shows which URLs are cited instead of yours, and flags crawler rules blocking retrieval. Run it alongside your own manual spot checks rather than instead of them, since comparing the two is exactly the data accuracy check step four calls for.

Summary

A trial that changes several things at once over thirty days and reports whatever moved is not evidence. Set decision criteria including a failure condition before you start, measure a baseline while nothing has changed, keep a holdout so external events do not get credited to the software, verify the vendor's data against your own manual checks, and log every confound as it happens. Then write the result up with its sample, period, baseline, method, confounds and denominator, and label any hypothetical clearly as illustrative. The result will be less dramatic than the alternative and it will be true, which is worth more to the people reading it.