TL;DR: Ask an AI assistant the same question twice and you will often get different answers, naming different brands and citing different sources. That is normal behaviour rather than a fault, and it has five distinct causes. It also means a single check tells you nothing: you need several runs per question before you have a rate, and you need to know your own variance before you can tell a real change from noise.
Key Takeaways
Five separate causes. Sampling during generation, live retrieval returning different results, personalisation and location, model and system updates, and prompt phrasing. They compound.
One check is an anecdote. Five runs per prompt is the practical minimum for a usable rate, and fewer than that will have you reporting noise as progress.
Measure your own variance first. Run one prompt twenty times and count. That range is the threshold below which nothing you observe is a result.
Some things are far more stable than others. Whether you are in the consideration set at all moves less than exactly which three brands get named in a given answer.
Hold the method constant. Same prompts, same number of runs, same assistants, same time window. Changing the method changes the number for reasons unrelated to your performance.
Variance is not a reason to skip measuring. It is a reason to measure properly, which means sample sizes and ranges rather than single numbers.
Anchor targets assume your CMS generates slug IDs from heading text. Confirm on staging and adjust the hrefs if your platform slugifies differently.
Why the same question gives different answers
Five causes, operating independently and compounding. Knowing which is which tells you what to hold constant when you measure.
1. Sampling during generation
Language models produce text by repeatedly choosing the next token from a probability distribution. Most consumer assistants do not simply take the highest-probability option every time; they sample, which is what makes output read naturally rather than mechanically.
That sampling is the first source of variation. Two runs of an identical prompt can diverge early and end up in different places, naming different examples.
This is a design choice rather than a defect. Deterministic output would be noticeably worse to read.
2. Live retrieval returning different results
When an assistant searches, it issues queries to a search backend. Those results change between runs: indexes update, ranking shifts, and the specific pages returned are not fixed.
Different retrieved documents means different source material, which means a different answer even if the model behaved identically.
This is usually the largest source of variation for the brand-recommendation questions marketers care about.
3. Personalisation and location
Assistants infer location from IP address and may use conversation history, account settings or memory. A query asked from two offices can return different local recommendations, and a query asked in a fresh session differs from one asked after related conversation.
4. Model and system updates
Providers update models, system prompts, retrieval configurations and safety behaviour continuously, often without announcement. A change in any of those can shift your results overnight for reasons entirely outside your site.
This is why a sudden drop is worth investigating before it is worth panicking about. Check whether competitors moved too; if everyone shifted, it was probably not your content.
5. Prompt phrasing
Small wording differences produce meaningfully different answers. Best project management tool and best project management software are not the same query to an assistant, and neither is the same as what should I use to manage projects.
This one is in your control: fix your phrasing and keep it fixed.
How they compound
Cause | Varies between | Can you control it? |
|---|---|---|
Generation sampling | Runs, seconds apart | No |
Retrieval results | Runs, hours apart | Indirectly, through your content |
Personalisation and location | Users and sessions | Partly, by fixing the location setting |
Model and system updates | Weeks to months | No |
Prompt phrasing | Whatever you type | Yes, fix it |
Three of five are entirely outside your control. That is the honest starting point for any measurement method in this category.
What is stable and what is not
Not everything moves equally. Some things are quite stable across runs, and those are the things worth building a programme around.
Roughly stable
What | Why it holds |
|---|---|
Whether you are in the consideration set at all | Driven by whether sources about you exist and are retrievable |
How your category is described | Reflects broad consensus across many sources |
Which two or three brands dominate a category | Market leaders appear consistently |
Whether your site is cited ever, across many runs | A function of retrievability, not chance |
How you are characterised when named | Comes from what sources say, which changes slowly |
Highly variable
What | Why it moves |
|---|---|
Exactly which brands appear in a given answer | Sampling plus retrieval variation |
The order they appear in | Largely incidental |
Which specific URLs are cited | Retrieval results differ run to run |
Whether a mid-tier brand appears in any single run | This is where most of the variance lives |
Exact wording about you | Regenerated each time |
Why this matters for strategy
If you are a market leader, your presence is relatively stable and your measurement will be too. If you are a challenger competing for the third slot in a three-name answer, you are measuring exactly the part of the system with the most variance.
That is not a reason to despair. It means a challenger needs more runs per prompt to get a reliable rate, and should expect month-to-month figures to bounce in ways a leader's do not.
The implication for what you track
Track this | Rather than this |
|---|---|
Rate across many runs | Whether you appeared today |
Trend across quarters | Month-to-month movement |
Whether cited ever, per prompt | Which URL was cited in one answer |
Share across the whole prompt set | Performance on a favourite prompt |
Aggregate across prompts and runs, and look at quarters rather than weeks. The aggregate is stable even though every individual observation is not, which is the normal relationship between a noisy measurement and a real underlying quantity.
One practical consequence
If a single prompt matters enormously to you commercially, track it with more runs than the rest. Twenty runs on one critical prompt gives you a usable figure for it, where five would not.
How much variance to expect
There is no universal figure, because it depends on your category's competitiveness and your position in it. What matters is measuring your own, which takes an hour.
The exercise
Pick one prompt that matters commercially. Run it twenty times, spaced across a day, in a fresh session each time. Record whether your brand was named and which competitors appeared.
That gives you two things: your rate on that prompt, and a sense of how wildly it moves.
What the result tells you
What you see across 20 runs | Interpretation |
|---|---|
Named in 19 or 20 | You are securely in the answer for this prompt |
Named in 12 to 17 | Present but contested. Normal for a strong challenger |
Named in 4 to 10 | On the edge of the consideration set. High variance territory |
Named in 1 to 3 | Occasional rather than present. Treat appearances as luck |
Never named | Not in the set. This is stable and it is a real finding |
The middle rows are where variance is highest, and where single checks mislead most. If you are named four times in twenty, a check on a lucky day tells you that you are winning and a check on an unlucky one tells you that you have vanished. Neither is true.
Expect competitors to move too
Record competitors in the same exercise. You will usually find that the first one or two names are consistent and the remaining slots rotate among a group of several brands.
That rotation is the real picture of a competitive category: not a fixed ranking but a pool of candidates with different probabilities of appearing. Knowing that changes how you read any single answer.
Variance across engines
Different assistants have different variance characteristics. Some are more consistent than others across runs, and the same brand can be stable in one and bouncy in another.
Measure each separately rather than blending them into one figure, because a blended number hides the engine where you have a real and fixable problem.
How often to redo this
Re-measure your variance quarterly. It changes as your position changes, and as providers update their systems. A threshold calculated a year ago is not the threshold today.
Measuring despite it
Variance does not make measurement impossible. It makes single observations useless and aggregates informative, which is an ordinary situation in measurement rather than a special problem.
The method
Decision | What to do |
|---|---|
Runs per prompt | Five minimum. More for prompts that matter commercially |
Timing | Spread across a day rather than clustered in a minute |
Session state | Fresh session each run, no conversation history |
Location | Set it explicitly and keep it fixed |
Phrasing | Exact, fixed, versioned. Never paraphrase between months |
Engines | Measured and reported separately |
Window | The same few days each month |
What to hold constant
Everything you can. Your measurement is a comparison between months, and anything that changes other than your performance pollutes that comparison.
Hold constant | Why |
|---|---|
The prompt list and its exact wording | Different words, different query |
The number of runs | Changes your sample size and precision |
The assistants | Different behaviour, not comparable |
The location setting | Changes local results |
The measurement window | Reduces timing effects |
What you cannot hold constant
Model updates, retrieval index changes and competitor activity. These will move your numbers and you will not always know when.
The mitigation is recording competitors in the same runs. If your rate dropped and theirs rose, something changed on your side. If everyone dropped, the system changed.
That single comparison turns an unexplained movement into a diagnosable one, and it costs nothing because you are already recording who appeared.
Sample size in practice
Prompts | Runs each | Total runs | Effort |
|---|---|---|---|
20 | 5 | 100 | Manageable manually, about two hours |
50 | 5 | 250 | The edge of manual |
120 | 5 | 600 | Needs tooling |
120 | 10 | 1,200 | Needs tooling, better precision |
More runs narrow your range, which lowers the threshold at which you can detect a real change. If your figures bounce too much to act on, the answer is usually more runs rather than more prompts.
The discipline that matters most
Write the method down and follow it exactly. A documented method applied consistently produces comparable numbers even when the method is imperfect. An undocumented one produces numbers nobody can defend six months later, including you.
Telling a real change from noise
This is where most AI visibility reporting goes wrong, and the fix is a single threshold you calculate once.
The threshold
Your observed run-to-run range is your noise floor. A movement smaller than that range is not evidence of anything.
If your citation rate measured across 600 runs came out at 23% with a range of 19% to 27%, then next month's 25% is inside the noise. It is not progress, and presenting it as progress will cost you credibility the first time someone looks closely.
What counts as signal
Observation | Verdict |
|---|---|
Rate moves less than your range | Noise |
Rate moves more than your range, competitors flat | Probably real |
Rate moves more than your range, everyone moved | System change, not yours |
Specific prompts flip from never to consistently | Real, and the most useful kind |
New URLs of yours start appearing in citations | Real, and traceable to specific pages |
One prompt behaves differently once | Noise |
Look at prompt level, not just the rate
The aggregate rate is the least informative view of your data. Far more useful: which specific prompts changed state this month, and does that change cluster around pages you worked on?
If you published four pages targeting six prompts and five of those six moved from never-cited to sometimes-cited, that is a result even if your headline rate barely moved. The aggregate dilutes real effects across prompts you did nothing about.
The investigation sequence
When something moves unexpectedly, in order:
Check competitors in the same data. Did everyone move, or just you?
Check whether the prompt set changed. Version drift is the most common explanation.
Check crawler access. A new block can remove you quickly.
Check for provider announcements. Model or system changes sometimes become public.
Check your own site. A migration, a redesign, a robots change.
Accept it may be unexplained. Sometimes the honest answer is that the system changed and nobody said so.
Step six is allowed. Inventing an explanation for a movement you cannot trace is worse than recording that it happened and watching whether it persists.
The reporting rule
When you report a change, state whether it exceeded your variance range. Three words in a footnote, and it is the difference between a number a colleague trusts and one they quietly discount.
Reporting this upward
Explaining variance to senior stakeholders is a real problem. Done badly it sounds like excuse-making in advance. Done well it is the thing that makes your numbers credible.
Lead with the method, once
The first time you present AI visibility measurement, spend two minutes on why the same question gives different answers and what you do about it. Not every month, once.
After that a footnote suffices, because the audience already knows the measurement is a sampled rate rather than a count.
The framing that works
Treat it as a survey, because that is what it is.
Nobody objects that a customer satisfaction survey has a margin of error. The expectation of a sample size and a confidence range is normal in that context. Presenting AI visibility the same way puts the audience in a frame they already understand, and it sets correct expectations about month-to-month movement.
What to put on the slide
Element | Example |
|---|---|
The rate | Citation rate 23% |
The sample | 120 prompts, 5 runs each, 600 runs |
The range | 19% to 27% |
The verdict | Up 2 points, inside normal variance. Not yet a result |
Prompt-level detail | 5 of 6 target prompts moved from never-cited to cited |
What shipped | 4 pages published, crawler block on resources template fixed |
That fourth row is the one that earns trust. A marketer who says this month's movement is not meaningful yet is far more believable the month they say it is.
Handling the inevitable question
Someone will ask why you cannot just report the number like a ranking. The short answer: there is no ranking to report. The assistant composes an answer each time and your brand either appears or does not, so the honest measurement is a probability rather than a position.
That is a one-sentence answer and it satisfies most people.
What not to do
Do not | Why |
|---|---|
Report a single run as a result | It is one sample of a noisy process |
Smooth the data without saying so | Hides real movement along with noise |
Model revenue from citation rate | The assumptions are not defensible |
Change the method to make a number look better | It will be noticed, and it ends the programme's credibility |
Hide months where nothing moved | Those months are normal, and reporting them is what makes the good months believable |
Semantic and keyword retrieval
One more source of apparent inconsistency worth understanding: retrieval systems commonly combine two different matching methods, and they fail in different ways.
The two methods
Method | How it matches | Strength | Weakness |
|---|---|---|---|
Keyword matching | Exact and near-exact term overlap | Precise on names, codes, specific terminology | Misses paraphrase entirely |
Semantic matching | Numeric representations of meaning, so similar concepts sit near each other | Catches paraphrase; stomach pain can match gastritis | Can drift to topically near but unhelpful results |
Many production systems run both and combine the results, usually called hybrid retrieval. That combination is one reason the same query can surface quite different documents on different runs: the balance between the two is not fixed and the candidate pool is larger than either alone.
Why this matters for your pages
It means two different things both help, and neither alone is sufficient.
For keyword matching | For semantic matching |
|---|---|
Use your actual product and category names consistently | Write naturally about the concept rather than repeating a phrase |
Include the specific terms people use, including technical ones | Cover the subject properly so related phrasings match |
Do not invent a category name nobody searches | Keep each page on one topic rather than scattering |
The practical upshot is unglamorous: write clearly about one thing, use the words people actually use for it, and do not stuff. That satisfies both methods, which is why keyword density tactics and vague conceptual writing both underperform.
What this does not mean
It does not mean you can optimise your position in a vector space. No external tool has access to a commercial system's embeddings, nothing can measure your coordinates in it, and nobody can tune them directly.
Claims about mapping your brand into the right vector coordinates describe something that cannot be done. The underlying advice, which is to write focused, clear content on your actual subject, is correct and does not require the mathematics.
The one practical test
Take a page and ask whether it answers one specific question in plain language, using the words a buyer would use, within the first 50 words.
If yes, it is well-formed for both retrieval methods. That test is worth more than any amount of theory about embedding spaces.
Refresh cycles and timing
You will see advice to time your publishing to each model's indexing cadence. It is appealing and it does not work.
What is actually knowable
Question | Can you know it? |
|---|---|
How often a provider re-crawls your site | Partly. Your server logs show when their agents visited |
How often their index updates | No. Not published, and it varies by site and page |
When a model is retrained | Roughly, from announcements, and it rarely matters for retrieval |
When your new page becomes retrievable | Only by testing it |
Your logs tell you when a crawler fetched a page. They do not tell you when that page became available to the retrieval system, and the gap between those two events is not published by anyone.
Why timing strategies fail
The cadence differs by provider, by site, and by page importance. It changes without notice. And most assistants retrieve through a search backend whose index updates continuously rather than on a schedule you could target.
Publishing on a Tuesday because a model supposedly indexes on Wednesdays is astrology with server logs.
What to do instead
Instead of | Do |
|---|---|
Timing publication to a cadence | Publish when the content is ready |
Guessing when a page is live in retrieval | Test it directly: ask the question it answers, weekly, until it appears |
Assuming slow indexing | Check your logs for crawler visits. If nobody came, the problem is access |
The direct test is the only reliable method. Pick the question your new page answers, ask it weekly, and record when your URL first appears. After a handful of pages you will have an empirical figure for your own site, which is worth more than anyone's general claim.
What that figure usually looks like
In practice, teams report anywhere from days to a couple of months between publishing and first reliable citation, varying by site authority and topic competitiveness. Treat that as a wide expectation rather than a benchmark, and measure your own.
The one timing thing that does matter
If a crawler cannot reach the page at all, none of this applies and the delay is infinite. Check server logs by user agent and status code before concluding anything about indexing speed.
That check resolves more apparent slow-indexing problems than any scheduling strategy ever will.
What you can and cannot control
After all of that, the picture of what is actually in your hands is clearer than it first appears, and shorter.
Not in your control
Factor | Why |
|---|---|
Generation sampling | A property of how the model produces text |
Which documents retrieval returns on a given run | Decided by a search backend you do not operate |
Model and system prompt updates | Provider decisions, often unannounced |
Retrieval index refresh timing | Not published, varies by site and page |
Whether an assistant searches at all for a query | Its own decision |
Personalisation for an individual user | Depends on their account and history |
In your control
Factor | How |
|---|---|
Whether crawlers can reach you | robots.txt, bot management, WAF, rendering |
Whether your content is indexed | Standard technical work |
Whether a passage answers the question directly | Writing the opener as an answer |
Whether one page covers one question | Avoiding near-duplicates that compete |
What third-party sources say about you | Profiles, directories, genuine coverage |
Whether your facts agree across sources | Entity consistency work |
Your measurement method | Prompts, runs, consistency |
The honest ratio
Six things outside your control, seven inside. The ones inside are unglamorous and they are the entire job.
Notice that none of the controllable items is a trick. There is no lever that makes a model prefer you. There is making your content retrievable, making it quotable, and making third-party sources agree about what you are.
What this means for expectations
You are influencing a probability rather than setting a position. Doing everything right raises the chance you appear; it does not guarantee any particular answer.
That is why a programme should be judged on rates across many prompts over quarters, and never on whether you appeared in a specific answer on a specific day.
The useful reframe
Stop asking why the answer changed. Start asking whether your rate across the whole set is trending up, and whether the specific prompts you worked on moved.
The first question has no useful answer. The second two do, and they are the ones that tell you whether the work is paying.
Six mistakes caused by variance
1. Checking once and drawing a conclusion. Someone asks ChatGPT about the category, sees a competitor, and declares a crisis. Or sees themselves and declares victory. Both are one sample of a noisy process, and neither is information.
2. Reporting month-to-month movement without a range. A move from 23% to 25% looks like progress on a chart and is probably noise. Without a stated range nobody can tell, including you.
3. Paraphrasing prompts between measurements. Small wording changes produce genuinely different queries. If the prompt text is not identical, you are measuring a different thing and calling it a trend.
4. Blaming your content for a system change. A provider updates something and your rate drops. Without competitor data in the same runs you will spend a month optimising pages that were never the problem.
5. Concluding that measurement is impossible. The opposite error. Variance means single observations are useless, not that aggregates are. Polling works despite individual unpredictability, and so does this.
6. Buying a tool expecting it to remove the variance. No tool can. What a good one does is run enough samples to give you a rate and a range. If a vendor's reporting shows a single number with no sample size, that is a reason for caution rather than reassurance.
The pattern
Four of six come from treating a probabilistic system as a deterministic one, which is a natural instinct coming from rank tracking where a position is a position.
The mental shift is from where do I rank to how often do I appear. Everything about correct measurement follows from that change.
The one to avoid most
Mistake four, because it wastes a quarter. Always record competitors in the same runs. It costs nothing, it is the only way to separate your performance from the system's behaviour, and almost nobody does it.
FAQs
Why does ChatGPT give different answers to the same question?
Five reasons, which compound: the model samples from a probability distribution as it writes, live retrieval returns different documents each time, location and personalisation differ, providers update models and systems continuously, and small wording changes produce different queries. It is normal behaviour, not a fault.
Does that mean AI visibility cannot be measured?
No. It means single observations are useless and aggregates are informative, which is true of any sampled measurement. Run each prompt several times, report a rate with its sample size and range, and compare across quarters rather than weeks.
How many times should I run each prompt?
Five is the practical minimum for a usable rate. Ten or more for prompts that matter commercially. More runs narrow your range, which lowers the threshold at which you can detect a real change, so if your figures bounce too much, add runs rather than prompts.
How do I know if a change is real?
Compare it against your own observed run-to-run range. If your rate varies between 19% and 27% across runs, a move to 25% is inside the noise. Also check competitors in the same data: if everyone moved, the system changed rather than your content.
Why did my visibility drop suddenly?
Check competitors first. If they dropped too, a provider changed something. If only you dropped, check crawler access, then whether your prompt set or site changed. Sometimes the honest answer is that it is unexplained, and you watch whether it persists.
Can I control which answer an assistant gives?
No. You influence the probability that you appear by being retrievable, having passages that answer questions directly, and having third-party sources agree about what you are. There is no lever that sets a particular answer.
Should I time publishing to a model's indexing schedule?
No. Refresh cadence is not published, varies by site and page, and changes without notice. Publish when content is ready, then test directly by asking the question your page answers weekly until your URL appears.
Does a paid tool remove the variance?
No tool can. What a good one does is run enough samples to produce a rate with a range. A vendor reporting a single number with no sample size has hidden the variance rather than handled it, which is a reason for caution.
Measuring a moving target
Start by finding your own variance. One prompt that matters, twenty runs across a day, fresh session each time, recording whether you were named and who appeared instead. An hour, and it gives you the threshold everything else depends on.
Then hold your method constant. Same prompts, same wording, same number of runs, same assistants, same window. Record competitors every time, because that is the only way to separate your performance from the system's behaviour.
The three rules
Never conclude from one run. It is a single sample of a probabilistic process.
Never report a rate without its range. The range is what makes the rate meaningful.
Always record competitors alongside yourself. It converts unexplained movements into diagnosable ones.
If the work is the constraint
Measuring properly usually reveals the same thing: you know which prompts you lose and cannot produce the pages to win them. That is a production problem rather than a measurement one, and Blazly SEO covers research, planning, drafting and direct publishing from about $39 a month.
It does not remove the variance, because nothing does. What it does is let you ship against the gaps faster, which is the only input you actually control.
Summary
AI assistants give different answers to the same question because they sample during generation, retrieve different documents each run, personalise by location and history, and sit on systems that change without notice. Three of those five causes are entirely outside your control. That makes single checks worthless and properly sampled rates informative, the same way a survey works despite individual unpredictability. Run each prompt at least five times, hold every other variable constant, measure your own run-to-run range, and treat any movement smaller than that range as noise. Record competitors in the same runs so you can tell a system change from your own. And judge the programme on rates across many prompts over quarters, never on whether you appeared in a particular answer on a particular day.