TL;DR: An AI search tracking system is four components: something that runs prompts on a schedule, something that detects your brand in the answer, something that extracts cited URLs, and somewhere to store it all. Building it is a weekend to a working prototype and an ongoing maintenance burden thereafter, because the hard parts are brand detection and the fact that provider interfaces change without notice.
Key Takeaways
The prototype is easy and the maintenance is not. Getting answers back is straightforward. Keeping it running for a year as interfaces change is the actual cost.
Brand detection is the hard component. Matching a company name in free text is deceptively difficult, especially for short, generic or multi-word names, and errors here contaminate every metric downstream.
Store raw answers, not just verdicts. Whatever you compute today, you will want to recompute differently in six months. Keep the full text.
Official APIs where they exist. Scraping consumer interfaces is fragile and often against terms of service. Check before you build.
Build if you have engineering capacity and unusual requirements. Buy if you want it working next week and your needs are standard, which they usually are.
Either way, the measurement discipline is the same. Fixed prompt set, multiple runs, variance recorded. The system is plumbing; the method is what produces a defensible number.
Anchor targets assume your CMS generates slug IDs from heading text. Confirm on staging and adjust the hrefs if your platform slugifies differently.
The four jobs
Strip away the dashboards and any tracking system does four things. Knowing which of the four is hard tells you whether building is sensible.
Job | What it does | Difficulty |
|---|---|---|
Run prompts | Send your questions to assistants on a schedule, several times each, and capture the answers | Easy to start, fragile over time |
Detect brands | Find your name and your competitors' in free-form answer text | The hard one |
Extract citations | Pull out every cited URL and attribute it to a domain | Moderate |
Store and query | Keep every run so you can compute rates, ranges and trends | Easy, and easy to get the schema wrong |
Why job two dominates
Everything downstream inherits detection errors. If your matcher misses your brand in one answer in six, your mention rate is wrong by that margin, your trend is wrong, your competitive comparison is wrong, and nothing in the dashboard reveals it.
This is also the component people underestimate most, because finding a company name in text sounds trivial until you try it on real answers.
What the system does not do
Worth stating plainly, because it shapes the build.
Not this | Why |
|---|---|
Observe real user conversations | No access exists. You are sampling your own prompts |
Measure your position in a model | There is no position to measure |
Read or influence training data | Not available to anyone outside the provider |
Report traffic from AI | That is your analytics, and AI referral attribution is incomplete |
A tracking system answers one narrow question: across the prompts I chose, how often do assistants name me and cite me, and who do they name instead. That is a useful question and it is the whole scope.
Before you build anything
Run twenty prompts manually, five times each, across three assistants, and log the results in a spreadsheet. Two hours.
You will learn what the answers actually look like, which tells you far more about the detection problem than any design discussion. Several teams stop here, because the spreadsheet turns out to be enough.
Build versus buy
The build cost is not the prototype. It is the year that follows.
The honest cost model
Phase | Effort | Note |
|---|---|---|
Working prototype | 2 to 5 engineering days | Prompts in, answers out, stored |
Reliable detection | 1 to 3 weeks | The hard part, and iterative |
Scheduling and retries | 3 to 5 days | Rate limits, failures, backoff |
Reporting people will use | 1 to 2 weeks | Or pipe into existing BI |
Initial total | 4 to 8 engineering weeks | For something trustworthy |
Ongoing maintenance | 2 to 6 days a month | Interfaces change; detection drifts |
These are practitioner estimates. Your figures depend on how much you already have and how unusual your requirements are.
Against buying
Trackers run roughly $29 to $500 a month at entry. At a loaded engineering rate, four to eight weeks of build is several years of subscription before you count maintenance.
That comparison favours buying for most teams, and it is not the whole picture.
When building genuinely wins
Situation | Why |
|---|---|
Unusual prompt volume | Thousands of prompts across brands, where per-prompt pricing becomes punitive |
You need the raw data in your own systems | Feeding a warehouse rather than reading a dashboard |
Non-standard requirements | Specific locations, languages, or a classification rule no vendor implements |
You are building a product | If tracking is part of what you sell, buying someone else's is awkward |
Data residency or privacy constraints | Regulated environments where sending prompts to a third party is a problem |
Engineering capacity is genuinely spare | Rare, and real in some organisations |
When buying wins
Situation | Why |
|---|---|
You want it working next week | A vendor has solved the fragile parts already |
Under a few hundred prompts | The economics do not justify a build |
No engineering capacity for ongoing maintenance | A half-maintained tracker is worse than none, because it reports confidently and wrongly |
Standard requirements | Which most are |
The middle path most teams miss
Buy the tracking, export the raw data, and build only your own reporting layer on top.
That gets you the vendor's maintenance of the fragile parts and your own analysis in your own warehouse. It requires that the vendor offers a usable export or API, which is worth confirming before you sign rather than after.
The decision in one line
If you cannot commit two to six engineering days a month indefinitely, buy. A tracker nobody maintains produces confident numbers from a drifting method, which is the worst of the available outcomes.
The architecture
Four components in a line. Answers flow left to right, and the one in the middle is where builds succeed or quietly fail.
[embed: node/b907892a-062b]
The prompt runner and the data store are ordinary engineering. The citation extractor is moderate. The brand detector is the one that needs real attention, and it is the one most prototypes treat as a string search.
What each component owns
Component | Responsibilities |
|---|---|
Prompt runner | Holds the frozen prompt set, schedules runs, handles rate limits and failures, records timestamps and settings |
Brand detector | Finds your brand and your competitors' in answer text, handling variants, abbreviations and false positives |
Citation extractor | Pulls every cited URL, resolves redirects, maps to domains, distinguishes your pages from others |
Data store | Keeps every raw answer plus the derived verdicts, queryable by prompt, date, engine and run |
The design decision that matters most
Store the raw answer text alongside whatever you computed from it.
Your detection rules will change. Your definition of recommendation will change. When they do, you want to recompute the last twelve months rather than start a new baseline. That is only possible if you kept the text.
Storage is cheap and this is the decision people regret most often.
A sensible build order
Prompt runner, writing raw answers to storage. Nothing else.
Run it for two weeks, so you have real answers to develop against.
Brand detection, developed against those real answers.
Citation extraction.
Reporting, last, once you know what the data actually looks like.
Building detection before you have a corpus of real answers means writing rules against imagined text, which is how detection ends up brittle.
The prompt runner
Easy to prototype and the component most likely to break six months later.
Access routes, in order of preference
Route | Stability | Note |
|---|---|---|
Official API | Good | Documented, versioned, supported. Use it wherever it exists |
Documented partner or enterprise access | Good | Some providers offer this for exactly this use case |
Browser automation of a consumer interface | Poor | Breaks on UI changes, and frequently against terms of service |
Scraping | Poor | Same objections, more so |
Check each provider's terms before building against a consumer interface. This is a real constraint rather than a formality, and it is worth having answered before engineering time goes in.
The API caveat worth understanding
An API call to a model is not always the same thing a person gets in the consumer app. The app may apply its own system prompt, its own retrieval configuration and its own search behaviour.
So API-based measurement can diverge from what your buyer actually sees. That does not make it useless; it makes it a proxy, and you should say so when reporting. Spot-check against the consumer interface manually to understand the gap.
This is also one of the things a vendor has usually already worked out, which is a genuine argument for buying.
What the runner must record
Every run, not just the verdict:
Field | Why |
|---|---|
Prompt text, exactly as sent | Paraphrase invalidates comparison |
Prompt set version | So you know the instrument did not change |
Timestamp | Answers vary through the day |
Engine and access route | Behaviour differs |
Any location or setting applied | Changes local answers |
Full raw answer | Everything downstream depends on this |
Whether retrieval appeared to run | Affects what citation rate means |
Error state, if the run failed | A failed run is not a zero |
That last row matters
A failed call is not evidence that you were not mentioned. If failures are recorded as absence, your rate drops whenever the provider has a bad day and you will investigate a content problem that does not exist.
Record failures explicitly and exclude them from the denominator.
Practical engineering notes
Rate limits are real; build backoff from the start. Spread runs across the day rather than firing them in a burst, both to respect limits and because clustering in one minute samples one moment rather than a day. Use a fresh session or conversation per run so history does not leak between them. And make the whole thing idempotent, because you will re-run it after failures and double-counting corrupts your rate.
Brand detection
Finding a company name in text sounds like a string search. It is not, and this is where home-built trackers quietly go wrong.
Why it is hard
Problem | Example |
|---|---|
Generic names | A brand called Notion, Monday, Linear or Arc appears in ordinary sentences constantly |
Variants | Acme Corp, Acme, Acme Inc, ACME, acme.com all refer to the same company |
Possessives and inflection | Acme's, Acmes |
Product versus company | The answer names your product but not your company, or the reverse |
Partial matches | Your brand is a substring of an unrelated word |
Misspellings | Models occasionally produce them |
Mentions in a negative frame | Named, and as the thing not to choose |
Mentions in a list of what the answer is not about | Technically present, practically absent |
A naive case-insensitive substring search will produce both false positives and false negatives, and the two do not cancel out.
A workable approach
Layered, cheapest first:
A variant list per brand. Hand-written: legal name, trading name, common abbreviations, product names, domain. Maintained deliberately.
Word-boundary matching rather than substring, which removes a class of false positives immediately.
A context window. Capture the surrounding sentence for every match, so you can inspect what kind of mention it was.
A classification pass over matches to separate recommended, mentioned in passing, and mentioned negatively. A model can do this well, with a written rule.
A manual review queue for low-confidence matches, at least while you are calibrating.
Calibrate against your own reading
The only way to know your detector works is to read a sample yourself.
Take 100 answers, mark them by hand for whether your brand was named and in what sense, and compare against what your detector produced. Record the agreement rate. Below about 95% agreement, keep working on it, because the errors are a direct bias in every figure you report.
Re-run that check quarterly. Detection drifts as the way models write about your category changes.
The competitor problem
You need the same quality of detection for every competitor you track, and you have less motivation to tune it for them. Resist the temptation to be sloppy here: your share-of-voice figures depend on competitor detection being as good as your own, and an undercounted competitor flatters you.
Why this argues for buying
A vendor has already hit every one of these problems across many customers. Unless your name is unusually clean or your requirements unusual, this component alone is often the reason buying wins.
If you do build it, budget more time here than anywhere else in the system.
Citation extraction
More tractable than brand detection and with its own traps.
What varies by interface
Assistants present sources differently: inline footnotes attached to specific claims, a list at the end, a sidebar, or links embedded in the prose. Some provide structured source data through an API; others only render them visually.
Your extractor needs a strategy per engine, and those strategies are among the first things to break when an interface changes.
The processing steps
Step | Why |
|---|---|
Extract every URL | Including competitors', which is half the value |
Resolve redirects and tracking wrappers | Many citations are wrapped; the wrapper is not the source |
Normalise to a canonical form | Strip parameters, unify protocol and subdomain handling |
Map to a registrable domain | Your comparison unit is usually the domain |
Classify as yours, a competitor's, or other | Your three buckets |
Keep the specific URL too | Domain tells you who won; URL tells you with what page |
That last row matters more than people expect. Knowing a competitor was cited is mildly useful. Knowing exactly which page of theirs was cited tells you what to write.
Decisions to make explicitly
Question | Why it matters |
|---|---|
Do subdomains count as you? | Docs, help centre, blog on separate hosts |
Do owned properties on other domains count? | Your G2 profile, your YouTube channel |
Does an uncited mention of your URL in prose count? | Different from a formal citation |
Do you count a citation once or per appearance? | Affects your denominator |
Write the answers down. These seem minor and they change your numbers, and six months later nobody will remember what the convention was.
The denominator question
Only answers that invoked retrieval can cite anything. If a third of runs produce no sources at all, citation rate over all runs and citation rate over sourced runs are quite different numbers.
Compute both if you like, and always state which one a figure uses. This is the single most common way citation rates get misreported.
What this gives you
Once extraction works, you can answer the question that actually drives content work: for the prompts I lose, which pages are being cited instead?
That output is a reading list and a content brief in one, and it is the most directly actionable thing the whole system produces.
The data store
The easiest component to build and a surprisingly easy one to design badly.
Store runs, not summaries
The atomic record is one run of one prompt against one engine at one moment. Everything else is computed from that.
Teams that store daily summaries instead cannot answer questions they have not anticipated, cannot recompute after a detection change, and cannot report a variance range because they threw away the individual runs.
A workable schema
Field | Notes |
|---|---|
| Unique |
| Text stored even though it is redundant, because sets get edited |
| The instrument version |
| API or interface |
| Timestamp with zone |
| Where applied |
| Success or the failure reason |
| The whole text. The most important field here |
| As returned, before processing |
| Derived, with the detector version that produced them |
| Array |
| Derived |
| So you can recompute when it changes |
The two fields people omit and regret
raw_answer. Without it you cannot recompute anything. Storage is cheap; a year of answers for a few hundred prompts is small.
detector_version. When you improve detection, you need to know which records were computed with which rules, or your trend line silently contains a methodology change.
Computed views on top
Keep the raw table immutable and compute rates in views or queries. Mention rate, citation rate and recommendation rate per prompt, per engine, per month, each with run counts and ranges.
Building rates into the stored data makes them impossible to revise. Computing them on read means a detector improvement can be applied retroactively to everything.
Scale
Small. 120 prompts, 5 runs, 3 engines, monthly is 1,800 rows a month. Even daily sampling at that size is manageable in an ordinary database, and the raw answers are the only bulky part.
This is not a big data problem, and anyone proposing significant infrastructure for it has misjudged the scale.
Retention
Keep everything. Your historical measurement is the one asset here you cannot rebuild, which is also why export rights matter so much when buying rather than building.
Scheduling, sampling and cost
How often to run
Cadence | Suits | Trade-off |
|---|---|---|
Daily | High-stakes, well-funded programmes | Most daily movement is noise; you will be tempted to react to it |
Weekly | Larger teams wanting early warning of big shifts | Reasonable middle ground |
Monthly | Most teams | Enough to see trends, not enough to chase noise |
Monthly is the right default. The useful unit of change here is a quarter, and daily sampling mostly produces opportunities to misread variance.
If you do sample more often, report monthly regardless. Collection frequency and reporting frequency are separate decisions.
Runs per prompt
Five is the practical minimum for a usable rate. Ten or more for prompts that matter commercially. More runs narrow your range, which lowers the threshold at which a change is detectable.
Spread them across a day rather than firing them together, because a burst samples one moment.
Cost
For an API-based build, the dominant costs are token usage and engineering time.
Component | Scale |
|---|---|
Model API calls | 120 prompts, 5 runs, 3 engines, monthly is 1,800 calls. Modest at current pricing |
Classification passes | One model call per detected mention, if you classify with a model |
Storage and compute | Negligible at this scale |
Engineering maintenance | 2 to 6 days a month. The real cost |
If you move to daily sampling the API cost rises thirtyfold and is still usually small. The maintenance line does not change, which is why frequency is rarely the thing that makes a build expensive.
A sampling detail worth getting right
Use the same measurement window each month, for example the first full week. Comparing a first-week reading to a month-end one introduces timing effects you cannot separate from real movement.
What to automate and what not to
Automate | Keep manual |
|---|---|
Running prompts, storing answers, extracting citations | Reading a sample of answers every month |
Rate computation | Deciding what a change means |
Alerting on movement beyond your range | Acting on the alert |
That first row on the right is the one teams drop once the system works, and it is how a drifting detector goes unnoticed for two quarters. Read twenty answers a month yourself. It takes fifteen minutes and it is the only check on the whole pipeline.
What breaks
The reason build estimates are wrong is that people price the build and not the decay.
The recurring failures
What breaks | How often | Symptom |
|---|---|---|
Interface or API changes | Every few months | Runs fail, or succeed and return a different shape |
Citation format changes | Periodically | Extraction silently returns nothing |
Model behaviour shifts | Unannounced | Your rate moves and nothing in your code changed |
Detection drift | Gradual | The way models write about your category changes; your matcher ages |
Rate limit changes | Occasionally | Partial data, which is worse than no data |
Competitor set goes stale | Quarterly | New entrants you are not tracking |
Prompt set ages | Quarterly | Your buyers' language moved |
The dangerous category
Silent failures. A crashed job gets noticed. An extractor that quietly returns zero citations because the source markup changed produces a confident chart showing your citation rate collapsing, and someone will spend a month investigating a content problem that does not exist.
Build these guards from the start
Guard | Catches |
|---|---|
Alert if the failure rate exceeds a threshold | Interface changes |
Alert if citations drop to zero across all prompts | Extraction breakage |
Alert if detection rate for a known-good prompt changes sharply | Detector drift |
A canary prompt you know you win | Whole-pipeline health |
Monthly human review of twenty answers | Everything else |
The canary is the cheapest and most effective of these. Pick a branded prompt you are confident about. If the system stops detecting you there, something broke in the pipeline rather than in your market.
The maintenance reality
Two to six days a month, indefinitely, and it is lumpy: nothing for six weeks, then two days when a provider changes something.
That pattern is hard to staff. It is too little work for a dedicated person and too interrupting for someone with other commitments, which is the practical reason many internal trackers are abandoned after a year rather than any technical failure.
Ask before you commit
Who owns this in eighteen months? If the honest answer is the person who built it, and they may not be here, factor that in. An abandoned tracker still produces numbers, and that is precisely the problem.
When to buy instead
For most teams, buying is the right answer, and the reason is maintenance rather than capability.
What to look for
The questions that matter map directly onto the components above.
Question | Why it matters | What good looks like |
|---|---|---|
How many runs per prompt per period? | Your sample size | Several, stated plainly |
Is variance visible in the reporting? | Whether you can tell signal from noise | A range alongside every rate |
Do I get full answer text? | Whether you can diagnose or only observe | Stored and exportable |
Do I get every cited URL, including competitors'? | The most actionable output | Yes, with the specific URL not just the domain |
Are mentions and citations reported separately? | Four diagnoses need different numbers | Separate metrics |
Which access route do you use? | Consumer behaviour versus API proxy | A straight answer |
How is brand detection done, and how accurate? | Everything inherits this | They can describe it |
Can I export everything, machine-readable? | Your switching cost | Contractual, not discretionary |
The test to run during a trial
Take twenty prompts, check them manually against the tool's data, and record the agreement rate. This is the same calibration you would do on your own detector, and it is the only way to know whether a vendor's numbers are right.
No vendor will do this for you. It is an hour and it occasionally saves a year of decisions made on a biased figure.
Cost comparison
Entry pricing runs roughly $29 to $500 a month. Compare on cost per tracked prompt at your real volume with the engines you need, since allowances vary far more than headline prices.
Against four to eight engineering weeks plus ongoing maintenance, buying wins on cost for almost everyone. The exceptions are in the build-versus-buy section and they are genuine but uncommon.
The hybrid worth considering
Buy the collection, export the raw data, build your own reporting. You get the vendor maintaining the fragile parts and your own analysis in your own systems.
Confirm the export is real before signing, in writing, including format and retention after cancellation.
If you have already built one
The question is not whether to throw it away. It is whether anyone is maintaining it. Run the canary check and the twenty-prompt calibration this week. If either fails, you have been reporting numbers from a drifting system, and that is worth knowing before the next board deck.
Six mistakes when building
1. Treating brand detection as a string search. The component that contaminates everything downstream, built in an afternoon with a case-insensitive substring match. Calibrate against 100 hand-read answers before trusting any number it produces.
2. Storing verdicts instead of raw answers. Your detection rules and metric definitions will change. Without the original text you cannot recompute, so every improvement starts a new baseline instead of correcting the old one.
3. Running each prompt once. Answers vary between runs, so a single sample is an anecdote. Five runs minimum, and the variance you observe is also the threshold below which nothing you see is a result.
4. Counting failed runs as absence. A rate limit error is not evidence you were not mentioned. Record failures explicitly and exclude them from the denominator, or your numbers drop whenever a provider has a bad day.
5. No silent-failure alerting. A crashed job gets noticed. An extractor returning zero citations because a format changed produces a confident chart showing collapse. Build the canary prompt and the zero-citation alert on day one.
6. Not budgeting maintenance. The prototype is days; keeping it honest is two to six days a month indefinitely. An unmaintained tracker still produces numbers, which is worse than producing none.
The pattern
Four of six produce numbers that look fine and are wrong. That is the characteristic failure of this category: the system keeps running, the dashboard keeps updating, and nothing announces that the method drifted.
That is also the strongest argument for buying. A vendor's failures are their problem to notice, and they have more signal to notice them with, because they see the same breakage across every customer at once.
The one to fix first
Mistake one. Everything else on this list degrades your data. That one biases it in a consistent direction, which is far harder to spot and far more likely to drive a wrong decision.
FAQs
What is AI search tracking?
Running a fixed set of prompts against AI assistants on a schedule, several times each, and recording whether your brand was named and whether your URLs were cited. It samples prompts you chose; no tool observes real user conversations, because that access does not exist.
Should we build our own tracker?
Build if you have unusual prompt volume, non-standard requirements, data residency constraints, or you need raw data in your own warehouse and have engineering capacity to maintain it. Buy otherwise. The deciding factor is maintenance, not capability.
How long does it take to build one?
Two to five days to a working prototype, four to eight engineering weeks to something trustworthy, then two to six days a month of maintenance indefinitely. The last figure is what makes or breaks the decision, and it is the one most estimates omit.
What is the hardest part?
Brand detection. Finding a company name in free text is deceptively hard with generic, short or multi-word names, and errors there bias every metric downstream without anything in the dashboard revealing it. Calibrate against 100 hand-read answers.
Can we use the official APIs?
Where they exist, yes, and prefer them. Note that an API call may not reproduce what a person sees in the consumer app, which applies its own system prompt and retrieval configuration. Treat API measurement as a proxy and spot-check manually.
What should we store?
One row per run, including the full raw answer text, the prompt set version, the detector version, timestamps, settings and failure states. Compute rates on read rather than storing them, so a detection improvement can be applied retroactively.
How often should we run it?
Monthly for most teams, with at least five runs per prompt spread across a day. Daily sampling mostly produces noise to misread. Collection frequency and reporting frequency are separate decisions; report monthly regardless.
How do we know the system is still working?
A canary prompt you are confident you win, an alert if citations drop to zero across all prompts, an alert if failure rates spike, and a human reading twenty answers a month. Silent failures are the real risk, not crashes.
Deciding how to track
Start manually. Twenty prompts, five runs each, three assistants, logged in a spreadsheet. Two hours, and it tells you what the answers actually look like, which is the thing you need before designing anything.
Several teams stop there, because at low prompt volume the spreadsheet genuinely is the system.
Build if
Thousands of prompts across brands, where per-prompt pricing becomes punitive
You need raw data in your own warehouse rather than a dashboard
Data residency or privacy constraints rule out a third party
Non-standard requirements no vendor implements
And you can commit two to six engineering days a month indefinitely
That last condition is the one that decides it. Without it, a build becomes an unmaintained system producing confident numbers from a drifting method.
Buy if
Your requirements are standard, you want it working next week, or nobody will own maintenance in eighteen months. Compare on cost per tracked prompt at your real volume, and run the twenty-prompt calibration during the trial regardless of which vendor you favour.
If execution is the real gap
Measuring properly usually reveals the same thing: you know which prompts you lose and cannot produce the pages to win them. That is a content constraint rather than a tracking one, and Blazly SEO covers research, planning, drafting and direct publishing from about $39 a month.
It does not track prompts or store AI answers, so it sits alongside whatever you measure with rather than replacing it.
Summary
An AI search tracking system is a prompt runner, a brand detector, a citation extractor and a data store. The runner and the store are ordinary engineering, the extractor is moderate, and the detector is where builds quietly fail, because matching a company name in free text is far harder than it looks and every metric inherits its errors. Store raw answers so you can recompute when your rules change, record failures separately from absence, run each prompt at least five times, and build silent-failure alerting from day one. The prototype takes days and the maintenance takes two to six days a month forever, which is the number that should decide build versus buy for you.