Build Your Own AI Visibility Tracker

Most homemade AI visibility trackers measure the wrong thing. They send a prompt to a chat completions endpoint, count brand mentions in the reply, and chart the result. That number describes what the model remembers from training, not what AI search retrieves and cites today. If your script never triggers a live search, you can spend a quarter rewriting pages that no retriever ever fetches and watch the chart stay flat. The fix is two lines of configuration, and the second fix, the one nobody mentions, is arithmetic: five runs against one prompt gives you a mention rate accurate to plus or minus 40 points, which is not a measurement at all.

This is the build. Prompt set, search-grounded call, scoring schema, sample size, and the cost of running it against what the tools charge.

Memory and retrieval are two different systems

A generative engine answers in two stages. It rewrites your question into one or more search queries, retrieves documents, then writes an answer conditioned on what came back. The founding GEO paper by Pranjal Aggarwal and colleagues at Princeton, published at KDD 2024, defines generative engines exactly this way: systems that "satisfy queries by synthesizing information from multiple sources and summarizing them using LLMs."

Skip the retrieval stage and you are testing stage two against a frozen snapshot of the web. The model will still name brands, confidently, from training data that may be a year old. It will name the brands that were famous when the snapshot was taken, which is why so many DIY dashboards show incumbents winning and never move.

Three checks tell you which system you just measured:

• Does the response object contain a list of search results or citations? No citations means no retrieval.
• Does the answer reference anything published after the model's training cutoff? If it never does, retrieval is off.
• Does the same prompt return different source URLs on different days? Retrieval is live; memory is not.

The engines expose this as an explicit switch. On the Anthropic API it is a server-side tool you declare in the request. On Google it is search grounding. Perplexity is search-grounded by default, which is why it is the cheapest engine to start with and the worst one to generalize from.

The diagram above shows the split. The top path is what most scripts do. The bottom path is the one that corresponds to what a buyer sees in ChatGPT, Perplexity or AI Overviews.

Design the prompt set before you write any code

The prompt set decides what your tracker can tell you. Get it wrong and no amount of engineering saves the output. Two rules matter more than the rest: most prompts should not contain your brand name, and each prompt should carry exactly one intent.

Brand-name prompts measure whether the model knows you exist. Category prompts measure whether it recommends you, which is the thing you are actually buying. Keep a few of the former for reputation checks and spend the budget on the latter.

Build the set from five families:

1. Category discovery. "Best [category] tools for [segment]" and its variants. This is where purchase intent concentrates.
2. Comparison. "[Competitor] vs [competitor]" and "alternatives to [competitor]". You want to know whether you get pulled into other people's comparisons.
3. Problem-first. The question a buyer asks before they know the category name. "How do I stop [problem]".
4. Brand-direct. "What is [your brand]" and "is [your brand] any good". Reputation, not acquisition.
5. Objection. "Is [category] worth it", "[category] pricing", "[competitor] complaints". These surface the sources that shape a shortlist.

Freeze the set once you start. Changing prompts mid-quarter resets your baseline, and a tracker with no baseline is a screenshot. Write the prompt text, the family, and the date added into a file you version control, then leave it alone.

Fifty prompts is a workable first set for one market. Below twenty you cannot see cluster-level movement. Above two hundred you are paying for precision you will not read.

The call that measures AI search

Here is the Claude version, using the official Anthropic SDK with the server-side web search tool declared. The tool type is what forces retrieval; without it the same call answers from memory.

```python import anthropic

client = anthropic.Anthropic()

def runprompt(prompt: str) -> dict: response = client.messages.create( model="claude-opus-5", maxtokens=2000, tools=[{ "type": "websearch20260209", "name": "websearch", "maxuses": 5, }], messages=[{"role": "user", "content": prompt}], )

answer, citations = [], [] for block in response.content: if block.type == "text": answer.append(block.text) for citation in getattr(block, "citations", None) or []: citations.append(citation.url) elif block.type == "websearchtool_result": # An error returns a single object here, a success returns a list. if isinstance(block.content, list): citations.extend(result.url for result in block.content)

return {"answer": "".join(answer), "citations": citations} ```

Two details in that snippet are easy to get wrong and expensive to debug. The web search tool returns HTTP 200 even when it fails, with an error object in place of the result list, so branch on the type before you iterate. And the tool type string is versioned: `websearch20260209` is the current variant on Opus 5, Opus 4.6 and later, and Sonnet 5. Older models take the earlier `websearch20250305` variant.

For the other engines, the same principle applies with different names:

Run every engine you sell into, and store the raw response. You will want to re-score old answers when you change your scoring rules, and you cannot do that if you only kept the score.

Score the answer, not the vibe

Counting mentions is not enough, because being named last in a list of nine is not the same result as being named first. Store five fields per run and you can answer almost every question a marketing team will ask.

• Mentioned. Boolean. Did the brand appear in the answer text.
• Ordinal position. Where in the list of named vendors it appeared, one-indexed, null if absent. First mention carries most of the click.
• Citation domains. Every URL the engine retrieved, normalised to the registrable domain. This is the most under-used field in AI visibility work, because it tells you which pages to earn rather than which pages to write.
• Competitor set. The other brands named in the same answer. Your real competitive set in AI search is often not the one in your deck.
• Sentiment of the mention. Three values are enough: positive, neutral, negative. Anything finer is noise at this sample size.

A single table holds all of it:

```sql create table runs ( id bigserial primary key, promptid text not null, engine text not null, runat timestamptz not null default now(), mentioned boolean not null, position smallint, sentiment text, competitors text[], citations text[], raw jsonb not null ); ```

Extract the structured fields with a second model call against the stored answer rather than with regular expressions. Brand names appear in possessive forms, inside URLs, and misspelled, and a regex that catches all of that is a weekend you will not get back.

How many runs you actually need

This is where most DIY trackers quietly fail, and it is pure arithmetic. A mention is a Bernoulli trial: the engine either names you or it does not, and the outcome varies run to run. The width of the confidence interval around your mention rate depends on how many runs you did.

For a proportion, the margin of error at 95% confidence is `1.96 × sqrt(p(1-p)/n)`. Put real numbers in it, assuming a brand that gets mentioned 30% of the time:

Detecting a change is harder still. To spot a move from 30% to 40% at 80% power, the two-proportion sample size works out to roughly 353 runs per prompt in each period. Nobody is running 353 calls per prompt per week, which means per-prompt week-over-week charts are decorative.

The way out is pooling. Precision comes from total observations, not observations per prompt, so measure at the level of the cluster instead of the prompt. Fifty prompts at five runs each is 250 observations, and 250 observations puts a 30% mention rate inside plus or minus six points. The same 250 calls spread as five runs on one prompt tells you nothing at all.

So the operating rule is: report category-level and family-level mention rates weekly, report per-prompt results only as examples, and never put a per-prompt sparkline in front of an executive. If you want one number to run the programme on, pool the category discovery family and track that.

Two more things that quietly bias the series. Run every engine at the same time of day, because retrieval indexes update and answers drift by hour. And fix the location parameter, because AI answers are geo-sensitive and a tracker that silently follows your VPN will invent trends.

What it costs, and when to just buy one

Do the multiplication before you build. Fifty prompts at five runs, across four engines, once a week, is 1,000 calls a week or roughly 4,300 a month.

The cost per call is dominated by the retrieved pages, not your prompt. Search results land in the input context, so a grounded call runs an order of magnitude larger than an ungrounded one. Assume 15,000 input tokens and 800 output tokens per call, which is realistic once five sources are pulled in:

• Claude Opus 5 at $5 per million input and $25 per million output: about $0.095 a call, so roughly $410 a month.
• Claude Haiku 4.5 at $1 and $5: about $0.019 a call, so roughly $82 a month.
• Perplexity Sonar at $1 per million tokens each way plus $8 per thousand requests at medium search context: about $0.024 a call before the request fee dominates at low token counts.

Now compare that to published tool pricing, checked on 16 September 2026. Otterly.AI starts at $29 a month for 15 prompts. Rankscale starts at $20. Athena has a free tier with $25 of credit and a $295 Starter plan. Scrunch AI starts at $300.

The honest conclusion: a statistically meaningful DIY tracker costs about what a mid-tier tool costs, and you also own the maintenance. Build it anyway when you need prompts, engines or markets the tools do not cover, when the raw answers have to stay inside your infrastructure, or when you want citation-domain data joined to your own CRM. Buy when you want a dashboard by Friday. We compared the options in the best AI visibility tools.

What to do next

Start with twenty prompts, one engine, the search tool switched on, and a single table. Run it daily for two weeks before you build any chart, then pool the results and see whether the number moves at all. Most teams discover their real problem in that fortnight: not that their mention rate is falling, but that the citation domains behind every answer belong to publications they have never pitched.

When you get to that point, the work stops being measurement and starts being earning the sources. That is what our GEO programme does, and you can see the shape of it in the insurance case study.