Meet an Expert

How to Audit AEO Monitoring Tools for Real Gaps

Two AEO monitoring tools run on the same prompts will report different counts. Here is how to audit the gap and separate signal from sampling noise.

Gilad Bechar Gilad Bechar
Nov 16, 2026 7 min read Tools

Load the same 40 prompts into two AEO monitoring tools, run them in the same week, and compare the mention counts for your brand. They will differ. Run one tool twice and its numbers will differ too. That second fact is the one most teams never check, and it is the key to reading the first. Auditing AEO monitoring tools side by side is mostly an exercise in measuring noise before you argue about signal.

Key Takeaways

  • In a University of St. In a University of St. Gallen study, identical prompts sent to the same AI engine within 24 hours shared well under half of their cited sources, so two tools sampling independently cannot be expected to match.
  • Disagreement between tools has two separable parts: different answers being sampled, and the same answer being parsed differently. Each needs its own test.
  • A gap between two tools only means something when it is larger than the gap between each tool and its own repeat runs.

Why Would Two Tools Report Different Numbers for the Same Prompts?

Because the engines they query rarely return the same answer twice. University of St. Gallen researchers sent identical prompts to ChatGPT, Gemini, Google AI Mode and Perplexity up to ten times within 24 hours, across four Swiss consumer verticals. The average Jaccard similarity of cited sources between runs ranged from 0.32 to 0.43. Two runs of the same prompt, close together in time, shared well under half of their sources.

Brand mentions held up somewhat better, with average brand overlap between 0.33 and 0.48. Day-to-day comparisons over several weeks showed similarly low source overlap, so the churn persists over time.

The prompt matters as much as the engine. In the same study some prompts produced near-identical source and brand sets on every run (Jaccard above 0.8), while others changed almost completely (below 0.2). Specific product questions were answered more consistently than broad ones, so two tools with slightly different prompt sets will drift apart for reasons unrelated to your brand.

Every AEO tool draws a small sample from a moving distribution. Two such samples will disagree, and that disagreement is the baseline.

Sampling Explains Only Part of the Gap

Even when two tools query the same engine on the same day, the pipeline from answer to dashboard number has several forks, and each one changes the count.

  • Collection method. Some tools query model APIs and others capture the consumer interface. The St. Gallen team restricted its analysis to one collection window because mixing API and interface data for ChatGPT would have created a methodological inconsistency.
  • Search triggering. ChatGPT turned on web search only for some queries, which left many of its runs in that study with zero citations. A tool that drops those runs and a tool that counts them as zeros will report very different citation rates from identical answers.
  • Artifacts. The ChatGPT data included an OpenAI image-delivery domain as a false cited source, and the researchers had to filter it out. Filtering it or not moves the denominator.
  • Entity matching. Substring matching against a fixed brand list misses synonyms, abbreviations and paraphrases, and can throw false positives when a generic word sits inside a brand name.
  • Location. All the St. Gallen data came from Swiss servers, and the authors note that IP address and locale may affect which sources an engine selects.

Definitions differ too. One dashboard counts a brand as present when it is named in the answer text. Another counts it only when one of its URLs appears as a cited source. Those measure different things, and a brand can be named often while its own pages are almost never cited. If one tool’s headline figure is a mention rate and the other’s is Share of Citation, comparing them directly tells you nothing.

How to Run a Side-by-Side Audit That Holds Up

The aim is to split total disagreement into its parts: how much comes from sampling, and how much comes from parsing. That needs a frozen setup and repeat runs.

  1. Freeze one prompt set and load it identically into every tool. Tag each prompt as specific or broad, since the two behave differently.
  2. Pin the engines, the country setting and the collection window, keeping runs within the same 24 hours wherever the tools allow it.
  3. Run each tool against itself. Repeat the full set several times per tool to establish each tool’s own noise floor.
  4. Export raw answer text and citation lists along with the scores. A tool that cannot export its responses can be trusted or distrusted, but it cannot be audited.
  5. Hand-code a sample of stored responses from each tool, marking mentions and citations yourself. Where your coding and the tool’s disagree on the same text, the problem is extraction, and sampling is ruled out.

Compare at three levels: per response (parsing), per prompt (sampling) and in aggregate (what the dashboard shows). For the per-prompt comparison, measure set overlap of brands and domains, then check rank order separately. In the St. Gallen data, rank-biased overlap scores were consistently lower than plain Jaccard, meaning the order of sources shifted even more than their membership.

Cadence matters for the same reason. A single collection is one draw, which is why we track client visibility repeatedly over months.

+64%
growth in AI brand presence for NewDay USA, tracked weekly across five engines over six months.See the case study.

When Is a Difference Between Tools Real?

A gap between tools is meaningful only when it exceeds the spread of each tool’s own repeat runs. If Tool A varies by several points from run to run and Tool B lands inside that band, the tools agree.

If two tools disagree with each other by less than each one disagrees with itself, they agree, and the variation belongs to the engines.

A paper by Ronald Sielinski on arXiv makes the point with confidence intervals. In a 200-query SearchGPT sample on running gear, one publisher held about 9.5% citation share and another about 6.0%. The 95% bootstrap intervals ran from roughly 5.5% to 12.5% and from 4.0% to 8.0%, so the apparent 3.5-point lead was indistinguishable from a tie. Across the platforms and topics he studied, overlapping intervals were the norm for domains separated by less than 5 to 7 percentage points.

Systematic gaps deserve more attention than random ones. If one tool always reports lower ChatGPT citation rates, check how it handles zero-citation runs. If one keeps missing a competitor, check its aliases.

Agreement on direction is worth more than agreement on level. Two tools that both show your Share of Citation rising over the same quarter give you a signal you can act on, whatever their absolute levels.

What to Do Before Your Next Tool Renewal

Send every vendor you use or are evaluating a written questionnaire. Ask whether they collect through the API or the interface, which region they query from, how many runs they make per prompt, how they treat runs with no citations, and how they match brand aliases. Vague answers are an answer.

Then run the self-against-self test this month. Take 40 prompts, run them through your current tool on several separate occasions within one week, and record the spread for your brand and your top three competitors. That spread is the smallest change your team should ever report as movement.

Change how numbers reach leadership: put the range from repeat runs next to every visibility figure, and stop presenting single-run rank changes as wins or losses. Pick one tool as the system of record for trends and use any second tool for quarterly spot checks on the same frozen prompt set.

FAQs

Why Do AEO Monitoring Tools Show Different Numbers for the Same Brand?

Two causes stack up. AI engines return different answers to identical prompts, so each tool samples a different set of responses, and each tool then parses those responses differently: API or interface collection, query location, handling of zero-citation runs, and brand alias matching all change the final count.

How Many Times Should Each Prompt Be Run?

More than once, and enough times that repeat runs settle into a stable range. University of St. Gallen researchers used up to ten repeats per prompt per engine and still found that identical prompts shared well under half of their cited sources. Start with several runs per prompt and add more if your ranges stay wide.

Is a Mention Count the Same as Share of Citation?

No. A mention means the brand is named in the answer text. Share of Citation measures how often AI models pull a brand into answers and name it as a source. A brand can be mentioned often and cited rarely, so check which one each tool reports before comparing them.

Should a Team Use More Than One AI Visibility Tool?

One tool should be the system of record for trends, because blending methodologies adds noise. A second tool is useful for periodic spot checks, provided both run the same frozen prompt set and you know each tool’s own repeat-run variance.

Gilad Bechar

About the author

Gilad Bechar

Gilad Bechar is the Founder & CEO of Moburst. Gilad serves as a mentor to rising startups at Microsoft Accelerator, The Technion, Tel-Aviv University, Unit 8200 and for strategic Moburst clients, and is the Academic Director of the Mobile Marketing and New-Media course at Tel-Aviv University.

Meet the whole team

Ready to make your brand the answer?

Our growth strategists take brands from overlooked to recommended. In one call you will see how the AI engines read your brand today, and leave with your own roadmap to dominate the new era of search.