Meet an Expert

When AI Citation Rate Drops, Is It the Model or You

Model updates move AI citation rates with no change on the brand side. Here is how to keep engine noise out of monthly AEO reporting and spot real gains.

Lior Eldan Lior Eldan
Nov 9, 2026 7 min read Engines

The monthly report lands and the brand’s citation rate on ChatGPT is down several points. Nobody published anything, nobody lost a link, and every competitor you track moved too. What changed was the engine. Answer engines are stochastic systems rebuilt on someone else’s schedule, and a monthly AI citation rate taken from one pass of prompts cannot separate an engine change from real AEO progress unless the measurement was designed to.

Key Takeaways

  • A model update can change how questions are rewritten into searches, which pages get retrieved and how many sources each answer cites, so a brand’s citation rate can move with zero change on the brand’s side.
  • A monthly citation rate is an estimate drawn from a random process, and without repeated runs and a reported range, a shift of a few points is often indistinguishable from noise.
  • The cleanest test of real progress is relative: compare prompts you worked on against prompts you did not, and your rate against a fixed competitor set, on the same engine version.

What Actually Changes When a Model Updates

A model update on a consumer answer engine is rarely one change. The weights may be new, which alters what the model believes about a category before it searches anything. The search layer may change as well: how the model rewrites the user’s question into queries, how many queries it fans out, which index it hits, and how results are reranked before they reach the context window.

Each of those layers moves citation rate independently of your content. A new rewriting step that splits “best payroll software for startups” into two narrower queries pulls a different candidate set. A model tuned to cite fewer sources per answer shrinks the pool of citations for every brand at once. A refreshed training cutoff can make a competitor whose press coverage grew last year the model’s default answer before retrieval even runs.

This maps onto the two layers of AEO. Training-data presence moves when the vendor retrains, which is slow and infrequent. Retrieval presence moves whenever the vendor touches search, and the vendor has no obligation to tell you which part it touched.

A Citation Rate Is an Estimate With an Error Bar

Even with no update, the same prompt asked twice gets different answers. A team from Northwestern University and Boston University studying LLM brand recommendations states plainly that repeated responses to the same query can produce different brands and rankings. Their design sent the same category prompt to each of six commercial models 40 times in independent sessions, with web search disabled, for 1,200 recommendation lists. They concluded that recommendation has to be evaluated as a stochastic retrieval-and-ranking process, and that individual generated lists cannot carry the analysis.

So a monthly citation rate is a sample statistic. If each prompt runs once a month, the figure in the report is one draw from a distribution, and next month’s figure is a draw from a distribution that may itself have moved. Two sources of movement are stacked on top of each other, and a single number per month cannot show which one you are looking at.

A monthly citation rate is one draw from a distribution that the engine vendor can redraw at any time.

The fix is more observations per period and a longer window before anyone calls a trend.

+64%
growth in AI brand presence for NewDay USA, tracked weekly across five engines over six months.See the case study.

How Many Runs Before a Number Means Anything?

A recent protocol paper on repeated-query auditing, the Dice Roll Method, breaks total response variance into four parts: sampling, prompt phrasing, run-to-run variation and model version. It reanalyses five brand-recommendation audits covering roughly 190,000 observations. Two of its findings matter for monthly reporting.

The first is about single prompts. In simulation, the Dice Roll paper put the statistical power to detect a large effect in a single prompt at 0.15 to 0.19 with five runs, and power reached 0.8 only at 15 to 25 runs. A prompt-level swing in a report built on a handful of runs tells you very little.

The second is about aggregation. Across 250 distinct questions, the reliability coefficient was 0.96 at five runs. For five phrasings of a single question on Gemini, it was 0.27. Portfolio-level citation rate across many prompts is stable enough to report. Individual prompts, and clusters of near-identical prompts, are not. The paper also declines to set a universal run count and recommends sizing runs from a pilot of your own prompts.

Separating the Engine’s Movement From Yours

Repeated runs shrink noise. They do nothing about a genuine engine change, because after an update the distribution itself is different. To attribute a shift, you need comparisons that the update affects equally on both sides.

The first is a competitor set. Track a fixed group of competitors on the same prompts and report your Share of Citation within that group alongside your absolute rate. If everyone fell three points and your share held, the engine moved. If your share fell while competitors held, something about your standing changed. Share of Voice will not answer this question, since web mentions do not move when ChatGPT changes its retrieval.

The second is an untreated prompt set. Tag each prompt by whether your team did work against it during the period: new structured content, digital PR, entity fixes. If treated and untreated prompts moved by the same amount, the change came from outside. If treated prompts gained relative to untreated ones on the same engine version, that gap is the closest thing to attributable AEO progress a monthly report can show.

The third is page-level citation data. A rising rate where the newly cited URLs are pages you shipped this quarter is evidence of progress. A rising rate built on the same old pages, appearing across all prompts at once, usually points to a retrieval change in your favour, which can reverse just as quickly.

What to Change in Next Month’s Report

  1. Freeze the core prompt panel for at least a quarter. Add new prompts as a separate panel so the trend line compares like with like.
  2. Run every prompt several times per period and size the run count from a short pilot of your own prompts. Report a range for the portfolio figure.
  3. Stop reporting prompt-level movement as findings. Use it to choose where to look.
  4. Add a competitor set and show your share within it next to the absolute rate.
  5. Tag prompts as treated or untreated against the work log, and report the gap between them.
  6. Keep an engine change log with dates. When a line breaks, check the log before writing the commentary, and split the month at the change date if one falls inside it.
  7. Where an engine offers dated API model snapshots, run a small reference panel against a pinned version. If the pinned panel holds while the consumer product moves, the product changed.

Put the treated-versus-untreated gap at the top of the report. It is the one number in the deck that a model update cannot fake.

FAQs

How Much Monthly Movement in Citation Rate Is Normal?

There is no universal threshold. It depends on how many prompts you track and how many times each one runs. Measure the spread on your own prompt set during a quiet period, and treat movement inside that spread as noise. Single-prompt swings mean little unless the prompt ran many times.

How Can I Tell if a Drop Came From a Model Update?

Check three things. Did tracked competitors fall by a similar amount? Did prompts you did no work on fall as much as the ones you did? Did the drop line up with a dated change on that engine and not appear on others? Yes to all three points to the engine.

Should We Report Weekly Instead of Monthly?

Weekly collection helps because it gives more observations and makes it easier to date an engine change. The reporting cadence can stay monthly. What matters is the number of runs behind each figure and a stable prompt panel.

Does Share of Voice Explain Swings in Citation Rate?

Rarely. Share of Voice measures how often a brand is mentioned across the web, and that does not change when an engine updates its retrieval. Share of Citation measures how often models name the brand as a source, which is the number a model update moves.

Lior Eldan

About the author

Lior Eldan

Lior Eldan is the Co-Founder of Moburst and serves as its COO. He works at the intersection of marketing, AI and growth, helping brands' teams adapt to AI-driven discovery and decision-making through data-informed strategy and systems thinking.

Meet the whole team

Ready to make your brand the answer?

Our growth strategists take brands from overlooked to recommended. In one call you will see how the AI engines read your brand today, and leave with your own roadmap to dominate the new era of search.