Meet an Expert

How Claude Reads robots.txt Differently Than ChatGPT

Claude's bots treat robots.txt as the gate even for user-initiated fetches. How that differs from OpenAI and Google, and what to change in your file.

Lior Eldan Lior Eldan
Nov 15, 2026 7 min read Engines

Take a robots.txt that ends with User-agent: * Disallow: / and makes exceptions only for Googlebot and Bingbot. A buyer pastes your pricing page into ChatGPT, then into Claude. OpenAI says robots.txt rules may not apply to fetches a user starts, so ChatGPT may still read the page. Claude follows Claude robots.txt rules for every one of its bots, including the one acting for the user. Your blanket rule shuts it out. The same file gives you two different outcomes.

Key Takeaways

  • Anthropic runs three crawlers, each with its own robots.txt token, and all three follow robots.txt. That includes Claude-User, which fetches pages when a person asks Claude something.
  • A robots.txt rule that names ClaudeBot overrides the general AI training preference signal Anthropic began honouring in October 2026, so one named group decides Claude’s behaviour.
  • OpenAI lets a user’s request override robots.txt for ChatGPT-User. Google ties AI Overviews to Googlebot. That leaves Claude as the engine where a wrong wildcard rule costs you the most.

Three Bots, Three Separate Permissions

Anthropic’s help center lists three robots. ClaudeBot collects web content that could be used to train models. Claude-SearchBot indexes content to improve Claude’s search results. Claude-User fetches pages when someone asks Claude a question. Anthropic says that disabling Claude-User stops its system from retrieving your content in response to a user query, which may reduce your visibility in user-directed web search.

These line up with the two layers of AEO. ClaudeBot feeds training-data presence, the slow layer you build through wide coverage. Claude-SearchBot and Claude-User feed retrieval presence, which you earn technically with structured, current pages. Under the robots.txt standard, a crawler obeys the group that names its token. If no group names it, the crawler falls back to the wildcard. So a rule written for ClaudeBot does nothing for the other two bots.

Anthropic also says its bots honour Crawl-delay and will not try to get past CAPTCHAs. It warns that blocking its IP addresses may not give a reliable opt-out, because the block also stops it reading your robots.txt. In other words, Anthropic wants robots.txt to be the place where you set its access.

Why a Named ClaudeBot Rule Outranks Your General Preference

On October 30, 2026, Anthropic started honouring a machine-readable signal that tells AI companies whether content may be used for training. The signal is based on an IETF draft from the aipref working group. A site uses it to state a general “no AI training” preference without naming any company.

The important detail is the order of precedence. Anthropic treats a robots.txt rule that names ClaudeBot as more specific than the general signal. If your file names ClaudeBot for a path, Anthropic follows that rule even when your general setting says otherwise. If the file never mentions ClaudeBot, the general setting applies.

That makes an explicit allowance meaningful. Say a CDN or CMS default sets a blanket no-training preference. One User-agent: ClaudeBot group with Allow: / overrides that default for Claude. It works in reverse too: a named Disallow beats a permissive general signal. Google says it plans to support the same AIPREF signal in early 2027, using the token __usage_ai-training. For now, Anthropic is the engine with a published rule for how the two signals interact.

With Claude, robots.txt decides whether your page can be read at the moment a buyer asks about it.

OpenAI Treats the User’s Request as the Permission

OpenAI also runs three agents. GPTBot is for training, OAI-SearchBot is for ChatGPT search, and ChatGPT-User handles actions a person triggers. The structure matches Anthropic’s. The rules attached to it are different.

PPC Land reported that on December 9, 2025, OpenAI revised its crawler documentation. The robots.txt tags are now listed only for OAI-SearchBot and GPTBot. For ChatGPT-User, the documentation now says that because these actions are user-initiated, “robots.txt rules may not apply.” The same revision says that if a site allows both GPTBot and OAI-SearchBot, OpenAI may use one crawl for both purposes. It also dropped the wording that linked OAI-SearchBot to navigational links, so blocking it no longer clearly keeps a site out of ChatGPT search.

The two companies disagree on where permission comes from. OpenAI treats the user’s request as enough permission for a live fetch. Anthropic treats the site owner’s file as the permission, even when a user is the one asking. For a brand that wants to be cited, OpenAI’s approach is the more forgiving one: a careless wildcard Disallow may not stop ChatGPT reading a page a user points it to. Claude gives you no such safety net.

Because each engine uses a different gate, you have to measure visibility one engine at a time. A single blended number hides a Claude-only failure.

+64%
growth in AI brand presence for NewDay USA, tracked weekly across five engines over six months.See the case study.

Google Has No AI Retrieval Crawler to Block

Google works differently again. According to Google’s crawler documentation, rules addressed to Googlebot apply to Google Search, including all Search features. AI Overviews are one of those features. You cannot block Googlebot to keep content out of AI Overviews without also leaving Search.

Google’s AI control is Google-Extended, and Google states that it has no user agent string of its own. It is a token that Google’s existing crawlers read. It controls whether crawled content can be used to train future Gemini models, and whether it can be used for grounding in Gemini Apps and in Grounding with Google Search on Vertex AI. Google says Google-Extended has no effect on inclusion in Google Search and is not a ranking signal.

Put side by side, the three policies show three views on who controls retrieval. Google folds AI answers into its search crawl, so for Search the choice is all or nothing. OpenAI lets the user’s intent override your file for live fetches. Anthropic lets your file govern every bot, including the one acting for a user, and lets a named rule outrank a general preference.

This matters more because of where Claude gets its information. We describe Claude as leaning on high-authority publications and documentation. If your own docs sit behind a wildcard Disallow, Claude will use someone else’s description of your product. Your Share of Voice can stay healthy while your Share of Citation in Claude falls toward zero.

What to Change in Your robots.txt This Quarter

Start with the live files. Anthropic asks for opt-outs on every subdomain, so pull robots.txt from each one, including docs, help and app subdomains. For each file, work out which group ClaudeBot, Claude-SearchBot and Claude-User actually fall into. Any token that lands in the wildcard group is a decision you have made by accident.

  1. Add named groups for Claude-SearchBot and Claude-User that allow your product, pricing, documentation and comparison pages. These two bots handle retrieval.
  2. Decide on ClaudeBot separately, as a training decision. If you want training presence in Claude, name it with an Allow, because a named rule beats any general no-training signal your CMS or CDN sends.
  3. Find out whether your CDN or a plugin already sends an AI training preference. Many teams have never checked.
  4. Take out IP-level blocks on Anthropic ranges unless you mean them. Use Anthropic’s published IP list to check that log entries are genuine.
  5. Decide on Google-Extended as a Gemini question, kept separate from your Search strategy. Put the early 2027 AIPREF support on your roadmap.

After you change the files, watch your server logs for Claude-User requests on your most important pages. Then track your citations engine by engine. A fix that only shows up in Claude is real, but an average across all engines will bury it.

FAQs

Does Blocking ClaudeBot Remove My Site From Claude’s Answers?

Not directly. ClaudeBot controls collection for training. Claude-SearchBot controls search indexing, and Claude-User controls retrieval when a user asks a question. Each needs its own rule, and a ClaudeBot rule does not apply to the other two.

Does ChatGPT-User Obey robots.txt?

OpenAI’s documentation, as revised in December 2025, says robots.txt rules may not apply to ChatGPT-User because its actions are user-initiated. Only GPTBot and OAI-SearchBot are listed with robots.txt tags.

Will Blocking Google-Extended Remove My Content From AI Overviews?

No. Google says Google-Extended does not affect inclusion in Google Search, and AI Overviews are a Search feature covered by Googlebot rules. Google-Extended controls Gemini training and grounding in Gemini Apps and Vertex AI.

What Happens if My robots.txt Names ClaudeBot and I Also Send a No-Training Signal?

Anthropic follows the named ClaudeBot rule for that path. The general training signal applies only when your robots.txt does not mention ClaudeBot.

Lior Eldan

About the author

Lior Eldan

Lior Eldan is the Co-Founder of Moburst and serves as its COO. He works at the intersection of marketing, AI and growth, helping brands' teams adapt to AI-driven discovery and decision-making through data-informed strategy and systems thinking.

Meet the whole team

Ready to make your brand the answer?

Our growth strategists take brands from overlooked to recommended. In one call you will see how the AI engines read your brand today, and leave with your own roadmap to dominate the new era of search.