Which crawler should I allow for ChatGPT search?

OpenAI's current publisher guidance says not to block OAI-SearchBot if you want content included in ChatGPT search summaries and snippets. GPTBot is a separate control for potential model training.

Can I allow AI search but block model training?

Some providers expose separate user agents for those purposes. For example, OpenAI distinguishes OAI-SearchBot from GPTBot, and Anthropic distinguishes Claude-SearchBot and Claude-User from ClaudeBot. Always verify the current official documentation.

Does Disallow in robots.txt remove a page from search?

Not reliably. robots.txt controls crawling, not guaranteed deindexing. A blocked URL may still be discovered through links. Use an index-control method such as noindex when removal is the goal, while allowing the crawler to read that directive.

Do I need special Allow rules for every AI crawler?

Not if your existing wildcard rule already allows public pages and your firewall permits the requests. Explicit rules are useful when you want different policies for search, user retrieval and training.

AI crawlers and robots.txt: a practical guide

Configure robots.txt for ChatGPT, Claude, Perplexity and Google while separating search visibility, user-requested retrieval and model training controls.

SeoWaveAugust 29, 20263 min read

AI engine crawler paths passing through allowed and blocked website routes

The fastest way to damage AI search visibility is to copy an old blocklist into robots.txt without checking what each agent does. OpenAI, Anthropic and Perplexity publish different user agents for search, user-requested fetches and model development. A single blanket rule can opt a site out of the answer surfaces it is trying to enter.

This guide explains the current separation and provides a conservative starting policy. Bot names and provider behavior can change, so treat the linked official pages as the source of truth.

First separate the three use cases

Search discovery

Search crawlers index or retrieve public content so an answer product can surface and cite it. OpenAI says OAI-SearchBot supports inclusion in ChatGPT search summaries and snippets. Anthropic describes Claude-SearchBot as the agent that improves search result quality. Perplexity says PerplexityBot is designed to surface and link websites in its search results.

User-requested retrieval

Some agents visit a URL because a user asked an assistant to open or analyze it. Anthropic names Claude-User for that purpose. Perplexity names Perplexity-User and notes that, because the fetch is user initiated, it generally ignores robots.txt. These agents are not the same as background search crawlers.

Model development

Training controls are separate for providers that expose them. OpenAI documents GPTBot for content that site owners may wish to exclude from potential training. Anthropic describes ClaudeBot as its model-development crawler.

Key takeaways

OAI-SearchBot: ChatGPT search discovery
GPTBot: OpenAI model-training control
Claude-SearchBot: Claude search indexing
Claude-User: user-requested Claude retrieval
ClaudeBot: Anthropic model-development control
PerplexityBot: Perplexity search discovery
Perplexity-User: user-requested retrieval

Sources: OpenAI publisher guidance, Anthropic crawler documentation, and Perplexity crawler documentation.

A conservative robots.txt example

The following policy allows search discovery and Claude user retrieval while opting out of the two documented training crawlers. It is an example, not a universal recommendation.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /
Disallow: /dashboard
Disallow: /api

Sitemap: https://example.com/sitemap.xml

If you are comfortable with model-development crawling, remove the specific Disallow groups instead of adding unnecessary Allow rules. If your wildcard policy already allows the public site, search crawlers are normally covered unless a CDN, WAF or application rule blocks them elsewhere.

robots.txt is not an access-control system

Never use crawler directives to protect customer data, staging environments or private documents. The file is public and compliant bots may choose not to fetch a disallowed URL, but the server still needs authentication and authorization.

Google also explains that robots.txt manages crawling rather than guaranteed removal from search. A disallowed URL may still be discovered from external links. If the goal is deindexing, use noindex or authentication and make sure the crawler can reach the page to read the directive. See Google's robots.txt guide and robots meta documentation.

Do not combine Disallow with a page-level noindex and assume both will be read. If crawling is blocked, the bot may never see the meta tag.

Check the WAF and CDN too

A correct file does not help when the request is rejected upstream. Test each important public route with the relevant user agent and inspect the final status, rendered content and headers.

curl -I -A "OAI-SearchBot" https://example.com/important-page
curl -I -A "Claude-SearchBot" https://example.com/important-page
curl -I -A "PerplexityBot" https://example.com/important-page

User-agent strings can be spoofed, so do not use the header alone to bypass meaningful security controls. Perplexity publishes current IP endpoints and recommends combining user-agent and IP checks when configuring a WAF. Refresh provider IP lists from official endpoints rather than hard-coding a copied range forever.

Verify the full crawler path

Use this sequence after every change:

Fetch /robots.txt without redirects or authentication.
Confirm the sitemap URL is current and returns valid XML.
Test representative public pages with each search user agent.
Confirm the page returns 200, a canonical URL and substantive server-rendered content.
Check CDN and application logs for 401, 403, 429 and challenge responses.
Verify that protected routes still require real authorization.
Recheck provider documentation quarterly because user agents evolve.

How this affects AI visibility

Crawl access is an eligibility condition, not a ranking guarantee. Google says public, crawlable content and standard SEO remain the foundation for its generative features. OpenAI and Perplexity similarly describe crawler access as a way to support discovery, not a promise of citation.

After technical access is clean, measure whether the engines actually mention the brand and which sources they choose. Follow the AI visibility audit framework, then use the GEO checklist to improve the pages tied to weak prompts.

Check your AI visibility

Frequently asked questions

Which crawler should I allow for ChatGPT search?: OpenAI's current publisher guidance says not to block OAI-SearchBot if you want content included in ChatGPT search summaries and snippets. GPTBot is a separate control for potential model training.
Can I allow AI search but block model training?: Some providers expose separate user agents for those purposes. For example, OpenAI distinguishes OAI-SearchBot from GPTBot, and Anthropic distinguishes Claude-SearchBot and Claude-User from ClaudeBot. Always verify the current official documentation.
Does Disallow in robots.txt remove a page from search?: Not reliably. robots.txt controls crawling, not guaranteed deindexing. A blocked URL may still be discovered through links. Use an index-control method such as noindex when removal is the goal, while allowing the crawler to read that directive.
Do I need special Allow rules for every AI crawler?: Not if your existing wildcard rule already allows public pages and your firewall permits the requests. Explicit rules are useful when you want different policies for search, user retrieval and training.