Free tool

robots.txt rules for AI crawlers

Each AI crawler has its own name you can allow or block in robots.txt. Training crawlers like GPTBot and ClaudeBot only affect whether your content trains future models. Search crawlers like OAI-SearchBot and PerplexityBot decide whether assistants can find and cite you. Pick a setting for each one below and copy the lines.

Start from:
Search and answer crawlers

These build the indexes AI assistants search when they answer. Block one and that assistant can't cite you.

  • OAI-SearchBot · OpenAI
    Removes your pages from ChatGPT search answers. Docs
  • Claude-SearchBot · Anthropic
    Stops Claude indexing your pages for search, which Anthropic says may reduce your visibility in its search results. Docs
  • PerplexityBot · Perplexity
    Keeps your pages out of Perplexity's search results. Docs
  • Bingbot · Microsoft
    Removes you from Bing, which Copilot and other assistants draw on. Almost no site should block it. Docs
  • Applebot · Apple
    Removes you from Apple's search features in Spotlight, Siri and Safari. Docs
  • Meta-WebIndexer · Meta
    Keeps your pages out of Meta AI search results; Meta says allowing it helps Meta AI cite and link you. Docs
  • Amzn-SearchBot · Amazon
    Keeps your pages out of search experiences in Amazon products such as Alexa. Docs
  • DuckAssistBot · DuckDuckGo
    Keeps your pages out of DuckDuckGo's AI-assisted answers. Its regular search rankings are not affected. Docs
  • MistralAI-Index · Mistral AI
    Keeps your pages out of Mistral's search index. Docs
User-triggered fetchers

These open a page when someone asks the assistant about it. Some providers say robots.txt may not apply to these requests; see their docs.

  • ChatGPT-User · OpenAI
    Stops ChatGPT from opening your pages when a user asks it to. Docs
  • Claude-User · Anthropic
    Stops Claude fetching your pages in response to a user's question. Docs
  • Perplexity-User · Perplexity
    Targets fetches Perplexity makes when a user asks about your page. Docs
  • Meta-ExternalFetcher · Meta
    Asks Meta not to fetch your pages when a user shares or requests a link. Meta says this fetcher may bypass robots.txt. Docs
  • Amzn-User · Amazon
    Asks Amazon not to fetch your pages for live answers to user questions. Amazon says it may not follow all rules. Docs
  • MistralAI-User · Mistral AI
    Stops Mistral's assistant fetching your pages when a user asks it to. Docs
Training crawlers and controls

These collect data for model training. Blocking them doesn't remove you from AI search answers.

  • GPTBot · OpenAI
    Opts your content out of OpenAI model training. ChatGPT search is not affected. Docs
  • ClaudeBot · Anthropic
    Opts your future content out of Anthropic model training. Docs
  • Google-Extended · Google
    Opts out of Gemini model training and grounding in Gemini Apps. Google Search and AI Overviews are not affected. Docs
  • Applebot-Extended · Apple
    Opts out of Apple's AI model training. It doesn't crawl; pages stay in Apple's search features. Docs
  • Meta-ExternalAgent · Meta
    Blocks Meta's crawler for AI model training and for indexing content in Meta products. Docs
  • Amazonbot · Amazon
    Opts out of Amazonbot, whose data Amazon says may be used to train its AI models. Docs
  • MistralAI-Training · Mistral AI
    Opts your content out of Mistral model training. Docs
  • CCBot · Common Crawl
    Keeps your pages out of Common Crawl's public web archive, a common source of AI training data. Docs
robots.txt
# AI crawler rules (built with geohats.com/tools/robots-txt-ai-rules)
# Every AI crawler listed is allowed. They follow your User-agent: * rules,
# so no extra lines are needed.

Add these lines to the robots.txt at the root of your domain (for example yoursite.com/robots.txt), next to your existing rules. Each subdomain needs its own file.

How it works

  1. For every crawler you block, the builder writes its own group with Disallow: /.
  2. Allowed crawlers need no lines: with no group of their own, they follow your User-agent: * rules.
  3. Paste the lines into robots.txt at the root of your domain. Crawlers pick up the change the next time they read the file.
  4. Test the live result with the AI crawler check, which requests your page as each bot. It catches firewall blocks that robots.txt can’t show.

The AI crawlers, and what blocking each one does

AI crawlers by robots.txt token, owner, type and effect of blocking
TokenOwnerTypeIf you block it
OAI-SearchBotOpenAISearch / answersRemoves your pages from ChatGPT search answers.
ChatGPT-UserOpenAIUser-triggeredStops ChatGPT from opening your pages when a user asks it to.
GPTBotOpenAITrainingOpts your content out of OpenAI model training. ChatGPT search is not affected.
Claude-SearchBotAnthropicSearch / answersStops Claude indexing your pages for search, which Anthropic says may reduce your visibility in its search results.
Claude-UserAnthropicUser-triggeredStops Claude fetching your pages in response to a user's question.
ClaudeBotAnthropicTrainingOpts your future content out of Anthropic model training.
PerplexityBotPerplexitySearch / answersKeeps your pages out of Perplexity's search results.
Perplexity-UserPerplexityUser-triggeredTargets fetches Perplexity makes when a user asks about your page.
BingbotMicrosoftSearch / answersRemoves you from Bing, which Copilot and other assistants draw on. Almost no site should block it.
Google-ExtendedGoogleTrainingOpts out of Gemini model training and grounding in Gemini Apps. Google Search and AI Overviews are not affected.
ApplebotAppleSearch / answersRemoves you from Apple's search features in Spotlight, Siri and Safari.
Applebot-ExtendedAppleTrainingOpts out of Apple's AI model training. It doesn't crawl; pages stay in Apple's search features.
Meta-WebIndexerMetaSearch / answersKeeps your pages out of Meta AI search results; Meta says allowing it helps Meta AI cite and link you.
Meta-ExternalFetcherMetaUser-triggeredAsks Meta not to fetch your pages when a user shares or requests a link. Meta says this fetcher may bypass robots.txt.
Meta-ExternalAgentMetaTrainingBlocks Meta's crawler for AI model training and for indexing content in Meta products.
Amzn-SearchBotAmazonSearch / answersKeeps your pages out of search experiences in Amazon products such as Alexa.
Amzn-UserAmazonUser-triggeredAsks Amazon not to fetch your pages for live answers to user questions. Amazon says it may not follow all rules.
AmazonbotAmazonTrainingOpts out of Amazonbot, whose data Amazon says may be used to train its AI models.
DuckAssistBotDuckDuckGoSearch / answersKeeps your pages out of DuckDuckGo's AI-assisted answers. Its regular search rankings are not affected.
MistralAI-IndexMistral AISearch / answersKeeps your pages out of Mistral's search index.
MistralAI-UserMistral AIUser-triggeredStops Mistral's assistant fetching your pages when a user asks it to.
MistralAI-TrainingMistral AITrainingOpts your content out of Mistral model training.
CCBotCommon CrawlTrainingKeeps your pages out of Common Crawl's public web archive, a common source of AI training data.

robots.txt isn’t the only gate

A clean robots.txt can still leave ChatGPT with an error page. Firewalls and CDNs decide who gets through, and Cloudflare can block AI crawlers at the edge: since September 15, 2026, new domains block AI training and agent bots on pages that show ads by default. If you use Cloudflare, check its AI crawler settings as well as robots.txt, then run the AI crawler check to confirm what each bot actually gets.

FAQ

Does blocking GPTBot remove me from ChatGPT answers?

No. GPTBot collects training data. ChatGPT search uses OAI-SearchBot, and OpenAI says sites that opt out of OAI-SearchBot won't be shown in ChatGPT search answers. You can block GPTBot and keep OAI-SearchBot allowed.

Does blocking Google-Extended remove me from AI Overviews?

No. Google says Google-Extended doesn't affect a site's inclusion in Google Search, and AI Overviews are part of Search. Google-Extended controls whether content is used for Gemini model training and for grounding in Gemini Apps.

Should I block AI crawlers?

It depends on what you want. If you want AI assistants to recommend and cite you, keep the search and answer crawlers allowed. Blocking training crawlers is a separate choice about whether your content trains future models, and it doesn't remove you from AI search answers.

Is robots.txt enough to block a crawler?

robots.txt is a request that well-behaved crawlers follow, not a lock. The major AI providers say their crawlers respect it, but some say user-triggered fetches may not. To enforce a block, use your firewall or CDN as well.

I allowed every AI crawler. Why are they still blocked?

Usually a firewall or CDN. Cloudflare can block AI crawlers at the edge regardless of robots.txt; since September 15, 2026, new domains block AI training and agent bots on pages with ads by default. The free AI crawler check requests your page as each bot to show the live result.

Learn more

More free tools

By Daniel Borodin, GEOHATS founder · Last updated