AI crawler user agents: the complete list

Crawlers without public documentation

Two names show up in server logs and AI bot lists without a vendor documentation page we could open on October 23, 2026, so we list no behavior for them:

• Bytespider (ByteDance). If you want it gone, add a robots.txt rule for Bytespider and, because robots.txt is voluntary, a firewall rule as well.
• xAI (Grok). We found no crawler page from xAI naming a robots.txt token, so we don’t list one.

We’ll add them once the vendors publish documentation.

How to allow AI search but block AI training

Most vendors now split their bots by job, so you can make two separate decisions. Blocking a training crawler keeps your content out of future models. Blocking a search crawler or user fetcher keeps you out of that assistant’s answers. OpenAI, for example, says sites that opt out of OAI-SearchBot won’t be shown in ChatGPT search answers, while GPTBot only covers training. This robots.txt allows the answer side and opts out of training:

# AI search and answer crawlers: allowed, so assistants can cite you
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: DuckAssistBot
Allow: /

# AI training crawlers and controls: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Meta-ExternalAgent
User-agent: MistralAI-Training
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

# Everyone else, including Googlebot and Bingbot
User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Three things to know before you copy it:

• Several User-agentlines in a row share the rules below them. That’s part of the robots.txt standard (RFC 9309). A bot follows the most specific group that names it, and only falls back to User-agent: * if none does.
• Google-Extendedalso covers grounding in Gemini Apps and Vertex AI, not just training. It doesn’t touch Google Search or AI Overviews: Google says robots.txt rules for Googlebot are the control there, and that you limit what AI Overviews show with nosnippet, data-nosnippet, max-snippet or noindex (Google’s AI features guide).
• Meta-ExternalAgent and Amazonbot are mixed: Meta describes Meta-ExternalAgent as covering both model training and product indexing, and Amazon says Amazonbot data may be used for training. Blocking either can cost you more than training. Use the robots.txt AI rules builder to pick bot by bot.

User-triggered fetchers are a special case. OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores them, and Meta and Amazon say the same of their user fetchers, because a person asked for the page. Anthropic says its bots, Claude-User included, honor robots.txt.

Why a firewall can block a bot that robots.txt allows

robots.txt is a request, not a lock. The standard itself says these rules “are not a form of access authorization”. The opposite also happens: your robots.txt says “welcome” and your CDN or firewall answers the bot with a 403 or a challenge page, and the assistant never sees your text.

Cloudflare is the common case. On July 1, 2025 it announced it was changing the default to block AI crawlers and that every new domain would be asked whether to allow them. Its current documentation sorts AI bots into Search, Agent and Training, and says that from September 15, 2026 new domains block Training and Agent bots on pages that display ads while Search stays allowed. Agent covers real-time fetchers such as chat assistants, so a user-triggered visit can be refused even when your robots.txt allows it. You change this under Security Settings in the Cloudflare dashboard.

Other CDNs, WAFs and hosting “bot protection” features can do the same. The only reliable test is to request your page as each bot and look at the response, which is what the free AI crawler check does.

One more reason a bot can “see” your page and still miss your content: JavaScript. In the documentation we checked, Google, Apple and Common Crawl are the vendors that say whether their crawler runs it. An independent Vercel analysis (December 2024) found that OpenAI’s, Anthropic’s and Perplexity’s crawlers did not render JavaScript, so text that only appears after scripts run may be invisible to them. Server-rendered HTML is the safe choice.

How to check a request really comes from the bot

Anyone can put GPTBot in a user agent. Before you allowlist a bot in your firewall, or trust a log line, check the IP address the way its vendor describes:

The IP lists change. If you allowlist ranges in a firewall, fetch the JSON files on a schedule instead of copying them once.

Test your own site

• AI crawler check
  Fetches your page as each AI bot and reads robots.txt, to catch firewall blocks.
• robots.txt AI rules builder
  Pick which AI crawlers to allow and copy the lines.

Being readable only makes you eligible to be cited. To see which buyer questions you’re losing and who wins them instead, read how to get cited by ChatGPT, look up a term in the GEO glossary, or see how the GEOHATS site optimizer runs these checks across every page.

By Daniel Borodin, GEOHATS founder · Last updated October 23, 2026. Spotted a change in a vendor’s docs? Email borodinplum@gmail.com and we’ll update the row.