How the log analyzer works
- Drop or paste an access log. Your browser reads it in chunks (up to 50 MB), so large files don’t freeze the page.
- Each line’s user agent is matched against 22 AI crawlers, from GPTBot and ClaudeBot to Bytespider.
- For each crawler you get hits, hits per day, the pages it reads most and the status codes it got.
- Crawlers whose owners publish IP ranges are checked against them, to flag fake bots.
No log at hand? The AI crawler checker requests your site as each AI bot and shows which ones get blocked right now.
AI crawlers it detects
| User agent | Feeds | Type | IP check |
|---|---|---|---|
| OAI-SearchBot | ChatGPT | Search / answers | Yes |
| ChatGPT-User | ChatGPT | User-triggered | Yes |
| GPTBot | ChatGPT | Training | Yes |
| Claude-SearchBot | Claude | Search / answers | - |
| Claude-User | Claude | User-triggered | - |
| ClaudeBot | Claude | Training | - |
| PerplexityBot | Perplexity | Search / answers | Yes |
| Perplexity-User | Perplexity | User-triggered | Yes |
| Bingbot | Bing / Copilot | Search / answers | Yes |
| Applebot | Apple / Siri | Search / answers | - |
| Meta-WebIndexer | Meta AI | Search / answers | - |
| Meta-ExternalFetcher | Meta AI | User-triggered | - |
| Meta-ExternalAgent | Meta AI | Training | - |
| Amzn-SearchBot | Alexa | Search / answers | - |
| Amzn-User | Alexa | User-triggered | - |
| Amazonbot | Alexa | Training | - |
| DuckAssistBot | DuckDuckGo | Search / answers | - |
| MistralAI-Index | Mistral | Search / answers | - |
| MistralAI-User | Mistral | User-triggered | - |
| MistralAI-Training | Mistral | Training | - |
| CCBot | Common Crawl | Training | - |
| Bytespider | ByteDance | Training | - |
IP ranges come from openai.com/gptbot.json, openai.com/searchbot.json, openai.com/chatgpt-user.json, perplexity.ai/perplexitybot.json, perplexity.ai/perplexity-user.json and bing.com/toolbox/bingbot.json.
How to read the results
- Search and user-triggered crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot) feed live AI answers. These are the ones to keep happy.
- Training crawlers (GPTBot, ClaudeBot, CCBot) collect data for future models. Blocking them is a policy choice, not a visibility fix.
- Refused hits (401, 403, 429) mean a firewall, CDN rule or rate limit turned the crawler away. Use the robots.txt generator to set clear rules, then loosen the CDN rule for the crawlers you want.
- Possibly fakehits use a crawler’s name from an IP its owner doesn’t list. Block those IPs, not the name.
FAQ
Is my log uploaded anywhere?
No. The analyzer runs in your browser: the file is read and parsed on your device, and nothing is sent to GEOHATS or anyone else. You can turn off your internet connection after the page loads and it still works.
Which log formats work?
Apache and Nginx access logs in the common combined format (and the vhost variant), Cloudflare Logpush and Vercel log drain JSON lines, and other JSON lines with the usual field names (user agent, client IP, path, status, timestamp). Gzipped .gz files open directly. For other formats the analyzer looks for the bot name, an IP, a date and a status code anywhere in each line.
Where do I find my access log?
On a server, Nginx usually writes to /var/log/nginx/access.log and Apache to /var/log/apache2/access.log (older days are rotated into .gz files). Most hosting panels have a "Raw access logs" download. On Cloudflare, use Logpush; on Vercel, a log drain. Shared hosts that only show analytics charts usually don't give raw logs.
How do you spot a fake GPTBot?
OpenAI, Perplexity and Microsoft publish the IP ranges their crawlers use. A request that calls itself GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User or Bingbot but comes from an IP outside those ranges is likely a scraper borrowing the name. The ranges are a snapshot taken when this tool was last updated, so a brand-new range can show up as unverified.
What does "refused" mean?
The site answered 401, 403 or 429: the crawler was turned away, usually by a firewall, a CDN bot rule or rate limiting. A refused search crawler can't read those pages, so the assistant it feeds can't quote them. Check your CDN's bot settings and robots.txt.
Should I block training crawlers like GPTBot and ClaudeBot?
That's your call. Blocking a training crawler opts your content out of model training but doesn't remove you from AI search answers, which use separate crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot). Our free robots.txt generator lets you allow search crawlers and block training ones.
If AI crawlers visit, will ChatGPT recommend me?
Not necessarily. Crawling means the assistant can read your pages; it doesn't mean it names you when a buyer asks for a recommendation. That takes pages that answer the question better than the ones it cites today. GEOHATS tracks whether ChatGPT, Claude, Perplexity, Gemini and Grok actually recommend you.
Learn more
- You rank #1 on Google but ChatGPT ignores you: why, and the fix
- How to show up as a source in Perplexity answers
- How to track brand mentions in AI answers
More free tools
- SaaS website auditCheck the pricing, comparison and alternatives pages AI quotes for software.
- AI prompt research toolFind the questions buyers ask about your category, by intent.
- Brand entity checkerDo Wikidata, Wikipedia and your schema know who you are?
- Website SEO checkerAudit 25 pages for SEO and AI readiness, with a 0-100 score.
- Broken link checkerFind dead internal and external links on 25 pages.
- Redirect checkerFollow every hop of a URL as a browser, Googlebot and AI crawlers.
- Indexability checkerCan Google, Bing, ChatGPT, Perplexity and Claude index this page?
- XML sitemap checkerValidate your sitemap and test the pages it lists.
- Title tag checkerCheck title and meta description length with a Google preview.
- AI crawler checkerCan GPTBot, ClaudeBot and PerplexityBot read your site?
- AI readiness page checkScore one page on the structure AI assistants quote.
- AI content graderGrade your writing on what AI assistants quote, problems highlighted.
- GA4 AI traffic regexTrack visits from ChatGPT, Perplexity and other assistants.
- llms.txt generator and validatorBuild an llms.txt from your sitemap, or check the one you have.
- robots.txt generatorBuild a robots.txt with rules for every AI crawler.
- Schema markup generator and checkerBuild JSON-LD with a live preview, or check a page's schema.
- Open Graph checkerSee your link preview on X, LinkedIn, Facebook, Slack and iMessage.
- SPF, DKIM and DMARC checkerGrade your domain's email authentication from A to F.
- Site optimizer (GEOHATS)Runs these checks on every page of your site and suggests the fixes.
By Daniel Borodin, GEOHATS founder · Last updated