Feature · Site optimizer
Site optimizer: an AI crawler audit and the technical fixes behind it
Last updated
Why AI crawlers need their own audit
Two things break AI visibility that a normal SEO audit can miss. First, the crawlers behind ChatGPT, Claude and Perplexity fetch your HTML but don’t execute JavaScript: Vercel’s crawler study found they download JavaScript files without running them, so text rendered in the browser is invisible to them. Second, a firewall can refuse a bot that robots.txt allows. Cloudflare can block AI crawlers at the edge: since September 15, 2026, new domains block AI training and agent bots on pages that show ads by default, and many site owners don’t know.
The first point is also why GEOHATS doesn’t apply fixes with a JavaScript snippet: changes made in the browser don’t reach the crawlers you are trying to win. Fixes land server-side, in your CMS or your code.
How the scan works
- 1
Find your pages
Sitemaps listed in robots.txt, then sitemap indexes and sitemaps. If there are none, it follows links from your homepage. Feeds, assets and pagination noise are skipped. - 2
Fetch raw HTML
Each page is fetched without running JavaScript, as an AI crawler would, with an honest user agent (GEOHATS-SiteScan). Up to 150, 300 or 500 pages per scan depending on your tier. - 3
Test every AI bot
robots.txt is evaluated for each AI crawler, and your homepage is requested as each one. If a browser gets the page but a bot gets a 403 or a challenge page, you are told exactly which assistant is blocked. - 4
Score and explain
Issues are grouped by severity in plain language, each with why it matters, how to fix it, and the affected URLs. Scans re-run weekly if you turned on automatic checks for the site.
The AI crawlers it tests
| Assistant | Crawlers | Checked how |
|---|---|---|
| ChatGPT | OAI-SearchBot (search), ChatGPT-User (live browsing), GPTBot (training) | robots.txt + live request |
| Claude | Claude-SearchBot, Claude-User, ClaudeBot (training) | robots.txt + live request |
| Perplexity | PerplexityBot, Perplexity-User | robots.txt + live request |
| Bing / Copilot | Bingbot | robots.txt + live request |
| Apple / Siri | Applebot (search), Applebot-Extended (training control) | robots.txt + live request for Applebot |
| Meta AI, Alexa, DuckDuckGo, Mistral | Meta-WebIndexer, Amzn-SearchBot, DuckAssistBot, MistralAI-Index (search) plus each vendor's user and training bots | robots.txt + live request for the search bot |
| Gemini | Google-Extended (training control) | robots.txt only |
A blocked search or browsing crawler is a critical issue. A blocked training crawler is only a notice, because many sites block training on purpose and stay visible in answers.
The 33 checks
Plus one alarm, shown first, when the scanner can’t load your site at all.
AI access (4)
- AI assistants are blocked from reading your site (robots.txt or firewall)
- AI training crawlers are blocked (reported separately, as a notice)
- AI sees an empty page because content loads with JavaScript
- No llms.txt file
Indexability (8)
- Site isn't on HTTPS
- No sitemap found
- Sitemap isn't listed in robots.txt
- Pages that return an error
- Redirect chains
- Pages hidden from search (noindex)
- Canonical points to a different page
- No canonical URL
Answer-readiness (6)
- Homepage has no Organization schema
- Long pages don't open with a direct answer
- Long pages with no FAQ or table
- No "last updated" date
- Content pages without structured data
- Walls of text (paragraphs over 150 words)
On-page (11)
- Pages with no title
- Duplicate titles
- Titles too short or too long
- Pages with no meta description
- Duplicate meta descriptions
- Meta descriptions too short or too long
- Pages with no H1 heading
- More than one H1
- Images without an alt attribute
- Image links with no text
- Thin pages (under 200 words)
Links (2)
- Links to broken pages
- Orphan pages (no links point to them)
Speed (2)
- Slow server response (over 1.5s)
- Very heavy HTML (over 1 MB)
How the AI-ready score is calculated
The score starts at 100. A site-wide issue costs a flat amount: 30 points if critical, 8 for a warning, 2 for a notice. A page issue costs 25, 10 or 3 points times the share of pages it affects, so one bad page out of 300 barely moves the score, while a problem on every page costs the full amount. Orphan pages are only reported when the crawl covered the whole site.
Fix suggestions
For the issues it can fix, GEOHATS suggests the new value next to the current one, with a one-line reason. You apply, copy or archive each suggestion; archived ones don’t come back.
| Fix | How it’s made | Cost |
|---|---|---|
| Page titles | AI, from the page's title, H1, text and your buyer questions | 1 run per batch |
| Meta descriptions | AI, same inputs, length-checked | 1 run per batch |
| Image alt text | AI, from the file name and surrounding text | 1 run per batch |
| Internal links | AI, using your link graph; the anchor text must already exist on the page | 1 run per batch |
| Canonical URLs | Rules | Free |
| Organization and Article schema | Rules, from facts on the page | Free |
| FAQ schema | Rules, only from questions and answers already on the page | Free |
Runs are the monthly usage unit in every tier; AI calls bill to your own key. See pricing for the limits.
Write-back to your CMS, with undo
Connect your site in Integrations and apply fixes in bulk. Every change records the value it replaced, so any applied fix can be undone in one click.
| Platform | Fixes it can write |
|---|---|
| WordPress (with the GEOHATS Connector plugin) | All: titles, meta descriptions, canonical, FAQ, Article and Organization schema, alt text, internal links |
| Webhook (Zapier, Make, n8n, your backend) | All, sent as signed events for your workflow to apply |
| Ghost | Titles, meta descriptions, canonical, FAQ and Article schema |
| Shopify | Titles and meta descriptions on products, pages, collections and articles |
| HubSpot | Titles and meta descriptions |
| Webflow | Titles and meta descriptions of static pages (then publish in Webflow) |
| Any other site | Copy each fix, export all as CSV, or let your coding agent apply them over MCP |
The WordPress plugin
WordPress doesn’t expose most SEO fields through its REST API, so write-back uses a small free plugin: download the GEOHATS Connector (.zip). It adds an authenticated endpoint (Application Password, edit permissions required) and:
- writes titles and descriptions into Yoast or Rank Math when they're active, and prints them itself when no SEO plugin handles them
- adds JSON-LD safely encoded, sets alt text in the Media Library and in the post
- adds internal links only around the first plain-text match, never inside headings, code or existing links
- saves a normal WordPress revision for content edits, and keeps everything undoable from GEOHATS
We tested it against a real WordPress install with no SEO plugin, with Yoast, and with Rank Math, including the full loop of scan, suggest, apply, check the live page, and undo. Write-back on Ghost, Shopify, HubSpot and Webflow is built to each platform’s documented API but hasn’t been tested on live customer accounts yet.
From your coding agent
Code-based sites (Next.js, Astro and similar) can use the MCP server: your agent reads the audit and the open fixes, edits your repo, and marks them applied.
It also informs your content plan
The crawl tells GEOHATS what pages you already have. Buyer questions are generated from what your site really sells, the content writer links to real pages with real titles, and when a question already has a matching page on your site, the plan tells you to improve that page instead of writing a new one that competes with it.
Try the free AI crawler check first
Want to know whether ChatGPT can read one page before you buy anything? The free AI crawler check tests a single URL against the same bots, with no account and nothing stored. The site optimizer does the same for your whole site, then fixes what it finds.
FAQ
What is an AI crawler audit?
A check of whether the crawlers behind ChatGPT, Claude and Perplexity can reach and read your pages: what robots.txt allows each of them, whether your firewall or CDN refuses them, and whether your text is in the HTML or only appears after JavaScript runs. GEOHATS runs it as part of a full site crawl.
Why not fix things with a JavaScript snippet?
Because the crawlers behind ChatGPT, Claude and Perplexity don't run JavaScript. A snippet that rewrites titles or adds schema in the browser can help Google, but those AI crawlers never see the change. GEOHATS writes fixes into your CMS or your code instead.
Does the site scan cost anything?
No. The crawl and all the checks use no AI model, so they cost nothing and don't use a run. AI fix suggestions for titles, meta descriptions, alt text and internal links use your own AI key, and each batch counts as one run. Canonical, schema and FAQ-schema fixes are rule-based and free.
How many pages does it crawl?
Up to 150 pages per scan on the 3-site tier, 300 on the 10-site tier and 500 on the unlimited tier. It finds pages through your sitemaps and falls back to following links from your homepage.
Do I need the WordPress plugin?
Only for write-back on WordPress. Most WordPress SEO fields aren't editable through the standard REST API, so the free GEOHATS Connector plugin adds a small, authenticated endpoint. Without it you can still copy fixes or export them as CSV.
Can I undo an applied fix?
Yes. GEOHATS stores the value before each change and every applied fix has an Undo button. On WordPress, content edits also save a normal post revision.
Can I check one page for free without an account?
Yes. The free AI crawler check tests one URL against the AI crawlers: robots.txt rules, a live request as each bot, and what the raw HTML contains. No account needed.
Further reading
Fix what the AI crawlers see
The site optimizer is included in every tier. The crawl and checks are free to run; you pay once for GEOHATS.
See pricing