How the indexability check works
- We load the page as a browser, follow any redirects, and read the status code, the X-Robots-Tag header and the raw HTML.
- We read every robots meta tag and the canonical (from the HTML and the Link header). If the canonical points elsewhere, we load that URL too.
- We read robots.txt and apply the rules for Googlebot, Bingbot, OAI-SearchBot, PerplexityBot and Claude-SearchBot to this path.
- We request the page as each AI search crawler, to catch firewalls that refuse them, and look for the URL in your sitemap.
What the indexability checker looks at
| Signal | What we look for | Why it matters |
|---|---|---|
| HTTP status | The page answers 200 (after any redirects) | Error pages are dropped from the index |
| X-Robots-Tag header | noindex, none, nosnippet in the response header, for all bots or one | Invisible in the page source, often set by a CDN |
| Meta robots | <meta name="robots">, plus googlebot, bingbot and AI bot meta tags | The usual way pages get hidden by accident |
| Canonical | Self, another URL, missing, or more than one; the target's status and noindex | Engines index the canonical URL, not yours |
| robots.txt | Rules for Googlebot, Bingbot, OAI-SearchBot, PerplexityBot and Claude-SearchBot on this path | A blocked crawler can't read the page |
| Firewall | A live request as each AI search crawler | CDNs often refuse bots that robots.txt allows |
| Sitemap | The URL is listed in your XML sitemap | Helps engines find and recrawl it |
| Raw HTML text | Words on the page before JavaScript runs | AI crawlers don't run JavaScript |
How to read the matrix
- Crawl allowed: robots.txt lets that engine’s crawler fetch this path, and for the AI crawlers, the live request wasn’t refused.
- Indexable: the page answers 200, no noindex applies to that engine, and (for the AI crawlers) the text is in the raw HTML.
- Canonical OK: the canonical points to this page, or there is none. A canonical to another URL means engines index that one instead.
Redirects in the way? The redirect checker shows every hop. To check a whole site rather than one page, run the website SEO checker.
FAQ
What makes a page indexable?
Four things: the page answers 200, crawlers are allowed to fetch it (robots.txt and the firewall let them in), nothing tells engines not to index it (no noindex in a meta tag or the X-Robots-Tag header), and the canonical points to the page itself, or is missing. If any of these fails, engines either skip the page or index a different URL instead.
Why check Bing for ChatGPT search?
ChatGPT search draws on Bing's index along with OpenAI's own crawler, OAI-SearchBot. A page that Bing can't index, or that blocks OAI-SearchBot, is much harder for ChatGPT to find and cite. That's why the Bing row checks both bingbot and OAI-SearchBot.
Where can a noindex hide?
In three places: a <meta name="robots"> tag, a bot-specific meta tag like <meta name="googlebot">, and the X-Robots-Tag HTTP header. The header is easy to miss because it never shows in the page source; it is often set by a CDN, a server rule or a plugin. This check reads all three.
My canonical points to another URL. Is that bad?
Not always. On a duplicate (a filtered product list, a print version, a tracking URL) it is exactly right. On a page you want found, it tells engines to index the other URL instead, so this one won't show up. A canonical that points to a broken, redirected or noindex page is always a mistake.
Does robots.txt stop a page from being indexed?
Not exactly. robots.txt stops crawling, not indexing. Google can still list a blocked URL without its content if other sites link to it, but it can't read the page, so it can't see a noindex tag on it and AI assistants can't quote it. To keep a page out of the index, allow crawling and use noindex.
Why does it matter if the text is in the raw HTML?
Google renders JavaScript before indexing. The crawlers behind ChatGPT, Claude and Perplexity mostly don't. If the text only appears after JavaScript runs, those crawlers see an almost empty page, even though Google indexes it fine.
Is this indexability checker free?
Yes. It doesn't use any AI model, you don't need an account, and nothing about the page is stored. To keep it fair, each visitor can run a few checks every 10 minutes.
Learn more
- You rank #1 on Google but ChatGPT ignores you: why, and the fix
- How to get cited by ChatGPT: a practical playbook
- Google AI Overviews optimization: how to get your site featured
More free tools
- SaaS website auditCheck the pricing, comparison and alternatives pages AI quotes for software.
- AI prompt research toolFind the questions buyers ask about your category, by intent.
- Brand entity checkerDo Wikidata, Wikipedia and your schema know who you are?
- Website SEO checkerAudit 25 pages for SEO and AI readiness, with a 0-100 score.
- Broken link checkerFind dead internal and external links on 25 pages.
- Redirect checkerFollow every hop of a URL as a browser, Googlebot and AI crawlers.
- XML sitemap checkerValidate your sitemap and test the pages it lists.
- Title tag checkerCheck title and meta description length with a Google preview.
- AI crawler checkerCan GPTBot, ClaudeBot and PerplexityBot read your site?
- AI readiness page checkScore one page on the structure AI assistants quote.
- AI content graderGrade your writing on what AI assistants quote, problems highlighted.
- AI crawler log analyzerSee how often GPTBot, ClaudeBot and PerplexityBot crawl you. Runs in your browser.
- GA4 AI traffic regexTrack visits from ChatGPT, Perplexity and other assistants.
- llms.txt generator and validatorBuild an llms.txt from your sitemap, or check the one you have.
- robots.txt generatorBuild a robots.txt with rules for every AI crawler.
- Schema markup generator and checkerBuild JSON-LD with a live preview, or check a page's schema.
- Open Graph checkerSee your link preview on X, LinkedIn, Facebook, Slack and iMessage.
- SPF, DKIM and DMARC checkerGrade your domain's email authentication from A to F.
- Site optimizer (GEOHATS)Runs these checks on every page of your site and suggests the fixes.
By Daniel Borodin, GEOHATS founder · Last updated