How the scan works
What we check, and what we cannot
The scan fetches public pages the way a crawler would. It does not log in, submit forms or follow links beyond the pages you enter.
How we fetch
- robots.txt, llms.txt and the XML sitemap, once per site.
- Each page with a browser user agent, then with crawler user agents (OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot) to compare responses.
- One rendered copy of each page through Cloudflare Browser Run, to compare text before and after JavaScript.
- Our own requests identify as
LBJO-AI-Visibility-Audit/1.0for robots.txt, llms.txt and sitemap fetches. Requests come from Cloudflare Workers. - Up to 3 pages per scan, on one site. Scans are rate limited.
The checks
| Check | Evidence | Why |
|---|---|---|
| AI search crawlers allowed in robots.txt | Supported | ChatGPT search, Claude and Perplexity can only cite pages their search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are allowed to fetch. OpenAI's publisher FAQ says sites that block OAI-SearchBot will not appear in ChatGPT search answers. |
| Training crawlers: a business choice, made on purpose | Supported | Blocking training crawlers does not remove you from AI search answers, and allowing them does not guarantee citations. It is a content-licensing decision. |
| CDN-managed robots.txt rules | Supported | Cloudflare can prepend its own managed block to robots.txt. Teams often do not know it is there, so AI crawlers get blocked by a setting rather than a decision. |
| User-initiated assistant fetches | Partly | ChatGPT-User, Claude-User and Perplexity-User fetch a page when a person asks an assistant about it. OpenAI says robots.txt rules may not apply to ChatGPT-User, so blocking it in robots.txt may not stop it. |
| Firewall and bot-rule behavior by user agent | Supported | Many sites allow AI crawlers in robots.txt but block them at the firewall. We fetched the page with several crawler user agents. Real crawlers come from published IP ranges, so this test can show a block but cannot prove access. |
| Main content in the server HTML | Supported | Vercel's server-log research found the major AI crawlers do not render JavaScript (Googlebot and Applebot do). Content that only appears after scripts run is invisible to them. |
| Key facts readable without JavaScript | Supported | Prices, contact details and other facts that answers quote need to be in the HTML a crawler receives. |
| Page can be indexed | Supported | A noindex meta tag or X-Robots-Tag header keeps a page out of Google and Bing, which also feed AI answers. |
| Page returns 200 | Supported | Crawlers skip pages that error, and long redirect chains waste crawl budget. |
| Visible or structured dates | Partly | Published and updated dates help readers and crawlers judge freshness. Original data with sources and dates is one of the few tactics with support. |
| Title, description and H1 | Partly | Basic search hygiene. Helps search engines and AI systems identify what the page is about. |
| Canonical URL | Partly | A wrong canonical can send crawlers to a different page than the one you want cited. |
| XML sitemap with lastmod | Partly | Sitemaps help search engines find and refresh pages. Bing and Google use lastmod when it is accurate. |
| Structured data (schema.org) | Unsupported | Google says there is no special schema markup needed for its AI features. Good hygiene for search features, but an Ahrefs test of 1,885 pages that added schema against 4,000 controls found no meaningful lift in AI Mode or ChatGPT citations. |
| llms.txt | Unsupported | Google says an llms.txt file neither helps nor harms visibility in Google Search, and no study so far shows an effect on AI citations. It is a cheap hedge, not a lever. |
What the scan cannot see
- Whether AI answers cite you today. That takes sampling real prompts in ChatGPT, Claude, Perplexity and Google AI Mode over time.
- Off-site evidence: reviews, press, directories and third-party pages that AI answers often cite instead of your own site.
- Your analytics and attribution: whether AI referrals are counted, and how much of your "direct" traffic they explain.
- Real crawler activity in your CDN or server logs, by verified IP, which is the only proof crawlers reach you.
Sources
Every evidence rating comes from AI Search Is Still Search, which lists the platform documentation and studies behind each one.
Questions or a site to exclude
Email logan@lbjoconsulting.com.