Training crawlers and search crawlers are different
AI companies run several crawlers with different jobs, and robots.txt treats each one separately. Training crawlers collect pages to train future models: GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Bytespider, Applebot-Extended and others. Search and answer crawlers fetch pages so that an assistant can find and cite them in answers: OAI-SearchBot for ChatGPT search, Claude-SearchBot, PerplexityBot. User-triggered fetchers such as ChatGPT-User, Claude-User and Perplexity-User load a page when a person asks the assistant to look at it.
That means blocking GPTBot does not remove you from ChatGPT search, and allowing it is not what gets you cited there — OAI-SearchBot is. Google-Extended is a separate token that controls whether Google may use your content for its Gemini models; it is not a Google Search ranking control, and AI Overviews in Search are crawled by the normal Googlebot. Whether you allow training is a business choice; the checker reports it neutrally and focuses on whether search and answer crawlers can get in.
What the checker tests
- Your robots.txt rules for 18 AI crawlers, from GPTBot and OAI-SearchBot to ClaudeBot, PerplexityBot, Google-Extended, Amazonbot, DuckAssistBot and MistralAI-User.
- Live fetches of your homepage with the OAI-SearchBot, ClaudeBot and PerplexityBot user agents, compared with a normal fetch, to catch firewall or CDN blocks and challenge pages.
- Whether your content is in the raw HTML or only appears after JavaScript runs.
- Organization, LegalService, ProfessionalService, Person, Service and FAQPage structured data, sameAs and Wikidata links.
- Machine-readable contact details, services, pricing, about and FAQ content, semantic HTML and heading outline.
- Canonical URL, XML sitemap, the optional llms.txt and llms-full.txt files and a Markdown version of the page.
About llms.txt
llms.txt is an optional, proposed convention published at llmstxt.org in 2024: a Markdown file at the root of your site with a title, a one-paragraph summary and lists of links to your key pages. Some developer-focused sites publish one. No major AI search provider has said it relies on the file, so we never mark its absence as a failure. If you have one, we check that it follows the proposed format.
How to improve your AI visibility
The fundamentals are the same as for good SEO: let the search crawlers in, serve real text in the HTML, state plainly who you are, what you do, where and for whom, and back it up with structured data. Short, factual paragraphs, clear headings, dated content and FAQ sections are easy for an assistant to quote accurately.