What robots.txt does — and does not do
robots.txt sits at the root of your domain and tells well-behaved crawlers which paths they may fetch. It is standardised as RFC 9309. Each group of rules starts with one or more User-agent lines followed by Allow and Disallow rules; a crawler follows the most specific group that names it and otherwise the group for the wildcard agent.
It controls crawling, not indexing. A page blocked in robots.txt can still appear in search results if other sites link to it, just without a description. To keep a page out of results, allow crawling and use a noindex meta tag or header instead. Google stopped supporting noindex rules inside robots.txt in 2019.
What we check
- That robots.txt exists, returns status 200 as plain text and parses without errors.
- A Sitemap line pointing to a sitemap that loads.
- Whether Googlebot is blocked from the homepage, important sections or the CSS and JavaScript needed to render pages.
- Explicit rules for 18 AI crawlers, separating training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended) from search and answer crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User).
- Live fetches with AI crawler user agents, to catch firewalls that block them regardless of robots.txt.
Common mistakes
The most damaging one is a leftover Disallow: / from a staging site, which blocks everything. Others are blocking /wp-content/ or asset folders so that Google cannot render the page, blocking OAI-SearchBot when the intention was only to opt out of training, and listing a sitemap URL on the wrong host or protocol.
If you want to opt out of AI training but remain visible in AI search, disallow the training crawlers by name and leave the search crawlers allowed. Our robots.txt generator writes those rules for you. Remember that robots.txt is a request: reputable crawlers honour it, but it is not access control.
A sensible default for most firms
For a typical firm website a short file is enough: one group for all crawlers that allows everything except admin or internal search paths, a Sitemap line with the full URL of your sitemap, and, if you have made a policy decision, separate groups for the AI training crawlers you want to exclude. Keep the file short and readable; complex rule sets are where accidental blocks hide. Review it whenever you redesign the site or change CMS.