What the generator creates
robots.txt is a plain text file at the root of your domain that tells crawlers which paths they may fetch, following RFC 9309. The generator writes a valid file from a few choices: which folders to keep crawlers out of, where your sitemap lives, and how to treat each AI crawler. The file updates as you click, ready to copy or download.
- A default group allowing all crawlers, with optional Disallow rules for admin, search-result or staging paths.
- A Sitemap line with your full sitemap URL.
- Separate toggles for AI training crawlers: GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Bytespider, meta-externalagent and others.
- Separate toggles for AI search and user-triggered crawlers: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot and Perplexity-User.
Training versus search: decide separately
Blocking a training crawler keeps your pages out of future training data collected by that crawler, if it honours robots.txt as the major ones say they do. Blocking a search crawler removes you from that assistant's search results and citations. Many firms choose to allow search crawlers, so they can be found and cited, and decide on training crawlers as a matter of policy.
Google-Extended only controls use of your content for Gemini models; it does not affect Google Search, and AI Overviews are crawled by Googlebot. Blocking Googlebot would remove you from Google Search entirely, so the generator never does that by default.
Before you publish
Never block CSS or JavaScript folders that pages need to render, and never publish a Disallow: / from a staging template on a live site. robots.txt is a public request, not security: do not list secret paths in it, and protect private areas with authentication. Upload the file to the root of your domain so it is available at /robots.txt, then test it with our robots.txt checker. The generator runs entirely in your browser.
An example for a firm website
A typical result has one group for all crawlers that disallows the admin area and internal search, separate groups that disallow selected AI training crawlers if you have decided to opt out, and a Sitemap line at the end. Crawlers you leave untouched follow the general group, so AI search crawlers such as OAI-SearchBot can still find and cite your pages.