Build a valid robots.txt with a form instead of syntax roulette, per-crawler rules, paths, and sitemap lines included.
robots.txt is a small file with outsized consequences: placed at a domain's root, it tells crawlers which paths they may fetch. A typo can quietly de-index a site (the classic accident is a staging 'Disallow: /' surviving a launch) or leave private areas exposed to indexing. This generator assembles the file from explicit choices, a default policy, specific disallowed paths, per-group rules for AI training crawlers where that matters to you, crawl-delay where supported, and the sitemap pointer that helps search engines find your canonical page list.
Three facts worth knowing alongside the tool: robots.txt is a request, not a lock, malicious bots ignore it, so it never substitutes for authentication on private content. Google ignores crawl-delay (use Search Console's settings there) while Bing and others honor it. And the file must live at the exact root (example.com/robots.txt) as plain text to count. Test the result with the Robots.txt Tester on this site before deploying, it parses the file the way a crawler would.
Does Disallow hide a page from Google completely? It stops crawling, but if other sites link to the URL, Google may still list the bare URL without content. For true exclusion use a noindex tag or password protection, robots.txt is about crawl budget and politeness, not secrecy.
Which AI crawlers does the block option cover? The well-known training/user-agent names (GPTBot, ClaudeBot, CCBot, Google-Extended and similar). The list evolves; the generator names the widely documented ones, and the generated file is plain text you can extend.
Where does the file go? At the root of the domain, https://example.com/robots.txt, as a plain-text file. Subdomains each need their own.