robots.txt generator

What the generated file does

• Every Disallow path is matched from the start of the URL path. /admin/ blocks /admin/users but not /blog/admin/. * matches any characters and $ anchors the end.
• An Allow line wins over a shorter Disallow, so it works as an exception inside a blocked folder.
• AI crawlers you block get their own User-agent group with Disallow: /. A crawler with its own group ignores the * group.
• Google-Extended and Applebot-Extended are usage tokens, not crawlers. Blocking them does not affect Google Search or Apple search. The AI bot checker lists what each crawler is for.

robots.txt does not remove pages from Google

It stops crawling, not indexing. A blocked URL that other pages link to can still show up in results without a description: Search Console then reports Indexed, though blocked by robots.txt. To keep a page out of the index, leave it crawlable and add noindex; check it with the noindex checker.

Already have a robots.txt? Test it with the robots.txt tester.

The file is built in memory from what you type. Nothing is fetched or stored.