NoindexWatch

Free robots.txt tester

Paste a robots.txt, or enter a site to fetch its live file. Add the URLs or paths you want to test, one per line. You get allowed or blocked for Googlebot, Bingbot and six AI crawlers, with the rule that decided it.

How matching works

• A crawler obeys only the most specific User-agent group that names it. If GPTBot has its own group, the * rules do not apply to it.
• Within that group the longest matching Allow or Disallow path wins. * matches any characters, $ anchors the end of the URL.
• robots.txt controls crawling, not indexing. A blocked URL can still be indexed from links, without its content. To keep a page out of Google, allow crawling and use noindex; the noindex checker tests that.

Pasted text is tested in memory and not stored. Live files are fetched once as NoindexWatchBot.