Search Console shows Indexed, though blocked by robots.txt when a URL is in Google's index but your robots.txt disallows Googlebot from crawling it. Google found the URL through links, internal or external, and indexed it without reading the page. In results it often shows with no description, or with "No information is available for this page".
robots.txt controls crawling, not indexing. A Disallow stops Googlebot from fetching the page, so Google never sees its content, its canonical or a noindex on it.
Fetches the live robots.txt and tests it for Googlebot, Bingbot and six AI crawlers. Add your own paths on the next page. Free, no signup.
| You want the page | Fix |
|---|---|
| In search results | Remove or narrow the Disallow rule that blocks it. Googlebot can then crawl the page and index it with its real content. This is the usual case after a migration or a staging robots.txt that went live. |
| Out of search results | Allow crawling and add noindex, either as <meta name="robots" content="noindex"> or as an X-Robots-Tag: noindex header. Once Google has recrawled and dropped the page you can leave it like that. Blocking it again in robots.txt is how the status started. |
| Not crawled, and you don't care if the bare URL shows up | Nothing to fix. Internal search results, cart and filter URLs often end up here. The status is a notice, not an error. |
For pages that must never appear, such as private documents, neither robots.txt nor noindex is the right tool. Put them behind a login.
Search Console's robots.txt report shows the file Google last fetched for each host. Test the URL against it: Googlebot obeys the most specific User-agent group that names it, then the longest matching path. A short rule like Disallow: /p also blocks /products/ and /pricing. Rules with * and $ catch more than they look like they do.
The robots.txt tester takes up to 25 URLs at once and shows the rule that decided each one. Check every host and protocol separately: https://www.example.com/robots.txt and https://example.com/robots.txt are different files.
After you change robots.txt, Google usually picks up the new file within a day; you can ask for a recrawl in the robots.txt report. Then run Test live URL in URL Inspection to confirm that crawling is allowed and, if you added one, that the noindex is seen. Use the noindex checker to see the meta robots tag, X-Robots-Tag and robots.txt result for one URL together.
For many URLs with the same cause, open the row in the Page indexing report and click Validate fix. Expect it to take days to weeks.
A wrong Disallow usually arrives with a deploy and shows up in Search Console days later. NoindexWatch checks your URLs once a day for robots.txt blocks, noindex in meta tags and headers, error status codes and canonicals that point elsewhere, and emails you when something changes. The free plan covers 1 URL, no card.
Sources: Google Search Central documentation on robots.txt, the robots.txt report, robots meta tags and the Page indexing report. Related: Excluded by 'noindex' tag.