Search Console lists a URL as Blocked by robots.txt under "Why pages aren't indexed" when Googlebot found the URL but your robots.txt disallows crawling it, and Google has not indexed it. Google knows the URL exists and has not read the page.
For cart, account, internal search and filter URLs this is usually what you want. If the list holds pages that should rank, a rule is wider than intended.
Fetches the live robots.txt and tests it for Googlebot, Bingbot and six AI crawlers. Add the blocked URLs on the next page. Free, no signup.
Open the Blocked by robots.txt row in the Page indexing report and look at the example URLs. Sort them into two groups: URLs you blocked on purpose, and URLs that should be in search. Only the second group needs a fix. If the report shows a sudden jump in blocked pages on a single day, look at what was deployed that day.
The usual causes, most common first:
| Cause | What it looks like |
|---|---|
| Staging robots.txt went live | User-agent: * with Disallow: /. Blocks the whole site. Often arrives with a launch, a migration or a new deploy pipeline. |
| Short prefix rule | Disallow: /p meant for one folder also blocks /products/ and /pricing. Rules match from the start of the path. |
| Wildcards | Disallow: /*? blocks every URL with a query string, including paginated or tracked URLs you may want crawled. * matches any characters, $ anchors the end. |
| A Googlebot group | If robots.txt has a User-agent: Googlebot group, Googlebot ignores the * group completely. A Disallow added only there, or an Allow added only to *, behaves differently than it reads. |
| Wrong host | Each host and protocol has its own file. https://shop.example.com/robots.txt and https://www.example.com/robots.txt are separate. |
Within the group that applies, the longest matching path wins, and on a tie Allow beats Disallow. The robots.txt tester takes up to 25 URLs at once and shows the rule that decided each one, so you can paste the example URLs straight from Search Console.
One more case: if robots.txt itself answers with a 5xx error, Google stops crawling the site for a while as if everything were disallowed. A 404 on robots.txt counts as no rules at all.
Remove or narrow the rule so the pages you want are allowed. To open one folder inside a blocked one, add an Allow with a longer path, for example Disallow: /shop/ plus Allow: /shop/products/. Re-test the full list of URLs before you publish the file, including the ones you want to keep blocked.
If a blocked page should stay out of Google, robots.txt is the wrong tool for that too. Google can still index the bare URL from links, which turns into Indexed, though blocked by robots.txt. Allow crawling and use noindex instead; the noindex checker shows whether Google can see it.
Google caches robots.txt for up to 24 hours. To speed it up, open the robots.txt report in Search Console settings and request a recrawl of the file. Then run Test live URL in URL Inspection on a few of the URLs to confirm crawling is allowed, and click Validate fix on the report row. Expect validation to take days to weeks.
A wrong Disallow usually arrives with a deploy and shows up in Search Console days later. NoindexWatch checks your URLs once a day for robots.txt blocks, noindex in meta tags and headers, error status codes and canonicals that point elsewhere, and emails you when something changes. The free plan covers 1 URL, no card.
Sources: Google Search Central documentation on robots.txt (how Google interprets the robots.txt specification), the robots.txt report and the Page indexing report. Related: Indexed, though blocked by robots.txt · Excluded by 'noindex' tag.