Updated
Search Console lists a URL under Server error (5xx), in the group These pages aren't indexed or served on Google, when Googlebot asked for it and got a 500-level status code back. Google's help page puts it in one line: "Your server returned a 500-level error when the page was requested."
That covers 500 (the application crashed), 502 and 504 (a proxy or CDN couldn't get an answer from your server), 503 (overloaded or in maintenance) and the rest of the 5xx range. The report tells you it happened when Google crawled. It doesn't tell you whether it is still happening, so start there.
Fetches the URL now and shows the HTTP status, every redirect, and whether robots.txt itself answers with an error. Free, no signup.
Step 1: is it still happening?
- Fetch the URL yourself. Use the checker above, or from a terminal:
- Run a live test in URL Inspection. It fetches the page as Googlebot from Google's side, so it also catches a firewall that only blocks Google. A live test that passes means the error has stopped, or comes and goes.
- Open Settings > Crawl stats > Host status. Google's fix list for server errors starts with "Look at your recent host availability". Failed requests clustered on one day point to an outage; a steady trickle points to an overloaded server or one slow template.
If everything answers 200 now and host availability shows a single bad day, the cause is gone: skip to Step 5 and validate. If the live test still gets an error, read on.
Step 2: what Google does with pages that return 5xx
Google's documentation on HTTP status codes:
- "5xx and 429 server errors prompt Google's crawlers to temporarily slow down with crawling." For 500 errors, "The decrease in crawl rate is proportionate to the number of individual URLs that are returning a server error."
- "Any content Google receives from URLs that return a 5xx status code is ignored." A pretty error page with your logo and text doesn't count as content.
- "already indexed URLs are preserved in the index, but eventually dropped."
- "Once the server starts responding with a 2xx status code, Google gradually increases the crawl rate for the site."
Google gives no number of hours or days before a page drops, and neither will we. A short outage usually costs nothing; one that lasts for days, or comes back every afternoon, costs pages. A 429 (too many requests) is treated the same way: "it's considered a server error."
Step 3: check robots.txt first
A 5xx on one page affects one page. A 5xx on /robots.txt affects the whole site. Google's robots.txt documentation:
| robots.txt keeps returning 5xx | What Google does |
|---|---|
| First 12 hours | "Google stops crawling the site but keeps trying to fetch the robots.txt file" |
| Up to 30 days | "Google will use the last good version". With no cached copy, "Google assumes there's no crawl restrictions" |
| After 30 days, rest of the site works | Acts "as if there is no robots.txt file" |
| After 30 days, site also has availability problems | Stops crawling the site, "while still periodically requesting a robots.txt file" |
This is easy to miss: the pages themselves answer 200 in a browser, yet Google crawls nothing. It happens when robots.txt is generated by the application (WordPress, many SEO plugins, Next.js robots.js) and the app errors on that one route, or when a firewall rate-limits the file with a 429. Test it with the robots.txt tester, which shows the status code it got. A 404 for robots.txt is harmless: Google treats every 4xx except 429 "as if a valid robots.txt file didn't exist", so everything is allowed.
Step 4: find the cause
| Cause | What you see | Check | Fix |
|---|---|---|---|
| Server overloaded | Errors at busy times, a mix of 500, 503 and timeouts | Host status in Crawl stats; CPU and memory graphs from your host for the same hours | Caching, a bigger plan, fewer heavy plugins. Google: "Make sure your site's hosting server is not down, overloaded, or misconfigured." |
| Slow dynamic pages | Search, filter or parameter URLs fail; normal pages work | Look at which URLs the report lists: mostly ? URLs? | Google: "Reduce excessive page loading for dynamic page requests." Cache or limit filter combinations, and don't link to endless parameter variants |
| Application error | One template always returns 500, for any visitor | The checker above on two URLs of that template; your PHP or app error log | Fix the code, plugin or theme update that broke it. On WordPress, turn on WP_DEBUG_LOG and read wp-content/debug.log |
| Firewall or CDN blocks Google | Pages work in your browser, but the URL Inspection live test fails | Your firewall or CDN event log for Googlebot requests | Google: "Check that you are not inadvertently blocking Google." Allow verified Googlebot; the 403 guide shows how. Rate limits that answer 429 belong here too |
| Origin down behind a CDN | 502 or 504 from Cloudflare or another proxy | The error page names the CDN; the origin server's own status | Restart or repair the origin; check the CDN points to the right origin address |
| Maintenance left on | 503 on every page for days | The checker above on the home page | Switch maintenance mode off, or see Step 6 |
Step 5: after the fix, validate
Run the checker on a few listed URLs and on robots.txt, then a URL Inspection live test on one of them. When all return 200, click Validate Fix in the Page indexing report. Google says validation "typically takes up to about two weeks, but in some cases can take much longer". Don't start a new validation while one is running.
Step 6: taking a site down for maintenance without losing pages
A 503 is the right answer for planned downtime, as long as it is short. From Google's page on pausing a business online:
- Returning
503is fine to disable the site urgently "for 1-2 days". Taking the whole site down is for "a very short period of time (a few days at most)". - "Don't return a 503 HTTP response status code for the robots.txt file because this blocks all crawling." Let robots.txt keep answering
200during maintenance. - While the site returns
503, "it's not possible for Google's systems to refresh titles, descriptions, metadata, or structured data".
Need longer than a few days? Keep the pages up with a 200 and a notice, or put up an indexable placeholder page, instead of leaving the 503 on.
Hear about errors before Search Console does
Search Console reports server errors days after Googlebot hits them. NoindexWatch fetches the URLs you choose and their robots.txt, and tells you by email, Slack or Discord when a page starts returning a 4xx or 5xx or robots.txt a 5xx, and again when it recovers. A timeout or unreachable server is reported after two failed checks in a row, so a one-off blip doesn't wake you. It isn't a minute-by-minute uptime monitor: the free plan covers 1 site with up to 20 URLs, checked once a day, no card; paid plans check every hour.
Sources: Google Search Console Help, Page indexing report; Google Search Central documentation on HTTP status codes and network errors, on how Google interprets the robots.txt specification, and on pausing or disabling a site. Related: Blocked due to access forbidden (403) ยท My site disappeared from Google.