How crawlers find a sitemap
Google and Bing read Sitemap: lines in robots.txt, and you can submit a sitemap in Search Console or Bing Webmaster Tools. Many crawlers also try /sitemap.xml. A Sitemap: line is the one place every crawler looks, and it may point to any path, including a sitemap index.
What makes a sitemap file unusable
- An HTML page or an error page served with HTTP 200 at the sitemap address, often from a CMS route that changed.
- A 404 or 5xx on a file that robots.txt or the index still lists.
- More than 50,000 URLs or 50 MB uncompressed in one file. Split it and list the parts in a sitemap index.
- URLs on another host or protocol, such as
http://URLs in the sitemap of anhttps://site, orwwwURLs on the bare domain.
Listed is not the same as indexed
A sitemap helps Google find and recrawl pages; it does not make them indexable. List only the canonical URL of each page, not redirects or noindexed pages. To see whether a listed page can actually be indexed, run it through the noindex checker.
We fetch robots.txt and up to 25 sitemap files, as NoindexWatchBot, for at most 20 seconds. Gzipped sitemaps are not read. Results are not stored.