Updated
Blocking GPTBot in robots.txt tells OpenAI not to use your pages to train its models. That is all it does. It has no effect on Google rankings, and ChatGPT search uses a different crawler, OAI-SearchBot, which you control separately. So the question is narrow: do you want your content used for model training? The answer depends on what your content is for.
Fetches your robots.txt and shows, for GPTBot, OAI-SearchBot and 12 other AI crawlers, whether they may fetch the URL and which rule decides it. Free, no signup.
What each crawler does
| Token | Used for | Follows robots.txt | If you block it |
|---|---|---|---|
GPTBot | Content that may be used to train OpenAI's foundation models | Yes | OpenAI: your content "should not be used in training generative AI foundation models". |
OAI-SearchBot | Showing and linking sites in ChatGPT search | Yes | OpenAI: sites opted out "will not be shown in ChatGPT search answers", though they can still appear as navigational links. |
ChatGPT-User | Fetching a page when a ChatGPT user asks for it | OpenAI says robots.txt rules "may not apply" | Not used to decide whether you appear in ChatGPT search. |
Google-Extended | Gemini training and grounding; a token only: Google's existing crawlers do the fetching | Yes | Google: it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal". |
OpenAI's bot documentation says each setting is independent of the others. If you allow both GPTBot and OAI-SearchBot, OpenAI may use one crawl for both purposes, so you can see fewer requests than you expect. Changes for search take about 24 hours after you update robots.txt.
GPTBot is OpenAI's crawler, so blocking it can't touch Google Search. Google's own AI opt-out, Google-Extended, doesn't either. Blocking Googlebot does.
Three questions
- Do you want to appear in ChatGPT search? If yes, keep
OAI-SearchBotallowed. This is independent of what you decide about GPTBot. - Is the content itself what you sell? Paid articles, original research, data, course material or documentation behind a login. Blocking training use is a reasonable choice here. For legal questions about how your content may be used, talk to a lawyer; robots.txt is a request, not a contract.
- Do you gain when models know about you? A brand, a product with public docs, a local business. Models that have read your pages can describe you correctly. Allowing GPTBot is a reasonable choice here.
If questions 2 and 3 both apply, there is no single right answer. You can also block GPTBot for part of the site only, such as Disallow: /research/ in its group, and leave the rest open.
The exact robots.txt lines, and why you shouldn't add a separate Allow group for OAI-SearchBot, are in How to block GPTBot and keep ChatGPT search. To refuse the requests at the server instead, see the nginx and Cloudflare section on the same page.
The mistake: blocking ChatGPT search by accident
Most sites that drop out of ChatGPT search didn't decide to. A few ways it happens:
- A
User-agent: *group withDisallow: /, or a broad section block, also applies toOAI-SearchBotwhen it has no group of its own. - A "block AI bots" toggle in a CDN, a security plugin or an SEO plugin that lists search crawlers next to training crawlers.
- A theme, platform or plugin update that rewrites robots.txt.
Nothing tells you. OpenAI applies the change about 24 hours later, and your pages stop showing in ChatGPT search answers (they can still appear as plain navigational links). Run the form above on your site and look at the OAI-SearchBot row.
Hear about it when your AI crawler rules change
robots.txt can change without anyone on your team editing it. NoindexWatch reads it every day and tells you by email, Slack or Discord when the list of AI crawlers it blocks changes, along with noindex, robots.txt blocks for Googlebot, canonicals and sitemap drops on the URLs you choose. The free plan covers 1 site with up to 20 URLs, checked daily, no card.
Sources: OpenAI developer documentation, Overview of OpenAI crawlers; Google Search Central documentation, Google's common crawlers. Related: robots.txt generator (blocks training crawlers in their own group) ยท Shopify robots.txt.