What is GPTBot?
GPTBot is OpenAI's crawler for collecting publicly available web content used in model training, distinct from OAI-SearchBot and ChatGPT-User.
Published
Definition
GPTBot is OpenAI’s web crawler for collecting publicly available content used in model training. It identifies itself with the user-agent string GPTBot and respects robots.txt rules.
The name is often used loosely to mean “OpenAI’s crawler”, which causes a specific and costly mistake.
Why It Matters for AI Visibility
OpenAI operates three separate agents, and they do different jobs:
| Agent | What it does |
|---|---|
GPTBot | Collects content for model training |
OAI-SearchBot | Builds the index behind ChatGPT search |
ChatGPT-User | Fetches a page live when a user’s prompt requires it |
Blocking GPTBot opts you out of training. It does not remove you from ChatGPT’s search results, because that is OAI-SearchBot. Many sites intending only to opt out of training block all three with a wildcard rule and disappear from AI search entirely — a decision they usually did not mean to make.
Example
Opt out of training, stay visible in AI search:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /