Training crawlers, like OpenAI’s GPTBot and the Common Crawl-affiliated CCBot, harvest pages to build or update model weights and run on their own schedule, unrelated to any specific user query. Retrieval crawlers, like OAI-SearchBot and the ChatGPT-User agent, Anthropic’s ClaudeBot and Claude-SearchBot, and PerplexityBot, fetch pages in real time to ground a specific answer, often within seconds of a user asking.
Sites increasingly split the decision: block training crawlers to keep content out of future model weights, while allowing retrieval crawlers so pages remain eligible for citation in live answers. This trade-off is enforceable per-bot in robots.txt, but it requires knowing which user-agent strings correspond to which purpose, since a blanket disallow rule blocks both indiscriminately.