AI crawler

An AI crawler is an automated bot operated by an AI company that fetches web pages, either to collect training data or to retrieve current content in response to a live user query. Different crawlers serve different purposes, and a site can often allow or block each one independently through robots.txt, which makes crawler management a real visibility lever.

Training crawlers, like OpenAI’s GPTBot and the Common Crawl-affiliated CCBot, harvest pages to build or update model weights and run on their own schedule, unrelated to any specific user query. Retrieval crawlers, like OAI-SearchBot and the ChatGPT-User agent, Anthropic’s ClaudeBot and Claude-SearchBot, and PerplexityBot, fetch pages in real time to ground a specific answer, often within seconds of a user asking.

Sites increasingly split the decision: block training crawlers to keep content out of future model weights, while allowing retrieval crawlers so pages remain eligible for citation in live answers. This trade-off is enforceable per-bot in robots.txt, but it requires knowing which user-agent strings correspond to which purpose, since a blanket disallow rule blocks both indiscriminately.