In 2023, blocking AI crawlers became a statement of principle for many sites. In 2026, with a measurable share of buying research happening inside answer engines, the same robots.txt line is a business decision with a visibility price. The good news is you no longer have to make it as one decision, because the crawling itself has split.
Three crawlers, three jobs
- GPTBot: collects content for training future models. Blocking it means future models learn less about you from your own site - though they still learn from every third-party page that mentions you.
- OAI-SearchBot: builds the index behind ChatGPT search. Blocking it removes your pages from search-backed answers.
- ChatGPT-User: fetches pages live when a user's question triggers browsing. Blocking it means even a user directly asking about your content gets a fetch failure.
Anthropic, Google, and Perplexity operate analogous crawlers with their own user-agent strings, and the same logic applies across vendors: training collection and answer-time retrieval are different pipes, controllable separately in robots.txt. Check each vendor's current documentation for exact user-agent names - they change, and a stale robots.txt quietly blocks more or less than you decided to.
The case for allowing everything
If your business depends on being found and recommended - which is what AEO is - your site is your primary evidence about yourself. Retrieval-backed answers are also the fastest feedback loop you have: a page improved today can be fetched tomorrow, while training-data effects take months to surface. Blocking retrieval crawlers cuts the one loop you can actually iterate on. And blocking training crawlers doesn't erase you from models; it just means the models' picture of you is assembled entirely from sources you don't control. For most B2B brands, that trade is all downside.
The case for the split
The block-training, allow-search position makes sense for businesses whose content is the product: publishers, research firms, paywalled data. Their concern is models reproducing the substance of paid work without sending the reader anywhere. Blocking GPTBot while allowing OAI-SearchBot and ChatGPT-User keeps them visible and citable in answers while withholding bulk training collection. It's a coherent stance - for that business model. A SaaS company copying a news publisher's robots.txt is applying someone else's tradeoff to a problem it doesn't have.
Decide, then verify
Whatever you choose, check that reality matches the decision. Read your actual robots.txt - we regularly see CDN defaults and old security plugins blocking AI crawlers on sites whose teams believe they're open. Check server logs for the crawler user-agents to confirm fetches are happening and returning 200s, not challenge pages from bot protection. One honest limit on measurement: you can verify crawlers can reach you, and you can track whether you appear in answers - ClerAEO does the latter across ChatGPT and Claude today - but attributing a visibility change to a robots.txt change alone is rarely clean, because it never ships in isolation. Treat crawler access as a precondition you verify once, not a lever you tune.