AI crawler.In plain English.

There are three kinds. OpenAI's GPTBot collects data for training, OAI-SearchBot finds sites for ChatGPT search, and ChatGPT-User fetches a page when a person asks. OpenAI says each setting is independent, so you can allow search and block training.

Anthropic runs ClaudeBot for training, and Claude-SearchBot and Claude-User for search and user requests, and says blocking the last two may reduce your site's visibility. Apple's Applebot powers search in Siri, Spotlight and Safari.

Some names aren't crawlers. Google-Extended and Applebot-Extended are robots.txt tokens that control whether content is used for AI training. Apple says Applebot-Extended does not crawl webpages, and Google says Google-Extended doesn't affect inclusion in Google Search.

User-triggered fetchers may not follow robots.txt: OpenAI says its rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. OpenAI also says robots.txt changes can take about 24 hours to reach ChatGPT search.

If your firewall or robots.txt blocks a search crawler, that engine may not be able to read your site when it answers. Perplexity says sites behind a firewall may need to allow its bots explicitly.

Damex's robots.txt names the main AI search crawlers, user fetchers and training tokens, and keeps the same private paths off limits for all of them.