robots.txt.In plain English.

The format is standardised as RFC 9309, published in September 2022. A crawler uses the group that names it, and if several groups name it they are combined. If none does, the * group applies, and within a group the longest matching rule wins.

If robots.txt can't be reached because of a server error, crawlers must assume everything is disallowed. The RFC also says the rules are not a form of access authorization, so don't rely on them for security.

Google says robots.txt is not a mechanism for keeping a page out of Google. Use noindex or a password for that.

For AI, Googlebot rules control Google Search, including AI Overviews and AI Mode, while Google-Extended only controls use in Gemini and Vertex AI and doesn't affect Search. Other AI companies publish their own user agents, such as OAI-SearchBot and PerplexityBot.

New signals are coming. Anthropic says that, as of October 30, 2026, it recognizes a machine-readable preference signal for AI training, and Google says it intends to support a training preference signal in early 2027.

Example: a site that puts 'Disallow: /api/' under * but gives GPTBot its own group with only 'Allow: /' lets GPTBot into /api/, because GPTBot follows its own group.