Skip to content

AI crawlers.Can they read your site?

Paste your robots.txt. We test it against every crawler the AI makers document, without fetching your site.

The short answer

Updated

The AI crawler checker reads a robots.txt file you paste in and shows whether each documented AI search crawler, such as OAI-SearchBot, PerplexityBot, Googlebot and bingbot, may fetch a page on your site. It uses the user agents each maker publishes, explains what blocking each one does, and runs entirely in your browser without visiting your site.

Your robots.txt

1 of 16 crawlers are blocked from this page.

  • OAI-SearchBot · ChatGPT

    Finds sites for ChatGPT search. OpenAI says sites that block it won't be shown in search answers, though they can still appear as plain links. Docs

    * group: Allow: /

    Allowed

  • GPTBot · ChatGPT

    Collects content for training. Blocking it doesn't remove you from ChatGPT search. Docs

    GPTBot group: Disallow: /

    Blocked

  • ChatGPT-User · ChatGPT

    Fetches a page when a user asks ChatGPT to. OpenAI says it isn't used to decide what appears in search. Docs

    * group: Allow: /

    Allowed

  • PerplexityBot · Perplexity

    Surfaces and links websites in Perplexity's search results. Perplexity says it isn't used to train AI foundation models. Docs

    * group: Allow: /

    Allowed

  • Perplexity-User · Perplexity

    Fetches a page when a user asks about it. Perplexity says this fetcher generally ignores robots.txt. Docs

    * group: Allow: /

    Allowed

  • Googlebot · Google AI Overviews and AI Mode, Gemini

    Crawls for Google Search, which includes AI Overviews and AI Mode. Google says robots.txt rules for Googlebot are the control. Docs

    * group: Allow: /

    Allowed

  • Google-Extended · Google AI Overviews and AI Mode, Gemini

    A robots.txt token, not a crawler. Covers Gemini training and grounding. Google says it doesn't affect inclusion in Search. Docs

    * group: Allow: /

    Allowed

  • bingbot · Microsoft Copilot

    Bing's standard crawler. Copilot draws on the Bing index, and Microsoft's guidelines say not to block it. Docs

    * group: Allow: /

    Allowed

  • Claude-SearchBot · Claude

    Indexes the web to improve Claude's search results. Anthropic says blocking it may reduce your visibility in user search results. Docs

    * group: Allow: /

    Allowed

  • Claude-User · Claude

    Fetches pages when someone asks Claude a question. Anthropic says blocking it may reduce your visibility in user-directed web search. Docs

    * group: Allow: /

    Allowed

  • ClaudeBot · Claude

    Collects content that may be used for training. Blocking it excludes your future content from training data. Docs

    * group: Allow: /

    Allowed

  • Meta-WebIndexer · Meta AI

    Improves Meta AI search results. Meta says allowing it helps Meta AI cite and link to your content. Docs

    * group: Allow: /

    Allowed

  • Meta-ExternalAgent · Meta AI

    Crawls for uses such as training AI models or indexing content directly. Docs

    * group: Allow: /

    Allowed

  • Meta-ExternalFetcher · Meta AI

    Fetches a link when a user asks. Meta says it may bypass robots.txt. Docs

    * group: Allow: /

    Allowed

  • Applebot · Apple Siri and Apple Intelligence

    Powers search in Spotlight, Siri and Safari. Apple says it may also help train models and add up-to-date context to AI answers. Docs

    * group: Allow: /

    Allowed

  • Applebot-Extended · Apple Siri and Apple Intelligence

    A robots.txt token, not a crawler. Disallowing it opts out of training Apple's foundation models; pages can still appear in search. Docs

    * group: Allow: /

    Allowed

robots.txt is a request, not a lock. Some user-triggered fetchers say they may ignore it, and a firewall can still block crawlers this file allows.

How it works.No hidden steps.

  1. 01Open yoursite.com/robots.txt, copy everything and paste it in.
  2. 02For each crawler, the checker uses the group that names it, or the * group if none does.
  3. 03Within that group, the longest matching rule wins, and Allow wins a tie, as the robots.txt standard says.
  4. 04Change the test path to check a specific page, such as a service page.

What we store: nothing. The checker never contacts your site, and your robots.txt stays in your browser.

Questions people ask.Short answers.

Where do I find my robots.txt?

At the root of your domain, for example yoursite.com/robots.txt. If there's nothing there, crawlers treat the whole site as allowed.

Should I block GPTBot?

It's your choice. GPTBot collects training data; ChatGPT search uses OAI-SearchBot. Blocking GPTBot doesn't remove you from ChatGPT search.

Why isn't Grok in the list?

Grok's maker doesn't document a crawler or robots.txt token, so there's nothing official to test.

Sources

  1. RFC 9309: Robots Exclusion Protocol · checked Oct 31, 2026
  2. OpenAI: Overview of OpenAI crawlers · checked Oct 31, 2026
  3. Perplexity: Perplexity crawlers · checked Oct 31, 2026
  4. Google: Overview of Google's common crawlers · checked Oct 31, 2026

By Shakhawat Ullah Badhon, Founder, Damex Digital. Published Oct 31, 2026, updated Oct 31, 2026.