An AI crawler checker answers one question: which AI systems may read your pages. To be cited in an AI answer, a page has to be readable by the crawler behind that answer: Googlebot for Google’s AI Overviews and AI Mode, OAI-SearchBot for ChatGPT search, PerplexityBot for Perplexity. A robots.txt rule that blocks them is the simplest reason a site never appears. Training crawlers such as GPTBot and Google-Extended are a separate choice: according to the companies’ own documentation, blocking them keeps your content out of model training without removing you from their search answers. Paste your robots.txt into the checker below to see where you stand.
- Three kinds of AI crawler: search crawlers that fetch pages to show and cite, training crawlers that collect data for models, and fetchers that read a page when a person asks.
- Google AI Overviews have no extra requirements. A page that is indexed and can show a snippet in Google Search is eligible, Google says; Google-Extended doesn’t affect Search.
- Blocking GPTBot doesn’t remove you from ChatGPT search. Blocking OAI-SearchBot does.
- robots.txt is a request, not a lock. Some on-request fetchers say it may not apply to them, and a firewall can block crawlers that robots.txt allows.
AI crawler checker: paste your robots.txt
Open yourdomain.com/robots.txt in a browser, copy all of it and paste it below, then enter the page you care about. The checker reads the rules the way the crawlers do and shows which rule decides for each one. Choose a policy underneath to get the lines to add. Nothing you paste leaves your browser.
Which AI crawler does what
Each company documents its own user-agent tokens, the names robots.txt rules refer to. These are the ones that matter for a site in the Gulf, grouped by what blocking them changes.
| Token | Company | What it does | Block it and |
|---|---|---|---|
Googlebot | Crawls for Google Search, which AI Overviews and AI Mode draw on | You leave Google Search and its AI features | |
Google-Extended | A control token for training Gemini models and grounding in Gemini apps | Search is unaffected: Google says it is not a ranking signal | |
OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT’s search features | You won’t be shown in ChatGPT search answers |
GPTBot | OpenAI | Collects content that may be used to train its models | Your content is marked as not for training |
ChatGPT-User | OpenAI | Fetches pages when a user asks ChatGPT to | OpenAI says robots.txt rules may not apply |
Claude-SearchBot | Anthropic | Indexes pages to improve search results | Your pages may be less visible in its answers |
ClaudeBot | Anthropic | Collects web content for model training | Future content is left out of training |
PerplexityBot | Perplexity | Surfaces and links websites in Perplexity’s results; not used for training | You won’t be linked in Perplexity answers |
Perplexity-User | Perplexity | Fetches pages a user asks about | Perplexity says it generally ignores robots.txt |
bingbot | Microsoft | Crawls for Bing search, which Microsoft’s Copilot also draws on | You leave Bing and its AI answers |
Applebot / Applebot-Extended | Apple | Search for Siri, Spotlight and Safari / opt-out from training | Applebot-Extended alone: you stay in Apple’s search results |
meta-webindexer / meta-externalagent | Meta | Meta AI search / crawling for uses such as training | Search and training are controlled separately |
CCBot | Common Crawl | Builds an open web archive often used to train AI models | Your pages are left out of future archives |
How robots.txt rules apply to AI crawlers
robots.txt follows the Robots Exclusion Protocol (RFC 9309). Three of its rules explain most surprises:
- A crawler follows only the group that names it. If a group names GPTBot, GPTBot ignores the
User-agent: *group entirely, including its Disallow lines. Only crawlers with no group of their own fall back to*. - Inside a group, the longest matching path wins, and when an Allow and a Disallow match equally, Allow wins.
- Each host has its own robots.txt. A blog on blog.example.ae is governed by blog.example.ae/robots.txt, not the main domain’s file.
The first rule catches people out. Writing a group that allows GPTBot everything also lets it into the folders your * group blocks, such as /cart/. To block training crawlers while leaving everything else to your existing rules, add a group like this to the end of the file:
# AI training crawlers: blocked
User-agent: GPTBot
User-agent: Google-Extended
User-agent: ClaudeBot
User-agent: Applebot-Extended
User-agent: meta-externalagent
User-agent: CCBot
Disallow: /Worked example: two robots.txt files
The checker’s example file blocks GPTBot and CCBot entirely and keeps every crawler out of /cart/ and /account/. Checked on a blog page, every AI search crawler can read it: 7 of 7. Four of the six training crawlers can too, and all four on-request fetchers.
A common pattern does the opposite: block everything, then allow Google.
User-agent: *
Disallow: /
User-agent: Googlebot
Allow: /| robots.txt | AI search crawlers allowed | Training crawlers allowed | On-request fetchers allowed |
|---|---|---|---|
| Checker example (blocks GPTBot and CCBot) | 7 of 7 | 4 of 6 | 4 of 4 |
| “Block all, allow Googlebot” | 1 of 7 | 0 of 6 | 0 of 4 |
The second file keeps the site in Google, AI Overviews included, and out of ChatGPT search, Perplexity, Bing and Copilot, Apple’s search and Meta AI. If that isn’t the intention, give each search crawler its own Allow group or remove the blanket Disallow.
Readable is not the same as cited
Access only makes a page eligible. Google says there are no additional requirements, special optimisations or structured data needed to appear in AI Overviews or AI Mode: the page must be indexed and able to show a snippet. What gets a page used is the same thing that gets it found: a clear answer to the question, early on the page.
- Answer first. Put the direct answer in the first paragraph under each heading, then the detail.
- Use tables for numbers and name the source and the date next to them.
- Keep the text in the HTML. If key content appears only after JavaScript runs, check that it is in the page source as well.
- Control snippets deliberately. In Google,
nosnippet,data-nosnippetandmax-snippetlimit what can be shown;noindexremoves the page. In Bing, NOCACHE limits its chat answers to the URL, title and snippet, and NOARCHIVE keeps content out of them. - Don’t count on llms.txt. None of the crawler documentation above mentions it.
For Arabic and English sites
Check both language paths, /en/ and /ar/, and every subdomain: each has its own rules. AI answers in Arabic draw on Arabic pages, so a rule that blocks /ar/ costs you the market where competition is weakest. For local searches, a complete Business Profile matters as much; see how to add business to Google Maps.
Crawler tokens and Google’s requirements checked on 24 October 2026. Companies add crawlers and change their documentation; we update the checker and this page when they do.
Found in search, and in the answers
We make sites in the UAE and Saudi Arabia readable and citable by search engines and AI answers, in Arabic and English: robots rules, structure, content and measurement. See SEO & AI Search, or ask us to check your site.
Questions and answers
How do I check if ChatGPT can crawl my site?
Look at your robots.txt for a group naming OAI-SearchBot (ChatGPT search) or GPTBot (training), or paste the file into the checker above. If neither is named, they follow your User-agent: * rules.
Does blocking GPTBot remove my site from ChatGPT?
No. OpenAI says GPTBot collects content for training, while OAI-SearchBot is used to show websites in ChatGPT search. Blocking GPTBot alone keeps your content out of training; blocking OAI-SearchBot removes you from ChatGPT search answers.
Do I need to do anything special to appear in Google AI Overviews?
No, according to Google. A page needs to be indexed and eligible to show a snippet in Google Search. There are no extra requirements, optimisations or special structured data for AI Overviews or AI Mode.
What is Google-Extended?
A robots.txt token Google uses to control whether content can be used to train Gemini models and to ground answers in Gemini apps. It doesn’t crawl by itself, and Google says it doesn’t affect inclusion or ranking in Google Search.
Does llms.txt help me appear in AI answers?
None of the AI companies’ crawler documentation we checked mentions it. Robots rules, indexable pages and clear answers are what they document.
Can AI assistants read pages my robots.txt blocks?
Some on-request fetchers may. OpenAI says robots.txt rules may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores it, and Meta says its user-requested fetcher may bypass it.
Sources
- AI features and your website, Google Search Central: what AI Overviews and AI Mode require
- Google’s common crawlers, Google Search Central: Googlebot and Google-Extended (updated 15 October 2026)
- Overview of OpenAI crawlers, OpenAI: OAI-SearchBot, GPTBot and ChatGPT-User
- Does Anthropic crawl data from the web?, Anthropic: ClaudeBot, Claude-SearchBot and Claude-User
- Perplexity crawlers, Perplexity: PerplexityBot and Perplexity-User
- About Applebot, Apple: Applebot and Applebot-Extended
- Meta web crawlers, Meta: meta-externalagent, meta-webindexer and meta-externalfetcher
- CCBot, Common Crawl
- New options for webmasters to control usage of their content in Bing Chat, Bing Webmaster Blog, September 2023
- RFC 9309: Robots Exclusion Protocol, IETF
