GeoIf ChatGPT or Perplexity can't fetch your pages, none of your other GEO work matters. Here's what to put in robots.txt so the right AI crawlers can read your site, and what a Cloudflare default change landing on September 15 means for you.
How do I allow GPTBot and ClaudeBot in robots.txt?
Add a User-agent line naming the bot, then Allow: / underneath it, in the robots.txt file at your site's root (yoursite.com/robots.txt). A minimal version that welcomes the main AI crawlers looks like this:
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /If you say nothing about a bot at all, most AI crawlers read silence as permission and proceed
anyway. The trouble is that a lot of default robots.txt files, especially ones generated by an old
WordPress plugin or a security tool, block everything with Disallow: / and never got
updated once AI crawlers showed up. Check yours before assuming it's fine.
Which crawlers actually matter, and what do they do?
Not every bot with "GPT" or "AI" in its name does the same job, and that distinction decides how you should treat it. Some fetch content to train a model months from now. Others fetch a page live, right when someone asks a question, to write the answer you might get cited in.
| Crawler | Company | What it does |
|---|---|---|
| GPTBot | OpenAI | Trains future models. Doesn't affect live ChatGPT answers. |
| OAI-SearchBot | OpenAI | Indexes pages for ChatGPT search results and citations. |
| ChatGPT-User | OpenAI | Fetches a page live when a user's prompt triggers a browse. |
| ClaudeBot | Anthropic | Crawls for training and retrieval. |
| Claude-SearchBot | Anthropic | Indexes for search-style answers in Claude. |
| PerplexityBot | Perplexity | Crawls and indexes pages Perplexity may cite. |
| Google-Extended | Controls use in Gemini and AI Overviews training, separate from normal Googlebot indexing. |
The practical split: if you want to be quoted, keep the search and retrieval bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot) allowed. Whether you also allow the pure training bots like GPTBot and ClaudeBot is a separate, more personal call about your content being used to train a model with no guarantee of a citation back. Blocking training bots while leaving search bots open is a common and reasonable middle ground.
What is Cloudflare changing on September 15?
This is worth knowing even if you've never touched your robots.txt, because it can override it. Cloudflare now lets every customer, including free-plan sites, control AI traffic by behavior instead of one blunt toggle: Search, Agent, and Training, each of which can be allowed, blocked everywhere, or blocked only on pages that carry ads.
Starting September 15, 2026, new domains onboarding to Cloudflare, plus existing free-plan sites, get new defaults: bots classified as Training or Agent are blocked on ad-carrying pages, while Search stays allowed. A crawler that does both search and training, which several of the big ones do, gets treated by the stricter Training rule. Existing paid Cloudflare customers keep whatever settings they already had. Anyone can opt out in Security settings before the date. (Source: Cloudflare changelog, July 1, 2026)
How do I check what's actually blocking a crawler?
Robots.txt is a request, not a lock; a well-behaved crawler reads it and honors it, but the real block usually happens somewhere else: a CDN rule, a WAF, a hosting provider's default bot protection, or a plugin that ships with "block AI scrapers" turned on. A few ways to check:
- Read your live robots.txt. Visit yoursite.com/robots.txt in a browser and look for Disallow lines under the bot names above.
- Check your CDN or firewall's bot management settings separately from robots.txt, since it can silently override what the file says.
- Look at server logs for 403s from GPTBot, ClaudeBot, or PerplexityBot user agents if you have access to them.
- Run a free GeoRoar audit, which fetches your robots.txt directly and tells you which AI crawlers are allowed or blocked as part of the Machine Access category.
Is allowing every AI crawler always the right move?
No, and we won't pretend otherwise. Allowing crawlers is a bet that being cited is worth more to you than keeping every scrap of content out of a training set. For most businesses that want customers to find them through an AI answer, that trade is an easy yes. If you run a subscription publication or have licensing concerns, blocking training bots while keeping search bots open is a defensible position, not a mistake. What isn't defensible is blocking everything by accident and never noticing, which is the far more common problem GeoRoar sees in audits.
How do I find out where my site stands?
GeoRoar's free audit fetches your robots.txt along with the rest of your public pages and scores Machine Access as one of six categories, worth 20 of 100 points. You get the exact bots that are allowed or blocked in about 30 seconds without creating an account. If you want the fixed file generated for you, that's the $99 Fix Pack.
Where's your site losing points?
Here's a sample GeoRoar scorecard. Yours takes about 30 seconds.
This is a sample. Your real scorecard is free, no sign-up.
Run your free GEO audit