GeoYour site can look perfect in a browser and still show up as a blank page to ChatGPT. Most owners never find out because nothing visibly breaks. This is the five-minute check for whether AI crawlers can read your website.
Can AI crawlers read my website?
Often not as well as you think. AI crawlers like GPTBot and ClaudeBot request your page as plain HTML. They can be stopped by a robots.txt rule, a firewall challenge, or content that only appears after JavaScript runs. Any one of these makes your site invisible to AI answers, whatever your Google rankings look like.
The tricky part is that you are testing with the wrong tool. Your browser runs scripts, accepts cookies, and passes bot checks. A crawler does none of that, so you have to look at your site the way it does.
Do AI crawlers run JavaScript?
Mostly no. In an analysis of its network traffic published in December 2024, Vercel found that none of the major AI crawlers executed JavaScript. ChatGPT's crawler fetched JavaScript files in 11.5% of requests and Claude's in 23.84%, but neither ran them. Apple's crawler was the exception, since it renders pages like Googlebot does.
That analysis is from late 2024, so treat it as a snapshot and not a promise. It matches what practitioners keep reporting, though. If your phone number, services, or prices load through a script, assume the crawler never saw them. Sites built as single-page apps are the usual victims. Many website builders and booking widgets work the same way.
How do I check what GPTBot actually sees?
Request your page as a crawler and read the raw response. Run one curl command with GPTBot's user agent, then check two things: the status code should be 200, and your name, address, services, and prices should appear in the text that comes back. If they do not, the crawler does not have them.
Here is the whole test:
- Fetch the page. In a terminal, run
curl -s -A "GPTBot" -o page.html -w "%{http_code}" https://yoursite.com. You want 200. A 403 or 503 means something is turning crawlers away. - Search the response. Run
grep -i "your business name" page.html, then repeat for your phone number and your main service. No match means that fact is invisible to a crawler. - Read your robots.txt. Open
yoursite.com/robots.txtand look forDisallowrules under GPTBot, ClaudeBot, PerplexityBot, or a blanketUser-agent: *followed byDisallow: /. - View the page source. In your browser, use View Source (not Inspect). Source shows what the server sent. Inspect shows what scripts built afterward.
- Check your CDN or firewall. If you use Cloudflare or a security plugin, open the bot settings and see whether AI crawlers are blocked.
If you do not use a terminal, steps three and four still work in any browser and catch most problems.
Why might a firewall or CDN block AI crawlers?
Because blocking is now the default in some places. On July 1, 2025, Cloudflare began blocking known AI crawlers by default for every new domain, according to its announcement. Cloudflare sits in front of roughly one in five websites, so a lot of owners may be affected without ever having chosen it.
Security plugins and managed hosts can do the same thing under names like "bot fight mode" or "AI scrapers." A publisher protecting its articles may be right to block. A local business that wants to be recommended is not, so make that choice yourself instead of inheriting it.
What should I fix first?
Fix access before content. The best-written page does nothing if the crawler gets a 403, so work in this order: remove blocks, then make sure key facts sit in the plain HTML, then add structured data. Each step only matters once the one before it passes.
For a local business, the facts to check in raw HTML are the ones customers ask about: what you do, where you are, when you are open, and what it costs. If a developer needs to change something, hand them the curl result as proof. It ends the "works fine for me" argument quickly.
A caveat worth stating: passing this check gets you read, and nothing more. No tool or tweak can promise an AI engine will cite you. Reachability gets you into the pool of sources an engine can pick from. It does not pick you.
How do I check all of this without the terminal?
Run an audit that does the fetching for you. GeoRoar's free audit requests your public pages, checks crawler access and robots.txt, and scores what it finds on a fixed 100-point rubric. If something fails, the $99 Fix Pack generates the corrected files, and Monitor ($19 a month) watches for regressions after a site update.
Is AI ignoring your site?
Here's a sample GeoRoar scorecard. Yours takes about 30 seconds.
This is a sample. Your real scorecard is free, no sign-up.
Run your free GEO audit