Are AI crawlers blocked by your server? A 5-minute check robots.txt will not give you

This is a documented guide. The bot facts come from OpenAI, Anthropic, Perplexity, Google and Cloudflare documentation, read in October 2026. The test is a simulation of the user agent only; the limits are listed below, and the worked example is a check we ran on our own site. Run it only on sites you own or manage.

Four places a crawler can be refused

Layers two to four are invisible to robots.txt checkers and SEO plugins, which is why a site can look correct and still return 403 to a crawler.

The crawlers worth testing, and what each is for

From OpenAI's bots page, Anthropic's Help Center article on its crawlers (updated April 2026) and Perplexity's bots guide. Each setting works independently, so you can appear in search and still opt out of training. Which side of that line to pick is the decision in the AI search checklist.

The 5-minute test

First, a normal request, so you have a baseline. Replace the address with your own:

curl -s -o /dev/null -w "%{http_code}\n" https://yoursite.com/

Then the same request with each bot's name in the user agent. This loop covers the bots in the table:

for ua in "OAI-SearchBot/1.0" "GPTBot/1.4" "ChatGPT-User/1.0" \
          "PerplexityBot/1.0" "Perplexity-User/1.0" \
          "ClaudeBot/1.0" "Claude-SearchBot/1.0" "Claude-User/1.0"; do
  printf "%-22s" "$ua"
  curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; $ua)" https://yoursite.com/
done

Repeat for an inner page, your CSS file, robots.txt and, if you publish one, llms.txt. Some rules are path-specific, and the worked example below found one that was.

If a bot is refused, find the layer

1. Test the origin directly. Send the same request straight to your server's IP, bypassing the CDN. If the origin answers 200 and the public site answers 403, the CDN or firewall is the layer. If both refuse, it is the host or the application. Use curl --resolve yoursite.com:443:ORIGIN_IP -k with the bot's user agent.
2. Check the CDN. On Cloudflare, look at Security events filtered by the bot's name, and at AI Crawl Control, whose Crawlers tab lists AI crawlers with a Block action. Cloudflare also has Configure AI bot policies under Security Settings, with Search, Agent and Training categories, each set to Block, Block on pages with ads, or Allow.
3. Check the host. Look at the control panel's firewall or security section and at the access and error logs for the refused requests.
4. Check the application. A security plugin or a user-agent rule in .htaccess can do this too.

Note a Cloudflare change that affects the decision. Its documentation says that from 15 September 2026, new domains block bots classed as Training or Agent on pages that show ads, and that its Block options apply to mixed-use crawlers such as Googlebot and Bingbot, so a Block setting can affect search as well as training. Read the options before you switch one on, and check what applies to your domain.

What this test cannot tell you

• It only sends a name. Real crawlers also come from published IP ranges, and some firewalls check both. A 200 on a simulated request does not prove the real crawler gets through, and a simulated 403 may not apply to it.
• It cannot prove a bot is who it says. A user agent is easy to fake. Google documents verifying Googlebot with a reverse and forward DNS lookup or its published IP ranges; Anthropic publishes a list of its crawler IPs; Perplexity recommends allowing its bots by user agent combined with its published IP ranges.
• Blocking by IP can backfire. Anthropic notes that IP-based blocking may not work reliably, because it stops the bot from reading robots.txt.
• Your real evidence is the log. Search the server access log for each bot's name and look at the status codes it actually received.

A worked example: our own check

On 16 October 2026 we ran this test on cheaperstack.com for more than 25 user agents, including every bot in the table. All of them received 200 and the full page, including GPTBot, ClaudeBot, PerplexityBot and CCBot. One did not: Bytespider, ByteDance's crawler, got a 403 on the home page and the CSS file, and at first on some inner pages, and 200 on robots.txt, llms.txt and the sitemap. The 403 page came from the web server. The same request sent straight to the origin returned 200, so the rule sits between the two and not in the site's files.

That is the situation the check exists for: our robots.txt allows Bytespider, and something below it refused. Whether to allow it is a policy choice; the lesson is that the choice should be made in robots.txt and in your firewall at the same time, not by accident in one of them.

What an SEO plugin can and cannot see

Rank Math's MCP tools include a read-only Get robots.txt tool and a Get llms.txt tool, so an assistant can read what your site declares. Neither shows what a firewall does with a request, which is why the curl test and the logs are the checks that matter. The connection steps are in Rank Math MCP with Claude, and the free plugin covers them. Get Rank Math (affiliate link).

Who can skip this

• Sites on a managed platform with no firewall settings. Ask the host what it blocks by default instead.
• Anyone who deliberately blocks AI crawlers. Then verify the block works, and record that it is on purpose.
• New sites with no pages indexed yet. Fix indexing first; see the 90-day plan.

The verdict, by situation

Frequently asked questions

Sources

OpenAI's bots documentation; Anthropic's Help Center article on its crawlers; Perplexity's bots guide; Google's guide to verifying Googlebot; and Cloudflare's Block AI Bots and AI Crawl Control documentation. These settings change often, so check the current pages and your own dashboard.

Related

Last reviewed 19 October 2026. Bot names and CDN settings change often, so this page is re-checked monthly. Spotted one out of date? Tell me.