AI crawlers in server logs: what a new site saw in its first 18 days

These are real numbers from our own web server's access log, not estimates and not a vendor's figures. The method and limits are below, because they decide how far the numbers can be used. No visitor IP addresses are published, and our own test requests were removed.

What each crawler did

Requests as logged by the web server between 30 September 2026 08:26 and 18 October 2026 08:04 (server time, UTC-4), after removing our own test requests. Counted by user agent name, which is not verified. A 304 means "not modified"; 301 is a redirect, mostly http to https.

Eight observations

1. OpenAI was the most active AI visitor. OAI-SearchBot, the crawler that surfaces sites in ChatGPT search, arrived on the first day. ChatGPT-User, which fetches a page when a user asks, appeared from 4 October on every one of the following days we logged, from 184 distinct IP addresses.
2. ChatGPT-User fetched almost only the home page. 187 of its 195 requests were for the home page, and the other eight went to five inner pages. We do not know why.
3. OAI-SearchBot read robots.txt before anything else. 62 of its 165 requests were for robots.txt. Its most fetched pages were the AI shopping visibility roundup (14), the Otterly vs Peec comparison (13) and the SEO MCP servers roundup (12).
4. ChatGPT sent measurable visits. 26 requests carried utm_source=chatgpt.com, from 16 distinct IP addresses, on two pages: the AI shopping roundup (15) and the Otterly vs Peec comparison (11). Those are also two of OAI-SearchBot's most fetched pages. That is a correlation in one small log, not proof of cause.
5. Anthropic's search crawlers made no requests. ClaudeBot and Claude-SearchBot never appeared, while Claude-User appeared twice. robots.txt allows all of them. Anthropic's documentation says its bots honor robots.txt; the log does not say why they did not visit.
6. Perplexity came late and lightly. PerplexityBot first appeared on 11 October, 11 days in, with 8 requests.
7. Rate limits hit some crawlers. GPTBot received 429 twice, and Meta's crawlers 23 times out of 41 requests. A 429 means too many requests, which points to a rate limit at the host or CDN. A crawler you want that keeps getting 429 deserves a look at those rules.
8. Search engines still dominated. Googlebot (202) and Bingbot (180) together made more requests than all AI crawlers except OpenAI's two. Google also sent 112 page views with a google.com referrer, led by the AI shopping roundup (42) and the Otterly vs Peec comparison (14). Googlebot's 11 404 responses were four for favicon.ico and seven for paths that do not exist here, such as /wp-json/ and /blog/.

Most of the log is not a reader or a crawler you want

Roughly half of all logged requests were 403 or 404 responses, mostly automated probes for files such as /.env, /.git/config and cloud credentials. They are not AI crawlers and they hit every public site; the point for anyone reading logs is to filter them out before counting anything. The 403 responses show the server protecting sensitive paths, which is the correct behavior.

Method and limits

• One site, 18 days, a new domain. Crawlers discover new sites in different ways and at different speeds, so this is not a benchmark for established sites.
• Counts are requests, not people or visits. One page view can be many requests.
• User agents are unverified. A name in the log can be faked. We did not verify crawler IP addresses against the vendors' published ranges.
• Own tests removed. About 550 of 12,528 logged requests were our own test requests and were excluded.
• CDN effects. The site sits behind a CDN. Requests answered by the CDN without reaching the server do not appear in this log. Our HTML pages are not cached there.
• ChatGPT tags went missing for about a week. Analysts reported that from about 9 October ChatGPT links stopped carrying utm_source=chatgpt.com, and that tagged clicks returned from 15 October. We found no official OpenAI statement. Our 26 tagged requests are therefore a floor: visits in that window may have arrived untagged, as direct traffic. See our summary, based on Search Engine Journal and Search Engine Roundtable.
• Cause is unknown. The log shows what arrived and what the server answered. It does not show why a crawler visited, or whether a page was cited in an answer.

Run it on your own site

With a standard access log (combined format), these commands count requests by crawler and status, and count ChatGPT-tagged visits. Remove your own test requests first.

# requests by status code for one crawler
zcat access.log.gz | awk -F'"' '$6 ~ /OAI-SearchBot/ {split($3, s, " "); print s[1]}' | sort | uniq -c

# visits that carry the ChatGPT tag, by page
zcat access.log.gz | grep 'utm_source=chatgpt.com' | awk '{print $7}' | sed 's/?.*//' | sort | uniq -c | sort -rn

If a crawler you want shows 403, 406 or 429, AI crawlers blocked by your server explains how to find the layer, and the AI search checklist covers which bots to allow. To measure the same thing from Google's and Microsoft's side, see Search Console and Bing AI reports.

Citing this data

You are welcome to quote the figures with a link to this page. Suggested wording: "In 18 days of server logs from a new site, OpenAI's OAI-SearchBot made 165 requests and ChatGPT-User 195, while ClaudeBot and Claude-SearchBot made none (CheaperStack, October 2026)." We will update this page if we publish a later period.

Frequently asked questions

Sources

Our own web server access log for cheaperstack.com, 30 September to 18 October 2026, analysed on 19 October 2026. Crawler descriptions are from the documentation of OpenAI, Anthropic and Perplexity.

Related

Last reviewed 22 October 2026. Spotted a mistake in the numbers? Tell me.