In the seven days from 6 to 13 November 2026, requests carrying the name of an AI crawler hit jsbcollaborative.com 472 times. A user agent is only a label, and anyone can type one, so we checked every request we could against the method each company publishes for proving its crawler is real. Three findings came out of it. OpenAI’s search crawler was the most frequent AI visitor we could verify. 16 of the 105 requests that called themselves Googlebot were fakes. And not one AI crawler asked for an llms.txt file.

Which AI crawlers visited, and how many could be verified?

OpenAI and Perplexity publish the IP ranges their crawlers use. Google, Bing and Apple let you confirm a crawler with a reverse DNS lookup. A request that passes is “verified.” A request whose user agent names one of those crawlers but whose address does not belong to the company is “fake.” For the other crawlers we had no published method to check against, so they are counted as “claimed.”

Crawler Operator Verified Fake Claimed (no way to verify)
ClaudeBot Anthropic – – 160
OAI-SearchBot OpenAI (ChatGPT search) 103 0 –
Googlebot Google 89 16 –
Applebot Apple 65 2 –
Bytespider ByteDance – – 59
GPTBot OpenAI (training) 54 0 –
Bingbot Microsoft 52 0 –
meta-externalagent Meta – – 26
Amazonbot Amazon – – 26
Claude-User Anthropic – – 19
PerplexityBot Perplexity 8 2 –
DuckAssistBot DuckDuckGo – – 7
ChatGPT-User OpenAI (a person asking ChatGPT) 3 4 –

This is one small business website over one week. It is a measurement, not a survey of the web.

How many “Googlebot” requests were fake?

16 of the 105 requests that said they were Googlebot came from addresses that do not belong to Google. That is about 15 percent. The fakes were not limited to Google. Four of the seven requests labelled ChatGPT-User were fake, and so were two of the ten labelled PerplexityBot.

Several of these requests were not crawling at all. Requests wearing ChatGPT-User, PerplexityBot, Claude-User and DuckAssistBot labels asked for files such as .env and /proc/self/environ, which hold passwords and server settings on a badly configured site. That is a security scanner borrowing a trusted name. If you count AI traffic by user agent alone, these scanners get counted as AI interest.

What did the AI crawlers actually read?

Mostly the rules, not the pages. 79 of OpenAI’s 103 verified search-crawler requests were for robots.txt, the file that tells crawlers what they may fetch. Among real pages, OAI-SearchBot went to our Cannabis SEO page more than any other: 14 requests to its current address and 10 more to an old address that redirects to it. Googlebot requested it 22 times across both addresses, more than it requested the homepage. GPTBot, OpenAI’s training crawler, went to the homepage most often, 11 of its 54 requests.

Did any AI crawler request llms.txt?

No. During the week, /llms.txt was requested by SEO tools, link checkers and a few web browsers. GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Googlebot never asked for it, even while they were fetching other files on the same site. We did not have an llms.txt file that week. We published one on 13 November 2026, and we will update this page with whether any AI crawler starts requesting it.

What this means if you run a small business website

  • If you want to appear in ChatGPT search, do not block OAI-SearchBot in robots.txt. On our site it was the most frequent AI crawler we could verify.
  • Verify before you trust a crawler report. A dashboard that counts user agents counts the fakes too, including scanners looking for passwords.
  • Do not expect an llms.txt file to be read yet. We saw no evidence of it this week, and we will report what changes.

How we checked

We read the web server’s own access logs for 6 November 2026 00:00 to 13 November 2026 00:00 UTC. Googlebot, Bingbot and Applebot requests were checked with a reverse DNS lookup and a matching forward lookup. OpenAI and Perplexity requests were checked against the IP lists those companies publish. Anthropic, ByteDance, Meta, Amazon and DuckDuckGo requests are reported as claimed because we had no published method to verify them against. Counts are requests, not unique visitors.