[invisit](https://invisit.dev) · [Docs](https://invisit.dev/docs) · [llms.txt](https://invisit.dev/llms.txt) · [llms-full.txt](https://invisit.dev/llms-full.txt)

# What AI crawlers did to our brand-new website

> Four days of crawler logs from invisit.dev: 1,912 bot requests, 41% from AI training crawlers, 127 assistant fetches and zero AI referral visits.

- Author: Miraç Berk
- Published: 2026-11-08
- Page: https://invisit.dev/blog/ai-crawlers-new-website
- Tags: ai-traffic, crawlers

![A small frosted-glass house with one indigo door, a queue of matte white robots at its step, on violet dithered ground.](https://invisit.dev/_astro/cover.DAj9uslV.png)

In about four days of logs, AI training crawlers made 41% of the 1,912 bot requests to invisit.dev, with [ClaudeBot](/bots/claudebot) and [GPTBot](/bots/gptbot) each reading more than 180 paths. Assistants fetched pages 127 times and sent 0 measured visits. Search crawlers spent most of their requests on robots.txt: 8 of OAI-SearchBot's 11 and 71 of Googlebot's 135.

This is one small new site, recording since 4 November 2026, so read it as a field report and nothing wider. Below: who showed up, what they read, what assistants fetched, why the zero needs care, how far the numbers can be trusted, and a checklist for your own first week.

## Which crawlers showed up on a brand-new website?

On invisit.dev, AI training crawlers were the largest group by far, at three times the share of classic search engines. The figures come from the 30-day report ending 8 November 2026; the Worker that records crawlers started on 4 November, so the report holds those few days.

| Kind of bot    | Share of 1,912 requests |
| -------------- | ----------------------- |
| AI training    | 41%                     |
| Other          | 30.75%                  |
| Search engines | 13.65%                  |
| AI assistants  | 6.64%                   |
| Social preview | 4.18%                   |
| SEO tools      | 2.51%                   |
| AI search      | 1.26%                   |

Other includes GoogleOther, which our classifier files there and which made 172 requests on its own.

| Bot           | Requests | Detail                |
| ------------- | -------- | --------------------- |
| ClaudeBot     | 379      | 182 different paths   |
| GPTBot        | 370      | 183 different paths   |
| GoogleOther   | 172      |                       |
| Googlebot     | 135      | 71 of them robots.txt |
| YandexBot     | 86       |                       |
| Claude-User   | 76       |                       |
| ChatGPT-User  | 51       |                       |
| Amazonbot     | 32       |                       |
| Applebot      | 28       |                       |
| PerplexityBot | 12       |                       |
| OAI-SearchBot | 11       | 8 of them robots.txt  |
| Bingbot       | 6        |                       |

Nothing stopped any of them. Our [robots.txt](https://invisit.dev/robots.txt) lists the major AI crawlers in one group with `Allow: /`, and the file has not changed since recording began. We cannot say what a site that blocks training crawlers would see.

## Did training crawlers read more than search crawlers?

Yes, by a wide margin on this site. ClaudeBot made 379 requests across 182 paths and GPTBot made 370 across 183, and GPTBot read each blog post twice. OAI-SearchBot made 11 requests in total, 8 of them for robots.txt, and the whole AI search group came to 1.26% of requests.

That gap is easy to misread. OpenAI [says](https://platform.openai.com/docs/bots) GPTBot crawls content "that may be used in training" and OAI-SearchBot is "used to surface websites in search results". It also says that if a site allows both, "we may use the results from just one crawl for both use cases to avoid duplicative crawling". So three page fetches by OAI-SearchBot does not prove ChatGPT search knows only three of our pages. It shows only what this log can show: which user agent made which request.

We do not have a robots.txt split for PerplexityBot's 12 requests, so "search crawlers mostly fetched robots.txt" rests on OAI-SearchBot and Googlebot.

## What did Googlebot do on a new website?

Just over half of Googlebot's requests were for robots.txt: 71 of 135, leaving 64 for everything else. Google [says](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt) its crawlers "download and parse the site's robots.txt file" before crawling a site, which fits the pattern. This log cannot tell us why the count is that high. GoogleOther made more requests than Googlebot, at 172, and Bingbot made 6.

In the same window Google sent 4 clicks. We draw no conclusion from a number that small except that it is small.

## What did AI assistants fetch on a new website?

Assistants mostly fetched the home page, the docs and the pricing page, 127 times in all. These are the fetchers that run when someone asks a question: OpenAI says ChatGPT-User "is not used for crawling the web in an automatic fashion", and Anthropic [says](https://support.claude.com/en/articles/8896518) that "when individuals ask questions to Claude, it may access websites using a Claude-User agent".

| Path                     | [Claude-User](/bots/claude-user) | [ChatGPT-User](/bots/chatgpt-user) |
| ------------------------ | -------------------------------- | ---------------------------------- |
| `/`                      | 14                               | 16                                 |
| `/docs`                  | 8                                | 3                                  |
| `/docs/web/installation` | 7                                |                                    |
| `/robots.txt`            | 6                                |                                    |
| `/pricing`               | 5                                | 9                                  |
| `/llms.txt`              | 3                                |                                    |

Blank cells mean the path is not among that bot's top paths in our report, which is not the same as zero.

The practical point is the shape of the list. No blog post is among these top paths; the fetches went to what the product is, how to install it and what it costs. Those are the pages worth checking as plain text. `/llms.txt` shows up 3 times, for Claude-User only, and that is all the evidence on llms.txt we have from this log.

> [!WARNING]
> Some of these fetches may be our own agents, and this log cannot separate them. Treat 127 as an upper limit on how often outsiders asked an assistant about us.

## Did AI assistants send any visitors?

No measured ones: 0 visits with an AI referrer against 127 assistant fetches in the same window. That is the number we can report, and it is weaker than it looks. A click from an assistant can arrive with no referrer and count as direct, and someone who reads a name in an answer and types it later leaves no trace at all. Our guide to [ChatGPT traffic](/blog/ai-assistant-traffic) goes through which clicks analytics can see.

Four days on a new site is not long enough to say anything about whether assistants send traffic. It says only that during these days we recorded none.

## How far can these numbers be trusted?

Only as far as one small site over about four days. The sample is 1,912 requests, and the smaller rows are tiny: one more day could move OAI-SearchBot's 11 by a large fraction.

- **One site, brand new.** Nothing here is a claim about websites in general. For that, see our [study of what sites block](/blog/how-many-sites-block-ai-crawlers), which covers robots.txt and not behaviour.
- **User agent only.** Reads were recorded by invisit, the tool we build, from a Cloudflare Worker in front of invisit.dev. It classifies each request by its user agent and does not check IP lists, so a fake GPTBot counts as GPTBot.
- **Our own agents.** Some assistant fetches may be ours, as above.
- **What the Worker sees.** It covers `invisit.dev` only. It records GET and HEAD requests for documents and skips static assets such as images and scripts, so a "request" here means a page or text file, not every byte fetched.
- **Our labels.** The kinds in the table are our classifier's. Amazonbot, for example, is filed under AI training.

## What would we do next?

We would change the way we measure before we change anything on the site. Our robots.txt says we welcome crawlers, training ones included, and nothing in this log argues for changing that. If you would rather opt out of training, the [robots.txt generator](/tools/robots-txt-generator) writes that file.

1. **Separate our own agents.** Until assistant fetches from our team can be told apart, 127 stays an upper limit.
2. **Check addresses against the operators' lists.** [How to track AI crawlers](/blog/track-ai-crawlers) has a script that does this from an access log, which our current setup cannot.
3. **Keep the fetched pages accurate.** Home, docs and pricing are what assistants read here. Every page on invisit.dev is already available as Markdown, so the next work is the wording, not the format.
4. **Rerun this over months.** The split between training and search on a four-day-old log tells us what happened that week and nothing about next quarter.

## Checklist for your own first week

- [ ] Record crawlers from day one, at the server or edge. Browser analytics will not see them, as [AI crawler user agents](/blog/ai-crawlers) explains.
- [ ] Split reads by kind: training, search and user fetches.
- [ ] Count robots.txt requests apart from page reads.
- [ ] List what assistants fetched, and read those pages as plain text.
- [ ] Set assistant fetches next to visits from AI referrers.
- [ ] Note your own agents' traffic, or say that you cannot.
- [ ] Write the site's age and the window next to every figure you share.