Who’s crawling your site.
The crawlers, AI bots and link preview fetchers that read websites: who runs each one, what it does with your pages, whether to block it and how to tell a real one from a fake. The same list invisit uses to sort crawler traffic.
No bot by that name. Paste its user agent into the user agent lookup instead.
AI training
Collects pages to train language models. Blocking it keeps your pages out of future training data, and does not remove you from AI search answers.
AI search
Indexes pages so an AI search product can cite and link them in answers. Blocking it usually means you stop showing up as a source there.
AI assistants
Fetches a page because a person asked an assistant about it, right now. Operators often say robots.txt does not apply to these user-triggered fetches.
Search engines
Crawls pages for a classic search index. Blocking it removes you from that search engine.
SEO tools
Crawls for an SEO product: backlink indexes, site audits, rank data. It sends no visitors; blocking it is common and harmless.
Link previews
Fetches a page when someone shares its link, to draw the preview card. Blocking it breaks your previews in that app.
Archivers
Saves copies of pages for a public archive.
Other
A bot that fits none of the groups above, such as an ad review crawler.