[invisit](https://invisit.dev) · [Docs](https://invisit.dev/docs) · [llms.txt](https://invisit.dev/llms.txt) · [llms-full.txt](https://invisit.dev/llms-full.txt) # Bot directory > The crawlers, AI bots and link preview fetchers that read websites: who runs each one, what it does with your pages, whether to block it and how to tell a real one from a fake. The same list invisit uses to sort crawler traffic. ## AI training Collects pages to train language models. Blocking it keeps your pages out of future training data, and does not remove you from AI search answers. - [AI2Bot](https://invisit.dev/bots/ai2bot.md): Allen Institute for AI - [Amazonbot](https://invisit.dev/bots/amazonbot.md): Amazon - [Bytespider](https://invisit.dev/bots/bytespider.md): ByteDance - [CCBot](https://invisit.dev/bots/ccbot.md): Common Crawl - [ClaudeBot](https://invisit.dev/bots/claudebot.md): Anthropic - [GPTBot](https://invisit.dev/bots/gptbot.md): OpenAI - [Meta-ExternalAgent](https://invisit.dev/bots/meta-externalagent.md): Meta - [MistralAI-Training](https://invisit.dev/bots/mistralai-training.md): Mistral AI - [Timpibot](https://invisit.dev/bots/timpibot.md): Timpi - [Webz.io](https://invisit.dev/bots/webzio.md): Webz.io ## AI search Indexes pages so an AI search product can cite and link them in answers. Blocking it usually means you stop showing up as a source there. - [Amzn-SearchBot](https://invisit.dev/bots/amzn-searchbot.md): Amazon - [Claude-SearchBot](https://invisit.dev/bots/claude-searchbot.md): Anthropic - [Google-CloudVertexBot](https://invisit.dev/bots/google-cloudvertexbot.md): Google - [Meta-WebIndexer](https://invisit.dev/bots/meta-webindexer.md): Meta - [MistralAI-Index](https://invisit.dev/bots/mistralai-index.md): Mistral AI - [OAI-SearchBot](https://invisit.dev/bots/oai-searchbot.md): OpenAI - [PerplexityBot](https://invisit.dev/bots/perplexitybot.md): Perplexity - [YandexAdditionalBot](https://invisit.dev/bots/yandex-additional.md): Yandex - [YouBot](https://invisit.dev/bots/youbot.md): You.com ## AI assistants Fetches a page because a person asked an assistant about it, right now. Operators often say robots.txt does not apply to these user-triggered fetches. - [Amzn-User](https://invisit.dev/bots/amzn-user.md): Amazon - [ChatGPT-User](https://invisit.dev/bots/chatgpt-user.md): OpenAI - [Claude-User](https://invisit.dev/bots/claude-user.md): Anthropic - [DuckAssistBot](https://invisit.dev/bots/duckassistbot.md): DuckDuckGo - [Google-Agent](https://invisit.dev/bots/google-agent.md): Google - [Google-GeminiNotebook](https://invisit.dev/bots/google-gemininotebook.md): Google - [Meta-ExternalFetcher](https://invisit.dev/bots/meta-externalfetcher.md): Meta - [MistralAI-User](https://invisit.dev/bots/mistralai-user.md): Mistral AI - [Perplexity-User](https://invisit.dev/bots/perplexity-user.md): Perplexity ## Search engines Crawls pages for a classic search index. Blocking it removes you from that search engine. - [360Spider](https://invisit.dev/bots/360spider.md): Qihoo 360 - [Applebot](https://invisit.dev/bots/applebot.md): Apple - [Baiduspider](https://invisit.dev/bots/baiduspider.md): Baidu - [Bingbot](https://invisit.dev/bots/bingbot.md): Microsoft - [Bravebot](https://invisit.dev/bots/bravebot.md): Brave - [coccocbot](https://invisit.dev/bots/coccocbot.md): Cốc Cốc - [Daum](https://invisit.dev/bots/daum.md): Kakao - [Diffbot](https://invisit.dev/bots/diffbot.md): Diffbot - [DuckDuckBot](https://invisit.dev/bots/duckduckbot.md): DuckDuckGo - [Googlebot](https://invisit.dev/bots/googlebot.md): Google - [Kagibot](https://invisit.dev/bots/kagibot.md): Kagi - [MojeekBot](https://invisit.dev/bots/mojeekbot.md): Mojeek - [PetalBot](https://invisit.dev/bots/petalbot.md): Huawei - [Qwantbot](https://invisit.dev/bots/qwant.md): Qwant - [SeekportBot](https://invisit.dev/bots/seekportbot.md): Seekport - [SeznamBot](https://invisit.dev/bots/seznambot.md): Seznam - [Sogou spider](https://invisit.dev/bots/sogou.md): Sogou - [Storebot-Google](https://invisit.dev/bots/storebot-google.md): Google - [Yahoo! Slurp](https://invisit.dev/bots/yahoo-slurp.md): Yahoo - [YandexBot](https://invisit.dev/bots/yandexbot.md): Yandex - [Yeti](https://invisit.dev/bots/naver-yeti.md): Naver ## SEO tools Crawls for an SEO product: backlink indexes, site audits, rank data. It sends no visitors; blocking it is common and harmless. - [AhrefsBot](https://invisit.dev/bots/ahrefsbot.md): Ahrefs - [Barkrowler](https://invisit.dev/bots/barkrowler.md): Babbar - [BLEXBot](https://invisit.dev/bots/blexbot.md): SE Ranking - [DataForSeoBot](https://invisit.dev/bots/dataforseobot.md): DataForSEO - [DotBot](https://invisit.dev/bots/dotbot.md): Moz - [Google-InspectionTool](https://invisit.dev/bots/google-inspectiontool.md): Google - [MJ12bot](https://invisit.dev/bots/mj12bot.md): Majestic - [Screaming Frog SEO Spider](https://invisit.dev/bots/screaming-frog.md): Screaming Frog - [SemrushBot](https://invisit.dev/bots/semrushbot.md): Semrush - [SerpstatBot](https://invisit.dev/bots/serpstatbot.md): Serpstat - [Sitebulb](https://invisit.dev/bots/sitebulb.md): Sitebulb ## Link previews Fetches a page when someone shares its link, to draw the preview card. Blocking it breaks your previews in that app. - [BingPreview](https://invisit.dev/bots/bingpreview.md): Microsoft - [Discordbot](https://invisit.dev/bots/discordbot.md): Discord - [facebookexternalhit](https://invisit.dev/bots/facebookexternalhit.md): Meta - [GoogleMessages](https://invisit.dev/bots/googlemessages.md): Google - [LinkedInBot](https://invisit.dev/bots/linkedinbot.md): LinkedIn - [MicrosoftPreview](https://invisit.dev/bots/microsoftpreview.md): Microsoft - [Pinterestbot](https://invisit.dev/bots/pinterestbot.md): Pinterest - [redditbot](https://invisit.dev/bots/redditbot.md): Reddit - [Slackbot](https://invisit.dev/bots/slackbot.md): Slack - [TelegramBot](https://invisit.dev/bots/telegrambot.md): Telegram - [Twitterbot](https://invisit.dev/bots/twitterbot.md): X - [WhatsApp](https://invisit.dev/bots/whatsapp.md): Meta ## Archivers Saves copies of pages for a public archive. - [Internet Archive](https://invisit.dev/bots/archive-org.md): Internet Archive ## Other A bot that fits none of the groups above, such as an ad review crawler. - [AdIdxBot](https://invisit.dev/bots/adidxbot.md): Microsoft - [AdsBot-Google](https://invisit.dev/bots/google-ads.md): Google - [APIs-Google](https://invisit.dev/bots/apis-google.md): Google - [FeedFetcher-Google](https://invisit.dev/bots/google-feedfetcher.md): Google - [Google-Read-Aloud](https://invisit.dev/bots/google-read-aloud.md): Google - [Google-Site-Verification](https://invisit.dev/bots/google-site-verification.md): Google - [GoogleOther](https://invisit.dev/bots/google-other.md): Google - [Meta-ExternalAds](https://invisit.dev/bots/meta-externalads.md): Meta - [OAI-AdsBot](https://invisit.dev/bots/oai-adsbot.md): OpenAI - [TikTokSpider](https://invisit.dev/bots/tiktokspider.md): ByteDance