What is AI2Bot?
It explores certain domains to find web content used to train the Allen Institute for AI's open language models.
Collects pages to train language models. Blocking it keeps your pages out of future training data, and does not remove you from AI search answers.
Ai2's crawler page does not mention robots.txt. It says the user agent can be used to filter or reject its traffic.
AI2Bot user agent
Allen Institute for AI documents this user agent for AI2Bot. Version numbers in it change over time, so match on the name, not the whole string.
Mozilla/5.0 (compatible) AI2Bot (+https://www.allenai.org/crawler)Should you block AI2Bot?
Block it if you don’t want your pages used to train models; nothing else changes. Allow it if you’d rather future models know your product, docs or writing.
How to block AI2Bot in robots.txt
Allen Institute for AI documents no robots.txt token for AI2Bot that we could confirm, so we don’t suggest one here. Block it at your server or firewall by user agent instead.
How to verify a request is really AI2Bot
Anyone can send any user agent, so the string alone proves nothing.
Allen Institute for AI publishes no IP ranges or DNS names for AI2Bot that we could find, so there is no reliable way to tell a real AI2Bot from a scraper using its user agent.
See when AI2Bot reads your site
Bots don’t run JavaScript analytics, so a script tag never sees them. invisit records crawler reads on the server instead, through its server SDK or a Cloudflare Worker, and shows which bots read which pages, next to your visitors.
Checked against Allen Institute for AI’s own documentation on . Something changed? Write to help@invisit.dev.