What is Internet Archive?
The Internet Archive's crawler for its wide crawl of the web, which preserves pages for the Wayback Machine.
Saves copies of pages for a public archive.
The Archive says some pages are not archived because of robots.txt, but in 2017 it stopped honoring robots.txt on US government and military sites and said it would go further. It documents archive.org_bot as its crawler name but no robots.txt token.
Should you block Internet Archive?
Allow it if you’re happy for old versions of your pages to be kept in a public archive.
How to block Internet Archive in robots.txt
Internet Archive documents no robots.txt token for Internet Archive that we could confirm, so we don’t suggest one here. Block it at your server or firewall by user agent instead.
To exclude existing captures, email info@archive.org with the URLs, the time period, and when you controlled the site; the Archive reviews each request without guarantees.
How to verify a request is really Internet Archive
Anyone can send any user agent, so the string alone proves nothing.
Internet Archive publishes no IP ranges or DNS names for Internet Archive that we could find, so there is no reliable way to tell a real Internet Archive from a scraper using its user agent.
See when Internet Archive reads your site
Bots don’t run JavaScript analytics, so a script tag never sees them. invisit records crawler reads on the server instead, through its server SDK or a Cloudflare Worker, and shows which bots read which pages, next to your visitors.
Checked against Internet Archive’s own documentation on . Something changed? Write to help@invisit.dev.