[invisit](https://invisit.dev) · [Docs](https://invisit.dev/docs) · [llms.txt](https://invisit.dev/llms.txt) · [llms-full.txt](https://invisit.dev/llms-full.txt)

# Internet Archive

> Internet Archive is a web archiver run by Internet Archive. The Internet Archive's crawler for its wide crawl of the web, which preserves pages for the Wayback Machine.

- Operator: Internet Archive
- Kind: Web archiver
- robots.txt token: none documented
- Follows robots.txt: partly
- Source: https://help.archive.org/help/using-the-wayback-machine/ (checked 2026-11-05)
- Page: https://invisit.dev/bots/archive-org

## What is Internet Archive?

The Internet Archive's crawler for its wide crawl of the web, which preserves pages for the Wayback Machine.

Saves copies of pages for a public archive.

The Archive says some pages are not archived because of robots.txt, but in 2017 it stopped honoring robots.txt on US government and military sites and said it would go further. It documents archive.org_bot as its crawler name but no robots.txt token.

- [Internet Archive’s documentation](https://help.archive.org/help/using-the-wayback-machine/)

## Should you block Internet Archive?

Allow it if you’re happy for old versions of your pages to be kept in a public archive.

## How to block Internet Archive in robots.txt

Internet Archive documents no robots.txt token for Internet Archive that we could confirm, so we don’t suggest one here. Block it at your server or firewall by user agent instead.

To exclude existing captures, email info@archive.org with the URLs, the time period, and when you controlled the site; the Archive reviews each request without guarantees.

- [Build a robots.txt for AI bots](https://invisit.dev/tools/robots-txt-generator)

## How to verify a request is really Internet Archive

Anyone can send any user agent, so the string alone proves nothing.

Internet Archive publishes no IP ranges or DNS names for Internet Archive that we could find, so there is no reliable way to tell a real Internet Archive from a scraper using its user agent.

## See when Internet Archive reads your site

Bots don’t run JavaScript analytics, so a script tag never sees them. invisit records crawler reads on the server instead, through its server SDK or a Cloudflare Worker, and shows which bots read which pages, next to your visitors.

- [Track crawlers with invisit](https://invisit.dev/docs/server/sdk#crawler-reads)

## Common questions

### What is Internet Archive?

Internet Archive is a web archiver run by Internet Archive. The Internet Archive's crawler for its wide crawl of the web, which preserves pages for the Wayback Machine.

### Is Internet Archive safe to block?

Allow it if you’re happy for old versions of your pages to be kept in a public archive.

### How do I block Internet Archive?

Internet Archive documents no robots.txt token for it that we could confirm, so block it at your server or firewall by its user agent.