[invisit](https://invisit.dev) · [Docs](https://invisit.dev/docs) · [llms.txt](https://invisit.dev/llms.txt) · [llms-full.txt](https://invisit.dev/llms-full.txt)

# Meta-WebIndexer

> Meta-WebIndexer is an AI search crawler run by Meta. It crawls the web to improve Meta AI search results. Meta says allowing it helps Meta cite and link to your content in Meta AI responses.

- Operator: Meta
- Kind: AI search crawler
- robots.txt token: Meta-WebIndexer
- Follows robots.txt: yes
- Source: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers (checked 2026-11-05)
- Page: https://invisit.dev/bots/meta-webindexer

## What is Meta-WebIndexer?

It crawls the web to improve Meta AI search results. Meta says allowing it helps Meta cite and link to your content in Meta AI responses.

Indexes pages so an AI search product can cite and link them in answers. Blocking it usually means you stop showing up as a source there.

Meta crawlers may cache robots.txt for up to 24 hours.

- [Meta’s documentation](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)

## Meta-WebIndexer user agent

Meta documents these user agents for Meta-WebIndexer. Version numbers in it change over time, so match on the name, not the whole string.

```
meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers)
meta-webindexer/1.1
```

## Should you block Meta-WebIndexer?

Usually allow it. It is how an AI search product finds pages to cite and link, so blocking it trades away being a source in those answers.

## How to block Meta-WebIndexer in robots.txt

Add a group for its token to robots.txt at the root of your site.

A bot follows only the group that names it, so it ignores your `User-agent: *` rules once it has a group of its own. Repeat any paths you also want kept from it there.

```
User-agent: Meta-WebIndexer
Disallow: /
```

- [Build a robots.txt for AI bots](https://invisit.dev/tools/robots-txt-generator)

## How to verify a request is really Meta-WebIndexer

Anyone can send any user agent, so the string alone proves nothing.

Meta publishes no IP ranges or DNS names for Meta-WebIndexer that we could find, so there is no reliable way to tell a real Meta-WebIndexer from a scraper using its user agent.

## See when Meta-WebIndexer reads your site

Bots don’t run JavaScript analytics, so a script tag never sees them. invisit records crawler reads on the server instead, through its server SDK or a Cloudflare Worker, and shows which bots read which pages, next to your visitors.

- [Track crawlers with invisit](https://invisit.dev/docs/server/sdk#crawler-reads)

## Common questions

### What is Meta-WebIndexer?

Meta-WebIndexer is an AI search crawler run by Meta. It crawls the web to improve Meta AI search results. Meta says allowing it helps Meta cite and link to your content in Meta AI responses.

### Is Meta-WebIndexer safe to block?

Usually allow it. It is how an AI search product finds pages to cite and link, so blocking it trades away being a source in those answers.

### How do I block Meta-WebIndexer?

Add “User-agent: Meta-WebIndexer” followed by “Disallow: /” to your robots.txt.