What is Meta-ExternalFetcher?
It fetches individual links at a user's request and supports agentic AI features, including helping AI navigate websites to complete tasks for users.
Fetches a page because a person asked an assistant about it, right now. Operators often say robots.txt does not apply to these user-triggered fetches.
Meta says this crawler may bypass robots.txt because the fetches are requested by users. Meta crawlers may cache robots.txt for up to 24 hours.
Meta-ExternalFetcher user agent
Meta documents these user agents for Meta-ExternalFetcher. Version numbers in it change over time, so match on the name, not the whole string.
meta-externalfetcher/1.1 (+/documentation/sharing/webmasters/web-crawlers)
meta-externalfetcher/1.1Should you block Meta-ExternalFetcher?
Usually allow it. Each fetch is a person asking about your page right now, which is close to a visit. Blocking it in robots.txt may not stop it, since operators often treat these fetches as the person’s own.
How to block Meta-ExternalFetcher in robots.txt
Add a group for its token to robots.txt at the root of your site. Meta says it follows robots.txt only in part, so this may not stop every fetch.
A bot follows only the group that names it, so it ignores your User-agent: * rules once it has a group of its own. Repeat any paths you also want kept from it there.
User-agent: meta-externalfetcher
Disallow: /How to verify a request is really Meta-ExternalFetcher
Anyone can send any user agent, so the string alone proves nothing.
Meta publishes no IP ranges or DNS names for Meta-ExternalFetcher that we could find, so there is no reliable way to tell a real Meta-ExternalFetcher from a scraper using its user agent.
See when Meta-ExternalFetcher reads your site
Bots don’t run JavaScript analytics, so a script tag never sees them. invisit records crawler reads on the server instead, through its server SDK or a Cloudflare Worker, and shows which bots read which pages, next to your visitors.
Checked against Meta’s own documentation on . Something changed? Write to help@invisit.dev.