What is facebookexternalhit?
It fetches apps and websites shared on Meta's apps, such as Facebook, Instagram and Messenger, to show their title, description and thumbnail.
Fetches a page when someone shares its link, to draw the preview card. Blocking it breaks your previews in that app.
Meta says it might bypass robots.txt for security or integrity checks, such as checking for malware.
facebookexternalhit user agent
Meta documents these user agents for facebookexternalhit. Version numbers in it change over time, so match on the name, not the whole string.
facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)
facebookexternalhit/1.1Should you block facebookexternalhit?
Allow it. Blocking it breaks the preview card when someone shares your link.
How to block facebookexternalhit in robots.txt
Add a group for its token to robots.txt at the root of your site. Meta says it follows robots.txt only in part, so this may not stop every fetch.
A bot follows only the group that names it, so it ignores your User-agent: * rules once it has a group of its own. Repeat any paths you also want kept from it there.
Meta's Sharing Debugger can force a fresh crawl of a URL once its content is available.
User-agent: facebookexternalhit
Disallow: /How to verify a request is really facebookexternalhit
Anyone can send any user agent, so the string alone proves nothing.
Meta publishes no IP ranges or DNS names for facebookexternalhit that we could find, so there is no reliable way to tell a real facebookexternalhit from a scraper using its user agent.
See when facebookexternalhit reads your site
Bots don’t run JavaScript analytics, so a script tag never sees them. invisit records crawler reads on the server instead, through its server SDK or a Cloudflare Worker, and shows which bots read which pages, next to your visitors.
Checked against Meta’s own documentation on . Something changed? Write to help@invisit.dev.