What is Twitterbot?
X's crawler that reads Cards markup on linked pages to build preview cards in posts.
Fetches a page when someone shares its link, to draw the preview card. Blocking it breaks your previews in that app.
These facts come from X's archived 2023 developer docs; the page has since been removed. Those docs said blocked pages get no card, blocked images get no thumbnail, and cards are cached for 7 days.
Twitterbot user agent
X documents this user agent for Twitterbot. Version numbers in it change over time, so match on the name, not the whole string.
Twitterbot/1.0Should you block Twitterbot?
Allow it. Blocking it breaks the preview card when someone shares your link.
How to block Twitterbot in robots.txt
Add a group for its token to robots.txt at the root of your site.
A bot follows only the group that names it, so it ignores your User-agent: * rules once it has a group of its own. Repeat any paths you also want kept from it there.
User-agent: Twitterbot
Disallow: /How to verify a request is really Twitterbot
Anyone can send any user agent, so the string alone proves nothing.
X publishes no IP ranges or DNS names for Twitterbot that we could find, so there is no reliable way to tell a real Twitterbot from a scraper using its user agent.
See when Twitterbot reads your site
Bots don’t run JavaScript analytics, so a script tag never sees them. invisit records crawler reads on the server instead, through its server SDK or a Cloudflare Worker, and shows which bots read which pages, next to your visitors.
Checked against X’s own documentation on . Something changed? Write to help@invisit.dev.