What is SemrushBot?
It collects web data for Semrush tools, including its Backlink Analytics index, Site Audit and other SEO tools.
Crawls for an SEO product: backlink indexes, site audits, rank data. It sends no visitors; blocking it is common and harmless.
Semrush uses a different token for each tool, so blocking SemrushBot alone does not block them all. It says not to block it by IP because it uses no consecutive IP blocks, and changes can take up to an hour or 100 requests to apply.
Should you block SemrushBot?
Block it freely if the load bothers you. It sends no visitors; the only cost is that the tool’s users, your competitors included, see less of your site.
How to block SemrushBot in robots.txt
Add a group for its tokens to robots.txt at the root of your site.
A bot follows only the group that names it, so it ignores your User-agent: * rules once it has a group of its own. Repeat any paths you also want kept from it there.
User-agent: SemrushBot
User-agent: SiteAuditBot
User-agent: SemrushBot-BA
User-agent: SemrushBot-SI
User-agent: SemrushBot-SWA
User-agent: SplitSignalBot
User-agent: SemrushBot-OCOB
User-agent: SemrushBot-FT
User-agent: RyteBot
User-agent: SemrushBot-ESI
Disallow: /How to verify a request is really SemrushBot
Anyone can send any user agent, so the string alone proves nothing.
Semrush publishes no IP ranges or DNS names for SemrushBot that we could find, so there is no reliable way to tell a real SemrushBot from a scraper using its user agent.
See when SemrushBot reads your site
Bots don’t run JavaScript analytics, so a script tag never sees them. invisit records crawler reads on the server instead, through its server SDK or a Cloudflare Worker, and shows which bots read which pages, next to your visitors.
Checked against Semrush’s own documentation on . Something changed? Write to help@invisit.dev.