[invisit](https://invisit.dev) · [Docs](https://invisit.dev/docs) · [llms.txt](https://invisit.dev/llms.txt) · [llms-full.txt](https://invisit.dev/llms-full.txt)

# Screaming Frog SEO Spider

> Screaming Frog SEO Spider is a SEO tool crawler run by Screaming Frog. Software that users run to crawl and audit websites for SEO issues.

- Operator: Screaming Frog
- Kind: SEO tool crawler
- robots.txt token: Screaming Frog SEO Spider
- Follows robots.txt: partly
- Source: https://www.screamingfrog.co.uk/guides/robots-txt/ (checked 2026-11-05)
- Page: https://invisit.dev/bots/screaming-frog

## What is Screaming Frog SEO Spider?

Software that users run to crawl and audit websites for SEO issues.

Crawls for an SEO product: backlink indexes, site audits, rank data. It sends no visitors; blocking it is common and harmless.

It obeys robots.txt by default, checking rules for its own token first, then Googlebot, then *. Licensed users can ignore robots.txt or change the user agent, which Screaming Frog says is the user's responsibility.

- [Screaming Frog’s documentation](https://www.screamingfrog.co.uk/guides/robots-txt/)

## Should you block Screaming Frog SEO Spider?

Block it freely if the load bothers you. It sends no visitors; the only cost is that the tool’s users, your competitors included, see less of your site.

## How to block Screaming Frog SEO Spider in robots.txt

Add a group for its token to robots.txt at the root of your site. Screaming Frog says it follows robots.txt only in part, so this may not stop every fetch.

A bot follows only the group that names it, so it ignores your `User-agent: *` rules once it has a group of its own. Repeat any paths you also want kept from it there.

```
User-agent: Screaming Frog SEO Spider
Disallow: /
```

- [Build a robots.txt for AI bots](https://invisit.dev/tools/robots-txt-generator)

## How to verify a request is really Screaming Frog SEO Spider

Anyone can send any user agent, so the string alone proves nothing.

Screaming Frog publishes no IP ranges or DNS names for Screaming Frog SEO Spider that we could find, so there is no reliable way to tell a real Screaming Frog SEO Spider from a scraper using its user agent.

## See when Screaming Frog SEO Spider reads your site

Bots don’t run JavaScript analytics, so a script tag never sees them. invisit records crawler reads on the server instead, through its server SDK or a Cloudflare Worker, and shows which bots read which pages, next to your visitors.

- [Track crawlers with invisit](https://invisit.dev/docs/server/sdk#crawler-reads)

## Common questions

### What is Screaming Frog SEO Spider?

Screaming Frog SEO Spider is a SEO tool crawler run by Screaming Frog. Software that users run to crawl and audit websites for SEO issues.

### Is Screaming Frog SEO Spider safe to block?

Block it freely if the load bothers you. It sends no visitors; the only cost is that the tool’s users, your competitors included, see less of your site.

### How do I block Screaming Frog SEO Spider?

Add “User-agent: Screaming Frog SEO Spider” followed by “Disallow: /” to your robots.txt. Screaming Frog doesn’t promise to follow it in every case, so block it at your server too if it matters.