How to appear in ChatGPT and Perplexity search answers

To appear in ChatGPT search, allow OAI-SearchBot in robots.txt and let OpenAI’s published IP ranges through your host or CDN. To appear in Perplexity, do the same for PerplexityBot. That is all either company tells site owners to do, and OpenAI adds that “Placement is not guaranteed”. Neither publishes its ranking factors, so nobody outside those companies knows what decides who gets cited. In our scan of 10,870 readable robots.txt files from the Tranco top 20,000 on 8 November 2026, 7.84% block OAI-SearchBot and 10.47% block PerplexityBot, by name or through User-agent: *.

Below: what each operator says, checked on 8 November 2026, how much of the “Bing index” story OpenAI actually confirms, commands that show why a site is absent and where those commands stop being useful, and a table of popular GEO claims set against the primary sources.

What do OpenAI and Perplexity say you have to do?

Both companies name one requirement: their search crawler must be able to reach your pages. OpenAI’s help page on web search says that to be eligible, “allow OAI-Searchbot to crawl the site and confirm that the website host or content delivery network allows traffic from OpenAI’s published searchbot IP addresses”. Perplexity’s crawler docs say to allow PerplexityBot in robots.txt and permit “requests from our published IP ranges”.

The quotes come from OpenAI’s crawler overview, its web search help page, Perplexity’s crawler docs and its robots.txt article, all checked on 8 November 2026. For every other AI bot, see AI crawler user agents.

That table is the entire public record. Neither company names a word count, a markup format, a freshness window or a domain authority measure. No page we read describes a way to request a crawl or a report of where your site appears.

Does blocking GPTBot keep you out of ChatGPT search?

No. OpenAI says each robots.txt setting “is independent of the others” and gives this exact case as its example: “a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot”. Most sites that block training already do this: of the 1,647 sites blocking GPTBot by name in our scan, 1,224 (74.3%) leave OAI-SearchBot unblocked. The reverse setup, blocking OAI-SearchBot while leaving GPTBot open, appears on 2 sites.

The accidental block is more common than the deliberate one. 416 sites (3.83%) block OAI-SearchBot by name, and the effective figure rises to 7.84% once sites that block it only through User-agent: * are counted. For PerplexityBot the numbers are 696 (6.40%) by name and 10.47% effective. If your file has a User-agent: * group with Disallow: /, every crawler you did not name is shut out, including these two. Our study of 20,000 sites has the method and limits.

Perplexity says PerplexityBot “is not used to crawl content for AI foundation models”, so blocking it costs you citations and protects nothing related to training.

Does ChatGPT use Bing, and do I need to be in Bing’s index?

OpenAI confirms that ChatGPT search “sometimes partners with other search providers” and does not say Bing is required. Its help page says that when this happens, ChatGPT “typically rewrites your query into one or more targeted queries” and sends them to those providers. The only provider names on the page appear in a pointer to their privacy policies: Microsoft and Shopify.

The page does not say which queries go to a provider, how those results combine with OAI-SearchBot’s crawl, or whether a page absent from Bing can still be cited. OpenAI’s 2024 launch post says search “leverages third-party search providers, as well as content provided directly by our partners”, and adds no more detail.

Perplexity says only that it “partners with third-party crawlers to help build our search index”, without naming them. Neither company says anything about Google’s index in the pages we read.

So being indexed in Bing is a sensible precaution. It costs an afternoon in Bing Webmaster Tools and helps you in Bing regardless. Anyone who says it is required, or sufficient, is going beyond what OpenAI has published.

Which technical problems can keep a site out?

Four things sit on your side of the fence, and each can fail while the site looks fine in a browser.

A firewall or CDN rule. This is the one both companies warn about. Perplexity’s docs give WAF steps for Cloudflare and AWS and warn that “Cloudflare’s Bot Fight Mode can’t be skipped with WAF custom rules”. Cloudflare’s AI crawler policies let you block Search, Agent and Training traffic separately. Its docs say new domains default to Allow as of November 2026, and they describe defaults for new domains only, so check what an existing zone is set to under Security > Settings.

A bad robots.txt status code. RFC 9309, the robots.txt standard, says a crawler that gets a 4xx “MAY access any resources”, and one that gets a server error “MUST assume complete disallow”. A CDN returning 503 for /robots.txt to bots therefore reads as a site-wide block. OpenAI and Perplexity do not document how their crawlers handle these codes, so the RFC is the best available guide.

Text that only exists after JavaScript runs. Neither OpenAI’s nor Perplexity’s crawler page mentions JavaScript or rendering. Google and Apple do say their crawlers render pages. With no such statement from these two, assume they read the HTML as it arrives, and put your content in it with server rendering or static generation.

A leftover noindex. Neither crawler page says how noindex is treated. Check for it anyway, since a tag left over from staging tells any crawler that honors it to drop the page.

How do I find out why my site is absent?

Run these in order and stop at the first failure. We ran every step against a test server, and steps 2 and 4 against live sites too. Set your addresses first:

SITE="https://example.com"
URL="https://example.com/your-page"
BROWSER='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36'
OAI="$BROWSER; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot"
PPLX='Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)'

The two bot strings are the example user agents each operator publishes; the version numbers may change.

1. Is robots.txt readable, and what does it say to these bots?

curl -sS -L -o /dev/null -w '%{http_code} %{content_type}\n' "$SITE/robots.txt"
curl -sS -L "$SITE/robots.txt" | grep -iE -A6 '^user-agent: *(oai-searchbot|perplexitybot|\*)'

You want 200 text/plain. If a group names the bot, only that group applies to it. If none does, the * group applies. Look for Disallow: /.

2. Does the page answer a browser and a bot the same way?

for ua in "$BROWSER" "$OAI" "$PPLX"; do
  curl -sS -L -o /dev/null --max-time 20 -A "$ua" -w '%{http_code}  %{size_download} bytes  ' "$URL"
  echo "${ua##* }"
done

On a test server that refuses these two user agents, the output is:

200  274 bytes  Safari/537.36
403  9 bytes  +https://openai.com/searchbot
403  26 bytes  +https://perplexity.ai/perplexitybot)

3. Is the response marked as a challenge or as noindex?

curl -sS -L -D - -o /dev/null -A "$OAI" "$URL" | grep -iE '^(HTTP/|cf-mitigated|x-robots-tag)'
curl -sS -L -A "$OAI" "$URL" | grep -ioE '<meta[^>]*robots[^>]*>'

Cloudflare documents that a challenge response carries cf-mitigated: challenge. An X-Robots-Tag header or a robots meta tag containing noindex is the leftover to remove.

4. Is the text in the HTML?

curl -sS -L -A "$OAI" "$URL" | perl -0pe 's/<(script|style)\b.*?<\/\1>//gis; s/<[^>]+>/ /g' | wc -w
curl -sS -L -A "$OAI" "$URL" | grep -cF 'a sentence copied from your page'

The first counts words outside scripts and styles. An empty app shell returned 1 in our test and a real pricing page returned 1,158. The second returns 0 if your sentence is not in the raw HTML, which is the plainest test there is.

5. Did the real crawlers come, and what did you answer?

awk -F'"' '$6 ~ /OAI-SearchBot|PerplexityBot/ { split($3, s, " "); match($6, /OAI-SearchBot|PerplexityBot/); print substr($6, RSTART, RLENGTH), s[1] }' /var/log/nginx/access.log \
  | sort | uniq -c | sort -rn

This reads the nginx combined log format, and prints each bot with each status it received:

      2 PerplexityBot 403
      2 OAI-SearchBot 200

A wall of 403s is your answer. Check that the IPs are genuine with the script in our crawler post, because anyone can send these user agents. No lines at all is ambiguous: the crawler may not have come, or your CDN may have refused it before the request reached the origin, in which case only the CDN’s logs show it.

What can a curl test not tell you?

A curl with a borrowed user agent shows only blocks keyed to the user agent. It does not come from the crawler’s IP addresses, and both companies tell you to allow those addresses precisely because firewalls can act on them. The test can be wrong in both directions.

It can pass while the real crawler fails. A rule that blocks the published IP ranges, or a bot category decided at the CDN, never sees your laptop. Cloudflare’s policies apply to “Verified bots with that behavior and matching unverified bots”, so a real crawler can land in a category your spoofed request does not.

It can fail while the real crawler passes. Perplexity’s own recommended rule matches the user agent and the IP range together. On a site set up that way, your request has the right user agent and the wrong address, so it is correctly refused.

Your own address matters too. A home connection, an office allowlist and a cloud VM can each get different treatment from the same rules. Treat a curl result as a strong hint, and treat your server or CDN logs as the evidence.

Which GEO claims have no primary source?

Most of them. Generative engine optimization advice is sold with more confidence than any public source supports, and these are the claims we see most, set against what the operators have published as of November 2026.

Correlation studies exist, and some are careful. They describe what happened in a sample, from outside, and the systems change without notice. Read them as hypotheses. None is a statement from the company that runs the ranking.

Google is the useful contrast: it says plainly that “there are no additional requirements to appear in AI Overviews or AI Mode, nor other special optimizations necessary”. OpenAI and Perplexity have not said that. They also have not said the opposite.

What should you do once the checks pass?

Publish pages that answer specific questions well, keep them crawlable, and accept that the rest is out of your hands. Nothing public says otherwise, and any paid promise of placement is a promise nobody can keep.

Expect modest numbers even when everything is right. invisit.dev allows every AI crawler, and in the 30 days ending 8 November 2026 (recording since 4 November) its log held 11 requests from OAI-SearchBot, 8 of them for robots.txt, and 12 from PerplexityBot. In the same window it saw 127 live fetches from assistants and 0 visits with an AI referrer. That is one small site, and some of those fetches may be our own agents, but it shows that being allowed and being read are different events, and being read and being clicked are different again.

One free tool runs most of these checks without a terminal. The AI visibility checker, from invisit, the tool we build, reads robots.txt for every AI bot, requests the page as a browser and as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot, counts the words in the raw HTML and reads noindex. It has the limit described above: it sends each crawler’s user agent from our servers, not from its IP addresses. It cannot tell you what an AI product does with the page afterwards.

For the real traffic in step 5, invisit’s server SDK records crawler reads, and the Crawlers page shows OAI-SearchBot and PerplexityBot under AI search, by bot and by page. It classifies by user agent and does not check IP lists, so a fake GPTBot counts as GPTBot.

Checklist: why is my site not in ChatGPT or Perplexity?

• /robots.txt answers 200 with text, on every host you care about.
• No group naming OAI-SearchBot or PerplexityBot disallows your pages, and User-agent: * has no Disallow: /.
• A browser and each bot’s user agent get the same status and a similar body size.
• Your CDN’s AI and bot settings allow search crawlers, and the published IP lists are let through.
• No noindex in a meta tag or X-Robots-Tag header.
• The page’s text appears in the HTML before any script runs.
• Your logs show genuine requests from those crawlers, answered with 200.
• Your site is verified and indexed in Bing, as a precaution OpenAI neither requires nor rules out.
• You waited at least 24 hours after changing robots.txt, the delay both companies quote.

If every box is ticked and you are still absent, you have run out of things the operators have told anyone to do.