AI crawlers and robots.txt: a playbook for deciding who gets in

Step 1: Inventory the agents and what each one is for

The first mistake we see in audits is treating “AI crawlers” as one thing. The major vendors document distinct user agents with distinct jobs. Here’s the current map, based on each vendor’s published documentation. These pages change, so re-check them before you ship a policy.

• OpenAI: GPTBot collects content that may be used to train models. OAI-SearchBot crawls for ChatGPT’s search features; OpenAI says sites that block it won’t be surfaced in ChatGPT search answers. ChatGPT-User makes fetches when a user’s request triggers them, such as a user asking ChatGPT to open a page.
• Anthropic: ClaudeBot collects training data, Claude-SearchBot crawls to improve search results, and Claude-User fetches pages when a Claude user asks a question that needs them.
• Perplexity: PerplexityBot indexes pages so they can appear in Perplexity’s answers. Perplexity-User handles user-initiated fetches.
• Google: Google-Extended is a product token, not a separate crawler. Crawling is still done by Googlebot. The token tells Google whether crawled content may be used to train Gemini models and for grounding in Gemini Apps and Vertex AI. Search inclusion and AI Overviews are governed by your normal Googlebot rules, noindex and snippet controls.

One nuance on user-initiated agents: vendors treat them differently. OpenAI’s documentation has said robots.txt rules may not apply to ChatGPT-User because a person initiated the request, and Perplexity says Perplexity-User generally ignores robots.txt for the same reason. Anthropic says its agents, including Claude-User, honor robots.txt. Read the current wording yourself before you rely on any of it.

Step 2: Understand how robots.txt actually matches

robots.txt is standardized in RFC 9309. Most mistakes come from a few rules in that spec.

1. One group wins. A crawler looks for the group whose User-agent line most specifically matches its own token, and follows only that group. If a GPTBot group exists, GPTBot ignores everything under User-agent: *.
2. Groups can share rules. Several User-agent lines stacked on top of one set of rules form one group. That’s valid and keeps files tidy.
3. Longest path match decides. Within a group, the most specific matching Allow or Disallow path applies. If an Allow and a Disallow match equally, the spec says to use the less restrictive rule.
4. Blocking crawling is not de-indexing. A disallowed URL can still be known, for example from links, and appear in some form. For Google Search, use noindex on a crawlable page if you want it removed.
5. Changes aren’t instant. Crawlers cache robots.txt. Google says it generally caches the file for up to 24 hours. Other vendors don’t all document their cache times.

Step 3: Decide by purpose, not by vendor

Work through three questions with whoever owns content licensing, legal review and organic growth. The answers are business decisions. There’s no universally correct setting.

1. Training: Are you comfortable with your content being used to train future models? This is the GPTBot, ClaudeBot and Google-Extended question. Blocking these affects future training collection by those agents. It does not remove anything already collected.
2. Search and retrieval: Do you want to be eligible for citation in AI search answers? This is OAI-SearchBot, Claude-SearchBot and PerplexityBot. For most marketing sites this is the visibility lever, and blocking it trades citations for control.
3. User-initiated fetches: Should an assistant be able to read your page when a user explicitly points it there? Blocking these can break the case where a prospect pastes your URL into an assistant.

A common outcome for brands that want AI visibility but are cautious about training is: allow search agents, disallow training agents, leave user fetches alone. It’s a reasonable default to start from, not a recommendation for every site.

Step 4: Write the file

Here’s what that policy looks like. Note that the search group repeats the private-path rules, because it won’t inherit them from *.

# Opt out of model training
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

# Allow AI search indexing, keep private paths out
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Disallow: /account/
Disallow: /cart/

# Everyone else
User-agent: *
Disallow: /account/
Disallow: /cart/

Sitemap: https://example.com/sitemap.xml

Before deploying, check each rule with a parser that follows RFC 9309, using real URLs from your site. Also check your CDN or WAF settings. Some providers offer AI bot blocking at the edge, and that overrides whatever robots.txt says, because blocked requests never reach your origin.

Practical tip: keep a dated changelog line at the top of robots.txt as a comment. When AI referrals or citations shift weeks later, you’ll want to know exactly what changed and when.

Step 5: Verify what actually shows up

Compliance with robots.txt is voluntary, and user-agent strings are easy to fake. Here’s what to check.

1. Pull raw access logs, not just analytics. Most AI crawlers don’t run your JavaScript tag, so they won’t appear in GA4.
2. Filter by user-agent token and group hits by path and status code. Look for agents requesting disallowed paths after the cache window has passed.
3. Validate the source. OpenAI publishes IP range files for its agents, and Google documents how to verify Googlebot via reverse DNS and published ranges. Other vendors publish verification details to varying degrees. Treat unverified “GPTBot” traffic from random hosting ranges as an impostor, not as OpenAI.
4. Confirm search agents get 200s. If OAI-SearchBot is allowed in robots.txt but receives 403s from your firewall, the allow rule has no effect.

What about llms.txt?

llms.txt is a community proposal for a Markdown file that summarizes a site for language models. It is not a standard, and no major assistant provider has documented using it for crawling, retrieval or ranking. It doesn’t control access the way robots.txt does. If it’s cheap to generate, it does no harm. Just don’t expect it to change your visibility, and don’t let it take time away from the basics in structuring content so LLMs can quote it.

Close the loop

A crawler policy is a hypothesis about what you’re trading. Once it’s live, track whether citations and AI referrals move, using the methods in tracking AI referrals and citations. Then make sure the pages you’ve let in describe your organization consistently. Our entity SEO playbook covers that layer.

If you’d like a second pair of eyes on your robots.txt, edge rules and logs, our AI SEO audits start there. Book an audit call and bring your current file.