How robots.txt Affects AI Crawlers
robots.txt uses per-crawler user-agent rules to control which AI bots can access your site — the same mechanism used for traditional search engines, applied to AI-specific user-agent tokens.
robots.txt is a plain text file at the root of a website (for example, example.com/robots.txt) that tells crawlers which parts of the site they're allowed to access. AI crawlers respect the same file and syntax as traditional search engine bots.
How the rules work
A robots.txt file is made of blocks, each starting with a User-agent line naming the crawler the block applies to, followed by Allow or Disallow rules:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Allow: /
When a crawler checks robots.txt, it looks for a block matching its own user-agent token first, and falls back to a User-agent: * block if no specific rule exists. When multiple rules could apply to the same path, the longest matching path wins; if there's an exact tie, Allow wins over Disallow.
Blocking or allowing specific AI crawlers
Because each AI crawler has its own user-agent token, you can write a rule for exactly the ones you care about:
User-agent: GPTBot
Disallow: /
User-agent: PerplexityBot
Allow: /
This blocks OpenAI's training crawler while still allowing Perplexity's. A generic User-agent: * rule affects any crawler without its own specific block, including AI crawlers that aren't individually named.
A common mistake
A Disallow: / under User-agent: * blocks every crawler that doesn't have its own more specific rule — including AI crawlers, even ones you didn't intend to block. If you want to allow general search engines but restrict specific AI crawlers, give each crawler you care about its own named block rather than relying only on the wildcard rule.
Checking your current rules
Rather than reading robots.txt manually and cross-referencing crawler names, the GPTBot Checker parses your live robots.txt and reports which of the 70 tracked AI crawlers are currently allowed or blocked.
Frequently Asked Questions
Does robots.txt apply to AI crawlers?
Yes — AI crawlers read and respect robots.txt the same way traditional search engine bots do, using the same User-agent/Allow/Disallow syntax.
How do I block just one AI crawler?
Add a User-agent block naming that crawler's specific token (for example, GPTBot) with a Disallow: / rule, leaving other crawlers unaffected.
What happens if there's no specific rule for an AI crawler?
It falls back to the User-agent: * block, if one exists. A wildcard Disallow: / blocks any crawler without its own specific rule, including AI crawlers.
How can I check my site's current AI crawler rules?
Use the GPTBot Checker to scan your live robots.txt against 70 tracked AI crawler tokens at once.