AI Crawlers Explained
AI crawlers are automated bots that fetch web content for AI model training, AI-powered search, or on-demand fetching by AI assistants — each identified by its own user-agent token in robots.txt.
"AI crawler" is a broad label for several different kinds of bots that fetch web content on behalf of AI systems. They don't all do the same thing, and a site's robots.txt can allow or block each one independently.
The four categories of AI crawlers
Based on the 70 AI crawler tokens tracked by ExtFoundry's GPTBot Checker, AI crawlers fall into four broad categories:
Model training (29 tokens) — bots that collect content to train AI models. Examples include GPTBot (OpenAI), ClaudeBot and anthropic-ai (Anthropic), Google-Extended (Google), Applebot-Extended (Apple), Bytespider (ByteDance), and CCBot (Common Crawl).
AI search index (17 tokens) — bots that index content specifically to power AI-driven search and answer results. Examples include OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic), and PerplexityBot (Perplexity).
User-triggered fetch (9 tokens) — bots that fetch a specific page on demand, usually because a person asked an AI assistant to read or summarize it. Examples include ChatGPT-User (OpenAI), Claude-User (Anthropic), and Perplexity-User (Perplexity).
Agent / research (15 tokens) — bots used by AI agents doing multi-step browsing or research tasks. Examples include GoogleOther and Gemini-Deep-Research (Google).
Why the distinction matters
A site might reasonably want to allow "user-triggered fetch" bots (so an AI assistant can summarize a page a visitor asked about) while blocking "model training" bots (to keep content out of training datasets). Because each category uses different user-agent tokens, robots.txt rules can be written to treat them differently — it isn't an all-or-nothing decision.
How to check which crawlers can access your site
The GPTBot Checker evaluates all 70 tracked tokens against your robots.txt in a single scan and groups the results by category, so you can see exactly which types of AI crawlers are allowed or blocked.
Frequently Asked Questions
What is an AI crawler?
An AI crawler is an automated bot that fetches web content on behalf of an AI system — for training a model, powering AI search, fetching a page a user asked about, or supporting an AI agent's research.
How many AI crawlers are there?
ExtFoundry's GPTBot Checker tracks 70 AI crawler tokens across four categories: model training (29), AI search index (17), user-triggered fetch (9), and agent/research (15).
Can I allow some AI crawlers and block others?
Yes — each AI crawler identifies itself with its own user-agent token in robots.txt, so you can allow or disallow them individually rather than as an all-or-nothing group.