Allowing AI Crawlers: How ChatGPT, Claude and Perplexity Read Your Website
AI systems can only mention what they are allowed to read. If you block GPTBot, ClaudeBot or PerplexityBot, you disappear from answers as a source. A look at your robots.txt is worthwhile.
Which AI crawlers are there?
The major providers use their own crawlers with their own user agent. Some collect training data, others fetch content live to answer a question.
| User agent | Provider | Purpose |
|---|---|---|
| GPTBot | OpenAI | Content for OpenAI models |
| OAI-SearchBot | OpenAI | Search in ChatGPT |
| ChatGPT-User | OpenAI | Retrieval when users request a page in ChatGPT |
| ClaudeBot | Anthropic | Content for Claude |
| PerplexityBot | Perplexity | Perplexity search index |
| Google-Extended | Controls usage for Gemini | |
| Bingbot | Microsoft | Bing index, foundation for Copilot |
How to allow AI crawlers in robots.txt
The robots.txt file is located in the root directory of your domain (e.g. ihre-domain.ch/robots.txt). An explicit allow looks like this:
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: PerplexityBot
Allow: /
If one of these bots has Disallow: /, your entire website is blocked for it. You can still exclude internal areas such as /admin/ specifically.
The hidden block: bot protection and firewalls
Even if robots.txt allows everything, security services such as Cloudflare can reject AI crawlers with an error message (e.g. HTTP 403). Some providers block AI bots by default. So check your CDN settings to see whether “AI Bots” or “Verified Bots” are allowed.
What else you can do
- Provide an up-to-date XML sitemap and reference it in robots.txt.
- Serve important content server-side so that it is readable without JavaScript.
Frequently asked questions
Is it harmful to allow AI crawlers?
For most companies, the benefits outweigh the risks: only what AI systems are allowed to read can be mentioned and linked in answers. Confidential content doesn't belong on the public web anyway.
How do I find out whether my website blocks AI crawlers?
Open your robots.txt and look for the user agents of AI crawlers. The free AI Check from Kamano AI checks robots.txt and bot protection automatically.