AI Crawlability Checker
Paste your robots.txt content to instantly see which AI crawlers — GPTBot, ClaudeBot, PerplexityBot, and more — can access your site. Understand the impact and get recommendations.
Tip: You can also leave this blank to see what a default (allow all) configuration looks like.
How to Check Your AI Crawlability
As AI-powered search engines — ChatGPT, Perplexity, Google Gemini, and Claude — become primary discovery channels for millions of users, whether or not these systems can access your content has real business implications. A misconfigured robots.txt could be silently blocking all AI traffic to your site.
This tool parses your robots.txt client-side (nothing is sent to any server) and evaluates access rules for every major AI crawler. It identifies which bots are fully allowed, fully blocked, or partially restricted, and explains the traffic and citation impact for each.
How to Get Your robots.txt Content
Find your robots.txt
Navigate to yoursite.com/robots.txt in your browser. This file is always publicly accessible at the root domain.
Copy the content
Select all text on the page (Ctrl/Cmd+A, then Ctrl/Cmd+C) to copy the full robots.txt content.
Paste & analyze
Paste the copied content into the text area above and click 'Analyze AI Crawlability' to see results instantly.
All AI Crawler User-Agent Strings
The following table lists all major AI crawlers, their official user-agent strings (as they appear in robots.txt), and the traffic impact of blocking each one. Use this as a reference when manually editing your robots.txt.
| Bot | User-agent string | Company | Purpose | Traffic impact |
|---|---|---|---|---|
| GPTBot | GPTBot | OpenAI | ChatGPT training data & Browse mode | High |
| ClaudeBot | ClaudeBot | Anthropic | Claude AI training & retrieval | Medium |
| PerplexityBot | PerplexityBot | Perplexity AI | Real-time AI answer generation | High |
| CCBot | CCBot | Common Crawl | Shared training dataset for many AI companies | Indirect |
| Bytespider | Bytespider | ByteDance / TikTok | TikTok search index & AI training | Low |
| Google-Extended | Google-Extended | Gemini AI model training (NOT search) | Medium | |
| Gemini-Web | Gemini-Web | Gemini real-time web browsing | Medium | |
| ChatGPT-User | ChatGPT-User | OpenAI | ChatGPT real-time browse requests | High |
| anthropic-ai | anthropic-ai | Anthropic | Alternate Anthropic crawler user-agent | Medium |
| YouBot | YouBot | You.com | You.com AI search engine | Low |
Understanding robots.txt Rules for AI Bots
Robots.txt rules are evaluated in order of specificity. A bot-specific rule (e.g., User-agent: GPTBot) overrides the wildcard rule (User-agent: *). This means you can allow all search engines by default while selectively blocking specific AI crawlers.
# Allow all search bots by default User-agent: * Allow: / # Block OpenAI training (does not affect ChatGPT browsing links) User-agent: GPTBot Disallow: / # Block CCBot (affects many AI training datasets) User-agent: CCBot Disallow: / # Block Bytespider User-agent: Bytespider Disallow: / # Safe to block - won't affect Google Search rankings User-agent: Google-Extended Disallow: / Sitemap: https://example.com/sitemap.xml
Frequently Asked Questions
How do I check if my site is blocking AI crawlers?
Open your robots.txt file (yoursite.com/robots.txt), copy the contents, and paste them into this tool. The checker parses your robots.txt and evaluates each AI crawler's access — GPTBot, ClaudeBot, PerplexityBot, CCBot, Bytespider, Google-Extended, and Gemini-Web — showing whether each is allowed, blocked, or partially restricted.
What is the difference between GPTBot and ChatGPT-User?
GPTBot is OpenAI's crawling bot for training data collection and web indexing. ChatGPT-User is the user-agent used when a ChatGPT user activates the real-time browsing feature. Both should be considered when configuring your robots.txt if you want to control OpenAI's access to your site.
Does blocking AI crawlers hurt my SEO?
Blocking AI crawlers does not directly affect traditional Google Search rankings. Googlebot (used for search rankings) is entirely separate from Google-Extended (used for Gemini training). However, blocking all AI crawlers will reduce your visibility in AI-powered search products like Perplexity, ChatGPT Browse, and Google AI Overviews, which increasingly drive referral traffic.
What happens if I accidentally block all crawlers with Disallow: /?
A wildcard rule 'User-agent: *' with 'Disallow: /' blocks ALL crawlers, including Googlebot, from indexing your site. This would cause your site to disappear from Google search results within a few weeks. This tool will flag this as a critical issue. Always check your robots.txt against all crawlers before deploying.
Which AI crawler sends the most traffic?
As of 2025, PerplexityBot drives the most measurable referral traffic from AI sources, followed by ChatGPT (GPTBot/ChatGPT-User). Google AI Overviews don't use a separate bot — they use the standard Googlebot. Allowing PerplexityBot and GPTBot gives you the best chance of appearing in AI-generated answers that users click through from.
Related Tools
Want a Full GEO Audit for Your Site?
Our specialists will audit your robots.txt, llms.txt, schema markup, and overall AI search visibility — and build a strategy to maximize your presence in ChatGPT, Perplexity, and Gemini.
Get a Free SEO Audit