Robots.txt Rule Tester
Paste your robots.txt and test whether any URL is allowed or blocked for Googlebot, GPTBot, ClaudeBot, Bingbot, or any custom user-agent. Understand exactly which rule matched.
How to Use the Robots.txt Rule Tester
Paste your robots.txt content into the large text area. Click Load Sample to try a demo file. Enter the URL path you want to test (e.g. /admin/ or https://example.com/blog/post-1), select a user-agent from the dropdown or type a custom one, then click Test Rule.
Understanding the Results
ALLOWED means the crawler is permitted to access this URL. Either an Allow rule matched, or no Disallow rule applies.
BLOCKED means a Disallow rule prevents the crawler from accessing this URL. The specific rule that caused the block is shown.
The Matched Rule field shows the exact Allow or Disallow directive from your robots.txt that determined the result.
The Matched Agent field shows which User-agent block the matching rule came from — the specific bot block or the wildcard (*) fallback.
Frequently Asked Questions
How does a robots.txt rule tester work?
A robots.txt rule tester parses your robots.txt content and simulates how a crawler evaluates a given URL against the declared rules. It checks User-agent blocks in order of specificity — a specific bot's block takes precedence over the wildcard (*) block. Within a block, the most specific (longest) matching rule wins. Allow and Disallow rules of equal length resolve in favor of Allow.
What is the correct order of precedence for robots.txt rules?
First, the tester checks for a User-agent block that matches the bot name exactly (case-insensitive). If a matching rule is found there, it applies and the wildcard (*) block is ignored for that bot. If no specific block exists, the wildcard (*) rules apply. Within any block, the longest (most specific) matching path wins. If an Allow and a Disallow rule match at the same length, Allow takes precedence — this is Google's behavior.
Why does Googlebot ignore my Disallow rule?
The most common reason is a more specific Allow rule that overrides the Disallow. For example, 'Allow: /blog/' with 'Disallow: /' means /blog/ is allowed even though everything else is blocked. Also check that you haven't defined a separate 'User-agent: Googlebot' block with its own rules — that block takes full precedence over the wildcard block for Googlebot.
Should I block AI bots like GPTBot and ClaudeBot?
Blocking AI training bots (GPTBot for OpenAI, ClaudeBot for Anthropic) prevents your content from being used in AI model training datasets. It does not affect your Google search rankings. To block them, add: 'User-agent: GPTBot' followed by 'Disallow: /' and repeat for ClaudeBot. These bots generally respect robots.txt, unlike scrapers. Decide based on whether you want your content used for AI training.
What is the difference between Googlebot and Google-Extended?
Googlebot is Google's main search crawler — it indexes your pages for Google Search. Google-Extended is a separate user-agent used by Google to train its AI products (Gemini, Vertex AI). You can block Google-Extended to opt out of AI training without affecting your search rankings. Add 'User-agent: Google-Extended' with 'Disallow: /' to block it while keeping Googlebot allowed.
Related Tools
Need a Full Technical SEO Audit?
Our SEO experts review your robots.txt, sitemap, crawl budget, and technical configuration to build a complete action plan for your site.
Get a Free SEO Audit