Robots.txt Tester

Paste a robots.txt, list one or more URLs, and check which paths each user-agent is allowed to crawl. Uses Google’s Robots Exclusion Protocol rules: longest matching pattern wins, with Allow beating Disallow on ties. Sitemap and non-group lines are surfaced separately.

How matching works

User-agents are matched case-insensitively. The longest matching agent token wins; the wildcard * group is the fallback.
Patterns support * (any sequence) and $ (end-of-path anchor).
For each path the rule with the longest matched pattern wins. On a tie, Allow beats Disallow (Google’s convention).
Empty Disallow: means “allow everything,” per the original 1994 spec.
Sitemap: and Host: directives are not part of crawl matching but are listed separately for visibility.

Per-URL verdict

Parsed rules

How matching works