Paste a robots.txt file, then test whether a specific path is allowed or blocked for a given crawler user-agent.
Robots.txt is a set of groups: each group names one or more user-agents and lists Disallow and Allow path patterns for them. To test a path, this tool finds the group or groups that apply to the user-agent entered — a group naming that agent specifically takes precedence over the wildcard group — then compares every matching pattern. The longest matching pattern decides the result, an Allow pattern wins a tie of equal length, and a path matched by no rule is allowed. Patterns use the standard wildcards: an asterisk matches any sequence of characters, and a trailing dollar sign anchors the pattern to the end of the path.
Does this tester fetch a robots.txt file from a website? No. Nothing is fetched. Paste the robots.txt contents into the box, and every test runs against that pasted text in the browser.
What happens when Allow and Disallow both match? The longest matching pattern wins, measured by the length of the pattern. If an Allow pattern and a Disallow pattern of the same length both match, the path is allowed, following the convention in RFC 9309.
What if no rule matches a path? The path is allowed. Robots.txt only blocks what a matching Disallow rule covers, and crawlers that are not named in any group fall back to the wildcard group, if one exists.