robots.txt Tester

How it works

1. Load the robots.txt — fetched from the site's host (/robots.txt, max 512 KB, redirects followed) or pasted; a missing file (4xx) means everything is allowed, a server error (5xx) means Google crawls nothing until it recovers
2. Parse it into groups: each run of User-agent lines starts a group holding the Allow and Disallow rules that follow; Sitemap lines are global; unknown directives are kept and flagged
3. Pick the group for the crawler: the most specific User-agent token that matches the crawler's product token (case-insensitive, e.g. Googlebot-Image over Googlebot); only if none matches, the User-agent: * group; if neither exists, everything is allowed
4. Match the path against every rule in that group — * matches any characters, $ anchors the end, otherwise prefix match on the percent-normalised path and query
5. The rule with the longest matching pattern decides; on a tie Allow wins; no match means allowed
6. Repeat for 20 known crawlers (search, AI training, AI answer engines, social previews, SEO tools) for the root and the tested path, and list syntax warnings and Sitemap lines

About robots.txt and how crawlers read it

robots.txt is a plain text file at the root of a host that tells crawlers which paths they may fetch. It is a request, not a lock: compliant crawlers — every major search engine, and the AI bots that publish a user-agent — read it before crawling, but nothing stops a crawler that ignores it. Its rules affect crawling only, not indexing: a blocked page can still appear in search results if other pages link to it, just without a description. To keep a page out of the index, it has to be crawlable and carry a noindex tag.

The matching rules are precise and mostly misunderstood. A crawler uses only one group: the one whose User-agent token matches it most specifically, or else the * group — never both. Within that group the longest matching path pattern wins regardless of the order the lines appear in, and when an Allow and a Disallow match with the same length, Allow wins. * matches any run of characters and $ pins a pattern to the end of the path; without $ every rule is a prefix. This tester implements those rules as RFC 9309 and Google's documentation describe them, and shows the line that made the decision rather than just the verdict.

The per-crawler summary exists because robots.txt has become the place where sites decide who may use their content for AI. GPTBot, ClaudeBot, CCBot, Google-Extended, Bytespider and meta-externalagent collect training data; OAI-SearchBot, ChatGPT-User, PerplexityBot and Claude-Web fetch pages to answer questions and can send traffic; Googlebot and Bingbot index for search. A single Disallow: / under User-agent: * blocks all of them at once, including the ones that send visitors. The table shows what each named bot would do with the file as written, so the trade-off is visible instead of accidental.

The warnings cover the mistakes that quietly change behaviour: a rule placed before any User-agent line (ignored), an empty Disallow (allows everything), a path without a leading slash, a relative Sitemap URL, Crawl-delay (which Google and Bing ignore), a Noindex directive (unsupported since 2019), a duplicate group, a byte-order mark, and a file over the size Google reads. None of this is advice about what to block — that depends on the site — only about whether the file says what its author meant.

Frequently asked questions

Does Disallow in robots.txt remove a page from Google?

No. It stops Googlebot from fetching the page, but the URL can still be indexed from links elsewhere and shown without a snippet. To remove or keep a page out of the index, let it be crawled and add a meta robots noindex tag or an X-Robots-Tag header. Blocking a page in robots.txt and adding noindex at the same time does not work, because the noindex is never seen.

Which rule wins when Allow and Disallow both match?

The one with the longer path pattern. Disallow: /admin/ and Allow: /admin/help both match /admin/help/start; the Allow pattern is longer, so the path is allowed. If two patterns have exactly the same length, Allow wins. Line order does not matter.

Does the User-agent: * group apply to Googlebot as well?

Only if there is no group naming Googlebot. A crawler follows exactly one group — the most specific match for its token, otherwise *. Rules are never merged across groups, which is why a site with a Googlebot group that lacks Disallow lines lets Googlebot crawl everything the * group blocks.

How do I block AI bots without blocking search engines?

Give each AI crawler its own group, or put several tokens in one group: User-agent: GPTBot, User-agent: CCBot, then Disallow: /. Leave the * group and any Googlebot or Bingbot groups as they are. The summary table shows which named bots end up blocked and which search crawlers are unaffected. Note that Google-Extended controls use for Gemini training but not Googlebot's crawl.

Why does the tester say my file allows everything when it has Disallow lines?

Most often because the Disallow lines sit before any User-agent line, or because the chosen crawler has its own group without those rules. The line highlight shows which group applied; the warnings list flags rules that belong to no group.

Is Crawl-delay supported?

Not by Google or Bing — Google ignores the directive entirely and Bing prefers the crawl-rate setting in Webmaster Tools. Yandex stopped honouring it in 2018; only some smaller crawlers still do. The tester parses it and flags it so you know it is not doing what you may expect for the two largest engines.

What happens when robots.txt returns an error?

A 404 or other 4xx means there is no robots.txt and crawlers assume everything is allowed. A 5xx or a timeout is treated by Google as a temporary full block: it stops crawling and can keep that state for up to 30 days before assuming no rules. A redirect to another host is followed, but the rules then apply to the original host only if the file is found.