Does Disallow keep a page out of search results?

No. robots.txt controls crawling, not indexing. A disallowed URL with inbound links can still appear in results as a bare URL with no description. To reliably keep a page out, the crawler must be able to read it and find a `noindex` directive — which means that page must not also be disallowed.

Why longest match rather than first match?

Google and Bing evaluate every rule and take the one with the longest — most specific — matching path, with Allow winning ties. So `Disallow: /admin` together with `Allow: /admin/public` leaves `/admin/public/x` allowed, even though the Disallow comes first. The tester here follows that rule.

SEO TOOLS · NOTHING IS FETCHED

robots.txt Generator and Tester

Build robots.txt rules and test whether a specific path is allowed, using longest-match precedence.

Input

User-agent

Crawl-delay

Disallow paths

Allow paths

Sitemap URLs

Result

User-agent: *
Disallow: /admin
Disallow: /private
Allow: /admin/public

Sitemap: https://example.com/sitemap.xml

Test path

Test crawler

Crawling allowedDeciding rule: Allow: /admin/public

About this tool

Set up User-agent groups with Disallow and Allow paths, add sitemap locations, and get a standard robots.txt. The same page tests it: enter a path and a crawler name and the tool reports allowed or blocked along with the rule that decided it. Precedence follows Google and Bing's **longest match** rule — the most specific matching path wins, and Allow beats Disallow at equal length — which differs from the intuitive "first matching rule" and is where mistakes usually happen.

How it works

Define User-agent groups with Disallow and Allow paths.
Add the sitemap URLs.
Enter a path in the tester to see the verdict and the deciding rule.

Limits and notes

Applies the longest-match rule used by Google and Bing. robots.txt controls crawling, not indexing.

Privacy

Pasted HTML, URLs, and text are analysed inside this browser only. Nothing is uploaded, stored, or written to the URL or any log.

Frequently asked questions

Does Disallow keep a page out of search results?: No. robots.txt controls crawling, not indexing. A disallowed URL with inbound links can still appear in results as a bare URL with no description. To reliably keep a page out, the crawler must be able to read it and find a `noindex` directive — which means that page must not also be disallowed.
Why longest match rather than first match?: Google and Bing evaluate every rule and take the one with the longest — most specific — matching path, with Allow winning ties. So `Disallow: /admin` together with `Allow: /admin/public` leaves `/admin/public/x` allowed, even though the Disallow comes first. The tester here follows that rule.

← All SEO tools