Skip to content
DomainVigil
Sign up

Free tool

Is this URL open to crawlers?

We read the domain's robots.txt and, if you give us a path, tell you whether a specific crawler is allowed to fetch it — along with the rule that decided, not just the raw file.

No sign-up, no email required. You can paste a full address — we'll pull the domain out of it.

Why test it at all

A misconfigured robots.txt costs you on both sides: one line too many and the crawler stops exploring pages you wanted ranked; one line too few and it walks through what you meant to keep quiet. The file never warns you, and nobody rereads their own.

How the tool decides

It reads the domain's robots.txt, splits it into groups, then applies the exact rule search engines use: first the most specific crawler group, then the rule with the longest path, and on a tie Allow wins. The same logic a crawler runs, without the guesswork of reading it by eye.

The winning rule

A robots.txt is not read top to bottom

A crawler does not take the first line that matches. It takes the most specific group, then, inside it, the rule with the longest path. Switch cases: the line that wins lights up.

Googlebot→/wp-admin/options.php

User-agent: *
Disallow: /wp-admin/← blocked
Allow: /wp-admin/admin-ajax.php
 
User-agent: AhrefsBot
Disallow: /

Googlebot has no group under its own name, so it follows the asterisk group. Only one rule matches this path, the Disallow on the folder. The page is closed.

Example file, on a domain name reserved for documentation. The precedence rule is the one from RFC 9309 and Google's own documentation.

Three lines

A whole robots.txt fits in three directives.

Everything else is just crawler names and fragments of paths.

  • Disallowa path crawlers are asked not to fetch
  • Allowthe exception that reopens a subpath closed above
  • Sitemapthe address of your sitemap, read by every engine

Reading the result

The verdict covers the path you gave; the findings below cover the whole file.

Path allowed
No rule closes this path in the selected group, or the longest matching one is an Allow. The crawler may go there.
Path blocked
The longest matching rule is a Disallow. The crawler will not fetch this address. Intended or not is your call — the tool only says which line won.
File found
The server answers 200 and the file splits cleanly into groups. This is the normal state.
No robots.txt
The site does not have one. To a crawler everything is allowed: perfectly valid, there is simply no sitemap declared.
Everything blocked
The group targeting every crawler forbids the root. On a production site, this is the first thing to check after a launch.
No sitemap
The file points to no sitemap. A single line declares it, and every search engine reads it.
Unreachable
No answer within the timeout: silent server, a name that does not resolve, or an upstream block. Nothing can be concluded.

Robots.txt syntax

Four directives and two wildcards. Anything not in this table is ignored by search engines.

SyntaxWhat it doesExample
User-agentOpens a group targeting one crawler, or all of them with an asteriskUser-agent: Googlebot
DisallowAsks crawlers not to fetch a pathDisallow: /cart
AllowReopens a subpath closed just aboveAllow: /cart/help
SitemapGives the full address of the sitemapSitemap: https://example.com/sitemap.xml
*Wildcard: any sequence of charactersDisallow: /*.pdf
$Anchors the end of the addressDisallow: /*.php$

Crawl-delay is common but not part of the standard: Google ignores it, Bing and Yandex read it.

Common questions

Does a Disallow hide my page?

No. It asks crawlers not to fetch it, not to leave it out of the index. A blocked page linked from elsewhere can still show up in results, without a description. To really remove it you need a noindex tag or a password — which means letting the crawler in so it can read that tag.

Where must the file live?

At the root of the domain and nowhere else: example.com/robots.txt. A file placed in a subfolder is never read. And every subdomain has its own, separate one.

Which group applies when several match?

The most specific one. If a crawler has a group under its own name, it ignores the catch-all group entirely. Within the selected group, the rule with the longest path decides, and on equal length Allow beats Disallow.

Should I declare my sitemap in it?

It is not required, but it helps: the Sitemap line is read by every search engine and hands them the address of your sitemap instead of making them look for it.

Do you keep what I test?

No. Neither the domain, nor the path, nor the result is stored. The file is read on the spot, shown, then forgotten.

0

The protection a Disallow gives a page. A robots.txt is a sign, not a lock: it asks a polite crawler not to come in. It hides nothing, protects nothing, and the file itself is public.

To pull a page out of search results you need a noindex tag or a password. A Disallow can even leave the address showing in Google, stripped of its content.

Four traps

What breaks a ranking in a single line

Four mistakes we find on a good share of the sites we take over. Each of them is one line too many, or one line in the wrong place.

  1. Blocking CSS and JavaScript

    A Disallow on the assets folder stops the crawler from loading your stylesheet and scripts. It then renders a broken page, and judges it exactly as it sees it. Display files must stay open.

  2. Confusing Disallow with noindex

    The first prevents fetching, the second prevents indexing. A blocked page linked elsewhere can still surface. And if the crawler is not allowed in, it will never read the noindex you placed inside.

  3. Shipping a staging robots.txt

    The Disallow on the root that protected your staging site sometimes ships to production with everything else. The site vanishes from search engines within days, with no warning at all.

  4. Putting it anywhere but the root

    A robots.txt is read at one address only, at the root of the domain. In a subfolder it does not exist as far as search engines are concerned. And a subdomain has its own, entirely separate.

Twenty checks, no sign-up