Robots.txt Checker
Read a site’s crawler rules the way a crawler does.
robots.txt is a request, not a control. Well-behaved crawlers read it and respect it; anything else ignores it completely. That distinction matters because the file is public, so every path listed in it is published to anyone who fetches it — including the scanners that look for exactly this. Using Disallow to hide an admin panel does the opposite of hiding it.
The syntax has one subtlety that trips up both people and parsers: consecutive User-agent lines form a single group sharing the rules that follow. A file naming Googlebot and Bingbot on two lines and then a Disallow applies that rule to both. A parser that ties rules only to the last agent reports the first one as unrestricted, which is wrong.
The failure worth checking for after every deployment is a staging configuration reaching production. Disallow: / under User-agent: * is exactly right for a staging site and removes a production site from search results within days. It is a one-line mistake with a long recovery.
What this tool does
Grouped correctly
Consecutive User-agent lines share one rule group. Parsers that miss this attribute rules to the wrong crawler.
Catches the disaster case
Disallow: / under User-agent: * asks every crawler to index nothing. Shipped to production by accident, it removes a site from search.
Flags what it advertises
robots.txt is world-readable and among the first files an attacker fetches. Listing /admin or /.env there points straight at them.
Finds declared sitemaps
Every Sitemap: line is extracted, which is the supported way to tell crawlers where your sitemap lives.
Frequently asked questions
Where does robots.txt have to live?
At the root of the host, exactly at /robots.txt. Crawlers do not look anywhere else, and a file at a different path has no effect. It is also per host and per scheme: a robots.txt on https://example.com does not govern https://sub.example.com.
Does Disallow stop a page appearing in search results?
Not reliably. Disallow asks crawlers not to fetch the page, but a URL that is linked from elsewhere can still be listed — without a description, because the crawler never read it. To keep a page out of results, allow the crawl and serve a noindex directive, or require authentication.
Is robots.txt a security control?
No, and treating it as one is actively harmful. The file is public, unauthenticated and among the first things fetched during reconnaissance. Listing sensitive paths tells an attacker precisely where to look. Protect those paths with authentication and keep them out of robots.txt entirely.
What happens if I have no robots.txt?
Crawlers treat it as permission to crawl everything, which is usually fine. You lose the ability to declare a sitemap or keep crawlers off expensive endpoints. A 404 is handled cleanly; a 500 is not, and some crawlers respond to a server error by pausing crawling of the site altogether.
Why does my robots.txt return an HTML page?
Almost always a catch-all route in a single-page application serving the app shell for any unmatched path. The file appears to exist but crawlers cannot parse it, so the site effectively has no robots.txt. It needs to be served as static text, ahead of the catch-all.
Does Crawl-delay work?
Only for some crawlers. Google ignores it entirely and uses its own rate adaptation, with a crawl-rate setting in Search Console; Bing and several others honour it. It is not part of the original specification, which is why support varies.
Monitor it, don't just check it
One-off checks catch what is broken today. CertNotify watches your certificates, domains, DNS and code continuously and tells you before something breaks.