Robots.txt Checker — Analyze Any Website's Crawl Rules
Check and analyze any website's robots.txt file instantly. Verify crawl rules, blocked paths, and sitemap references. Free robots.txt checker. Essential for SEO audits.
Use Robots.txt Checker
Launch the Robots.txt Checker tool — fully free, no signup required.
What Is a Robots.txt File?
robots.txt is a small text file placed at the root of a domain (example.com/robots.txt) that tells web crawlers — Googlebot, Bingbot, AI training crawlers, and so on — which paths they may or may not fetch. It's part of the Robots Exclusion Protocol and is the very first thing well-behaved crawlers request from a new domain.
How to Read a Robots.txt File
Each block starts with one or more User-agent lines naming the crawler(s) the rules apply to. "*" matches all crawlers. Disallow lines block paths; Allow lines re-open subpaths inside a blocked area. Sitemap lines point crawlers to your XML sitemap. Order matters within a block — the most specific matching rule wins.
Common Robots.txt Mistakes That Hurt SEO
The classic disaster is "Disallow: /" left over from a staging environment — it tells Google to skip the entire site. Other frequent issues: blocking CSS/JS so Google can't render the page properly, blocking pages that should also be noindex'd (Google can't see the noindex if it can't fetch the page), and forgetting to declare the sitemap.
Robots.txt Best Practices
Use the cheat sheet below for the directives you'll actually need. Keep the file short, declare your sitemap, never block CSS/JS, and re-check after every deploy that touches site infrastructure.
| Directive | Meaning | Example |
|---|---|---|
| User-agent: * | Applies to all crawlers | User-agent: * |
| Disallow: / | Blocks entire site | Disallow: / |
| Allow: / | Allows crawling | Allow: /blog/ |
| Sitemap: | Points to sitemap | Sitemap: /sitemap.xml |
| Crawl-delay: | Sets crawl rate | Crawl-delay: 10 |
What It Is
Robots.txt is a small plain-text file served at the root of a domain — example.com/robots.txt — that tells web crawlers which paths they may and may not fetch. It's the cornerstone of the Robots Exclusion Protocol, formalized as RFC 9309, and it's the very first request a well-behaved crawler makes when visiting a new site. Googlebot, Bingbot, AI training crawlers, SEO tools, and security scanners all respect it (or at least claim to). The Robots.txt Checker fetches that file, parses every directive, and tells you exactly what each crawler is allowed and forbidden to do. SEO teams use it to prevent accidental deindexing. Security teams use it to spot leaked admin paths. AI-conscious site owners use it to opt out of model training.
How It Works
The tool issues an HTTP GET against /robots.txt at the domain you specify and parses every line according to the Robots Exclusion Protocol. User-agent blocks group rules by crawler. Allow and Disallow lines control path access. Sitemap lines point crawlers at your XML sitemap. Crawl-delay and Request-rate (non-standard but widely supported) throttle aggressive crawlers. The parser flags syntactic mistakes — unknown directives, blocks with no user-agent, conflicting rules — and renders the effective policy so you can verify intent against implementation.
Real-World Examples
A site owner who just relaunched on a new CMS runs the check and discovers the dev team's staging robots.txt ('Disallow: /') shipped to production — they fix it before Google's next crawl. An SEO manager auditing a new client confirms /wp-admin and /cart are correctly blocked, while /blog and /products are fully open. A privacy-focused publisher adds explicit Disallow rules for GPTBot, Google-Extended, ClaudeBot, CCBot, and PerplexityBot. A developer chasing a 'page not indexed' error in Search Console runs the checker and finds the page is blocked by a wildcard rule they forgot about. An ecommerce admin notices their faceted-search URLs are crawled even though they thought they'd blocked them — the rule's path didn't match the actual URL pattern.
Common Mistakes to Avoid
Don't use robots.txt to keep pages out of Google search results — Google can still index a blocked URL based on inbound links, just without the snippet. Use a noindex meta tag for that. Don't block CSS or JavaScript; Google needs both to render and rank the page properly. Don't put robots.txt on a subdomain expecting it to apply to the root — each (sub)domain needs its own. Don't leave staging configurations in production; staging-style 'Disallow: /' is the most common SEO disaster on relaunch day. Don't use absolute URLs in Disallow paths; only path components are honored.
Security Implications
Robots.txt is a polite request, not a security control. Listing /admin in robots.txt tells search engines to skip it — and tells attackers exactly where to look. The safer pattern is to keep sensitive paths out of robots.txt entirely, behind authentication and not linked anywhere public. If you need to block crawlers from a path, do it with HTTP-level auth, network ACLs, or noindex headers, not with a public Disallow rule. And treat robots.txt as part of your perimeter: monitor it for unexpected changes the same way you monitor DNS and certificates.
Best Practices
Keep robots.txt short — under 50 lines is plenty for most sites. Always declare your sitemap with a full absolute URL. Use User-agent: * for general policy and add specific blocks only when you mean to differentiate. Test before deploying with this checker and with Search Console's robots.txt tester. Use crawl-delay sparingly; Google ignores it and other crawlers may interpret it differently. Re-audit after every CMS upgrade, theme change, or platform migration — defaults differ wildly. And document why each Disallow rule exists in a comment so the next engineer doesn't undo it.
Troubleshooting Guide
Tool returns 404? Your site isn't serving robots.txt at all — create one. Returns 500? The route is broken (often a CMS plugin gone wrong). Pages you expected to be indexed are missing from Google? Run the checker and confirm they aren't blocked. Pages you expected to be blocked are showing up in search results? Robots.txt only blocks crawling, not indexing — switch to noindex. Crawler ignoring the file entirely? It's either a malicious bot (use rate-limiting or a WAF) or your file returns the wrong Content-Type (it should be text/plain). Sitemap URL marked invalid in Search Console? Confirm the URL in your Sitemap: directive is exact, absolute, and HTTPS.
Frequently Asked Questions
What is a robots.txt file?+
A robots.txt file is placed in the root directory of a website and tells search engine crawlers which pages they are allowed or not allowed to access and index.
Does blocking a page in robots.txt remove it from Google?+
No. It prevents Googlebot from crawling it, but does not remove it from results if other pages link to it. To remove a page, use the noindex meta tag instead.
Where should robots.txt live?+
Always at the root of the domain — example.com/robots.txt. Crawlers don't look anywhere else. A robots.txt on a subdomain only applies to that subdomain.
Should I block AI crawlers in robots.txt?+
Optional. Add lines like "User-agent: GPTBot" + "Disallow: /" to block OpenAI's training crawler, and similar entries for Google-Extended, ClaudeBot, etc. Well-behaved AI crawlers respect these rules; bad ones won't.
How do I test changes safely?+
Use this checker and Google Search Console's robots.txt tester before pushing live. A wrong rule can deindex an entire site within days, and recovering takes weeks.
Where exactly should robots.txt live?+
At the root of each domain or subdomain: https://example.com/robots.txt. Crawlers never look anywhere else.
Can a single robots.txt control multiple domains?+
No. Each domain and each subdomain needs its own file. A robots.txt at example.com does not apply to api.example.com.
Will Disallow remove a page from Google's index?+
No. Disallow stops Google from crawling the page, but the URL can still appear in search results based on external links. Use noindex to remove it.
What's the difference between Disallow and noindex?+
Disallow lives in robots.txt and blocks crawling. Noindex is a meta tag (or X-Robots-Tag header) and removes a crawled page from the index. They solve different problems.
Should I block ChatGPT and other AI crawlers?+
Optional. To block training, add User-agent blocks for GPTBot, Google-Extended, ClaudeBot, CCBot, and PerplexityBot with Disallow: /. Well-behaved AI crawlers respect these directives.
Does case matter in Disallow rules?+
Paths are case-sensitive in URLs and therefore in robots.txt rules. /Admin and /admin are different paths to a crawler.
How do I block a specific file type?+
Use the $ end-of-URL anchor: 'Disallow: /*.pdf$' blocks every URL that ends in .pdf. Most modern crawlers honor the * and $ wildcards.
How to audit a robots.txt file
- 1
Enter the domain
Type the bare domain — the tool fetches /robots.txt automatically. There's no separate URL to provide.
- 2
Read the user-agent blocks
Each block begins with one or more User-agent lines. Confirm the right crawlers are addressed and that '*' fallback rules match your intent.
- 3
Audit Disallow rules
Walk every Disallow line and confirm none of them block paths you actually want indexed. A single 'Disallow: /' will deindex an entire site.
- 4
Verify Allow exceptions
Allow lines re-open subpaths inside a Disallow. Order and specificity matter — the more specific rule wins.
- 5
Confirm the Sitemap directive
Make sure 'Sitemap: https://yourdomain.com/sitemap.xml' is present. It's how crawlers discover your sitemap on first visit.
Learn More — Related Guides
Related Tools — More Security & Privacy
SSL Checker
Check SSL certificate validity, issuer, and expiry date for any website. Free online SSL checker.
Password Generator
Generate strong, secure random passwords instantly. Free password generator with custom options.
Password Strength Checker
Test the strength of your password and get tips to make it more secure. Free online tool.
Email Validator
Validate any email address format instantly. Free online email checker, no signup required.
IP Blacklist Checker
Check if your IP address is listed on any spam or blacklist database. Free online tool.
Encrypt / Decrypt
Encrypt and decrypt text online for free. Simple, fast and secure encryption tool.