Free Robots.txt + Sitemap Validator
Drop in your domain. We fetch and parse your robots.txt and sitemap.xml — flagging the common silent killers (accidental disallows, blocked AI bots, dead sitemap URLs, malformed XML).
Want it in your inbox?
A senior strategist will email a written read of what your output means, plus a free one-domain audit. We don't sell the list. We don't pre-load a sequence.
Three steps. About a minute end to end.
- 01
Enter your domain
Just the bare hostname. We fetch robots.txt and the sitemap declared inside it.
- 02
Read the issue list
Each issue comes with severity (block, warn, info) and the line number in your file.
- 03
Fix and re-test
Robots.txt edits go live the moment you deploy. Re-run this check to confirm the fix.
Running this on a live engagement? A strategist will walk it with you.
Schedule a BriefingWhat this tool does — and what it doesn't.
What does robots.txt do?
It tells crawlers (Googlebot, Bingbot, GPTBot, ClaudeBot, etc.) what they can and cannot fetch. A misconfigured robots.txt can silently deindex your entire site — we see this in production audits more often than you would believe.
Should I block AI crawlers?
Probably not. AI engines (ChatGPT, Perplexity, Claude, Gemini) are now a meaningful share of discovery traffic — for some B2B sites, more than Google. Blocking GPTBot or ClaudeBot is leaving inbound traffic on the table. The exception is sites with proprietary content that should not be reproduced.
What is a valid sitemap?
A well-formed XML file at a stable URL, declared in robots.txt and submitted to Search Console / Bing Webmaster Tools. Each URL is canonical (no trailing-slash drift), reachable (not 404), and current (lastmod accurate). Stale sitemaps waste crawl budget.
Can I have multiple sitemaps?
Yes. Most large sites use a sitemap index file that points to multiple child sitemaps (articles, products, categories, etc.). Each child sitemap caps at 50,000 URLs or 50MB uncompressed.
How often should I update my sitemap?
Automatically, on every content change. A sitemap that has not been updated in six months tells Google your site is not being maintained — even if you are publishing weekly.
A tool tells you what's broken.
A strategist tells you what to do about it.
Fifteen minutes with a senior strategist. We bring your audit, your competitors' weak spots, and one growth move worth running this quarter. No deck. No fee.
Begin Private Audit