• SEO
Fetch and parse any website's robots.txt file. See allowed/disallowed paths, crawl delays, and sitemap references.
Domain or URL only — e.g. example.com or https://example.com/page → checks example.com
• About this tool
Analyze any website's robots.txt file. See which paths are blocked for crawlers, crawl-delay directives, and sitemap URLs.
• FAQ
A robots.txt file tells search engine crawlers which pages or sections of a site should not be crawled or indexed.
No. robots.txt is a convention, not enforcement. Malicious bots ignore it. Never rely on robots.txt to protect sensitive content.
Disallow tells crawlers to skip specific URL paths. Disallow: / blocks all pages; Disallow: /admin/ blocks only paths starting with /admin/.
Crawlers assume all content is allowed and will crawl the entire site.
• Related
Sitemap Analyzer
Fetch and parse any website's XML sitemap. See URL count, priorities, and last-modified dates.
HTTP Security Headers Checker
Audit the HTTP security headers of any website. Score CSP, HSTS, X-Frame-Options, and more.
Redirect Checker
Trace the full redirect chain for any URL. Detect redirect loops, mixed-content downgrades, and unnecessary hops.
• automated security
PandaONE runs autonomous agents against your app, validates findings with proof, and opens fixes as pull requests.
Request Access