• SEO
Analyze any website's robots.txt file. See which paths are blocked for crawlers, crawl-delay directives, and sitemap URLs.
Fetch and parse any website's robots.txt file. See allowed/disallowed paths, crawl delays, and sitemap references.
Domain or URL only — e.g. example.com or https://example.com/page → checks example.com
• About this tool
A robots.txt validator fetches /robots.txt for a domain and reports which paths each user-agent may crawl, plus syntax problems that cause crawlers to ignore rules. The file must be named robots.txt — plural.
| What this checks |
|---|
| Fetch live robots.txt |
| Per user-agent Allow/Disallow |
| Sitemap references |
| Syntax / structure checks |
• How to
Enter the site origin
Use the apex or www host you want to inspect.
Fetch robots.txt
The tool requests /robots.txt from the origin.
Review rules
Check Disallow/Allow groups and Sitemap lines.
Fix staging leaks
Ensure Disallow: / never shipped from staging to production.
User-agent selects the crawler. Disallow and Allow define path prefixes. Sitemap points crawlers at XML indexes. Crawl-delay is a hint — not universally honored.
| Directive | Purpose |
|---|---|
| User-agent | Which crawler the following rules apply to |
| Disallow | Paths the agent should not fetch |
| Allow | Exceptions inside a broader Disallow |
| Sitemap | URL of your XML sitemap |
| Crawl-delay | Optional delay hint |
Blocking /_next/ or CSS/JS can break rendering for crawlers. Disallow: / on a host promoted from staging removes the whole site. Blocking the sitemap path while listing it elsewhere creates confusion.
robots.txt can advertise your sitemap; the sitemap lists URLs to crawl. Analyze both with the sitemap analyzer after validating robots rules.
No. It is a courtesy protocol. Sensitive paths need authentication. Pair this check with AI VAPT on verified domains for real access-control findings.
Major AI crawlers request /robots.txt. Use explicit User-agent groups if you want to allow or disallow training or retrieval bots separately from Googlebot.
• FAQ
A robots.txt file tells search engine crawlers which pages or sections of a site should not be crawled or indexed.
No. robots.txt is a convention, not enforcement. Malicious bots ignore it. Never rely on robots.txt to protect sensitive content.
Disallow tells crawlers to skip specific URL paths. Disallow: / blocks all pages; Disallow: /admin/ blocks only paths starting with /admin/.
Crawlers assume all content is allowed and will crawl the entire site.
Always robots.txt (plural) at the domain root. robot.txt singular is ignored by Google, Bing, and AI crawlers.
Use this checker to fetch the live file and inspect directives. Fix typos, conflicting Allow/Disallow pairs, and missing Sitemap URLs.
Only if those crawlers honor your Disallow rules. Explicit User-agent blocks for GPTBot and similar are clearer than relying on * alone.
Next step
This free check is point-in-time. PandaONE maps the full surface, then runs proof-backed AI testing on domains you verify.
• Related
Sitemap Analyzer
Fetch and parse any website's XML sitemap. See URL count, priorities, and last-modified dates.
HTTP Security Headers Checker
Audit the HTTP security headers of any website. Score CSP, HSTS, X-Frame-Options, and more.
Redirect Checker
Trace the full redirect chain for any URL. Detect redirect loops, mixed-content downgrades, and unnecessary hops.
• automated security
PandaONE runs autonomous agents against your app, validates findings with proof, and opens fixes as pull requests.
Scan your domain