Robots.txt Validation: Every Directive, Every Common Mistake

Robots.txt Validation: Every Directive, Every Common Mistake

1 min read

robots.txt · seo · crawlers · tools

Authors

PandaONE Team

Validate robots.txt syntax, understand Allow/Disallow/Sitemap, and learn why robot.txt (singular) is ignored — plus how AI crawlers read your rules.

Staging shipped with Disallow: / still in place, or the file is named robot.txt and nobody notices until search traffic flatlines. A validator fetches /robots.txt and shows which paths each user-agent may crawl, plus syntax errors that make crawlers ignore the file.

robots.txt vs robot.txt

The only valid name is robots.txt — plural, lowercase — at the site root (https://example.com/robots.txt). robot.txt is a frequent typo; major crawlers (including GPTBot, ClaudeBot, and PerplexityBot when configured) will not treat it as your policy file.

Validate in the browser

  1. Open the robots.txt analyzer.
  2. Enter the origin (https://example.com, no path).
  3. Read Allow/Disallow blocks per User-agent, Sitemap lines, and parser errors.
  4. Confirm production did not inherit a blanket Disallow: / from staging.

Directives

DirectiveRole
User-agentWhich crawler the next rules apply to
DisallowPaths the agent should not fetch
AllowExceptions inside a broader Disallow
SitemapPointer to your XML sitemap
Crawl-delayHint — many crawlers ignore it

robots.txt is not access control

Well-behaved bots honor the file. Attackers and misconfigured scrapers do not. Sensitive paths need authentication and authorization, not a Disallow line. When you have fixed auth, AI VAPT can exercise the app — robots rules alone prove nothing about exploitability.

Featured on ScrollLaunchFeatured on CodeHype