Robots.txt Tester, Validator and Generator

Enter your root domain and the tester fetches /robots.txt, parses it into user-agent groups, and records the line number of every rule it finds. Pick the crawlers you care about, enter the paths you want to check, and each row of the results table answers one question: can this crawler fetch this URL?

The column that matters most is the last one. Knowing a URL is blocked is only half an answer — you need to know which rule blocked it before you can fix anything. Every row names the directive and the line it sits on, so you can open your file and go straight to it.

Paths accept full URLs as well as bare paths. Paste https://example.com/blog/post and the tester uses the path portion, so you can copy a URL out of the browser bar without editing it first.

Most robots.txt checkers do naive prefix matching and read the file top to bottom, first match wins. Real crawlers do neither. The behaviour is specified in RFC 9309 and it has four rules that trip up simple implementations:

• Wildcards. * matches any sequence of characters and $ anchors the end of a path, so Disallow: /*.pdf$ blocks PDFs anywhere on the site.
• Longest match wins, not first match. Rule order in the file is irrelevant. The rule with the longest matching path decides the outcome.
• Allow beats Disallow on a tie. When two rules match with the same specificity, the crawl is allowed.
• One group applies, not all of them. A crawler uses the single most specific user-agent group that names it. If GPTBot has its own group, the * group is ignored for GPTBot entirely — including any rules you assumed were global.

That last one causes the most damage in practice. Adding a narrow GPTBot group to a file with strict * rules does not add restrictions to GPTBot; it replaces them, often opening up paths you meant to keep closed.

AI crawlers are not interchangeable, and the names do not map to products the way people expect. The distinction that catches most sites out is between Googlebot and Google-Extended.

• Googlebot — Google Search. Blocking it removes you from Search and from AI Overviews together.
• Google-Extended — Gemini and AI Overviews grounding. Separate from Googlebot, so you can stay in Search while opting out of AI use.
• GPTBot — OpenAI's training and browsing crawler. Blocking it has no effect on Google rankings.
• PerplexityBot, ClaudeBot, Applebot-Extended, CCBot — Perplexity, Claude, Apple Intelligence and Common Crawl respectively. CCBot is worth attention because Common Crawl feeds many downstream models, so blocking it has a wider effect than its traffic suggests.

If you want a fuller picture of which AI crawlers can currently reach your site, the AI Crawler Checker runs the same parse across the full AI bot roster in one pass.

Almost every serious robots.txt problem is silent. Crawlers skip lines they do not understand rather than reporting an error, so a typo does not fail loudly — it just stops protecting whatever it was meant to protect.

• Misspelled directives. Dissallow is ignored entirely, so the path you thought was blocked is fully crawlable.
• An HTML page served at /robots.txt. Many sites return their styled 404 page with a 200 status. Crawlers receive markup instead of directives and the site effectively has no robots.txt.
• Rules before any User-agent line. A Disallow that appears before the first User-agent belongs to no group and applies to nobody.
• Missing leading slash. Disallow: admin does not block /admin.
• No Sitemap directive. Not an error, but it is the first place AI crawlers look. If yours is missing, the Sitemap Health Checker will tell you whether the sitemap itself is in good shape first.
• Blocking CSS and JS. Crawlers that cannot render your page cannot evaluate its content properly. Blocking /assets/ or /_next/ is a common own goal.

Everything else in GEO — the structure of your answers, your schema markup, the entities you establish — assumes a crawler can read the page. Robots.txt is the gate in front of all of it, and it is the cheapest thing on the list to get wrong, because a single mistyped line takes effect immediately and reports nothing.

There is a subtlety worth understanding. Blocking a URL stops the fetch, not the mention. An engine that learned about your page from somewhere else can still cite it — it simply cannot read your words, so the description it gives is assembled from third-party sources. For a brand that is the worst outcome available: discussed everywhere, described by everyone else. If you genuinely want a page out of AI answers, allow the crawl and control it at the page level instead.

Once crawler access is confirmed, the next constraint is usually whether your content is structured in a way engines can quote. The Content Structure Analyzer and the Schema Markup Checker cover that ground.