02 · Bench
Free robots.txt Checker for AI Crawlers
Check which AI and search crawlers robots.txt allows or blocks, inspect the matching rules, and understand what each directive controls.
Check one public robots.txt
Enter a domain or page URL. The checker normalizes it to the public origin and retrieves only /robots.txt.
Free, no account required. One robots file is fetched; no site crawl and no LLM call occurs.
How the result is produced
The checker parses robots.txt by user agent and explains the documented purpose of each crawler without collapsing them into one AI-access score.
Check the full site in Site AuditA robots.txt looks like a list. It is a selection procedure.
Open a robots.txt and you read it top to bottom like house rules. A crawler does something else: it first finds the block matching its own name, and reads only that. Everything above and below does not exist for it.
That single property produces the most common mistake in technical SEO. Someone adds a block for GPTBot to stop training and assumes the general rules under `User-agent: *` still apply. They do not. The directory that was meant to be closed to everyone is, from that moment, open to GPTBot - not because anyone opened it, but because a second block replaced the first.
The second classic concerns order. Google does not decide by which rule comes first, but by which describes the longer path. An `Allow: /blog/guides/` far down the file overrides a `Disallow: /blog/` at the top. Both rules are correctly written; the outcome is rarely the intended one.
Which is why this tool checks the result rather than the syntax: what may this specific bot do with this specific path? A syntactically flawless file can achieve the exact opposite of what it appears to say.
What AI robots.txt Checker checks
A robots.txt is not one verdict about your site. It is a set of rule blocks evaluated per crawler. The checker reads the file the way a bot reads it: it finds the block matching its own name and applies only that one.
This is where the most common misunderstanding lives. Many people expect the rules under `User-agent: *` to apply on top of everything else. They do not: once a crawler finds a block bearing its own name, it ignores the wildcard block entirely. A well-meant general rule can therefore be inert for precisely the bots it was written for.
The check also works out which rule wins when several match. Google does not decide by order but by length: the longest matching path prevails, and where two are equally long, `Allow` beats `Disallow`. A rule further down the file can override one further up.
The terms, briefly
- User-agent
- Opens a rule block for one crawler. A bot uses only the block that matches its name most precisely - every other block might as well not exist for it.
- Disallow
- Forbids fetching paths with this prefix. An empty value means the opposite: everything is allowed.
- Allow
- Carves an exception out of a broader disallow. Needed when a whole directory is blocked but individual files inside it should stay reachable.
- Specificity
- When several rules match, the one with the longest path pattern wins, not the one listed first. At equal length, Allow takes precedence over Disallow.
- Crawl-delay
- A wait between fetches. Google ignores the directive entirely and other crawlers interpret it differently, which makes it unreliable as a control.
Reading the result
Allowed for search
That provider's search crawler finds no block excluding it and may fetch the path you checked.
Nothing.
Blocked for other AI systems
A training or grounding crawler is blocked while the same provider's search crawler is not. That is a valid and often deliberate separation.
Only verify the split was intended rather than inherited from a template.
Not covered by the wildcard block
A block exists specifically for this bot, so the general rules under User-agent: * do not apply to it.
Repeat the intended rule inside that bot's own block. There is no inheritance here.
No robots.txt found
The server returns nothing usable. Without a robots.txt everything counts as allowed.
Only act if you actually meant to restrict certain areas.
Three mistakes we see constantly
Relying on inheritance from the wildcard block
As soon as a crawler finds a block with its name, that block is the only one that applies to it. Rules under `User-agent: *` become inert for it - every restriction you still want has to be repeated in the bot's own block.
Assuming order beats length
A rule further up does not automatically win. Google evaluates the longest matching path pattern. An `Allow: /blog/guides/` further down overrides a `Disallow: /blog/` further up - often unintentionally, in both directions.
Blocking a page to get it out of the index
A URL blocked by robots.txt can stay in the index, because Google knows it from links and merely may not read it. Removal needs a noindex - and for that, the fetch has to be allowed.
Questions about AI robots.txt Checker
Does User-agent: * apply in addition to my bot-specific block?
No. A crawler picks exactly one block - the one matching its name most precisely - and ignores all others. If you add a block for GPTBot, the rules under * no longer apply to GPTBot. Anything that should still hold has to be repeated there.
Is robots.txt enough to remove a page from Google?
No, and this is one of the most expensive misunderstandings in technical SEO. robots.txt prevents crawling, not indexing. Google can still list a blocked URL if links point to it - just without a description. To remove it, apply a noindex and allow the fetch so Google can actually see it.
How do I block AI training without losing visibility?
Block the training crawlers and allow the search crawlers. For OpenAI: block GPTBot, allow OAI-SearchBot. For Anthropic: block ClaudeBot, allow Claude-SearchBot. For Google, Google-Extended governs AI use while Googlebot continues to serve ordinary search.
What happens if my robots.txt has a syntax error?
Crawlers skip lines they do not understand and evaluate the rest. That is more dangerous than a hard failure: a misspelled directive does not announce itself, it simply has no effect. Which is why this check evaluates the actual outcome per bot, not just the syntax.
Does Crawl-delay belong in a modern robots.txt?
Not for Google - the directive is ignored. Other crawlers treat it inconsistently. If server load is the real problem, rate limiting at the server or CDN and a clean sitemap are the reliable instruments.
Primary sources
The claims on this page come from the providers' own documentation, not from secondhand summaries. Every source is linked so you can read what it actually says.
Continue with Contextter
Rules change without anyone changing them
Most robots.txt problems are not created when the file is written but afterwards - during a relaunch, a CDN migration, a plugin update.
- Re-evaluates access rules on every run and reports when the outcome changes for a bot.
- Shows the effect on URLs that actually exist rather than only validating the file.
- Maps crawl blocks to the affected pages so you can see what the mistake actually costs.
No payment details required. The free check above stays usable without an account.
Related free tools
Where this continues in Contextter
Use Site Audit to verify whether robots, CDN, or application controls block important pages at scale.
Continue in ContextterReady when you are.
Start the check in the form above. Resource-intensive runs require a short browser verification and use strict limits.
Inputs are used only for the selected check. Browser-only tools never upload pasted text or files.
No payment details required. The free check above stays usable without an account.