01 · Bench
Free AI Crawler Checker
Check whether a public page can be fetched, indexed, and meaningfully extracted by search and AI crawlers.
Check one public page
Enter the exact page URL. The checker fetches bounded HTML and the final origin's robots.txt without rendering JavaScript.
The URL is used only for this check. No LLM, paid crawler, proxy bypass, account data, or authenticated content is used.
How the result is produced
One bounded HTML and robots.txt fetch, evaluated with deterministic rules. The tool will not invent a citation probability.
Check the full site in Site AuditAlmost nobody blocks AI systems on purpose
Few companies deliberately decide that OpenAI should not crawl their pages for ChatGPT search. Yet sites make that choice by accident: a robots.txt copied from an old article, a bot-protection rule switched on at the CDN, or a staging configuration shipped to production. The result can limit how their content is discovered and used in ChatGPT search without anyone having made a conscious decision.
It happens easily because providers run several crawlers with different jobs and confusingly similar names. GPTBot crawls for model training. OAI-SearchBot crawls public pages for ChatGPT search. ChatGPT-User fetches a page because a person requested it. Anyone who blocks all three as “the AI bots” has made three separate choices at once, usually without noticing.
That is also why an access check cannot usefully answer “allowed” or “blocked”. The useful answer is a breakdown: who may do what, and was that the intention? Anything less reproduces the exact confusion that caused the problem.
What AI Crawler Checker checks
An AI system can only use your page if it can reach it, read it, and make sense of it. The checker walks those three stages separately: does a request get as far as the HTML, does the page itself permit indexing, and once navigation, banners and scripts are stripped away, is there enough connected prose left to quote a statement from?
The distinction that matters most here is between crawler roles. OpenAI, Google and Anthropic each run several bots with different purposes: one for training data, one for the search index, and one for the live fetch when someone pastes a link into a chat. Blocking those three has three entirely different consequences - and folding them into a single rule usually locks you out of something you meant to keep.
The check is deliberately narrow: a bounded HTML fetch plus the robots.txt of the final destination, without executing JavaScript. It tells you whether access is open. It does not tell you whether your content is good enough to be cited - no fetch test decides that.
The terms, briefly
- GPTBot
- OpenAI's crawler for training data. Blocking it keeps your content out of future model training. It does not remove you from ChatGPT search - a different bot handles that.
- OAI-SearchBot
- Crawls public pages for ChatGPT search. Blocking it prevents OpenAI from crawling the page for summaries and snippets, but a link and page title may still appear if the URL is discovered elsewhere. Access is a prerequisite, not a citation guarantee.
- ChatGPT-User
- The live fetch: a user names a URL in the chat, or the model follows a link. No training, no index - a single request on behalf of one specific person.
- Google-Extended
- Controls whether your content is used for Gemini and Google's AI answers. It has no effect on your ranking in ordinary Google Search - Googlebot is untouched by it.
- ClaudeBot / Claude-SearchBot / Claude-User
- Anthropic's split follows the same pattern: training, search index, live fetch. Here too, a blanket block hits all three purposes at once.
Reading the result
200 OK · indexable
The page responds, and neither a meta robots tag nor an X-Robots header forbids indexing.
Nothing. Access is open - move on to the content itself.
Allowed for search
That provider's search crawler may fetch the page. It can appear as a source in their answers.
Nothing. Just check that this holds for every provider that matters to you, not only one.
Blocked for training
A deliberate choice, if it was made deliberately. Your content stays out of model training while remaining visible in search.
Only act if the block was unintended - for instance inherited from a copied robots.txt.
Blocked for search
The search crawler cannot get through. The page cannot appear as a source in that AI system, however good it is.
Check robots.txt for an over-broad disallow block and separate training from search.
No extractable text
After navigation and boilerplate are removed, too little connected prose remains. Common with content rendered purely on the client.
Consider server-side rendering or static HTML for the sections that carry the content.
Three mistakes we see constantly
Blocking every AI bot in one rule
A `Disallow: /` under a lumped-together user-agent block hits training, search index and live fetch at once. Anyone who only meant to stop training also loses their visibility as a source.
Confusing robots.txt with noindex
robots.txt controls crawling, not indexing. A blocked URL can still end up in the index - just without its content being read. To keep a page out, you need a noindex, and you have to allow the fetch so it can be seen.
Only checking the homepage
Access rules often differ by path. An open homepage says nothing about whether your guides, product pages or blog are reachable.
Questions about AI Crawler Checker
Does blocking GPTBot hurt my Google rankings?
No. GPTBot is an OpenAI crawler and has nothing to do with Googlebot. Your Google Search rankings are unaffected. Likewise, Google-Extended only governs AI use of your content, not classic ranking.
What is the difference between GPTBot and OAI-SearchBot?
GPTBot crawls content for model training. OAI-SearchBot crawls public pages so they can be considered in ChatGPT search. You can block GPTBot while allowing OAI-SearchBot to separate training from search crawling. That access does not guarantee a citation.
Why can't ChatGPT see my page even though robots.txt allows it?
The usual causes: the content is loaded by JavaScript and is not in the served HTML; a meta robots tag or X-Robots header forbids indexing; or the page is reachable but simply is not chosen as the most relevant source for that question. Access is the precondition, not the guarantee.
Do I have to allow AI crawlers at all?
That is a business decision, not a technical one. If you want reach through AI answers, allow at least the search crawlers. If you want to protect your work from training, block the training crawlers. You can do both at once - that is exactly why the user agents are separate.
How often should I check this?
After any deployment that touches robots.txt, CDN rules, WAF settings or rendering. In practice most unintended blocks come not from a decision but from an inherited configuration or a bot-protection rule.
Primary sources
The claims on this page come from the providers' own documentation, not from secondhand summaries. Every source is linked so you can read what it actually says.
Continue with Contextter
Checking one URL is diagnosis. Checking a domain is control.
This check tells you whether one page is reachable. The question that follows is usually: does that hold for the other two thousand? And did I know when it changed?
- Crawls the whole domain instead of a single URL, and shows where access rules differ by path.
- Compares each run to the last - a new block surfaces instead of going unnoticed for months.
- Ties the technical finding to the pages that actually carry traffic, rather than weighting every error equally.
No payment details required. The free check above stays usable without an account.
Related free tools
Where this continues in Contextter
The free checker inspects one public URL. Open a full Site Audit to verify the same blockers across the entire site.
Continue in ContextterReady when you are.
Start the check in the form above. Resource-intensive runs require a short browser verification and use strict limits.
Inputs are used only for the selected check. Browser-only tools never upload pasted text or files.
No payment details required. The free check above stays usable without an account.