AI Crawler Checker Free
Enter your domain. We read the live robots.txt, then show which AI crawlers can read your site and which ones you have shut out without meaning to.
How it works
- We fetch the file, you do not. A browser is not allowed to read a file on another domain, so the read happens on our server and never through anyone else. Enter the domain and the live rules come back. A file that is not live yet still goes in the paste tab, where it is parsed in your browser and nothing is uploaded.
- It reads the file the way a crawler does. Directives are grouped by user agent, the longest matching path wins, and Allow beats Disallow on a tie, with * and $ wildcards supported.
- Each crawler gets its own verdict. A named group is used when it exists, otherwise the crawler falls back to User-agent: *, which is where most accidental blocks come from.
- The live read is the home page rule. The engine reads the rules that apply to /. To test /pricing or any other path, paste the file and set the path.
Answer crawlers and training crawlers are two decisions
Blocking them is not one policy. One costs you citations, the other is a licensing choice. Score them separately or you will make the wrong call.
Answer crawlers
They build the index an assistant cites and links. Block one and you disappear from that surface's source list. This is a traffic decision, and the answer is almost always to allow.
Training crawlers
They collect pages for model training. Refusing them is a legitimate commercial position with no direct traffic cost today. It is a licensing decision, not an SEO one.
The line is blurring
Several operators use one token for both, and the split changes without notice. Re-check the file every quarter rather than setting it once and forgetting it.
The mistakes that cost the most
| Mistake | What it looks like | What it costs |
|---|---|---|
| Wildcard block | User-agent: * / Disallow: / | Every AI crawler, and every search engine, is shut out. Usually a staging file that shipped. |
| Blocking Googlebot to stop Gemini | User-agent: Googlebot | Removes you from Google Search entirely. Google-Extended is the Gemini control, and it is a different token. |
| Two tokens, one brand | ClaudeBot allowed, anthropic-ai blocked | A contradictory policy nobody intended. Keep paired tokens identical. |
| Blocking a live fetch token | ChatGPT-User: Disallow: / | A person pasted your link into a chat and the assistant cannot open it. That is a lost visit, not a training question. |
Questions
How does the tool read my robots.txt?
Our server fetches it. A browser is blocked from reading a file on another domain unless that domain returns an Access-Control-Allow-Origin header, and almost no site sets one on robots.txt, so the read happens server side and never through a third party proxy. Entering a domain runs a full site audit, which is what produces the crawler rows, and the report is saved to your project so the analyzer and the workspace show the same read. If you already ran an audit on that domain, the tool reuses it and tells you when it ran. The paste tab is still there for a file that is not live yet, and anything you paste is parsed in your browser and never uploaded.
Does blocking AI crawlers hurt my visibility in AI answers?
It depends on which crawler. Answer engine crawlers such as OAI-SearchBot and PerplexityBot build the index those products cite, so blocking them removes you from their source lists and their links. Training crawlers such as GPTBot, ClaudeBot and CCBot feed model training instead, which is a separate commercial decision. The line between the two is blurring, so treat a full block as a real cost to how often your brand gets named, not a free privacy win.
Does blocking Google-Extended affect my Google Search rankings?
No. Google-Extended is a separate control for whether Google uses your pages for Gemini training and grounding. Google Search crawling and ranking run on Googlebot, which is a different token. Blocking Google-Extended leaves Googlebot untouched, and blocking Googlebot removes you from Search entirely.
Do AI crawlers actually obey robots.txt?
The major operators publish their tokens and say they honour the file, and in practice the named ones do. It is a request, not a wall: robots.txt has no enforcement, so it will not stop a scraper that ignores it. If you need an actual block, that is a server rule or a firewall rule by user agent and IP range, not a line in this file.
Next step
Letting the crawlers in is step one. Being worth citing is the rest.
Run the free analyzer and see your SEO and GEO position in full: what AI engines can read, what they will quote, and what is missing.