Free tools · No signup
Free AI crawler checker
See which AI bots your robots.txt lets in: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and more.
What the three results mean
Blocked
A Disallow rule in your robots.txt stops this crawler from reading your homepage. We show the exact line so you can find it.
Allowed
Your robots.txt names this crawler and lets it in. That's a choice your site has made.
Not mentioned
No rule names this crawler, so the User-agent: * group (or nothing) decides. It may be allowed today, but the next robots.txt edit can change that without anyone noticing.
The check reads the rules that apply to your homepage. A rule that only blocks one folder, like /private/, won't show as blocked here.
Answer crawlers and training crawlers
Some crawlers fetch pages so an AI tool can answer questions and cite sources. Others only collect data to train models. Blocking a training-only crawler keeps your pages out of that data set. It doesn't stop the answer engines that read your pages for search results.
How to allow or block an AI crawler
Add a group for the crawler to the robots.txt file at the root of your site. For example:
# Let ChatGPT search read your site
User-agent: OAI-SearchBot
Allow: /
# Keep a training-only crawler out
User-agent: CCBot
Disallow: /A crawler with its own group ignores the User-agent: * group, so repeat any * rules you still want it to follow. Run the check again after you change the file.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT?+
Not by itself. OpenAI uses GPTBot to collect training data and OAI-SearchBot to find pages for ChatGPT search answers. If you want to show up in ChatGPT search, OAI-SearchBot needs access.
Does Google-Extended affect Google Search or AI Overviews?+
No. Google says Google-Extended doesn't affect your inclusion or ranking in Google Search, and AI Overviews are part of Search. Google-Extended controls whether Google can use your content to train Gemini models and to ground answers in Gemini apps.
Why aren't ChatGPT-User and Perplexity-User listed?+
They open pages when a person asks a question, instead of crawling on their own. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them, so a robots.txt check can't tell you what they'll do.
What happens if I don't have a robots.txt?+
Crawlers treat a missing robots.txt as permission to crawl everything. Nothing is blocked, but you haven't made a choice about any crawler either.
My robots.txt returns a server error. Is that a problem?+
Yes. Under the robots.txt standard (RFC 9309), when robots.txt can't be loaded because of a server error, crawlers must assume they can't crawl anything until it loads again.
Does robots.txt actually stop AI bots?+
It's a request, not a lock. The major AI companies say their crawlers follow it, but it can't stop a crawler that chooses to ignore it.
Is this tool free?+
Yes, with no signup. It reads your public robots.txt and nothing else.
