Free tools · No signup
Free robots.txt generator for AI crawlers
Choose which AI bots can read your site, then copy the rules into your robots.txt.
AI answer crawlers
OAI-SearchBotOpenAIFinds pages for ChatGPT search answers
Claude-SearchBotAnthropicIndexes pages for Claude's search results
Claude-UserAnthropicOpens pages when someone asks Claude
PerplexityBotPerplexityFinds and links pages in Perplexity answers
Google-ExtendedGoogleGemini app answers and training (not Google Search)
BingbotMicrosoftFeeds Copilot as well as Bing
ApplebotAppleSiri, Spotlight and Safari suggestions
AmazonbotAmazonAlexa and Rufus answers
Training-only crawlers
GPTBotOpenAITraining data for OpenAI models
ClaudeBotAnthropicTraining data for Claude models
Applebot-ExtendedAppleTraining Apple's AI models (doesn't crawl)
CCBotCommon CrawlBulk corpus used to train many models
anthropic-aiAnthropicOlder Anthropic training token
BytespiderByteDanceTraining crawl, not answers
Your robots.txt
# AI crawler rules, made with the free generator at raechal.ai/tools/robots-txt-generator User-agent: OAI-SearchBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Google-Extended User-agent: Bingbot User-agent: Applebot User-agent: Amazonbot User-agent: GPTBot User-agent: ClaudeBot User-agent: Applebot-Extended User-agent: CCBot User-agent: anthropic-ai User-agent: Bytespider Allow: / User-agent: * Allow: /
How the rules work
Each crawler reads the group that names it. A crawler with its own group ignores the User-agent: * group, which is why any private folders you add here are repeated in every allowed group. Crawlers you leave out follow the * group.
Answer crawlers fetch pages so an AI tool can answer questions and cite sources. Training-only crawlers collect data to train models. You can allow one kind and block the other.
Where to put your robots.txt
Any site
Upload the file to the root of your site so it opens at yourwebsite.com/robots.txt.
WordPress
Most SEO plugins, such as Yoast and Rank Math, include a robots.txt editor.
Webflow and Wix
Edit robots.txt in the site's SEO settings.
Shopify
Edit the robots.txt.liquid template in your theme.
Then run the AI crawler checker to confirm each bot sees what you expect.
Frequently asked questions
Will blocking AI crawlers hurt my Google rankings?+
Not on its own. Google Search crawls with Googlebot, and blocking Google-Extended or GPTBot doesn't change what Googlebot can read. Keep the User-agent: * group so Googlebot stays allowed.
Should I block training crawlers?+
It's your choice. Blocking them asks those companies not to use your pages to train their models. Vendors treat their search and training crawlers separately: OpenAI, for example, says blocking GPTBot doesn't block OAI-SearchBot.
What does "Leave out" do?+
Nothing is written for that crawler, so your User-agent: * group decides what it can read.
I already have a robots.txt. Should I replace it?+
Add the new groups to your existing file instead of replacing it, so you keep the rules you already rely on. Then check the result with the free AI crawler checker.
Does robots.txt stop every bot?+
No. It's a request, not a lock. The major AI companies say their crawlers follow it, but a crawler that chooses to ignore it can still read your pages.
Is this generator free?+
Yes, with no signup. The file is built in your browser.
