The short answer
To get content indexed by AI search engines, make sure three things are true: the AI search crawlers are allowed to fetch your pages, your pages are in the Google and Bing indexes those engines search, and the content is present in the HTML your server sends, not added later by JavaScript. Then help discovery with an accurate sitemap, IndexNow and internal links, and confirm in your server logs that the crawlers are actually arriving.
This is the first step of making your business show up in AI search. Nothing else on that list works until it is done.
Which AI crawlers exist, and which ones matter
Every major AI company runs more than one crawler, and they do different jobs. Mixing them up is the most common way sites block themselves from AI answers by accident.
| Company | Search and answers | User-requested fetches | Training |
|---|---|---|---|
| OpenAI | OAI-SearchBot | ChatGPT-User | GPTBot |
| Anthropic | Claude-SearchBot | Claude-User | ClaudeBot |
| Perplexity | PerplexityBot | Perplexity-User | (not used for training, per Perplexity) |
| Googlebot (AI Overviews and AI Mode use the Search index) | Google user-triggered fetchers | Google-Extended token (Gemini) | |
| Microsoft | Bingbot (Copilot uses Bing) | n/a | n/a |
The rule of thumb: if you want to appear in AI answers, allow the crawlers in the first two columns. Blocking a training crawler like GPTBot or ClaudeBot does not remove you from search answers, and Google states that Google-Extended does not affect inclusion in Google Search, which is where AI Overviews come from.
Our robots.txt rules for GPTBot, PerplexityBot and the rest show how to allow search crawlers while opting out of training.
How AI search engines find your pages
Most AI answers are built from a search that runs at the moment someone asks. The engine queries an index, reads some of the results and writes an answer. So "indexed by AI" mostly means indexed by the search systems behind each engine:
- Google AI Overviews and AI Mode use Google's own Search index.
- Microsoft Copilot uses Bing.
- ChatGPT search uses OpenAI's own crawl (OAI-SearchBot) plus third-party search providers, of which Bing is the best known.
- Perplexity runs its own index built by PerplexityBot and also fetches pages live.
- Claude uses web search when it needs current information, with Claude-SearchBot and Claude-User fetching pages.
These systems discover pages the usual ways: links from pages they already know, XML sitemaps, and, for engines that support it, IndexNow notifications. A page with no internal links and no sitemap entry can sit undiscovered for a long time.
The Bing connection
Many sites have worked on Google for years and never touched Bing. That matters more now, because Bing powers Copilot and is one of the providers behind ChatGPT search. If Bing has not indexed a page, you have removed one route by which ChatGPT could find it. Bing indexing is not a guarantee of citation, but it is cheap and it eliminates a common reason for being missing.
- Verify the site in Bing Webmaster Tools and submit your XML sitemap. You can import a verified site from Google Search Console.
- Turn on IndexNow, a protocol Bing supports, so new and updated URLs are announced the moment they publish. Many CMS platforms and CDNs offer it as a setting or plugin.
- Check Bing-specific blocks. Some robots.txt files allow Googlebot by name and block everything else by wildcard, which shuts Bingbot out.
- Keep canonicals clean. Pages whose canonical points elsewhere are usually left out of the index.
For why this matters to ChatGPT specifically, see how Bing indexing feeds ChatGPT search.
Content that only appears after JavaScript runs
Googlebot renders JavaScript. Most AI crawlers do not: Vercel's analysis of AI crawler traffic found that the major AI crawlers fetch JavaScript files but do not execute them (Vercel, "The rise of the AI crawler"). If your headings, body text, prices or reviews are injected by client-side code, those crawlers receive a page without them.
- Test what the server sends. View the page source (not the inspector) or fetch the URL with a command-line tool. If the main text is not in that HTML, AI crawlers cannot read it.
- Render on the server. Server-side rendering or static generation puts the content in the first response. Most modern frameworks support it per page.
- Do not hide answers behind clicks. Tabs and accordions are fine if their text is in the HTML; content loaded only when a user clicks is not.
Indexing code examples and documentation
Developer documentation and technical pages have their own failure modes. Best practices for getting code content indexed by AI search engines:
- Put code in text, not images. Use
preandcodeelements so the snippet can be read and quoted. - Explain each snippet in a sentence. Engines extract the prose around a code block to decide what it answers.
- One task per URL. "How to authenticate with the API" on its own page is easier to retrieve than one long reference page.
- Label versions. State which version a page applies to and keep old versions on separate URLs, so answers do not mix them.
- Server-render docs sites. Many docs platforms render in the browser; check the page source as above.
How to speed up indexing
- Keep an accurate sitemap with real
lastmoddates, submitted to both Google Search Console and Bing Webmaster Tools. - Ping IndexNow on every publish and update.
- Link new pages from existing ones. A link from a page crawlers already visit often is the fastest discovery route.
- Request indexing for important new pages with the URL inspection tools in Search Console and Bing Webmaster Tools.
- Keep responses fast and stable. Timeouts and server errors during a crawl leave pages unindexed.
Monitoring AI crawlers in your server logs
Analytics tools usually filter out bots, so your server or CDN logs are the only reliable record of which AI crawlers visit and what they get back. Filter the user agent for OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Perplexity-User, Googlebot and bingbot.
Look at three things: which URLs they request, how often they return, and the status codes they receive. Repeated 403 or 5xx responses to a search crawler usually mean a firewall rule or an overloaded server is blocking it. User agents can be spoofed, so verify important findings against the IP ranges each company publishes.
AI indexing checklist
- robots.txt allows the search and user-request crawlers you want.
- Firewall and CDN bot rules do not challenge those crawlers.
- Main content is present in the server HTML.
- Site verified in Google Search Console and Bing Webmaster Tools, sitemap submitted to both.
- IndexNow enabled.
- Every important page linked from at least one other page.
- Logs show search crawlers receiving 200 responses.
Frequently asked questions
Does blocking GPTBot remove me from ChatGPT answers?
No. GPTBot collects training data. ChatGPT search relies on OAI-SearchBot and on search providers, and ChatGPT-User fetches pages for users. Block GPTBot only if you want to opt out of training.
Can AI search engines read JavaScript-rendered pages?
Google renders JavaScript, so AI Overviews can use rendered content. Most other AI crawlers do not run scripts, so put important content in the HTML your server returns.
How do I know if my pages are indexed for AI search?
Check Google Search Console and Bing Webmaster Tools for each important URL, search for a distinctive sentence from the page on both engines, and check your logs for visits from the AI search crawlers.
How long does indexing take?
With a submitted sitemap, IndexNow and internal links, new pages are often indexed within days. Pages with no links and no sitemap entry can take much longer or never be found.
Do I need an llms.txt file to be indexed?
No. Indexing depends on crawler access, sitemaps and links. Some AI tools read llms.txt as a page map, but Google Search ignores it.
Check your own site
The quickest first step is to check which AI crawlers can read your pages: the free check reads your robots.txt and response headers for each AI crawler and flags the ones you are blocking.
