--- title: "AI bots are blocked from your site - Serpwise" description: "A robots.txt or WAF rule can limit AI training, search inclusion, or on-demand retrieval. Audit each crawler separately before changing access." url: "https://serpwise.ai/learn/ai-bots-blocked/" source: "https://serpwise.ai/learn/ai-bots-blocked/" --- [← Back to Learn](/learn/) AI & Bot Visibility Critical # AI bots are blocked from your site A robots.txt or WAF rule can limit AI training, search inclusion, or on-demand retrieval. Audit each crawler separately before changing access. A `Disallow: /`, a 403 from your WAF, or an automated bot rule can stop a crawler from retrieving public pages. The effect depends on which crawler is blocked: model training, search inclusion, and user-triggered retrieval use different controls. ## What this actually blocks Major AI products publish named crawlers with different purposes. Treat them as separate controls: | Crawler or token | Primary purpose | |-----------------------------------|-----------------------------------------------------------------------------| | `GPTBot` | OpenAI foundation-model training | | `OAI-SearchBot` | Inclusion in ChatGPT search results | | `ChatGPT-User` | User-triggered page visits and actions, not search inclusion | | `ClaudeBot` | Anthropic model development | | `Claude-User`, `Claude-SearchBot` | Claude user-triggered retrieval and search | | `PerplexityBot` | Perplexity search index | | `Perplexity-User` | Perplexity user-triggered retrieval | | `Googlebot` | Google Search, including AI Overviews and AI Mode | | `Google-Extended` | A Google control for certain Gemini model uses, not Google Search inclusion | | `CCBot` | Common Crawl collection | `Google-Extended` is a robots.txt product token, not a separate crawler. Blocking it does not remove a page from Google Search and does not control whether the page can appear in AI Overviews or AI Mode. Those search features are governed by Googlebot and the normal Google Search controls. ## How to detect it Two layers. Check both. ``` GET /robots.txt HTTP/1.1 Host: example.com ``` Look for: ``` User-agent: GPTBot Disallow: / ``` Or a global block: ``` User-agent: * Disallow: / ``` Then verify the live HTTP response per bot - a permissive robots.txt is meaningless if the WAF returns 403. ``` for ua in \ "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot" \ "Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)" \ "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" do echo "== $ua" curl -sI -A "$ua" https://example.com/ | head -1 done ``` An expected successful status and a normal page response are evidence that the crawler can reach the URL. A `403`, repeated `429`, or an interstitial challenge page means the delivery layer is overriding the robots.txt policy. ## The fix ### Universal robots.txt ``` User-agent: * Allow: / User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / User-agent: CCBot Allow: / Sitemap: https://example.com/sitemap.xml ``` `Allow: /` is the explicit form. Removing the `Disallow` lines works too - by spec, anything not explicitly disallowed is allowed. ### WordPress (Yoast / Rank Math) Both plugins auto-generate `robots.txt`. Override in the dashboard (Yoast: SEO → Tools → File editor; Rank Math: General Settings → Edit robots.txt). Or drop a static `robots.txt` in the WordPress root - a real file beats the plugin’s virtual one. ### Shopify Shopify generates `robots.txt` from `robots.txt.liquid`. Edit it in the theme code editor: ``` {% for group in robots.default_groups %} {{- group.user_agent }} {%- for rule in group.rules -%} {{ rule }} {%- endfor -%} {%- endfor %} User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / ``` ### Cloudflare Two things to check, both in **Security → Bots**: 1. **AI Scrapers and Crawlers** - set to off, or set to “Allow” for the specific bots you want. 2. **Block AI Bots** managed rule - off. Then audit any custom WAF rules for `cf.client.bot` matches. The default Cloudflare bot fingerprint marks AI crawlers as “bot,” and a rule like `Block when (cf.client.bot)` blocks them all. ### Akamai / Imperva Akamai Bot Manager → AI Bot category → set policy to Allow. Imperva Advanced Bot Protection → AI Bot category → Allow. ## Pitfalls **Don’t filter by user-agent string alone for sensitive paths.** User agents are trivial to spoof. OpenAI, Anthropic, and Perplexity publish their egress IP ranges - verify by IP if access matters. For public marketing content, allow openly and don’t overthink it. **Don’t use one policy for every AI crawler.** Training, search, and user-triggered retrieval are separate purposes. Decide which uses you support, then configure the matching crawler or token. **Don’t confuse `Google-Extended` with Google Search.** Googlebot controls inclusion in Google Search, including AI Overviews and AI Mode. Google-Extended does not affect Search inclusion or ranking. ## Fix at the edge with Serpwise The block usually lives in a layer the SEO team doesn’t own - a WAF rule from years ago, a CMS-generated robots.txt that fights manual edits, a hosted vendor. Serpwise can manage robots.txt and edge access rules for known crawler user agents before traffic reaches your origin. That gives the domain owner one place to review and change crawler access without waiting for an origin release. [See pricing](/pricing) or run a [free AI visibility audit](/audit). From diagnosis to deployment ## Find the issue. Ship the fix. Use Learn to understand the problem, then run Serpwise against your own site to see what can be approved and deployed. [Run free audit](/audit/) [Book a demo](/demo/)