When the page a visitor sees and the page an AI crawler receives differ in meaningful content — title, headings, body copy, structured data — the crawler indexes what it received. Which is usually almost nothing.
This is rarely intentional cloaking. It’s almost always a rendering pipeline assumption: “modern crawlers execute JS.” Googlebot does, mostly. GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, CCBot — do not, as of 2026.
The two failure modes
1. JS-only rendering. Your page is a thin HTML shell that hydrates content client-side. To a browser, it looks fine. To curl with a bot user-agent, the response is <div id="root"></div> and a script tag.
2. Geo, A/B, or personalization branches that key on user-agent or IP. A WAF returns a stripped homepage for “suspicious” traffic. An A/B test serves a control variant based on a cookie that bots don’t have. A geo-redirect sends bots to a default-locale page with no content.
Both produce the same crawler experience: stripped-down, non-canonical, lower-ranking.
How to detect it
Compare the raw HTML the bot receives against what a real browser renders.
# What GPTBot actually sees
curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot" \
https://example.com/ > bot.htmlIn Chrome, right-click → “View Page Source” gives you the raw response, the same thing the bot sees. Inspect Element shows the post-JS DOM, which is not what the bot sees. Diff source vs Inspect — that delta is what AI crawlers are missing.
The faster signal: word count. If bot.html is under 500 words and the rendered page is 2,000+, the bot is receiving a shell.
Google’s URL Inspection (Search Console) shows the rendered HTML Googlebot stored. Divergence between (raw source) ↔ (Googlebot rendered) ↔ (live page) is a problem.
The fix
The fix is the same regardless of platform: render meaningful content server-side.
Universal — what HTML must contain before any JS runs
<!doctype html>
<html lang="en">
<head>
<title>Page topic — Brand</title>
<meta name="description" content="155-char summary." />
<link rel="canonical" href="https://example.com/page" />
</head>
<body>
<h1>The page's primary topic</h1>
<p>The actual content, as real text — not data injected later.</p>
<a href="/related">Discoverable internal links</a>
<script type="application/ld+json">{ "@context": "https://schema.org", "@type": "Article" }</script>
</body>
</html>If curl shows this much, the bot sees this much.
WordPress
Server-rendered by default. The risk is plugins that move content client-side — sliders, “reveal on scroll” galleries that hydrate into empty containers, AJAX-loaded tabs. Audit any plugin that says “lazy-load” or “AJAX content.” For each, confirm the first paint contains real text.
Shopify
Server-rendered by default. Two pitfalls: apps that inject product descriptions via JS (cross-sell, dynamic pricing), and themes that defer “below the fold” sections behind JS. Test with curl + a bot UA.
Next.js (App Router)
Use Server Components. Avoid "use client" for components that render primary content.
// app/products/[slug]/page.tsx
export default async function ProductPage({ params }: { params: { slug: string } }) {
const product = await getProduct(params.slug); // server-side fetch
return (
<article>
<h1>{product.title}</h1>
<p>{product.description}</p>
</article>
);
}Avoid ssr: false dynamic imports for SEO-critical UI. If you need a client component for interactivity, pass server-rendered HTML through children so the text reaches the response.
Nuxt / Astro / SvelteKit
Same principle. Static or server rendering for content routes. Client hydration is fine for interactivity, but the first HTML response must contain the words.
Cloudflare Workers / edge
If a Worker injects personalization, branch on bot UA and bypass:
export default {
async fetch(request: Request) {
const ua = request.headers.get("user-agent") ?? "";
const isBot = /GPTBot|ClaudeBot|PerplexityBot|OAI-SearchBot|Googlebot/i.test(ua);
const origin = await fetch(request);
if (isBot) return origin; // crawlers always get the canonical version
return personalize(origin);
},
};Cloaking vs rendering bug — Google’s line
Google’s spam policy bans intent-based cloaking: showing different content to rank for terms you don’t actually cover. It does not ban serving identical content over a different transport.
Serving server-rendered HTML to bots and the same content (post-hydration) to users is fine. Serving “Top 10 Watches” to Googlebot and a payday-loan landing page to users is not.
Same words → safe. Different topic, copy, or links → cloaking, regardless of intent.
Pitfalls
Don’t user-agent sniff to inject content for bots only. That is cloaking. The fix is to render real content server-side for everyone.
Don’t trust Inspect Element as proof your bot view is correct. Inspect shows the post-JS DOM. Right-click → View Page Source shows the raw response — what most AI crawlers actually see.
Don’t assume Google ranking proves you’re safe with AI. Googlebot defers rendering by hours to days. GPTBot and ClaudeBot don’t render at all. A page that ranks on Google can be invisible inside ChatGPT.
Fix at the edge with Serpwise
Moving a hydration-heavy SPA to SSR is a multi-quarter project. The faster path is to prerender at the edge — only for AI crawlers.
Serpwise can detect the AI user agent, run a headless render of your origin, cache the resulting HTML, and serve that to bots while regular visitors continue to receive the SPA. Same content, different transport, no origin code change.
See pricing or run a free AI visibility audit.