---
title: "AI bots see different content than your visitors - Serpwise"
description: "Do not assume every crawler will render client-side JavaScript. Compare the raw response with the rendered page and keep primary content in the HTML."
url: "https://serpwise.ai/learn/ai-bots-see-different-content/"
source: "https://serpwise.ai/learn/ai-bots-see-different-content/"
---
[← Back to Learn](/learn/)
AI & Bot Visibility Critical
# AI bots see different content than your visitors
Do not assume every crawler will render client-side JavaScript. Compare the raw response with the rendered page and keep primary content in the HTML.
When the page a visitor sees and the page a crawler receives differ in meaningful content - title, headings, body copy, structured data - discovery becomes less reliable. A client-rendered page can look complete in a browser while its initial HTML contains little useful content.
This is rarely intentional cloaking. It is usually a rendering pipeline assumption: “modern crawlers execute JavaScript.” Google can render JavaScript, but rendering capabilities and timing vary across crawlers. The safe implementation is to include the primary content and metadata in the initial HTML response.
## The two failure modes
**1. JS-only rendering.** Your page is a thin HTML shell that hydrates content client-side. To a browser, it looks fine. To `curl` with a bot user-agent, the response is `` and a script tag.
**2. Geo, A/B, or personalization branches that key on user-agent or IP.** A WAF returns a stripped homepage for “suspicious” traffic. An A/B test serves a control variant based on a cookie that bots don’t have. A geo-redirect sends bots to a default-locale page with no content.
Both produce the same crawler experience: stripped-down, non-canonical, lower-ranking.
## How to detect it
Compare the raw HTML the bot receives against what a real browser renders.
```
# What GPTBot actually sees
curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot" \
https://example.com/ > bot.html
```
In Chrome, “View Page Source” shows the initial HTML response. Inspect Element shows the post-JavaScript DOM. Compare both with a fetch using the crawler’s documented user agent. A meaningful difference identifies content that depends on rendering.
Word count is a useful first signal, but not a pass or fail threshold. A large difference between the crawler response and the rendered page deserves investigation.
Google’s URL Inspection (Search Console) shows the rendered HTML Googlebot stored. Divergence between (raw source) ↔ (Googlebot rendered) ↔ (live page) is a problem.
## The fix
The fix is the same regardless of platform: render meaningful content server-side.
### Universal - what HTML must contain before any JS runs
```
Page topic - Brand
The page's primary topic
The actual content, as real text - not data injected later.
Discoverable internal links
```
If `curl` shows this much, the bot sees this much.
### WordPress
Server-rendered by default. The risk is plugins that move content client-side - sliders, “reveal on scroll” galleries that hydrate into empty containers, AJAX-loaded tabs. Audit any plugin that says “lazy-load” or “AJAX content.” For each, confirm the first paint contains real text.
### Shopify
Server-rendered by default. Two pitfalls: apps that inject product descriptions via JS (cross-sell, dynamic pricing), and themes that defer “below the fold” sections behind JS. Test with curl + a bot UA.
### Next.js (App Router)
Use Server Components. Avoid `"use client"` for components that render primary content.
```
// app/products/[slug]/page.tsx
export default async function ProductPage({ params }: { params: { slug: string } }) {
const product = await getProduct(params.slug); // server-side fetch
return (
{product.title}
{product.description}
);
}
```
Avoid `ssr: false` dynamic imports for SEO-critical UI. If you need a client component for interactivity, pass server-rendered HTML through `children` so the text reaches the response.
### Nuxt / Astro / SvelteKit
Same principle. Static or server rendering for content routes. Client hydration is fine for interactivity, but the first HTML response must contain the words.
### Cloudflare Workers / edge
If a Worker injects personalization, branch on bot UA and bypass:
```
export default {
async fetch(request: Request) {
const ua = request.headers.get("user-agent") ?? "";
const isBot =
/GPTBot|ClaudeBot|PerplexityBot|OAI-SearchBot|Googlebot/i.test(ua);
const origin = await fetch(request);
if (isBot) return origin; // crawlers always get the canonical version
return personalize(origin);
},
};
```
## Cloaking vs rendering bug - Google’s line
Google’s spam policy bans **intent-based cloaking**: showing different content to rank for terms you don’t actually cover. It does not ban serving identical content over a different transport.
Serving server-rendered HTML to bots and the same content (post-hydration) to users is fine. Serving “Top 10 Watches” to Googlebot and a payday-loan landing page to users is not.
Same words → safe. Different topic, copy, or links → cloaking, regardless of intent.
## Pitfalls
**Don’t user-agent sniff to inject content for bots only.** That is cloaking. The fix is to render real content server-side for everyone.
**Don’t trust Inspect Element as proof your crawler view is correct.** Inspect shows the post-JavaScript DOM. Compare the raw response and an actual fetch using the crawler’s documented user agent.
**Don’t assume Google visibility proves every other crawler receives the same page.** Different products use different crawlers and processing systems. Test the response delivered to each one you care about.
## Fix at the edge with Serpwise
Moving a hydration-heavy SPA to SSR can be a substantial project. An edge-rendered transition can make the same primary content available in the initial response while the origin is improved.
Serpwise can detect the AI user agent, run a headless render of your origin, cache the resulting HTML, and serve that to bots while regular visitors continue to receive the SPA. Same content, different transport, no origin code change.
[See pricing](/pricing) or run a [free AI visibility audit](/audit).
From diagnosis to deployment
## Find the issue. Ship the fix.
Use Learn to understand the problem, then run Serpwise against your own site to see what can be approved and deployed.
[Run free audit](/audit/) [Book a demo](/demo/)