--- title: "AI Markdown Serving - Serpwise Documentation" description: "Serve clean Markdown to AI crawlers instead of HTML so ChatGPT, Claude, Perplexity, and other AI systems can understand and cite your content." url: "https://serpwise.ai/docs/ai-markdown/" source: "https://serpwise.ai/docs/ai-markdown/" --- [Back to Serpwise](/) Serpwise Documentation Connect · Configure · Execute Search documentation `/` No documentation found. ↑↓ Navigate Enter to open Browse documentation **AI Markdown Serving** detects AI crawler requests (GPTBot, ClaudeBot, PerplexityBot, CCBot, and others) and responds with a clean Markdown version of your page instead of the full HTML. It now lives inside **AI Access → Markdown**, alongside crawler policy, attribution, and discovery settings. Regular visitors and search engine crawlers continue to receive the normal HTML response - nothing changes for them. The Markdown output strips navigation, sidebars, footers, scripts, advertisements, and other page chrome. What remains is the meaningful content that AI models can index, understand, and cite effectively. * * * ## How It Works ``` AI Bot Request → Proxy → Origin Response (HTML) ├─ IsAIBot? + Markdown Serving enabled? │ YES → Strip chrome → Convert to Markdown → Serve as text/markdown └─ NO → Normal HTML pipeline (rules → modifications → HTML) ``` 1. The gateway receives the HTML response from your origin 2. Bot detection identifies the request as an AI crawler 3. The HTML is parsed and page chrome is stripped (nav, footer, sidebar, scripts, styles, ads, etc.) 4. If a **Content Selector** is configured, only the matching element’s content is extracted 5. Additional **Remove Selectors** are applied 6. The cleaned HTML is converted to Markdown 7. Optional YAML frontmatter (title, description, URL) is prepended 8. The response is returned with `Content-Type: text/markdown; charset=utf-8` Header-modification rule actions (`set_response_header`, `remove_response_header`) still apply to Markdown responses. HTML-modification actions (`inject_html`, `set_meta_robots`, `set_canonical`, etc.) are skipped. * * * ## Enabling AI Markdown Serving 1. Navigate to your domain in the dashboard 2. Open **GEO → AI Access** 3. Click the **Markdown** tab 4. Toggle **Enable AI Markdown Serving** 5. Configure any advanced options (see below) 6. Click **Save Markdown Settings** The gateway cache is invalidated automatically when you save. Changes take effect on the next AI crawler request. * * * ## Configuration Options ### Content Selector A CSS selector that targets the main content area of your pages. Only the HTML inside the matched element is included in the Markdown output. ``` article main #content .post-body ``` If the selector matches nothing on a given page, the gateway falls back automatically: `main` → `article` → `[role="main"]` → full ``. Leave blank to let the gateway use this fallback chain on every page. * * * ### Remove Selectors Additional CSS selectors to strip from the page before conversion, one per line. Use this for site-specific elements that aren’t caught by the default chrome-stripping. **Example:** ``` .author-bio .related-posts #comments .newsletter-cta ``` The gateway always strips the following elements regardless of this setting: `nav`, `footer`, `header`, `aside`, `script`, `style`, `noscript`, `iframe`, `svg`, `form`, `.sidebar`, `.menu`, `.nav`, `.footer`, `.header`, `.ads`, `.advertisement`, `.cookie-banner`, `.popup`, `.modal` Up to 20 selectors, 200 characters each. * * * ### Include YAML Frontmatter When enabled (default: on), the Markdown output starts with a YAML frontmatter block containing page metadata: ``` --- title: "Getting Started with Serpwise" description: "A step-by-step guide to setting up your first proxy domain." url: "https://example.com/docs/getting-started" source: "https://example.com/docs/getting-started" --- # Getting Started with Serpwise ... ``` AI models use frontmatter to understand the document’s context and provenance. Disable this if your content pipeline already includes structured metadata. * * * ### Include Links When enabled (default: on), hyperlinks are preserved as Markdown link syntax: `[link text](https://example.com)`. When disabled, link text is retained but URLs are removed - useful if your content contains many internal navigation links you don’t want AI models to follow. * * * ### Include Images When enabled (default: on), images are preserved as Markdown image syntax: `![alt text](https://example.com/image.jpg)`. When disabled, image references are removed entirely from the output. * * * ## Markdown Output Example Given an article page with navigation and sidebars, the output looks like: ``` --- title: "10 SEO Tips for E-Commerce Sites" description: "Practical SEO advice for online stores." url: "https://example.com/blog/seo-tips" source: "https://example.com/blog/seo-tips" --- # 10 SEO Tips for E-Commerce Sites Good product page SEO starts with structured data. Here are ten techniques that consistently improve visibility in both traditional and AI-powered search. ## 1. Use Schema.org Product Markup Add `Product` structured data to every product page. Include `name`, `description`, `price`, `availability`, and `image` at minimum. { "@context": "https://schema.org", "@type": "Product", "name": "Wireless Headphones", "price": "79.99" } ## 2. Write Descriptive Alt Text Every product image should have alt text that describes the item clearly. Avoid generic text like "image1.jpg" - AI models use alt text to understand what the image depicts and associate it with your content. ``` * * * ## Which AI Crawlers Does This Apply To? AI Markdown Serving activates for any request the gateway identifies as an AI bot, including: | Bot | User-Agent Pattern | |-------------------|---------------------| | GPTBot | `GPTBot` | | ChatGPT-User | `ChatGPT-User` | | ClaudeBot | `ClaudeBot` | | CCBot | `CCBot` | | PerplexityBot | `PerplexityBot` | | Google-Extended | `Google-Extended` | | Applebot-Extended | `Applebot-Extended` | Regular search engine crawlers (Googlebot, Bingbot, etc.) are **not** affected - they receive the normal HTML response with all rule modifications applied. * * * ## Analytics When AI Markdown Serving is enabled, the **AI Markdown** tab shows: - **Total Markdown views** - number of AI bot requests served as Markdown over the selected period - **Bot breakdown** - which AI crawlers are requesting your content most frequently Each Markdown-served request is also logged in the standard request log with a `markdown_serving` modification entry. For crawler fetchability, origin persona controls, and robots.txt content signals, use [AI Access](/docs/ai-access). For a broader view of how AI crawlers interact with your content - including AI readiness scoring and optimization recommendations - see [AI Visibility Tracking](/docs/ai-visibility). * * * ## Limitations - **HTML only** - PDF, images, JSON, and other non-HTML content types are passed through unchanged - **Domain-wide** - no per-page or per-path filtering; use the Rules Engine to block specific AI bots from specific paths - **No caching** - Markdown is converted on each request; caching of converted output may be added in a future release - **20 remove selectors maximum** - each up to 200 characters - **Content Selector: 200 character maximum** - **Graceful fallback** - if HTML parsing fails for any reason, the original HTML response is returned unchanged; no errors or 500s * * * ## AI Markdown vs. Other Features | Feature | Who sees it | What it changes | |----------------------------------|------------------------------|-------------------------------------------------------------------| | [**AI Access**](/docs/ai-access) | AI and search crawlers | Fetch policy, origin persona, content signals, and serving format | | **AI Markdown Serving** | AI crawlers only | Response format (HTML → Markdown) | | **AI Attribution Tracker** | AI crawlers only | Rewrites links with tracking parameters | | **Code Injection** | All visitors | Injects CSS/JS into every HTML response | | **Rules Engine** | Configurable (any condition) | Meta tags, canonicals, headers, structured data | You can enable AI Markdown Serving and AI Attribution together - the attribution tracker runs only for HTML responses, so there is no conflict.