AI Markdown Serving

Serve clean Markdown to AI crawlers instead of HTML — improving how ChatGPT, Claude, Perplexity, and other AI systems understand and cite your content.

AI Markdown Serving detects AI crawler requests (GPTBot, ClaudeBot, PerplexityBot, CCBot, and others) and responds with a clean Markdown version of your page instead of the full HTML. Regular visitors and search engine crawlers continue to receive the normal HTML response — nothing changes for them.

The Markdown output strips navigation, sidebars, footers, scripts, advertisements, and other page chrome. What remains is the meaningful content that AI models can index, understand, and cite effectively.

AI Markdown Serving is a domain-level feature. When enabled, it applies to all AI bot requests for that domain — there is no per-page filtering. Use the Rules Engine with request.bot_name conditions if you need to block specific bots from specific paths.

How It Works

AI Bot Request → Proxy → Origin Response (HTML)
  ├─ IsAIBot? + Markdown Serving enabled?
  │     YES → Strip chrome → Convert to Markdown → Serve as text/markdown
  └─     NO → Normal HTML pipeline (rules → modifications → HTML)
  1. The gateway receives the HTML response from your origin
  2. Bot detection identifies the request as an AI crawler
  3. The HTML is parsed and page chrome is stripped (nav, footer, sidebar, scripts, styles, ads, etc.)
  4. If a Content Selector is configured, only the matching element's content is extracted
  5. Additional Remove Selectors are applied
  6. The cleaned HTML is converted to Markdown
  7. Optional YAML frontmatter (title, description, URL) is prepended
  8. The response is returned with Content-Type: text/markdown; charset=utf-8

Header-modification rule actions (set_response_header, remove_response_header) still apply to Markdown responses. HTML-modification actions (inject_html, set_meta_robots, set_canonical, etc.) are skipped.

Enabling AI Markdown Serving

  1. Navigate to your domain in the dashboard
  2. Click the AI Markdown tab
  3. Toggle Enable AI Markdown Serving
  4. Configure any advanced options (see below)
  5. Click Save Markdown Settings

The gateway cache is invalidated automatically when you save. Changes take effect on the next AI crawler request.

Configuration Options

Content Selector

A CSS selector that targets the main content area of your pages. Only the HTML inside the matched element is included in the Markdown output.

article
main
#content
.post-body

If the selector matches nothing on a given page, the gateway falls back automatically: main → article → [role="main"] → full <body>.

Leave blank to let the gateway use this fallback chain on every page.

Remove Selectors

Additional CSS selectors to strip from the page before conversion, one per line. Use this for site-specific elements that aren't caught by the default chrome-stripping.

Example:

.author-bio
.related-posts
#comments
.newsletter-cta

The gateway always strips the following elements regardless of this setting:

nav, footer, header, aside, script, style, noscript, iframe, svg, form, .sidebar, .menu, .nav, .footer, .header, .ads, .advertisement, .cookie-banner, .popup, .modal

Up to 20 selectors, 200 characters each.

Include YAML Frontmatter

When enabled (default: on), the Markdown output starts with a YAML frontmatter block containing page metadata:

---
title: "Getting Started with SerpWise"
description: "A step-by-step guide to setting up your first proxy domain."
url: "https://example.com/docs/getting-started"
source: "https://example.com/docs/getting-started"
---

# Getting Started with SerpWise
...

AI models use frontmatter to understand the document's context and provenance. Disable this if your content pipeline already includes structured metadata.

When enabled (default: on), hyperlinks are preserved as Markdown link syntax: [link text](https://example.com).

When disabled, link text is retained but URLs are removed — useful if your content contains many internal navigation links you don't want AI models to follow.

Include Images

When enabled (default: on), images are preserved as Markdown image syntax: ![alt text](https://example.com/image.jpg).

When disabled, image references are removed entirely from the output.

Markdown Output Example

Given an article page with navigation and sidebars, the output looks like:

---
title: "10 SEO Tips for E-Commerce Sites"
description: "Practical SEO advice for online stores."
url: "https://example.com/blog/seo-tips"
source: "https://example.com/blog/seo-tips"
---

# 10 SEO Tips for E-Commerce Sites

Good product page SEO starts with structured data. Here are ten techniques that
consistently improve visibility in both traditional and AI-powered search.

## 1. Use Schema.org Product Markup

Add `Product` structured data to every product page. Include `name`, `description`,
`price`, `availability`, and `image` at minimum.

    {
      "@context": "https://schema.org",
      "@type": "Product",
      "name": "Wireless Headphones",
      "price": "79.99"
    }

## 2. Write Descriptive Alt Text

Every product image should have alt text that describes the item clearly.
Avoid generic text like "image1.jpg" — AI models use alt text to understand
what the image depicts and associate it with your content.

Which AI Crawlers Does This Apply To?

AI Markdown Serving activates for any request the gateway identifies as an AI bot, including:

BotUser-Agent Pattern
GPTBotGPTBot
ChatGPT-UserChatGPT-User
ClaudeBotClaudeBot
CCBotCCBot
PerplexityBotPerplexityBot
Google-ExtendedGoogle-Extended
Applebot-ExtendedApplebot-Extended

Regular search engine crawlers (Googlebot, Bingbot, etc.) are not affected — they receive the normal HTML response with all rule modifications applied.

Analytics

When AI Markdown Serving is enabled, the AI Markdown tab shows:

  • Total Markdown views — number of AI bot requests served as Markdown over the selected period
  • Bot breakdown — which AI crawlers are requesting your content most frequently

Each Markdown-served request is also logged in the standard request log with a markdown_serving modification entry.

For a broader view of how AI crawlers interact with your content — including AI readiness scoring and optimization recommendations — see AI Visibility Tracking.

Limitations

  • HTML only — PDF, images, JSON, and other non-HTML content types are passed through unchanged
  • Domain-wide — no per-page or per-path filtering; use the Rules Engine to block specific AI bots from specific paths
  • No caching — Markdown is converted on each request; caching of converted output may be added in a future release
  • 20 remove selectors maximum — each up to 200 characters
  • Content Selector: 200 character maximum
  • Graceful fallback — if HTML parsing fails for any reason, the original HTML response is returned unchanged; no errors or 500s

AI Markdown vs. Other Features

FeatureWho sees itWhat it changes
AI Markdown ServingAI crawlers onlyResponse format (HTML → Markdown)
AI Attribution TrackerAI crawlers onlyRewrites links with tracking parameters
Code InjectionAll visitorsInjects CSS/JS into every HTML response
Rules EngineConfigurable (any condition)Meta tags, canonicals, headers, structured data

You can enable AI Markdown Serving and AI Attribution together — the attribution tracker runs only for HTML responses, so there is no conflict.

Page Studio

Visually edit any existing page on your domain — click elements, modify styles and SEO tags, use AI for changes, then deploy edits as proxy rules.

AI Meta Generation

Generate optimized meta titles and descriptions using AI, review suggestions with SERP previews, and apply them as per-URL overrides served by the gateway.

On this page

How It Works
Enabling AI Markdown Serving
Configuration Options
Content Selector
Remove Selectors
Include YAML Frontmatter
Include Links
Include Images
Markdown Output Example
Which AI Crawlers Does This Apply To?
Analytics
Limitations
AI Markdown vs. Other Features