AI Access
Control how AI crawlers and AI tools fetch, read, discover, and attribute your domain.
AI Access is the GEO control center for your domain. It brings crawler fetchability, AI Markdown, AI attribution, and discovery surfaces into one workspace so you can make your site usable by AI systems without weakening your origin security.
Use AI Access when Cloudflare, a WAF, bot protection, robots.txt, or origin behavior prevents AI crawlers from reading the same content real users and search engines can reach.
AI Access is domain-level. It applies to recognized crawler requests for the selected Serpwise domain and does not change the experience for regular visitors.
Workspace Tabs
AI Access is split into four tabs:
| Tab | Purpose |
|---|---|
| Access | Configure crawler policy, origin fetch persona, response format, robots.txt signals, and fetchability tests. |
| Markdown | Serve clean Markdown to AI crawlers instead of full HTML. |
| Attribution | Add tracking parameters to links seen by AI crawlers so AI-referred visits can be measured. |
| Discovery | Enable MCP and /llms.txt discovery surfaces for AI tools. |
Access Controls
The Access tab decides how Serpwise handles recognized AI and search crawlers.
Mode
| Mode | Behavior |
|---|---|
| Monitor | Record what crawlers see without changing the origin fetch persona. Use this first to understand current blocking. |
| Allow | For bots marked Allow, Serpwise can normalize the upstream request sent to your origin while preserving the original crawler identity in logs. |
Origin Fetch Persona
The origin fetch persona controls the User-Agent Serpwise sends upstream when an allowed crawler request is proxied to your origin.
| Persona | Use When |
|---|---|
| Preserve client | Your origin already allows the crawler and you only want visibility and policy logging. |
| Serpwise gateway | Your origin or CDN should allow a stable Serpwise-controlled fetch identity. |
| Browser-like | Your origin blocks crawler user agents but serves browser requests normally. |
Bot Policy
Each recognized crawler can be set to:
| Policy | Behavior |
|---|---|
| Monitor | Observe and log requests without changing origin fetch behavior. |
| Allow | In Allow mode, apply the selected origin fetch persona and AI serving behavior. |
| Block | Return a Serpwise-managed forbidden response for that crawler. |
Core search and user-triggered agents are intentionally not blockable from this panel. Training crawlers and selected AI search crawlers can be blocked when the site owner does not want that class of crawler to access the domain.
Served Format
AI Access coordinates with AI Markdown Serving:
| Format | Behavior |
|---|---|
| Auto | Serve Markdown to allowed AI crawlers when AI Markdown is enabled; otherwise serve HTML. |
| Markdown | Prefer Markdown for allowed crawlers when AI Markdown is enabled. |
| HTML | Force regular HTML responses even for allowed AI crawlers. |
If Markdown conversion fails, the gateway falls back to the original HTML response.
robots.txt and Content Signals
When AI Access is enabled, Serpwise can append policy and content-use declarations to robots.txt responses.
Examples:
User-agent: OAI-SearchBot
Allow: /
User-agent: Google-Extended
Disallow: /
Content-Signal: search=yes, ai-input=yes, ai-train=noUse content signals to declare whether the domain permits search indexing, AI input use, and AI training use. These signals are advisory and depend on crawler support, but they make the site policy explicit.
Fetchability Test
The Fetchability Test runs the AI visibility audit against the selected domain using browser, search, and AI crawler personas. It helps identify whether AI systems are seeing useful content or being blocked by CDN rules, WAF settings, robots.txt, JavaScript-only rendering, or origin firewall behavior.
Run the test after changing AI Access settings, Cloudflare bot rules, or origin firewall rules.
Related Features
Page Studio
Visually edit any existing page on your domain — click elements, modify styles and SEO tags, use AI for changes, then deploy edits as proxy rules.
AI Markdown Serving
Serve clean Markdown to AI crawlers instead of HTML — improving how ChatGPT, Claude, Perplexity, and other AI systems understand and cite your content.