---
title: "Edge Crawler & Indexer - Serpwise Documentation"
description: "Automatically warm your edge cache and build a structural index of your pages for AI features."
url: "https://serpwise.ai/docs/edge-crawler/"
source: "https://serpwise.ai/docs/edge-crawler/"
---
[Back to Serpwise](/)

Serpwise Documentation

Connect · Configure · Execute

Search documentation

`/`

No documentation found.

↑↓ Navigate Enter to open

Browse documentation

The **Edge Crawler & Indexer** is an asynchronous background system powered by Bastio AI (Firecrawl). It systematically navigates your website through the Serpwise Edge Proxy to accomplish two major goals:

1. **Cache Warming:** By visiting your URLs, the crawler forces a “Cache Miss”, causing Serpwise to run your rules and store the final HTML in memory. When real visitors or Googlebot arrive later, they get instant “Cache Hits”.
2. **Page Indexing:** The crawler extracts clean, LLM-ready Markdown, raw HTML, and Meta Titles from every page it visits. This structured index powers future Serpwise features (like AI Product Feeds, automatic XML Sitemap generation, and cross-site context for Gemini).

## Setting Up Crawlers

You can configure multiple independent crawlers for a single domain (e.g., a “Daily Products Crawl” and a “Weekly Blog Crawl”).

1. Navigate to your Domain Dashboard and click the **Edge Crawler** tab.
2. Click **Add Crawler**.

### Configuration Options

- **Name:** A friendly identifier (e.g., “XML Sitemap Warmer”).
- **Schedule:** Choose how often this crawler should run automatically (`Daily`, `Weekly`, `Monthly`, or `Manual Only`).
- **Start URL:** The entry point for the crawler. We highly recommend using an **XML Sitemap URL** (e.g., `https://example.com/sitemap_products.xml`), as this is the most efficient way to discover all your important pages. You can also use your homepage (`/`).
- **Max Pages Limit:** A safety ceiling. The crawler will stop once it hits this many pages, preventing you from accidentally burning through all your credits on an infinitely deep website.
- **Max Depth:** How many “clicks” deep from the Start URL the crawler should go. If you use a Sitemap, a Depth of `1` is usually sufficient.
- **Include / Exclude Paths:** You can restrict the crawler to only visit specific sections of your site (e.g., Include `*/products/*`, Exclude `*/checkout/*`).

## Manual Execution

Even if a crawler is on a schedule, you can trigger it instantly at any time:

1. Find the crawler in the **Saved Crawlers** table.
2. Click the **Run Now** (Play) button.
3. A banner will appear indicating the crawler is running, and you’ll see the “Pages” count update in real-time as webhooks arrive.

## The Page Index

Below the crawler configurations, you will find the **Indexed Pages** table. This is a unified view of every URL the crawler has successfully discovered, fetched, and saved to the database.

It displays:

- The URL Path
- The final HTTP Status Code (returned *after* Serpwise rules are applied)
- The extracted `<title>` tag
- The relative time since it was last updated

## What Crawling Enables

The page index built by the Edge Crawler powers a wide range of URL Intelligence features:

[**SEO Audit Engine**  
→  
\
Automated 50+ point checks run on every crawled page - scoring, categories, and actionable fixes.](/docs/seo-audit)

[**Content Analysis**  
→  
\
Word count, readability, keyword extraction, and duplicate detection for every indexed page.](/docs/content-analysis)

[**Internal Link Graph**  
→  
\
Map internal link structure, find orphan pages, and measure link depth.](/docs/link-graph)

[**Change Detection**  
→  
\
Detect SEO-critical changes between crawls - title, canonical, robots, and content modifications.](/docs/change-detection)

[**Sitemaps**  
→  
\
Auto-generate XML and HTML sitemaps from your crawled page index.](/docs/sitemaps)

[**Product Feeds**  
→  
\
Generate product feeds from pages with Product schema markup.](/docs/product-feeds)