--- title: "Crawl Guard: Stop Bots Crawling Your Filters | Serpwise" description: "Robots.txt is advice. Crawl Guard enforces it at the edge: Google and Bing get priority, scrapers stay off your filter, sort and search URLs. No regex." url: "https://serpwise.ai/product/crawl-guard/" source: "https://serpwise.ai/product/crawl-guard/" --- Crawl Guard · New # Take control of bot traffic. Reduce the load on your site. Bots can crawl thousands of filter combinations and search results on your website. Crawl Guard stops unwanted traffic before it reaches your server. You decide who can access what. [Book a demo](/demo/) [See how it works](#how-it-works) - Google and Bing can access your content by default - Test your rules in Preview before blocking traffic See how Crawl Guard can be configured for your website. [Hosting provider? See what it does for your servers](#hosting) Crawl Guard Enforcing Magento 2 pack Block, limit and send robots signals. Off Preview Enforce 24 hours 7 days 30 days Example: about 3.2 hours of server time and 11,800 origin requests saved Who gets where Filter URL /women/shoes.html?color=49&size=167 - Google Search Allow - Bing Allow - SEO crawlers Block - AI crawlers Block Sort and search URLs /women/shoes.html?product\_list\_order=price /catalogsearch/result/?q=boots - Google Search Allow - Other bots and scripts Block Product, category and content pages /women/shoes/ankle-boot.html - SEO crawlers Allow - AI crawlers Allow Illustrative example with Magento 2 pack defaults. Server time saved is estimated from your own traffic. Decided at the edge Blocked requests and requests over your limit never reach your server Managed packs We keep the URL patterns and bot lists up to date Search engines protected Verified Google and Bing keep their access by default The crawl budget leak ## Every filter creates new URLs. Bots crawl all of them. Color, size, price and sort order multiply into a near-endless set of URLs. Search engines, SEO tools, AI crawlers and scrapers walk through them, and every new combination is a page your server has to build. [Source: Google Search Central, Crawling December: Faceted navigation (17 December 2024)](https://developers.google.com/search/blog/2024/12/crawling-december-faceted-nav) 01 ### Filters are the top source of overcrawl Google says faceted navigation is by far the most common source of overcrawl issues that site owners report to it. 02 ### Robots.txt is only a request Google caches it for up to 24 hours. Scrapers and fake search bots ignore it completely. Nothing in it is enforced. 03 ### Your server pays The first time a bot asks for a filter combination, it is a cache miss. Your shop builds the page from scratch, for a visitor that never buys. 04 ### The usual fix is regex Hosting teams juggle .htaccess rules and regex to slow bots down. Every change is a ticket, and one wrong pattern blocks real shoppers. Safe by default ## See the impact. Stay in control. You do not need to block anyone to find out what Crawl Guard could do for your website. ### Google has access by default Google and Bing can still visit your content. You decide whether to restrict access to particular URL types. ### Preview before enforcing Preview shows which requests your rules would reject. No requests are blocked while you test your settings. ### Turn it off at any time Switch back to Preview or turn Crawl Guard off from the dashboard. How it works ## Three steps. No engineering ticket. Crawl Guard runs in the Serpwise edge in front of your site. You set it up in the dashboard, and changes go live when you publish them. 1. 1 ### Pick a pack Magento 2, WooCommerce, WordPress or Any website. The pack sorts every page request by what it asks for (filters, search, cart actions and more) and by who is asking. 2. 2 ### Preview Crawl Guard counts what it would block or limit, and blocks nothing. Read the Report, test real URLs and adjust any cell. 3. 3 ### Enforce Blocked bots get a 403. Limited ones get a 429 with Retry-After. Both stop at the edge, before your server does any work. Switch back to Preview or Off at any time. The control grid ## Who is asking, and what they ask for. Rows are bot groups. Columns are kinds of URLs. Every cell is Allow, Limit (requests per minute) or Block. The defaults keep Google and Bing on every page that matters and keep scrapers off crawl traps. Product, category and content pages stay open. You can change any cell. Allow Passes through 60/min Requests per minute, then 429 Block Refused with 403 | | Crawl traps | | | | | | | | | Rest of the shop | |--------------------------------------------|-------------|---------|---------|---------|-------------|--------------|---------|------------|--------|------------------| | Bots / URLs | Filters | Sorting | Search | Session | System URLs | Cart actions | Account | Pagination | Feeds | Rest of site | | Google Search | Allow | Allow | Allow | Allow | Allow | Block | Allow | Allow | Allow | Allow | | Google Shopping and Ads | Allow | Allow | Allow | Allow | Allow | Allow | Allow | Allow | Allow | Allow | | Bing | Allow | Allow | Allow | Allow | Allow | Block | Allow | Allow | Allow | Allow | | Other search engines | Block | Block | Block | Block | Block | Block | Block | Allow | Allow | Allow | | Social and catalog previews | 60/min | 60/min | 60/min | 60/min | Block | Block | Block | Allow | Allow | Allow | | Audit and monitoring tools | 120/min | 120/min | 120/min | 120/min | 120/min | Block | Allow | 120/min | Allow | Allow | | SEO crawlers | Block | Block | Block | Block | Block | Block | Block | 30/min | 30/min | Allow | | AI crawlers | Block | Block | Block | Block | Block | Block | Block | Allow | Allow | Allow | | Cloud and datacenter | 120/min | 60/min | 30/min | Allow | 30/min | 20/min | Allow | 120/min | 30/min | Allow | | Other bots and scripts | Block | Block | Block | Block | Block | Block | Block | 30/min | 30/min | Allow | | Fake search bots | Block | Block | Block | Block | Block | Block | Block | Block | Block | Block | | Visitors | 60/min | 60/min | 30/min | Allow | 60/min | 30/min | Allow | 120/min | 30/min | Allow | | Your sources | | | | | | | | | | | | Python scripts UA contains python-requests | Block | Block | Block | Block | Block | Block | Block | 30/min | 30/min | Allow | Magento 2 pack defaults. Limits are requests per minute. Rest of site means product, category and content pages. The last row is an example of a source you add yourself. Magento 2 pack defaults. Limits are requests per minute. Rest of site means product, category and content pages. The last row is an example of a source you add yourself. Scroll sideways to see every column. ### Limiting Google is your call By default Google and Bing reach every URL except add-to-cart links. If they overcrawl your filters, limit them per column. Crawl Guard shows the cost before you save. ### Add your own sources Create a row from user-agent text, an IP range, a network (ASN) or a country. Match any of them, or require all. ### Named bots share one budget Limits on named bot groups and the cloud row count for the whole group. Visitors and other scripts are counted per IP address, per edge location. Beyond the grid ## The tools you reach for when the bots get loud. Crawl Guard helps you find what to stop, react to an attack, and show what it saved. ### Under attack mode One click makes every bot group stricter for 1, 6, 24 or 72 hours, then it ends on its own. Google is slowed, not blocked. Robots.txt stays as it is. Ends by itself Under attack. Stricter limits are live. Ends in 5h 42m · End now 1 hour 6 hours 24 hours 72 hours ### 404 storm guard Scanners crawl missing pages looking for weak spots. When one client asks for too many different missing pages, Crawl Guard pauses it for a while. Default: 30 missing pages in 1 minute pauses that client for 5 minutes 404 /old-shop/ 404 /backup/ 404 /test/ 404 /shop-v1/ Paused 5 min ### Your own sources Turn a user agent, IP range, network or country into a row in the grid. Verified Google, Bing and Google Shopping crawlers are excluded from your custom source rules. User agent, IP range, ASN, country User agent IP range ASN Country Match any · Match all ### Cloud and datacenter row Traffic from major cloud and hosting networks that doesn't admit to being a bot gets its own row. It shares one budget, so rotating IPs can't multiply the limit. AWS, Google Cloud, Azure and more Cloud and datacenter Cart 20/min Search 30/min Sorting 60/min Filters 120/min ### What we tell good bots Per URL group, choose nothing, a robots.txt Disallow or a noindex header. Crawl Guard writes a marked section into the robots.txt your site serves. Your origin file is untouched. Applied in Enforce only `User-agent: * # BEGIN SerpWise Crawl Guard Disallow: /checkout/cart/add Disallow: /catalogsearch/ Disallow: /*?product_list_order= Disallow: /*&product_list_order= # END SerpWise Crawl Guard` ### Filters we saw on your site See the query parameters your pages received in the last 7 days, with a flag on the ones that look like filters. Add one to a group in a click. Key names only, never values. Finds your shop's real filter names - color Looks like a filter - price Filters - product\_list\_order Sorting - utm\_source Neutral ### Server time saved The Report estimates server time and origin requests saved from your own traffic, lists your top sources, and turns a busy network into a rule with Make a rule. An estimate, clearly labelled as one Top sources · 7 days - Example Cloud Ltd AS64500 Make a rule - Example Hosting AS64501 Make a rule - Example Networks AS64502 Make a rule For hosting providers ## Lighter servers. Fewer support tickets. Crawl Guard enforces bot rules at the Serpwise edge. Blocked requests and traffic over your configured limits are rejected before they reach origin. Control access by bot group and URL type without maintaining .htaccess rules or installing an agent on your customer’s server. ### Control traffic before it reaches origin - Rules by bot group and URL type - Shared limits across IP addresses - Preview rules against real traffic - Estimated savings per website [Book a technical walkthrough](/demo/) [Talk to us about a partnership](/contact/#contact-form) - ### 403 and 429 responses at the edge Blocked requests receive HTTP 403. Requests over the configured limit receive HTTP 429 with Retry-After. Rejected requests are not forwarded to origin. - ### Shared limits across IP addresses Each named bot group and the cloud group share a group-wide limit. Rotating IP addresses does not give the same crawler a fresh allowance. - ### Test rules against real traffic Preview shows which requests your rules would reject without blocking them. Check the impact before switching to Enforce. - ### Temporary rules that expire automatically Tighten bot rules for up to 72 hours with Under attack. The stricter rules expire automatically and do not tighten access to account and checkout pages. - ### No agent on origin Traffic to your customer’s domain must route through Serpwise. Manage rules in the dashboard without plugins or application code changes. - ### Track the impact per website The report estimates saved origin requests and server time from your customer’s traffic. Compare these estimates with your own server measurements. Crawl Guard questions ## What shop owners and hosts ask first. 01 Can Crawl Guard affect my visibility on Google? + - That depends on your settings. By default, Google and Bing can access your content, including filter, sorting and search URLs. Cart, wishlist and compare actions are excluded. Google Shopping and Ads retain full access. Restricting Google or adding robots.txt or noindex directives can affect crawling or indexing. 02 Does it block SEO tools like Ahrefs or Semrush? + - Only on crawl traps. By default SEO crawlers such as AhrefsBot and SemrushBot can't reach filter, sort, search, session and cart URLs, but product, category and content pages stay open to them, and pagination is paced. Audits you run yourself, such as Ahrefs Site Audit or Screaming Frog, count as Audit and monitoring tools: they are paced on crawl traps, not blocked. 03 Does it change my robots.txt? + - In Enforce, yes, unless you choose otherwise. For each URL group you decide what good bots are told: nothing, a Disallow in robots.txt, or a noindex header. The shop packs start with Disallow lines for cart, account, search and sorting URLs, among others, inside a clearly marked Crawl Guard section of the robots.txt your site serves. Your origin file is never edited, Preview changes nothing, and Under attack never touches robots.txt. 04 Do I need to change any code? + - No. Crawl Guard runs in the Serpwise edge that already sits in front of your site. Once your domain routes through Serpwise, you turn it on in the dashboard. No plugin, no .htaccess, no regex and no deploy. 05 What if my site isn't Magento, WooCommerce or WordPress? + - Pick the Any website pack. It covers common login, cart and account paths, plus search, sorting, filter and pagination parameters. Add your own URL groups by path or parameter, and use Filters we saw on your site to find the parameters your site really uses. Shopify stores are not covered. 06 Can real visitors be affected? + - Yes, if they reach a limit you set. The Visitors group can be rate limited but cannot be set to Block. Multiple visitors may share an IP address. Preview shows which requests would have been limited before you enforce your rules. Payment callbacks and POST requests are excluded. 07 Is Crawl Guard available now? + - Crawl Guard is rolling out and must be enabled for your account. Book a demo to discuss your website and access. Creating a Serpwise account does not automatically enable Crawl Guard. 08 What happens to checkout and payments? + - Under attack does not tighten rules for account or checkout pages, and they are excluded from 404 storm protection. POST requests, payment callbacks and authenticated API calls are also excluded. Start in Preview ## Stop paying for crawls you never asked for. Book a demo to see the controls, Preview and reporting. We will help you assess whether Crawl Guard fits your website and explain how to get access. [Book a demo](/demo/) [Contact us](/contact/)