Pages returning HTTP errors

HTTP status codes are the only language crawlers actually trust. A page that returns the wrong code at the wrong time can disappear from the index for weeks. A page that returns the right code recovers in a single crawl.

This guide covers the codes that matter for SEO, what each signals to Googlebot, and how to find the patterns behind error clusters before they pile up.

The codes that matter

Code	Meaning	What Google does
200	OK	Index normally
301	Moved permanently	Pass signals to target, eventually drop old URL
302	Found (temporary)	Keep old URL indexed, do not pass signals strongly
304	Not modified	Reuse cached copy, no re-fetch needed
404	Not found	Drop from index after a few retries
410	Gone	Drop from index faster than 404
451	Unavailable for legal reasons	Drop and mark; respect the legal signal
500	Server error	Keep page, retry — but extended 5xx tanks crawl rate
503	Service unavailable	Treat as temporary if Retry-After is set

Two practical rules:

1. If a page is gone forever and you don’t have a replacement, use 410. It tells Google “stop checking” faster than 404.
2. If your origin is briefly down, return 503 with Retry-After. Persistent 500s convince Google the site is broken and slow the crawl. 503 is the polite signal.

How to detect it

The signal is in your access logs, filtered to crawler user-agents.

# Top error URLs hit by Googlebot in the last 7 days
awk '$9 ~ /^[45]/ && $0 ~ /Googlebot/ { print $9, $7 }' access.log \
  | sort | uniq -c | sort -rn | head -30

For a richer view, use the Pages report in Google Search Console:

• “Not found (404)” — actual 404s Google encountered
• “Server error (5xx)” — sustained 5xx
• “Soft 404” — your origin returned 200 but the page is empty or says “not found” in the body. Soft 404s are worse than real 404s because they waste crawl on dead pages.

The fix

Universal — pick the right code

HTTP/1.1 410 Gone
Content-Type: text/html

Returned for URLs that are gone for good (discontinued products, deleted user profiles).

HTTP/1.1 503 Service Unavailable
Retry-After: 3600

Returned during planned maintenance. Tells Google to come back in an hour rather than re-checking aggressively.

HTTP/1.1 301 Moved Permanently
Location: https://example.com/new-url

Returned when there’s a true replacement URL. Don’t redirect everything to the homepage — that’s interpreted as a soft 404.

WordPress

WordPress returns 200 for deleted pages by default (a “Page not found” template with a 200 status). That’s a soft 404. Fix with the wp_die template override or via a plugin like Yoast (Tools → Crawl optimization → 404 handling).

For permanent removals — discontinued products, deleted authors — use a redirect manager to issue a 410, not a 301 to the homepage.

Shopify

Deleted products redirect to a 404 template that returns the right status. The problem is variants: deleting variant URLs that were previously crawled leaves them in Google’s index with no replacement. Use 301 redirects from the deleted variants to the parent product (Online Store → Navigation → URL redirects).

Nginx

# Permanently gone — use 410
location = /products/discontinued-widget {
  return 410;
}

# Maintenance window — use 503 with Retry-After
location / {
  return 503;
  add_header Retry-After 3600 always;
}

Apache

# Permanently gone
Redirect 410 /products/discontinued-widget

# Maintenance window
RewriteEngine On
RewriteCond %{REQUEST_URI} !=/maintenance.html
RewriteRule ^.*$ /maintenance.html [R=503,L]
Header always set Retry-After "3600"

Next.js (App Router)

// app/products/[slug]/page.tsx
import { notFound } from "next/navigation";

export default async function Page({ params }: { params: { slug: string } }) {
  const product = await getProduct(params.slug);
  if (!product) notFound(); // Next emits 404 with the proper status
  return <Product product={product} />;
}

For 410 Gone, set the status in a route handler:

// app/products/discontinued-widget/route.ts
export function GET() {
  return new Response("This product has been discontinued.", { status: 410 });
}

Pitfalls

Don’t redirect every 404 to the homepage. Google treats large numbers of redirects-to-root as soft 404s. The signal is “this page didn’t exist, and you have nothing to replace it with” — return 410 instead.

Don’t rely on meta robots noindex to remove a page. It works, but only after Google re-crawls. For permanent removals, a 410 is faster and more decisive.

Don’t 301 discontinued products to a similar product without thinking. If the user intent doesn’t match, you’ll get a soft 404 anyway. Either redirect to a real replacement (same product, new SKU) or return 410.

500 errors are the silent killer. A flaky origin returning intermittent 500s teaches Google to slow the crawl. Fix the root cause; don’t paper over it with retries at the CDN.

Fix at the edge with Serpwise

Most teams discover an HTTP error pattern after damage is done — a deploy that 500’d for 90 minutes during a Googlebot surge, a CDN misconfig returning 404 for a section of the site, a CMS update that broke every URL containing a slash.

Serpwise can return the right status code for any URL pattern at the edge, immediately, without an origin deploy. Convert a cluster of 404s to 410. Issue a 503 with Retry-After during a maintenance window. Redirect a section of URLs based on a CSV import. The fix is live in seconds; the long-term cleanup lands in your stack on its own schedule.

See pricing or run a free AI visibility audit.