What is Crawl Error Rate?

Why It Matters

Crawl error rate is one of the earliest warning signs that something technical has gone wrong on a site, often before rankings or traffic visibly drop. Every crawl spent hitting a broken page is a crawl not spent on a valid one, so a rising error rate compounds the crawl budget problem on top of the immediate issue of broken links. It's especially important to watch right after a site migration, redesign, or URL structure change, since those are the events most likely to introduce a batch of broken links or misconfigured redirects at once. Left unaddressed, a high and sustained crawl error rate can cause search engines to crawl the rest of the site less thoroughly, slowing down indexing for entirely unrelated, healthy pages.

Example

A search engine crawler attempts to access 10,000 pages on a site over a month and encounters errors, such as 404s or server timeouts, on 350 of them. Crawl error rate is 350 divided by 10,000, times 100, which equals 3.5%. If a site migration introduces a batch of broken internal links and crawl error rate spikes to 12% the following month, that spike typically causes search engines to crawl the site's legitimate pages less thoroughly during that period, which is why crawl error rate is monitored closely immediately after any major site change.

Frequently Asked Questions

• What counts as a crawl error?

  Common examples include 404 not found errors, 5xx server errors, DNS resolution failures, and excessively long redirect chains. Some tools also flag pages blocked by robots.txt or returning a soft 404 as a separate category worth reviewing.

• How is crawl error rate different from crawl budget?

  Crawl budget is how much crawling activity a search engine allocates to a site overall, while crawl error rate measures what portion of that activity is being spent hitting errors instead of valid pages. A high error rate effectively shrinks the usable portion of an otherwise healthy crawl budget.

• What's a healthy crawl error rate?

  There's no fixed universal benchmark, but a rate under 1 to 2% is generally considered healthy for most established sites, while anything climbing into the double digits usually warrants immediate investigation.

• How often should crawl error rate be checked?

  Most sites review it weekly through tools like Google Search Console, with more frequent checks immediately after a migration, redesign, or any change to URL structure or navigation.

• What typically causes a spike in crawl error rate?

  Site migrations, URL structure changes, expired content that wasn't redirected, server outages, and broken internal links introduced during a redesign are the most common triggers.