SEO & GEO
What is Crawl Budget?
Definition
The number of pages search engine crawlers will visit on your site within a given period — a resource that must be managed carefully on large or programmatic sites.
In more detail
Googlebot has finite capacity to crawl the web. For large sites — e-commerce stores, news publishers, programmatic SEO sites with tens of thousands of pages — if that capacity is wasted on low-value or duplicate pages, important content may not get crawled or indexed promptly.
Common crawl budget wasters: faceted navigation creating millions of URL combinations, paginated parameter URLs (`?page=2`, `?sort=price`), soft 404 pages that return 200 status codes, redirect chains that waste crawl hops, and orphaned pages with no internal links pointing to them.
For sites under ~10,000 pages, crawl budget is rarely a limiting factor. It becomes critical for large e-commerce sites with filters and sorting, programmatic SEO implementations, and any site experiencing indexing delays where new pages take weeks to appear in search results.
Why it matters
If important pages aren't being crawled, they won't be indexed — and they won't rank. Crawl budget optimisation is a core technical SEO discipline for sites operating at scale, and one of the first things to audit when indexing is slow.
Related service
Working with Crawl?
I offer SEO Services for businesses ready to move from understanding to implementation.
Learn about SEO Services →