What is crawl budget?

Crawl budget is the number of pages a crawler will fetch from one site over a period. The term comes from search engines, where Google describes it as the product of two things: the rate a site can serve without straining, and how much of that capacity Google thinks the site's content deserves.

The concept transfers to your own crawls, and it is more useful there than it first appears, because the constraint is the same even when the numbers are yours.

The two halves

What the site can bear. Google raises its rate while a site responds quickly and lowers it when responses slow or errors appear. Your crawler should behave the same way, and mostly does not: a fixed rate that ignores a rising 503 count keeps pushing a site that is already struggling, which ends in a block that was avoidable.

What is worth fetching. For a search engine this is a quality judgement. For you it is a coverage question. Budget spent on duplicates, filter permutations and pages you will discard is budget not spent on the pages you actually wanted.

Where a crawl budget goes to waste

Nearly always in the same places:

• Faceted navigation. Every combination of colour, size and price filter is a distinct URL serving substantially the same products. A catalogue of a thousand items can generate hundreds of thousands of URLs this way.
• Query parameters that change nothing. Tracking parameters, session IDs and sort orders produce different URLs for identical content. Normalising them is the fix.
• Pagination without limits. Deep pagination that continues past any useful content.
• Calendars. An infinite next-month link is the classic trap.
• Redirect chains. Each hop costs a request, and long chains multiply across a crawl.

Spending it deliberately

The practical moves, in rough order of effect: normalise URLs before enqueuing so duplicates collapse; exclude parameter patterns that do not change content; set a depth limit; prefer sitemap URLs over discovered ones where a sitemap exists; and adapt your rate to the site's response times rather than holding it fixed.

Crawling in priority order helps too. If the crawl is interrupted, which it will be, you would rather have fetched the pages that mattered than an arbitrary slice.

Where Zenrows fits

The rate side is partly infrastructure. Concurrency limits are per plan and are the number to size a worker pool against, rather than discovering your own ceiling by hitting it. Batch handles pacing and retries for a submitted URL list, and lets you check a job's cost before it runs, which turns budget from something you discover afterwards into something you decide in advance.