What is the best way to scrape a large website?

TL;DR

Separate discovery from fetching, run the fetching as a managed job rather than a loop on your own machine, and pick the fetch method per URL instead of globally. At small volumes none of this matters. Past roughly ten thousand pages, the things that break are cost, partial failure and state, in that order, and each has a different fix.

Split discovery from fetching

The first mistake is one loop that finds links and downloads pages at the same time. It works until it stops halfway, and then you cannot tell which URLs were visited, which are pending and which failed.

Two phases fix it. Discovery walks the site and produces a URL list, from a sitemap where one exists, from category pages where one does not. Fetching consumes that list. The list is a checkpoint: if the second phase dies at page 40,000, you restart from a known position instead of from the beginning.

Do not render every page

This is where large crawls become expensive. Running a headless browser on every URL is the single biggest cost multiplier, and most pages do not need it. Zenrows prices this explicitly: a standard request costs one credit, JavaScript rendering costs five, and a residential proxy costs ten, so rendering everything by reflex is a five times bill for no additional data.

Check what a plain request returns first. Many sites that look client-rendered still ship the content in the initial HTML or in a JSON payload you can request directly. Reserve the browser for the pages that genuinely need it.

Expect partial failure and design for it

At a hundred pages you notice each failure individually. At a million you are managing a failure rate. Some fraction of requests will time out, get blocked or return an empty page, and a run that stops on the first error will never finish.

What that means in practice: retry with exponential backoff and jitter rather than a fixed interval, treat 503s and 429s as backoff signals rather than failures, write results incrementally instead of holding everything in memory until the end, and keep a dead-letter list of URLs that failed repeatedly so a human can look at a sample rather than at all of them.

Where Zenrows fits

This is what Batch is for. You submit the URL list as a job and Zenrows runs it, so retries, concurrency and result collection are handled rather than being orchestration you write and maintain. Jobs report progress while running, results are collected when they finish, and you can check the cost of a job before it spends anything, which matters when the difference between rendering and not rendering is five times the bill.

For the discovery phase, the seed URL crawling guide covers building the URL list. If you would rather drive it yourself, concurrency limits are per plan and are the number to size your own worker pool against.

Key takeaways

Treat discovery and fetching as separate phases with the URL list as the checkpoint between them. Choose the fetch method per URL, because rendering everything is usually the reason a large crawl costs more than it should. Assume a failure rate rather than a failure event, and let a batch service own retries and orchestration unless running that yourself is the actual job.