What is concurrency in web scraping?

Concurrency is the number of requests in flight simultaneously. Scraping is dominated by waiting: most of a request's duration is network latency, with the client idle. Sending one request at a time means being idle almost the whole run.

Raising concurrency from one to twenty on a workload of ten thousand pages changes a run from hours to minutes, which is why it is the first lever people reach for. It is also the one that gets people blocked.

Two limits, and they are different

Yours. Memory, file descriptors, and whatever your provider allows. Exceeding it produces timeouts, connection errors and 502s that look like the target's fault and are not.

Theirs. What one site tolerates from one client. Exceed it and you get 429s, 503s, or a block that outlasts the run.

The important distinction is that the second limit is per host, not global. Two hundred concurrent requests spread across fifty domains is four per site, which is unremarkable. The same two hundred against one domain will end the run early. Concurrency should always be capped per host, not just in total.

Signs you have gone too far

Watch the ratio rather than the raw rate. Error rate rising with concurrency, response times climbing as you add workers, 429s appearing where there were none, or 502s and timeouts clustering at peak parallelism all mean you have passed the useful point.

Past that point, throughput actually falls: more requests fail, each failure is retried, and retries add load that causes more failures. The fastest setting is not the highest one.

Finding the right number

Start low, around five per host. Raise it while error rates stay flat and response times stay stable. Stop at the first sign of either moving, and back off from there.

Better still, adapt rather than fix. Reduce concurrency when errors rise and increase it when they clear, which handles a site that is fast at 3am and struggling at noon. A fixed setting is always wrong at one of those times.

Two other details worth getting right: add jitter so workers do not synchronise into bursts, and remember that a browser-rendered request is far heavier than a plain one, so the safe concurrency for rendering is much lower than for fetching.

Where Zenrows fits

Concurrency limits are set per plan and documented, so you can size a worker pool against a known number rather than discovering the ceiling by hitting it. Batch removes the question for list-shaped work: pacing and parallelism are managed inside the job, so there is no pool to tune.