Crawl: the most reliable web crawler APISee what’s new
Zenrows
Talk to sales Start free

What is crawl delay?

Crawl delay is the time a crawler waits between requests to the same website. Sites can ask for a specific delay with a Crawl-delay line in robots.txt, and responsible crawlers apply a delay even when none is requested. It is the simplest way to control crawl rate, the number of requests per second a crawler sends to one host.

Crawling with a sensible delay, a per-host concurrency limit and respect for robots.txt is often called polite crawling.

The Crawl-delay directive

User-agent: *
Crawl-delay: 10

This asks every crawler to wait ten seconds between requests. Support varies:

Crawler Honours Crawl-delay?
Googlebot No. Google adjusts its rate automatically from server response times and errors
Bingbot Yes
Yandex No longer; it uses settings in its webmaster tools
Open-source crawling frameworks Varies; many do not apply it automatically

The directive is not part of the RFC 9309 robots standard, so treat it as a request from the site owner rather than a guaranteed behaviour.

Crawl delay vs crawl rate vs concurrency

Term Meaning Unit
Crawl delay Pause between consecutive requests to one host Seconds
Crawl rate How many requests per unit of time reach one host Requests per second
Concurrency How many requests are in flight at the same moment Parallel requests

They interact. Ten parallel workers each waiting one second still send ten requests per second to the host. The rate the site experiences is what counts, so delays need to apply per host across all workers, not per worker.

How to choose a crawl delay

  • Follow Crawl-delay when the site sets one.
  • Adapt to the server. If response times rise or 503 and 429 errors appear, back off. If responses stay fast, the site has headroom.
  • Honour Retry-After headers on 429 and 503 responses.
  • Add jitter. A random variation around the delay looks less mechanical and spreads load more evenly than a fixed interval.
  • Crawl off-peak for smaller sites, when their real visitors are asleep.
  • Spread hosts. A crawler visiting many sites can stay fast overall while being slow on each one.

Why it matters

Too fast, and a crawler can degrade a small site for its real users, trigger IP rate limiting, and get blocked, which ends the crawl anyway. Too slow, and a large crawl takes weeks. The right delay is the fastest rate the site handles comfortably, discovered by watching its responses. The blog covers rate limits in web scraping in more depth.

Where Zenrows fits

With Zenrows, the request rate you can send is set by your plan's concurrency, and that is the number to size a worker pool against. A Zenrows Crawl job runs in the background on Zenrows' infrastructure: you set its depth and limits, and every page is fetched past anti-bot protection while you read results as they arrive.

Frequently asked questions

Does Google respect crawl delay?

No. Googlebot ignores the Crawl-delay directive and sets its rate from how quickly and reliably a server responds. Site owners who need to slow it down can return 503 or 429 responses temporarily.

What is a good crawl delay for web scraping?

There is no universal number. Start with the site's Crawl-delay if it has one, otherwise a conservative rate per host, and adjust based on response times and errors.

What is polite crawling?

Crawling that respects the site: following robots.txt, limiting requests per host, honouring Retry-After, identifying the crawler where appropriate, and avoiding peak hours on small sites.

What is crawl rate?

The number of requests a crawler sends to a site per unit of time. Crawl delay is one way to control it; concurrency limits are another.

Go deeper on the blog

In the docs

Last updated: Oct 9, 2026

Get reliable web data in minutes.

Free plan, 5,000 credits every month, no credit card required.