What is crawl delay?
Crawl delay is the time a crawler waits between requests to the same website. Sites can ask for a specific delay with a Crawl-delay line in robots.txt, and responsible crawlers apply a delay even when none is requested. It is the simplest way to control crawl rate, the number of requests per second a crawler sends to one host.
Crawling with a sensible delay, a per-host concurrency limit and respect for robots.txt is often called polite crawling.
The Crawl-delay directive
User-agent: *
Crawl-delay: 10
This asks every crawler to wait ten seconds between requests. Support varies:
| Crawler | Honours Crawl-delay? |
|---|---|
| Googlebot | No. Google adjusts its rate automatically from server response times and errors |
| Bingbot | Yes |
| Yandex | No longer; it uses settings in its webmaster tools |
| Open-source crawling frameworks | Varies; many do not apply it automatically |
The directive is not part of the RFC 9309 robots standard, so treat it as a request from the site owner rather than a guaranteed behaviour.
Crawl delay vs crawl rate vs concurrency
| Term | Meaning | Unit |
|---|---|---|
| Crawl delay | Pause between consecutive requests to one host | Seconds |
| Crawl rate | How many requests per unit of time reach one host | Requests per second |
| Concurrency | How many requests are in flight at the same moment | Parallel requests |
They interact. Ten parallel workers each waiting one second still send ten requests per second to the host. The rate the site experiences is what counts, so delays need to apply per host across all workers, not per worker.
How to choose a crawl delay
- Follow
Crawl-delaywhen the site sets one. - Adapt to the server. If response times rise or 503 and 429 errors appear, back off. If responses stay fast, the site has headroom.
- Honour
Retry-Afterheaders on 429 and 503 responses. - Add jitter. A random variation around the delay looks less mechanical and spreads load more evenly than a fixed interval.
- Crawl off-peak for smaller sites, when their real visitors are asleep.
- Spread hosts. A crawler visiting many sites can stay fast overall while being slow on each one.
Why it matters
Too fast, and a crawler can degrade a small site for its real users, trigger IP rate limiting, and get blocked, which ends the crawl anyway. Too slow, and a large crawl takes weeks. The right delay is the fastest rate the site handles comfortably, discovered by watching its responses. The blog covers rate limits in web scraping in more depth.
Where Zenrows fits
With Zenrows, the request rate you can send is set by your plan's concurrency, and that is the number to size a worker pool against. A Zenrows Crawl job runs in the background on Zenrows' infrastructure: you set its depth and limits, and every page is fetched past anti-bot protection while you read results as they arrive.
Frequently asked questions
Does Google respect crawl delay?
No. Googlebot ignores the Crawl-delay directive and sets its rate from how quickly and reliably a server responds. Site owners who need to slow it down can return 503 or 429 responses temporarily.
What is a good crawl delay for web scraping?
There is no universal number. Start with the site's Crawl-delay if it has one, otherwise a conservative rate per host, and adjust based on response times and errors.
What is polite crawling?
Crawling that respects the site: following robots.txt, limiting requests per host, honouring Retry-After, identifying the crawler where appropriate, and avoiding peak hours on small sites.
What is crawl rate?
The number of requests a crawler sends to a site per unit of time. Crawl delay is one way to control it; concurrency limits are another.
Go deeper on the blog
In the docs
Last updated: Oct 9, 2026