How does a web crawler work?

A crawler is a loop around a queue. It starts from one or more seed URLs, fetches a page, pulls out the links, adds the ones it has not seen to the queue, and goes again. Everything else is refinement.

Crawling discovers URLs. Scraping extracts data from them. The two usually run together, and the blog covers the distinction properly.

The loop, and what each step actually involves

Take a URL from the frontier. The frontier is the queue of URLs waiting to be visited. Which one comes next decides the shape of the crawl: first-in-first-out spreads broad across a site, last-in-first-out dives deep down one branch.

Check whether you should fetch it. Have you seen it already? Is it on a host you are allowed to crawl? Does robots.txt permit it? Is it too deep?

Fetch it, with the same considerations as any request: rendering when needed, proxies when needed, retries on transient failure.

Extract links. Resolve relative URLs against the current page, and normalise them so /page, /page/ and /page?utm_source=x do not become three separate entries.

Enqueue what is new, and record the URL as seen.

The three problems every crawler hits

Duplicates. Without normalisation and a seen-set, a crawler revisits the same content endlessly through different URLs. Query parameters, trailing slashes, session IDs and fragments all produce the same page at different addresses.

Traps. Calendars with an infinite next-month link, faceted search generating combinations without end, deliberately hidden links planted to catch crawlers. A depth limit and a page cap are what stop an infinite crawl, and they should be there from the first version.

Politeness. A crawler that requests as fast as it can will degrade a small site and get itself blocked. A delay between requests to the same host, a concurrency limit per host, and honouring robots.txt and Retry-After are the minimum.

State is what makes it resumable

A crawl of any size will be interrupted. The frontier and the seen-set are what let it resume, so holding them only in memory means an interruption costs the entire run.

Persisting both, and recording each URL's status as it is processed, turns a restart into a continuation. At small scale a file is enough; past that a database or a queue service, especially once several workers share one frontier.

Where Zenrows fits

Discovery and fetching are separable, and the seed URL crawling guide walks through building a crawler that starts from a seed, finds internal links and scrapes at scale.

Once you have the URL list, Batch runs it as a managed job, which handles the concurrency, retries and result collection that would otherwise be the bulk of your crawler's code.