What is a web crawler?
A web crawler is a program that finds web pages on its own. It fetches a starting page, collects the links on it, visits those links, and repeats, building up a list of URLs (and often their content) that nobody had to write down in advance. Web crawling is the name for that process, and you will also see crawlers called spiders, spider bots or bots.
Search engines are the best known example: Googlebot and Bingbot crawl the public web so there is something to index. Most crawlers are much smaller and narrower. A price monitoring team crawls one retailer's category pages to find every product URL. A research group crawls a news site's archive. An AI team crawls documentation to feed a retrieval pipeline.
How a web crawler works
Every crawler is built around the same loop, which the page on how a web crawler works walks through step by step:
- Start from seeds. One or more seed URLs go into a queue, often called the frontier.
- Fetch the next URL from the queue, rendering JavaScript when the site needs it.
- Extract links from the page and resolve them into absolute URLs.
- Filter and clean them. Drop links outside the scope, apply URL normalization, and skip anything already seen.
- Enqueue what is new, record the current URL as done, and go back to step 2.
The crawl ends when the queue is empty or a limit is reached: a maximum crawl depth, a page cap, or a time budget. The order in which the queue is drained decides the crawl's shape, which is the subject of breadth-first vs depth-first crawling.
Types of web crawlers
| Type | What it crawls | Typical goal |
|---|---|---|
| Search engine crawler | The open web, continuously | Build and refresh a search index |
| Focused crawler | One topic or one set of sites | Collect only pages relevant to a subject |
| Site crawler (SEO audit) | A single domain you own | Find broken links, redirects and indexing issues |
| Listing or URL crawler | A category, search or listing page | Collect every item URL behind it, such as all products in a category |
| Incremental crawler | A site already crawled before | Fetch only what is new or changed since the last run |
Web crawler vs web scraper
A crawler discovers URLs. A scraper extracts data from them. Most real projects need both, and the crawler's output, a clean URL list, becomes the scraper's input. The blog covers web crawling vs web scraping in depth.
What makes a crawler hard to get right
- Duplicates. The same page appears under many URLs because of URL parameters, trailing slashes and tracking tags. Without deduplication, the crawler fetches the same content again and again.
- Traps. Calendars, faceted navigation and session IDs can generate endless URLs. A spider trap is the general name.
- Pagination. Listings spread items across many pages, and each site paginates differently. See pagination.
- Rules and politeness. robots.txt states what a site allows, and a crawl delay keeps the load reasonable.
- Blocking. Sites protected by anti-bot systems treat a fast, regular crawler as a bot, because it is one.
How Zenrows Crawl handles this
Zenrows Crawl is a web crawler API. Give it any public URL: a homepage, a category page or a search page. It opens the page, follows the links on it as deep as you choose, keeps only the URLs you want, drops duplicates, and fetches every page past anti-bot protection. You get back every URL it found, and each page's HTML if you want it. You don't write, host or maintain a crawler.
Results can be read while the crawl runs or downloaded as one NDJSON file, and each crawl reports pages fetched, pages failed, URLs found and duplicates removed. For structured data, send the URLs to Extract. The Crawl docs cover the full flow.
Frequently asked questions
What is the difference between web crawling and a web crawler?
Web crawling is the activity: systematically discovering and visiting pages by following links. A web crawler is the program that does it. People also say spider, spider bot or bot for the same thing.
Is web crawling legal?
Crawling publicly available pages is common practice, but the answer depends on the site's terms, the data involved and the jurisdiction. Respect robots.txt, avoid personal data you have no basis to collect, and keep your request rate reasonable. For anything sensitive, ask a lawyer.
How do websites detect web crawlers?
Through the user agent, request rate, IP reputation, missing browser signals and fingerprinting. The page on how websites detect scrapers covers the signals in detail.
Do I need a crawler if I already have the URLs?
No. If you already hold the URL list, the job is fetching, and a batch scraping tool is simpler than a crawler. Crawling is the discovery step for when you do not know the URLs yet.
Go deeper on the blog
In the docs
Last updated: Oct 9, 2026