What is a URL crawler?
A URL crawler is a tool that collects URLs. It starts from a page, follows the links and pagination it finds, and returns a list of the URLs behind it, such as every product in a category, every article in an archive or every job in a search. It is also called a link crawler, link extractor or URL extractor, depending on whether it follows links across many pages or only lists the links on one.
A URL crawler is the discovery half of a data pipeline. Its output, a deduplicated URL list, is what a scraper or a batch job then fetches.
URL crawler vs link extractor vs web crawler
| Tool | Scope | Output |
|---|---|---|
| Link extractor / URL extractor | One page | Every link on that page |
| URL crawler / link crawler | A listing, a section or a site, across many pages | A filtered, deduplicated URL list |
| Web crawler | Any scope, up to the whole web | URLs, and usually page content too |
| Web scraper | Known URLs | Structured data from each page |
A link extractor is a single step. A URL crawler repeats that step across pages, which is where the hard parts begin: pagination, duplicates, filtering and blocking.
How a URL crawler works
- Open the start page, typically a category, search results or index page.
- Extract every link and resolve relative URLs.
- Follow the pagination so pages 2, 3 and onward are walked too.
- Filter by pattern. Keep only the URL shapes you want, such as
/product/or/jobs/, and drop navigation, footer and filter links. - Normalize and deduplicate, so the same item reached twice, or carrying a tracking parameter, is listed once.
- Optionally go one level deeper, opening each item to collect the links it holds.
- Return the list, ideally with a count you can check against what the site says it holds.
What makes a good URL crawler
- Pagination handled for you, across page numbers, offsets and next links, without counting as depth.
- Include and exclude patterns that keep the output to the URLs you actually need.
- Normalization that strips
utm_*,gclidand other tracking parameters. - Access to protected sites. Listings on large retailers, travel sites and property portals are often behind anti-bot systems, and a crawler that gets blocked on page 3 returns a partial list without saying so.
- A completeness signal, so you know whether the list is whole.
- Output you can stream, such as NDJSON, for lists in the tens of thousands.
Common uses
- Collecting every product URL in a category for price and assortment monitoring.
- Building a list of all listings in an area on a real estate portal.
- Gathering every job posting for a search on a job board.
- Seeding a batch scraping job or a RAG ingestion pipeline.
- Auditing your own site's internal links.
How Zenrows Crawl works as a URL crawler
Zenrows Crawl is a URL crawler as an API. Give it any public URL: a homepage, a category page or a search page. It opens the page, follows the links on it as deep as you choose, keeps only the URLs you want, drops duplicates, and fetches every page past anti-bot protection. You get back every URL it found, and each page's HTML if you want it. You do not write a link loop, a filter or a deduplication step, and you don't host or maintain a crawler.
- Any starting point. A homepage, a category page or a search page. Crawl stays on the start URL's domain, subdomains included.
- Depth you choose. At depth 1 you get the links on the start page; each extra level opens the kept links and collects theirs.
- Patterns keep it focused. Include patterns such as
/product/keep only the URLs you want; exclude patterns drop the rest. - No duplicates. Each URL appears in the results once, and the crawl reports how many duplicates it dropped.
- Anti-bot handled. Every page is fetched by Fetch, which picks the settings page by page to get past anti-bot protection.
- URLs, with HTML if you want it. Read results while the crawl runs or download them as one NDJSON file, then send the URLs to Extract when you need structured JSON.
The Crawl guide walks through a full crawl.
Frequently asked questions
What is a URL extractor?
A tool that lists the links on a page. Run it across many pages, with pagination and deduplication, and it becomes a URL crawler.
How do I get all URLs from a website?
Check the sitemap and robots.txt first, then crawl the site's category and listing pages. The blog's guide to finding all URLs on a domain compares the methods.
What is the difference between a URL crawler and a web scraper?
A URL crawler finds the pages; a scraper extracts data from them. A URL crawler's output is usually a scraper's input.
Can a URL crawler handle JavaScript sites?
Only if it renders pages or reads the underlying data. Links built by JavaScript are invisible to a crawler that reads raw HTML. See JavaScript crawling.
Go deeper on the blog
In the docs
Last updated: Oct 9, 2026