What is web crawling?
Web crawling is the automated process of discovering web pages by following links. A program visits a page, collects the links on it, decides which ones to follow, and repeats, gradually mapping which URLs exist on a site or across the web. The program that does this is a web crawler; web crawling is the activity itself, and the word is used for everything from Google indexing the web to a script collecting every product URL in one category.
The output of a crawl is a set of URLs, often with the content of each page. That set is what search indexes, price monitors, research datasets and AI retrieval systems are built from.
What web crawling is used for
| Use | What gets crawled | What the crawl produces |
|---|---|---|
| Search engines | The public web, continuously | An index of pages to rank |
| Ecommerce and price monitoring | Category and search pages of retailers | Every product URL, then product data |
| Real estate and jobs data | Listing pages on portals and boards | Every property or job posting |
| SEO auditing | Your own site | Broken links, redirects, orphan pages |
| Research and archiving | News sites, forums, public records | A dataset or a snapshot |
| AI and RAG pipelines | Documentation, knowledge bases | Pages to chunk and embed |
The web crawling process
- Seed. Choose where to start: a homepage, a category, a search results page or a sitemap.
- Fetch. Request the page, rendering JavaScript when the site needs it.
- Parse links. Pull out every link and resolve it to an absolute URL.
- Filter. Keep links inside the scope, drop tracking parameters, and deduplicate.
- Schedule. Add new URLs to the queue in an order, usually breadth-first.
- Repeat until the queue is empty or a limit on depth, pages or time is reached.
Each step has its own failure modes. The fetch step meets anti-bot systems, the parse step meets JavaScript-rendered links, the filter step meets duplicates and spider traps, and the schedule step has to respect a site's robots.txt and a sensible crawl delay.
Broad crawling vs focused crawling
Broad crawling tries to cover as much of the web as possible, which is what search engines do. It needs enormous infrastructure and careful prioritisation.
Focused crawling stays within a defined scope: one site, one section, one topic or one listing. Almost every business crawl is focused, because the goal is a specific dataset. Starting from the right page, such as a product listing page, and following only its pagination and item links is the most efficient form of focused crawling.
Web crawling vs web scraping
Crawling finds pages; scraping extracts data from them. A crawler answers "which URLs exist?" and a scraper answers "what is on this page?". Most data projects run both in sequence: crawl a listing to get item URLs, then scrape each item. The blog compares web crawling and web scraping in depth.
Best practices
- Define the scope before you start: domains, URL patterns and a depth limit.
- Normalize and deduplicate every URL before it enters the queue.
- Be polite: follow robots.txt, limit requests per host and back off on errors.
- Persist state so an interrupted crawl resumes instead of restarting.
- Measure completeness against something the site states, such as a result count.
How Zenrows Crawl handles this
Zenrows Crawl turns web crawling into an API call. Give it any public URL: a homepage, a category page or a search page. It opens the page, follows the links on it as deep as you choose, keeps only the URLs you want, drops duplicates, and fetches every page past anti-bot protection. You get back every URL it found, and each page's HTML if you want it. The proxies, browsers, retries and deduplication logic that usually make up a crawler are handled for you, and each crawl reports pages fetched, URLs found and duplicates removed. Send the URLs to Extract when you need structured JSON. The Crawl docs show the full flow.
Frequently asked questions
What is web crawling in simple terms?
A program visiting web pages and following their links to find more pages, the way you might click through a site, only automatically and at scale.
What is the difference between web crawling and a web crawler?
Web crawling is the process. A web crawler, also called a spider or bot, is the program that carries it out.
Is web crawling the same as indexing?
No. Crawling discovers and fetches pages. Indexing, which search engines do afterwards, analyses and stores them so they can be retrieved for queries.
Is web crawling legal?
Crawling public pages is widespread, but legality depends on the site's terms, the data collected and the jurisdiction. Respect robots.txt, avoid personal data without a lawful basis, and get legal advice for sensitive projects.
Go deeper on the blog
In the docs
Last updated: Oct 9, 2026