Web crawling
Before you can extract anything you need the URLs, and on most sites nobody hands you the list. A crawler finds it: it starts from a page, follows pagination and links, decides which URLs are worth keeping, and stops before it wanders into a trap. The terms below cover how crawlers work and walk a site, the listings and navigation patterns they meet, the rules and signals sites publish for them, how URLs get cleaned and deduplicated, and how a long crawl delivers its results safely.
Crawlers
Breadth-first or depth-first crawling? Breadth-first crawling visits every page at one link distance from the start before going one level deeper, using a first-in-first-out queue, while depth-first crawling follows one chain of links as far as it goes before backtracking, using a stack. What is a URL crawler? A URL crawler, also called a link crawler or URL extractor, is a tool that starts from a web page, follows its links and pagination, and returns a clean list of the URLs it finds, such as every product or article URL behind a category page. What is a web crawler? A web crawler is a program that discovers web pages automatically by fetching a page, collecting the links on it, and visiting those links in turn, so it can build a list of URLs, or a copy of their content, without anyone supplying the list. What is a web spider? A web spider, or spider bot, is another name for a web crawler, a program that moves through the web by following links from page to page to discover and download pages, named for the way it travels the web's network of links. What is an AI web crawler? An AI web crawler is either a crawler run by an AI company to collect web pages for training models or answering questions, such as GPTBot or ClaudeBot, or a crawler that uses AI models to decide what to follow and to turn pages into structured data or text for AI applications. What is crawl depth? Crawl depth is how many link hops a page sits from the crawl's starting point, and the depth limit is the maximum number of hops a crawler will follow before it stops going further. What is JavaScript crawling? JavaScript crawling is crawling a website with a crawler that executes the page's JavaScript before reading it, so links and content that only appear after the page renders in a browser are discovered too. What is web crawling? Web crawling is the automated process of discovering web pages by visiting a page, collecting its links and following them in turn, so that a program can map which URLs exist on a site or across the web without being given the list. Which web crawler should I use? The right web crawler depends on the job. Open-source frameworks such as Scrapy and Crawlee suit teams that want full control and will run their own infrastructure, hosted crawler APIs suit teams that want URLs or data back without operating crawlers, and SEO site auditors such as Screaming Frog suit checking your own website.Listings and discovery
How do I scrape every product in a category? Start from the category's product listing page, follow its pagination to the end, keep only links that match the product URL pattern, deduplicate them, check the total against the count the page shows, and then fetch each product page for its data. What is a product listing page? A product listing page (PLP) is a page that shows a list of products, such as a category, search results or brand page, with each item linking to its own product detail page, and it is the usual starting point for crawling a catalogue. What is a spider trap? A spider trap, or crawler trap, is a set of web pages that generates an endless or enormous number of URLs, such as an infinite calendar or unlimited filter combinations, causing a crawler to keep fetching pages without ever finishing. What is faceted navigation? Faceted navigation is the set of filters on a listing page, such as brand, size, colour and price, that let visitors narrow results by several attributes at once, and each combination usually produces its own URL. What is pagination in web scraping? Pagination is splitting a long list of results across several pages, reached by page numbers, offsets, a "next" link or a cursor, and in web scraping it means following those pages so you collect every item rather than only the first page.Rules sites set
What is a sitemap? A sitemap is a file, usually XML at /sitemap.xml, that lists the URLs a website wants crawlers to find, often with the date each page last changed, so search engines and other crawlers can discover pages without following every link. What is crawl delay? Crawl delay is the time a crawler waits between requests to the same website, often requested by the site through a Crawl-delay line in robots.txt, to keep the crawl rate low enough that the server is not overloaded. What is robots.txt? robots.txt is a plain text file at the root of a website, such as example.com/robots.txt, that tells crawlers which paths they may and may not request, using the Robots Exclusion Protocol.Clean URLs
What are URL parameters? URL parameters, also called query parameters or query strings, are key and value pairs added to a URL after a question mark, such as ?page=2&sort=price, that pass extra information to the page, like a filter, a page number or a tracking tag. What is a canonical URL? A canonical URL is the one address a website declares as the main version of a page that is reachable at several URLs, usually with a link rel="canonical" tag, so search engines and crawlers treat the duplicates as one page. What is deduplication in web crawling? Deduplication is detecting and removing repeated records, and in web crawling it means making sure each page is fetched and stored once, even when it appears under several URLs or is linked from many places.Running a crawl
What is an idempotency key? An idempotency key is a unique value a client sends with a request, usually in an Idempotency-Key header, so that if the request is retried the server recognises it and returns the original result instead of performing the action a second time. What is NDJSON? NDJSON (newline-delimited JSON), also called JSON Lines or JSONL, is a text format where every line is one complete JSON object, so a file can be written, streamed and read one record at a time.Get reliable web data in minutes.
Free plan, 5,000 credits every month, no credit card required.