Crawl: the most reliable web crawler APISee what’s new
Zenrows
Talk to sales Start free

How do I scrape every product in a category?

To scrape every product in a category, start from the category's product listing page, follow its pagination to the last page, keep only the links that point to product pages, deduplicate them, and check the total against the count the page shows. Then fetch each product page for its data. That two-phase pattern, discover the URLs, then extract from them, is the core of most ecommerce web scraping.

The steps are simple to describe. The difficulty is doing them reliably on large retail sites, where pagination varies, sponsored items repeat, filters multiply URLs and anti-bot systems watch for exactly this kind of traffic.

Step 1: pick the right start page

Use the broadest listing that matches your scope: the unfiltered category rather than a filtered view, and the category rather than the homepage. Note the result count the page states, such as "1,284 products". That number is your target.

Step 2: walk every page of the listing

Find how the listing paginates: a page number (?page=2), an offset (?start=48), a "next" link, or a "load more" button backed by an API. Follow it until a page returns no new items or there is no next link, rather than assuming a page count.

A listing page links to navigation, filters, related categories and footer pages as well as products. Filter by the product URL pattern, such as /product/, /p/ or /dp/, and drop faceted navigation links so the crawl does not explode into filter combinations.

Step 4: normalize and deduplicate

Strip tracking parameters, collapse different spellings of the same URL, and keep each product once. Sponsored and featured products often appear on every page; counting them repeatedly inflates the total.

Step 5: check completeness

Compare the number of unique product URLs with the count the listing stated. A large gap usually means a pagination step failed, a page was blocked, or the site caps how many pages a listing shows, in which case split the category with one filter, such as brand, and crawl each part.

Step 6: extract the product data

Fetch each product page and extract the fields you need: name, price, currency, availability, rating, images, variants. Selectors written per site break when the markup changes; schema-based extraction returns the same fields across sites.

Common mistakes

Mistake What happens Fix
Scraping only page one A fraction of the catalogue, with no error Follow pagination to the end
Following every link Thousands of filter and navigation URLs Filter by product URL pattern
No deduplication Totals higher than the site's count Deduplicate on a normalized URL
Fixed page count Misses new pages, or loops on the last one Stop on content, not on a number
Ignoring blocks Silent gaps where pages returned a CAPTCHA Check status and content of every page
Treating pagination as depth A depth limit cuts the listing short Count only item links as depth

Doing it with Zenrows Crawl

Zenrows Crawl handles discovery, and Extract handles step 6. Send the category URL as the starting page, a depth, and an include pattern for product URLs. To walk more pages of the category, raise depth and use include patterns to keep only the URLs you want. Add output_format: "html" if you also want each page's HTML:

curl -X POST "https://api.zenrows.com/v1/crawls" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.scrapingcourse.com/ecommerce/product-category/clothing/men/tops/hoodies-sweatshirts/",
    "depth": 1,
    "max_items": 100,
    "include_patterns": ["/product/"]
  }'

The crawl starts at once and runs in the background, returning a crawl_id. Crawl opens the category page, reads its links, keeps the ones that match /product/ and drops duplicates, fetching every page it opens past anti-bot protection. When you want the whole result, download it as a single NDJSON file, even while the crawl is still running:

curl "https://api.zenrows.com/v1/crawls/YOUR_CRAWL_ID/download" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
  -o crawl.jsonl

Each line is one product URL:

{"url":"https://www.scrapingcourse.com/ecommerce/product/abominable-hoodie/"}
{"url":"https://www.scrapingcourse.com/ecommerce/product/ajax-full-zip-sweatshirt/"}

For the product data itself, send each URL to Extract with extract=auto, which returns the page as structured JSON without selectors:

import json, os, requests

with open("crawl.jsonl") as f:
    urls = [json.loads(line)["url"] for line in f if line.strip()]

for url in urls:
    response = requests.get("https://api.zenrows.com/v1/", params={
        "url": url,
        "apikey": os.environ["ZENROWS_API_KEY"],
        "mode": "auto",
        "extract": "auto",
    })
    product = response.json().get("parsed", {})
    print(url, product)

For a long list, Batch runs the same requests as one managed job. The crawl reports pages fetched, pages failed, URLs found and duplicates removed, and comparing the final list with the count the category shows (step 5) tells you when the list is whole. The Crawl guide covers reading results while the crawl runs, stopping a crawl and reading each page's HTML.

Frequently asked questions

How do I scrape an ecommerce website?

Crawl its category pages and search results to collect product URLs, then extract data from each product page. Handle pagination, deduplication and blocking explicitly, and check your totals against what the site shows.

Collecting publicly displayed product information such as prices is common practice, but terms of service and local law vary. Avoid personal data, respect robots.txt, keep request rates reasonable, and get legal advice for commercial projects.

How do I scrape a category with more products than the site shows?

Some sites cap a listing at a fixed number of pages. Split the category with a filter, such as brand or price range, crawl each slice, and deduplicate the combined results.

Should I scrape the listing page or each product page?

The listing gives you every product URL and summary fields such as name and price quickly. Product pages give the full data: specifications, variants, stock and reviews. Most projects use both.

Go deeper on the blog

In the docs

Last updated: Oct 9, 2026

Get reliable web data in minutes.

Free plan, 5,000 credits every month, no credit card required.