--- title: "Product data scraping" description: "Product data scraping for whole catalogs: titles, variants, specs, images, prices and sellers from Amazon, Walmart, eBay and Shopify stores in one feed." canonical: https://www.zenrows.com/use-cases/product-data-scraping updated: 2026-10-18 --- # Product data scraping Product data scraping is the automated extraction of complete product records, such as titles, variants, specs, images, prices and sellers, from marketplaces and retailer sites. Zenrows maps every product URL in a catalog with Crawl, pulls each page as structured JSON with Extract, and runs the full list through Batch, so you get one normalized catalog feed. ## Sample output: one catalog record ```json { "source": "marketplace", "url": "https://www.example-marketplace.com/p/trail-runner-gtx/48213", "product_id": "48213", "title": "Trail Runner GTX Men's Running Shoe", "brand": "Example Outdoor", "category_path": ["Sports", "Footwear", "Running", "Trail"], "gtin": "0194715623408", "price": 129.95, "currency": "USD", "variants": [ { "sku": "48213-BLK-10", "color": "Black", "size": "10", "available": true }, { "sku": "48213-BLK-11", "color": "Black", "size": "11", "available": false }, { "sku": "48213-OLV-10", "color": "Olive", "size": "10", "available": true } ], "specs": { "weight_g": 310, "drop_mm": 8, "upper": "Gore-Tex mesh" }, "images": 7, "seller": { "name": "Example Sports Store", "fulfilled_by_marketplace": true }, "rating": 4.4, "review_count": 1287, "scraped_at": "2026-10-17T06:12:44Z" } ``` An illustrative record from a marketplace product page, with variants kept as rows under one parent product instead of separate listings. ## What catalog teams collect with product data scraping Product data scraping usually starts as one job, ingesting a catalog, and grows into several once the feed exists. The page types stay the same: category listings, product detail pages and seller storefronts. | Job | Data you collect | Typical sources | |---|---|---| | Marketplace catalog ingestion | Title, brand, category path, specs, images, variants | Amazon, Walmart, eBay, Allegro, Bol.com, Mercado Libre | | Aggregator and comparison feeds | Product records with identifiers and current offer | Retailer sites, Shopify and WooCommerce stores | | Assortment analysis | Products per category, brand mix, price bands, new arrivals | Category pages at Target, Wayfair, Zara, IKEA | | Seller catalog mapping | Which sellers list which products, at what offer | Seller storefronts on Amazon, eBay, Etsy, Shopee | | Agentic commerce catalogs | Product data mapped to one schema for shopping agents | Brand sites such as Nordstrom, Sephora, Nike | | New product detection | Product URLs that were not in the previous crawl | Category and "new in" pages, refreshed weekly | ## Why catalog scraping breaks at full scale A scraper that handles one product page is a weekend job. A scraper that keeps 2 million records current across ten marketplaces runs into different problems, and most of them show up as missing rows rather than errors. | What breaks | What it looks like | How Zenrows handles it | |---|---|---| | Incomplete discovery | Category pagination stops at page 20, so half the catalog never gets a URL | Crawl follows links across the site and keeps only URLs that match your product pattern | | Variant explosion | Colour and size options load by script, so you capture one variant per product | Adaptive Stealth Mode escalates to browser rendering and returns the page after the variant selector has loaded | | Rate limits and blocks | A large retailer answers the first thousand requests, then serves 403s or CAPTCHA pages | Zenrows' stealth gets through the protections large retailers use, escalating only when a page needs it | | Schema drift across sites | Ten marketplaces, ten field layouts, ten parsers to maintain | Extract returns structured JSON without per-site selectors | | Run time | A full refresh takes longer than the window between refreshes | Batch submits the whole URL list as one job and reports when it finishes | ## How to scrape product data with Zenrows 1. **Map the catalog.** Start a crawl from a category or brand page with an include pattern for product URLs, such as `/p/` or `/dp/`. 2. **Test a product page.** Run Extract on one URL to see which fields come back, and decide your target schema. 3. **Submit the full URL list.** Send every product URL to Batch with Adaptive Stealth Mode and Extract. 4. **Normalize and load.** Map each source's fields to your schema, group variants under a parent ID, and load the feed into your catalog database. 5. **Re-crawl on a schedule.** Compare each crawl's URL list with the previous one to add new products and retire delisted ones. ```python Python # pip install requests import requests crawl = requests.post( "https://api.zenrows.com/v1/crawls", headers={"X-API-Key": "YOUR_ZENROWS_API_KEY"}, json={ "url": "CATEGORY_URL", # e.g. a category or brand page "max_items": 1000, "max_pages": 1000, "include_patterns": ["/p/"], "exclude_patterns": ["/reviews/", "/questions/"], }, ).json() # Later: read the product URLs the crawl found result = requests.get( f"https://api.zenrows.com/v1/crawls/{crawl['crawl_id']}", headers={"X-API-Key": "YOUR_ZENROWS_API_KEY"}, ).json() print(result) ``` ```bash cURL curl -X POST "https://async.api.zenrows.com/v1/jobs" \ -H "X-API-Key: YOUR_ZENROWS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "type": "regular", "status": "closed", "zenrows_params": { "mode": "auto", "extract": "auto" }, "tasks": [ { "url": "PRODUCT_URL_1", "external_id": "sku-48213" }, { "url": "PRODUCT_URL_2", "external_id": "sku-48214" } ] }' ``` ```bash CLI npx -y @zenrows/cli init zenrows crawl create CATEGORY_URL --include-patterns /p/ --max-pages 1000 --follow zenrows crawl results --out product-urls.jsonl # Check which fields one product page returns before you run the batch zenrows extract PRODUCT_URL --autoparse zenrows batch create product-urls.jsonl --wait ``` ```text Claude Code (MCP) # Connect once claude mcp add zenrows -e ZENROWS_API_KEY=YOUR_ZENROWS_API_KEY -- npx -y @zenrows/mcp # Then ask: Use Zenrows to crawl the cordless drills category on homedepot.com, then extract title, brand, model number, price, rating and every variant from each product page, and return one JSON record per product. ``` ## Which Zenrows product fits | If you need to | Use | Why | |---|---|---| | Find every product URL in a category or store | [Crawl](/products/crawl) | Follows links on one domain and filters by URL pattern | | Get product fields as JSON from many different sites | [Extract](/products/extract) | Structured JSON with no selectors to write per marketplace | | Pull one product page with your own parser | [Fetch](/products/fetch) | Returns rendered HTML from protected sites; name fields with CSS selectors to get JSON | | Refresh a catalog of millions of URLs | [Batch](/products/batch) | Thousands of URLs per job, results by webhook or download | | Try targets from a terminal | [CLI](/cli) | `extract` and `batch` commands with no code to write | ## What success looks like | Measure | What good looks like | How to check it | |---|---|---| | Catalog coverage | Your record count per category matches the count the site shows | Compare crawled product URLs with the category's own result total | | Variant completeness | Every colour and size option exists as a row, not only the default | Sample products and compare variant counts with the live page | | Required fields | Title, brand, price, image and identifier present on nearly every record | Report empty fields per source after each run | | Duplicate rate | One parent record per product, even when it appears in several categories | Count records sharing a product ID or GTIN | | Refresh window | A full catalog run finishes well before the next one starts | Track Batch job duration against your schedule | | Failed URLs per run | Few product URLs fail or come back without required fields | Count failed and incomplete records per source after each job | ## Product data scraping FAQ ### What is product data scraping? Product data scraping is collecting product information from websites automatically: names, descriptions, specifications, variants, images, prices, availability, ratings and seller details. It differs from price scraping in scope. Price scraping refreshes a few fields on known URLs, while product data scraping builds and maintains the full record for every product in a catalog. ### How is product data extraction different from a product data API? A product data API from a retailer only covers what that retailer chooses to expose, usually for its own sellers or partners. Product data extraction reads the public product page, so you get the same fields a shopper sees, from any site you need, in one format. A common approach is to use official feeds where they exist and scrape the rest. ### How do I get every product on a marketplace, not just the first pages? Category pagination often caps results, so a single listing page will not reach the whole catalog. Crawl from several entry points, such as brand pages, subcategories and sitemaps, with an include pattern for product URLs. Merge the URL lists and deduplicate by product ID before you fetch the pages. ### How do I handle product variants? Variants are usually loaded by JavaScript from a selector on the page. Fetch the page with Adaptive Stealth Mode, which escalates to browser rendering when variants need it, and either read the variant list from the Extract output or name the variant fields with CSS selectors. Store each size or colour as a row that points to one parent product so prices and stock stay per variant. ### Can I scrape product data from Amazon and Walmart? Yes. Both sites are protected and render key fields with JavaScript. Adaptive Stealth Mode starts with the lightest setup and escalates to browser rendering and Zenrows' stealth when a page needs it. Use geotargeting when you need a specific marketplace's catalog, such as a UK request for amazon.co.uk or a German one for amazon.de. ### How do I keep a product catalog up to date? Split the refresh in two. Re-crawl category and brand pages on a slower schedule, weekly for most stores, to find new product URLs and ones that disappeared. Re-fetch the known product URLs on a faster schedule for the fields that move, such as price and stock. Mark a product delisted only after it is missing from two crawls in a row, so a single failed page does not delete a live record. ### Is it legal to scrape product data? Catalog fields such as names, prices, identifiers and specifications are facts shown on public pages, and collecting them for analysis or comparison is common practice. The questions usually arise from what you do next: republishing a marketplace's photos or copy in your own store, or ignoring a site's terms of use. Decide the reuse first, then confirm it with your legal team. ## Related use cases - [Ecommerce web scraping](https://www.zenrows.com/use-cases/ecommerce-web-scraping.md): Product, price, stock, review and seller data from Amazon, Walmart, eBay and Shopify stores, as JSON. - [Product data enrichment](https://www.zenrows.com/use-cases/product-data-enrichment.md): Specs, images, descriptions and GTINs from manufacturer and retailer pages, written into the gaps in your PIM. - [Competitor price scraping](https://www.zenrows.com/use-cases/price-scraping.md): Competitor prices, promotions, shipping and stock per SKU from retailers and marketplaces, refreshed on your schedule. - [Digital shelf analytics](https://www.zenrows.com/use-cases/digital-shelf-analytics.md): Search rank, share of shelf, content compliance, ratings and availability for CPG brands, per retailer and region. - [Retail data scraping](https://www.zenrows.com/use-cases/retail-data.md): Store locations, store-level prices, promotions and click-and-collect stock from grocery and high-street chains. Start free: https://app.zenrows.com/register. Docs: https://docs.zenrows.com