---
title: "Web scraping for price comparison"
description: "Web scraping for price comparison sites: merchant prices, stock, shipping and GTINs from retailer catalogs, Google Shopping and marketplaces, kept fresh."
canonical: https://www.zenrows.com/use-cases/web-scraping-for-price-comparison
updated: 2026-10-26
---

# Web scraping for price comparison

Web scraping for price comparison builds the offer feed behind a comparison site or app: price, availability, shipping and product identifiers from every merchant you list. Zenrows crawls each merchant's catalog, returns product pages as structured JSON, and refreshes every offer with Batch, so the price you show matches what the merchant charges.

## Sample output: one product with merchant offers

```json
{
  "product_id": "cmp-000418",
  "title": "Daily Comfort Contact Lenses, 30 pack",
  "brand": "Example Optics",
  "gtin": "07612345000418",
  "offers": [
    {
      "merchant": "lens-shop.example",
      "url": "https://lens-shop.example/p/daily-comfort-30",
      "price": 24.9,
      "shipping": 0.0,
      "total": 24.9,
      "currency": "EUR",
      "in_stock": true,
      "captured_at": "2026-10-17T04:10:00Z"
    },
    {
      "merchant": "optic-direct.example",
      "url": "https://optic-direct.example/daily-comfort-30pk",
      "price": 22.5,
      "shipping": 3.95,
      "total": 26.45,
      "currency": "EUR",
      "in_stock": true,
      "captured_at": "2026-10-17T04:12:00Z"
    }
  ],
  "cheapest_total": "lens-shop.example"
}
```

Illustrative: offers from Extract, matched on GTIN, ranked by price with shipping.

## What price comparison sites collect

A product catalog with merchant offers under each product. Feeds cover some merchants; scraping covers the rest and checks the feeds.

| Job | Data you collect | Typical sources |
|---|---|---|
| Merchant catalog discovery | Every product URL in your categories | Merchant category pages and sitemaps |
| Offer refresh | Price, stock and currency per offer | Merchant product pages, Amazon, eBay, Allegro, bol.com |
| Product matching data | GTIN, EAN, MPN, brand, model, size, pack count | Product pages and their structured data |
| Total price | Shipping fee, free-shipping threshold, delivery time | Product and shipping info pages |
| Feed verification | Page price against affiliate feed price | Merchant product pages |
| Market benchmark | Offers on other comparison services | Google Shopping, idealo, Ceneo, PriceRunner |

## Why price comparison data goes stale

Shoppers expect the listed price at checkout; if the merchant shows more, they leave and your click earns nothing.

| What breaks | What it looks like | How Zenrows handles it |
|---|---|---|
| Merchants without a feed | No affiliate feed, or a weekly one | Crawl collects product URLs; Extract turns each into an offer |
| Same product, many names | Six titles for one lens pack, impossible to group | Extract returns GTIN and MPN where shown, for matching |
| Price shown without shipping | The cheapest listing is the dearest delivered | Collect shipping with the price; rank by total |
| Bot protection on large merchants | Crawler challenges on marketplaces and big retailers | Zenrows' stealth gets through Cloudflare, Akamai, AWS WAF, PerimeterX (HUMAN) |
| Country storefronts | Foreign visitors redirected or shown another currency | Geotargeting requests each storefront from its market |
| Refresh at catalog scale | A million offers from hundreds of merchants outgrow a homegrown scheduler | One Batch job, results per URL |

## Web scraping for price comparison with Zenrows

1. **Discover each merchant's catalog:** Crawl category pages with a product URL include pattern.
2. **Turn product pages into offers:** automatic extraction returns price, stock, shipping and identifiers as JSON.
3. **Match offers to your products:** GTIN or MPN first, then brand, model and pack size; review uncertain matches by hand.
4. **Refresh every offer** in Batch at least daily, more for top categories; hide offers whose last good check is too old.

```bash cURL
curl -X POST "https://api.zenrows.com/v1/crawls" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "MERCHANT_CATEGORY_URL",
    "max_items": 2000,
    "include_patterns": ["/p/"],
    "exclude_patterns": ["/reviews", "/compare"]
  }'
# No output_format, so the crawl returns URLs only.
# Read them with GET https://api.zenrows.com/v1/crawls/CRAWL_ID
```

```python Python
# pip install requests
import requests

offer_urls = load_offer_urls()  # merchant product URLs from your database
job = {
    "type": "regular",
    "status": "closed",
    "zenrows_params": {"mode": "auto", "extract": "auto"},
    "tasks": [{"url": o["url"], "external_id": o["offer_id"]} for o in offer_urls],
}
response = requests.post(
    "https://async.api.zenrows.com/v1/jobs",
    headers={"X-API-Key": "YOUR_ZENROWS_API_KEY"},
    json=job,
)
print(response.json()["job_id"])  # read results at /v1/jobs/JOB_ID/results
```

```bash CLI
npx -y @zenrows/cli init
zenrows crawl create MERCHANT_CATEGORY_URL --include-patterns /p/ --max-pages 2000 --follow
zenrows extract MERCHANT_PRODUCT_URL --autoparse
# offers.jsonl: one {"url": "..."} per offer URL
zenrows batch create offers.jsonl --wait
```

```text Claude Code (MCP)
# Connect once
claude mcp add zenrows -e ZENROWS_API_KEY=YOUR_ZENROWS_API_KEY -- npx -y @zenrows/mcp

# Then ask:
Use Zenrows to read these three product pages for the Sony WH-1000XM5 headphones on bestbuy.com, walmart.com and bhphotovideo.com, and return a JSON table with merchant, price, currency, in-stock status, shipping cost and any GTIN or UPC shown on the page.
```

## Which Zenrows product fits

| If you need to | Use | Why |
|---|---|---|
| Find every product a merchant sells | [Crawl](/products/crawl) | Follows categories and pagination, keeps product URLs |
| Read offers from hundreds of merchant layouts | [Extract](/products/extract) | Price, stock and identifiers without per-merchant selectors |
| Refresh every offer daily or hourly | [Batch](/products/batch) | One job; your IDs map results to offers |
| Recheck an offer a shopper reports | [Fetch](/products/fetch) | The merchant page as it is now |

## What success looks like

| Measure | What good looks like | How to check it |
|---|---|---|
| Click-through price match | Your price equals the merchant's on arrival | Daily refetch of a sample against listed prices |
| Offer freshness | No live offer older than your limit | Age of each live offer's latest `captured_at` |
| Match rate | Most offers tied to a canonical product | Offers matched on GTIN, MPN or brand plus model |
| Merchant coverage | Each merchant's full category catalog | Offers per merchant against its category counts |
| Dead offers | Delisted and out-of-stock items gone within one refresh | Live offers whose URL 404s or shows out of stock |
| Refresh failure rate | Few failures per run, retried before the next | Failed tasks per Batch run over offers submitted |

## Web scraping for price comparison FAQ

### How do price comparison sites get their data?

Merchant feeds, affiliate network feeds and web scraping. Feeds often cover part of a catalog and lag the live site; scraping fills the gaps and checks feed prices against the pages.

### How do I match the same product across merchants?

GTIN, EAN, UPC or part number first, from specs or structured data; then brand, model, size and pack count, with hand review for best sellers. See product matching .

### How often should a price comparison site refresh prices?

At least daily; several times a day for top products in electronics, travel and fast-moving consumer goods. Hide or flag offers past your check limit so shoppers never see a stale price.

### Can I compare prices across countries?

Yes. Geotarget each request to its market for local price, currency and stock, and store the country with every offer: many merchants run separate storefronts per country.

### Should I show the price with or without shipping?

Show both and rank by delivered total where you can: low price plus high shipping can cost more. Collect shipping fees and free-shipping thresholds with each offer.

### What is a price comparison scraper?

It reads merchant product pages on a schedule and returns each offer as a record: title, price, currency, stock, shipping, identifiers, URL. With Zenrows, no per-merchant parser: Crawl finds URLs, Extract returns offers as JSON, Batch refreshes them.

### Is it legal to scrape prices for a comparison site?

An established business on public prices, and many merchants want the traffic. Review each merchant's terms, link to its product page, follow affiliate rules, and do not copy images or long descriptions you cannot reuse. Ask your legal team.

## Related use cases

- [Product matching](https://www.zenrows.com/use-cases/product-matching.md): GTINs, model numbers, pack sizes and specs from every retailer, so one product resolves to one record.
- [Competitor price scraping](https://www.zenrows.com/use-cases/price-scraping.md): Competitor prices, promotions, shipping and stock per SKU from retailers and marketplaces, refreshed on your schedule.
- [Ecommerce web scraping](https://www.zenrows.com/use-cases/ecommerce-web-scraping.md): Product, price, stock, review and seller data from Amazon, Walmart, eBay and Shopify stores, as JSON.
- [Product data scraping](https://www.zenrows.com/use-cases/product-data-scraping.md): Whole product catalogs, variants, specs, images and sellers from marketplaces, normalized into one feed.
- [Travel data scraping](https://www.zenrows.com/use-cases/travel-data-scraping.md): Fares, room rates, car hire quotes, tour prices and reviews from airlines, OTAs and travel marketplaces.

Start free: https://app.zenrows.com/register. Docs: https://docs.zenrows.com