Crawl: the most reliable web crawler APISee what’s new
Zenrows
Talk to sales Start free

Web scraping for price comparison

Web scraping for price comparison builds the offer feed behind a comparison site or app: price, availability, shipping and product identifiers from every merchant you list. Zenrows crawls each merchant's catalog, returns product pages as structured JSON, and refreshes every offer with Batch, so the price you show matches what the merchant charges.

Updated

Sample output: one product with merchant offers
{
  "product_id": "cmp-000418",
  "title": "Daily Comfort Contact Lenses, 30 pack",
  "brand": "Example Optics",
  "gtin": "07612345000418",
  "offers": [
    {
      "merchant": "lens-shop.example",
      "url": "https://lens-shop.example/p/daily-comfort-30",
      "price": 24.9,
      "shipping": 0.0,
      "total": 24.9,
      "currency": "EUR",
      "in_stock": true,
      "captured_at": "2026-10-17T04:10:00Z"
    },
    {
      "merchant": "optic-direct.example",
      "url": "https://optic-direct.example/daily-comfort-30pk",
      "price": 22.5,
      "shipping": 3.95,
      "total": 26.45,
      "currency": "EUR",
      "in_stock": true,
      "captured_at": "2026-10-17T04:12:00Z"
    }
  ],
  "cheapest_total": "lens-shop.example"
}
Illustrative: offers from Extract, matched on GTIN, ranked by price with shipping.

What price comparison sites collect

A product catalog with merchant offers under each product.

Feeds cover some merchants; scraping covers the rest and checks the feeds.

JobData you collectTypical sources
Merchant catalog discovery Every product URL in your categories Merchant category pages and sitemaps
Offer refresh Price, stock and currency per offer Merchant product pages, Amazon, eBay, Allegro, bol.com
Product matching data GTIN, EAN, MPN, brand, model, size, pack count Product pages and their structured data
Total price Shipping fee, free-shipping threshold, delivery time Product and shipping info pages
Feed verification Page price against affiliate feed price Merchant product pages
Market benchmark Offers on other comparison services Google Shopping, idealo, Ceneo, PriceRunner

Why price comparison data goes stale

Shoppers expect the listed price at checkout; if the merchant shows more, they leave and your click earns nothing.

What it looks like

No affiliate feed, or a weekly one

How Zenrows handles it

Crawl collects product URLs; Extract turns each into an offer

Web scraping for price comparison with Zenrows

  1. Discover each merchant's catalog: Crawl category pages with a product URL include pattern.
  2. Turn product pages into offers: automatic extraction returns price, stock, shipping and identifiers as JSON.
  3. Match offers to your products: GTIN or MPN first, then brand, model and pack size; review uncertain matches by hand.
  4. Refresh every offer in Batch at least daily, more for top categories; hide offers whose last good check is too old.
curl -X POST "https://api.zenrows.com/v1/crawls" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "MERCHANT_CATEGORY_URL",
    "max_items": 2000,
    "include_patterns": ["/p/"],
    "exclude_patterns": ["/reviews", "/compare"]
  }'
# No output_format, so the crawl returns URLs only.
# Read them with GET https://api.zenrows.com/v1/crawls/CRAWL_ID

What success looks like

MeasureWhat good looks likeHow to check it
Click-through price match Your price equals the merchant's on arrival Daily refetch of a sample against listed prices
Offer freshness No live offer older than your limit Age of each live offer's latest captured_at
Match rate Most offers tied to a canonical product Offers matched on GTIN, MPN or brand plus model
Merchant coverage Each merchant's full category catalog Offers per merchant against its category counts
Dead offers Delisted and out-of-stock items gone within one refresh Live offers whose URL 404s or shows out of stock
Refresh failure rate Few failures per run, retried before the next Failed tasks per Batch run over offers submitted

Web scraping for price comparison FAQ

  • Merchant feeds, affiliate network feeds and web scraping. Feeds often cover part of a catalog and lag the live site; scraping fills the gaps and checks feed prices against the pages.

  • GTIN, EAN, UPC or part number first, from specs or structured data; then brand, model, size and pack count, with hand review for best sellers. See product matching.

  • At least daily; several times a day for top products in electronics, travel and fast-moving consumer goods. Hide or flag offers past your check limit so shoppers never see a stale price.

  • Yes. Geotarget each request to its market for local price, currency and stock, and store the country with every offer: many merchants run separate storefronts per country.

  • Show both and rank by delivered total where you can: low price plus high shipping can cost more. Collect shipping fees and free-shipping thresholds with each offer.

  • It reads merchant product pages on a schedule and returns each offer as a record: title, price, currency, stock, shipping, identifiers, URL. With Zenrows, no per-merchant parser: Crawl finds URLs, Extract returns offers as JSON, Batch refreshes them.

  • An established business on public prices, and many merchants want the traffic. Review each merchant's terms, link to its product page, follow affiliate rules, and do not copy images or long descriptions you cannot reuse. Ask your legal team.

Turn the live web into usable data.