Web scraping for price comparison
Web scraping for price comparison builds the offer feed behind a comparison site or app: price, availability, shipping and product identifiers from every merchant you list. Zenrows crawls each merchant's catalog, returns product pages as structured JSON, and refreshes every offer with Batch, so the price you show matches what the merchant charges.
Updated
{
"product_id": "cmp-000418",
"title": "Daily Comfort Contact Lenses, 30 pack",
"brand": "Example Optics",
"gtin": "07612345000418",
"offers": [
{
"merchant": "lens-shop.example",
"url": "https://lens-shop.example/p/daily-comfort-30",
"price": 24.9,
"shipping": 0.0,
"total": 24.9,
"currency": "EUR",
"in_stock": true,
"captured_at": "2026-10-17T04:10:00Z"
},
{
"merchant": "optic-direct.example",
"url": "https://optic-direct.example/daily-comfort-30pk",
"price": 22.5,
"shipping": 3.95,
"total": 26.45,
"currency": "EUR",
"in_stock": true,
"captured_at": "2026-10-17T04:12:00Z"
}
],
"cheapest_total": "lens-shop.example"
}
What price comparison sites collect
A product catalog with merchant offers under each product.
Feeds cover some merchants; scraping covers the rest and checks the feeds.
| Job | Data you collect | Typical sources |
|---|---|---|
| Merchant catalog discovery | Every product URL in your categories | Merchant category pages and sitemaps |
| Offer refresh | Price, stock and currency per offer | Merchant product pages, |
| Product matching data | GTIN, EAN, MPN, brand, model, size, pack count | Product pages and their structured data |
| Total price | Shipping fee, free-shipping threshold, delivery time | Product and shipping info pages |
| Feed verification | Page price against affiliate feed price | Merchant product pages |
| Market benchmark | Offers on other comparison services |
Why price comparison data goes stale
Shoppers expect the listed price at checkout; if the merchant shows more, they leave and your click earns nothing.
No affiliate feed, or a weekly one
Crawl collects product URLs; Extract turns each into an offer
Six titles for one lens pack, impossible to group
Extract returns GTIN and MPN where shown, for matching
The cheapest listing is the dearest delivered
Collect shipping with the price; rank by total
Crawler challenges on marketplaces and big retailers
Zenrows' stealth gets through Cloudflare, Akamai, AWS WAF, PerimeterX (HUMAN)
Foreign visitors redirected or shown another currency
Geotargeting requests each storefront from its market
A million offers from hundreds of merchants outgrow a homegrown scheduler
One Batch job, results per URL
Web scraping for price comparison with Zenrows
- Discover each merchant's catalog: Crawl category pages with a product URL include pattern.
- Turn product pages into offers: automatic extraction returns price, stock, shipping and identifiers as JSON.
- Match offers to your products: GTIN or MPN first, then brand, model and pack size; review uncertain matches by hand.
- Refresh every offer in Batch at least daily, more for top categories; hide offers whose last good check is too old.
curl -X POST "https://api.zenrows.com/v1/crawls" \
-H "X-API-Key: YOUR_ZENROWS_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "MERCHANT_CATEGORY_URL",
"max_items": 2000,
"include_patterns": ["/p/"],
"exclude_patterns": ["/reviews", "/compare"]
}'
# No output_format, so the crawl returns URLs only.
# Read them with GET https://api.zenrows.com/v1/crawls/CRAWL_ID# pip install requests
import requests
offer_urls = load_offer_urls() # merchant product URLs from your database
job = {
"type": "regular",
"status": "closed",
"zenrows_params": {"mode": "auto", "extract": "auto"},
"tasks": [{"url": o["url"], "external_id": o["offer_id"]} for o in offer_urls],
}
response = requests.post(
"https://async.api.zenrows.com/v1/jobs",
headers={"X-API-Key": "YOUR_ZENROWS_API_KEY"},
json=job,
)
print(response.json()["job_id"]) # read results at /v1/jobs/JOB_ID/resultsnpx -y @zenrows/cli init
zenrows crawl create MERCHANT_CATEGORY_URL --include-patterns /p/ --max-pages 2000 --follow
zenrows extract MERCHANT_PRODUCT_URL --autoparse
# offers.jsonl: one {"url": "..."} per offer URL
zenrows batch create offers.jsonl --wait# Connect once
claude mcp add zenrows -e ZENROWS_API_KEY=YOUR_ZENROWS_API_KEY -- npx -y @zenrows/mcp
# Then ask:
Use Zenrows to read these three product pages for the Sony WH-1000XM5 headphones on bestbuy.com, walmart.com and bhphotovideo.com, and return a JSON table with merchant, price, currency, in-stock status, shipping cost and any GTIN or UPC shown on the page.Which Zenrows product fits
- Find every product a merchant sells CrawlFollows categories and pagination, keeps product URLs
- Read offers from hundreds of merchant layouts ExtractPrice, stock and identifiers without per-merchant selectors
- Refresh every offer daily or hourly BatchOne job; your IDs map results to offers
- Recheck an offer a shopper reports FetchThe merchant page as it is now
What success looks like
| Measure | What good looks like | How to check it |
|---|---|---|
| Click-through price match | Your price equals the merchant's on arrival | Daily refetch of a sample against listed prices |
| Offer freshness | No live offer older than your limit | Age of each live offer's latest captured_at |
| Match rate | Most offers tied to a canonical product | Offers matched on GTIN, MPN or brand plus model |
| Merchant coverage | Each merchant's full category catalog | Offers per merchant against its category counts |
| Dead offers | Delisted and out-of-stock items gone within one refresh | Live offers whose URL 404s or shows out of stock |
| Refresh failure rate | Few failures per run, retried before the next | Failed tasks per Batch run over offers submitted |
Web scraping for price comparison FAQ
-
Merchant feeds, affiliate network feeds and web scraping. Feeds often cover part of a catalog and lag the live site; scraping fills the gaps and checks feed prices against the pages.
-
GTIN, EAN, UPC or part number first, from specs or structured data; then brand, model, size and pack count, with hand review for best sellers. See product matching.
-
At least daily; several times a day for top products in electronics, travel and fast-moving consumer goods. Hide or flag offers past your check limit so shoppers never see a stale price.
-
Yes. Geotarget each request to its market for local price, currency and stock, and store the country with every offer: many merchants run separate storefronts per country.
-
Show both and rank by delivered total where you can: low price plus high shipping can cost more. Collect shipping fees and free-shipping thresholds with each offer.
-
It reads merchant product pages on a schedule and returns each offer as a record: title, price, currency, stock, shipping, identifiers, URL. With Zenrows, no per-merchant parser: Crawl finds URLs, Extract returns offers as JSON, Batch refreshes them.
-
An established business on public prices, and many merchants want the traffic. Review each merchant's terms, link to its product page, follow affiliate rules, and do not copy images or long descriptions you cannot reuse. Ask your legal team.