---
title: "Market research web scraping"
description: "Market research web scraping for category sizing and consumer studies: products, prices and reviews from Amazon, Walmart, Reddit and niche retailers."
canonical: https://www.zenrows.com/use-cases/market-research-web-scraping
updated: 2026-10-18
---

# Market research web scraping

Market research web scraping collects public web data for market sizing, trend tracking and consumer research, covering product assortments, prices, reviews and forum discussion across many sites. Zenrows crawls category pages, returns products and reviews as structured JSON or Markdown, and reruns the same study as a recurring tracker, so each wave compares cleanly with the last.

## Sample output: one product in a category wave

```json
{
  "study": "home-espresso-machines-uk",
  "wave": "2026-10",
  "retailer": "example-retailer.co.uk",
  "category_path": ["Kitchen", "Coffee", "Espresso machines"],
  "product": "Example Brand Bean-to-Cup 300",
  "brand": "Example Brand",
  "model_number": "EB-BTC300",
  "price": 449.0,
  "list_price": 499.0,
  "currency": "GBP",
  "on_promotion": true,
  "rank_in_category": 7,
  "rating": 4.3,
  "review_count": 812,
  "attributes": { "type": "bean-to-cup", "milk_frother": true, "pump_pressure_bar": 15 },
  "review_themes": ["easy to clean", "noisy grinder"],
  "collected_at": "2026-10-16T08:12:00Z"
}
```

An illustrative record from one wave of a category study: the same schema for every retailer is what lets you add up the market and compare one wave with the next.

## What market research data the web holds

Web data gives researchers observed behaviour next to what panels and surveys report: what is on the shelf, at what price, and what buyers say once they own it. Most studies combine several of these jobs.

| Job | Data you collect | Typical sources |
|---|---|---|
| Market sizing | Product counts, price points and brands per category | Amazon, Walmart, Target, Argos, specialist retailers |
| Price and promotion tracking | Price, list price, promotion depth and frequency per wave | Retailer category and product pages |
| Consumer sentiment | Review text, ratings, recurring praise and complaints | Amazon reviews, Best Buy reviews, Trustpilot |
| Trend spotting | New entrants, rising products, discussion volume over time | Reddit, Product Hunt, Kickstarter, niche forums |
| Share of shelf | Share of listings and top-ranked positions by brand | Marketplace search results and category rankings |
| Spec benchmarking | Product attributes, specs, bundles and variants | GSMArena, Notebookcheck, manufacturer spec sheets |

## Why market research scraping is hard to repeat

A one-off study survives a few manual fixes. A tracker that reruns every month needs each wave to be complete and comparable, or the trend line shows your scraper's bad days instead of the market.

| What breaks | What it looks like | How Zenrows handles it |
|---|---|---|
| Incomplete waves | A retailer blocks halfway through the category, and the wave shows a false drop in assortment | Zenrows' stealth keeps requests going through retailer protections, and each wave reports failed URLs so you rerun only those |
| Category trees change | The retailer splits one category into two, and half the products vanish from your URL list | Crawl the category root each wave with include patterns, so new subcategory URLs join the study |
| A different schema per site | Twelve retailers label price, rating and specs twelve ways | Extract returns structured fields per page, and you map them to one schema once |
| Regional markets | Assortment, prices and language change by country | Geotargeting collects each market from inside its country |
| Text at volume | Thousands of reviews and forum threads to read and code | Markdown output returns clean text, ready for a coding frame or an LLM classifier |

## How to use web scraping for market research

1. **Scope the category and the sources.** Pick the retailers, marketplaces and forums that together cover the market you are sizing, and agree the schema before collecting.
2. **Crawl the category tree.** Run Crawl from each category root with an include pattern for product URLs.
3. **Collect products and reviews.** Fetch each product page with Extract for structured fields, and forum threads and review pages as Markdown.
4. **Normalise and deduplicate.** Map fields to your schema and match the same product across retailers by model number or GTIN.
5. **Rerun as a tracker.** Resubmit the URL list, plus anything new the crawl found, as a Batch job on the same day each month.

```bash cURL
# Collect product URLs from a category; read them from GET /v1/crawls/{crawl_id}
curl -X POST "https://api.zenrows.com/v1/crawls" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "CATEGORY_URL",
    "max_items": 5000,
    "max_pages": 5000,
    "include_patterns": ["/product/"],
    "exclude_patterns": ["/write-a-review"]
  }'
```

```python Python
# pip install requests
import json, requests

PRODUCT_URLS = ["PRODUCT_URL_1", "PRODUCT_URL_2"]  # from the crawl results

with open("wave-2026-10.jsonl", "w") as out:
    for url in PRODUCT_URLS:
        params = {
            "apikey": "YOUR_ZENROWS_API_KEY",
            "url": url,
            "mode": "auto",
            "proxy_country": "gb",
            "extract": "auto",
        }
        r = requests.get("https://api.zenrows.com/v1/", params=params)
        if r.ok:
            out.write(json.dumps({"url": url, "data": r.json()}) + "\n")
```

```bash CLI
npx -y @zenrows/cli init
zenrows crawl create CATEGORY_URL --include-patterns /product/ --max-pages 500 --follow
zenrows crawl results <crawl-id> --out product-urls.jsonl
# Forum threads and review pages for this wave, returned as Markdown for coding
zenrows batch create forum-threads.jsonl --output markdown --wait
```

```text Claude Code (MCP)
# Connect once
claude mcp add zenrows -e ZENROWS_API_KEY=YOUR_ZENROWS_API_KEY -- npx -y @zenrows/mcp

# Then ask:
Use Zenrows to crawl the robot vacuum category on argos.co.uk, then extract brand,
model, price, rating and review count from every product page, and give me the
results as a CSV with one row per product.
```

## Which Zenrows product fits

| If you need to | Use | Why |
|---|---|---|
| List every product in a category or on a retailer | [Crawl](/products/crawl) | Follows category pages and pagination, and keeps only URLs that match your patterns |
| Get product fields from many retailers in one schema | [Extract](/products/extract) | Returns structured JSON without writing selectors for each site |
| Pull a single page or a forum thread | [Fetch](/products/fetch) | Returns HTML or Markdown, including from protected retail sites |
| Rerun a 20,000-URL study every month | [Batch](/products/batch) | One job per wave, with results by webhook when it finishes |
| Run waves from a script or a scheduled task | [CLI](/cli) | One command submits the file and waits for the results |

## What success looks like

| Measure | What good looks like | How to check it |
|---|---|---|
| Category coverage | Your product count per retailer matches the count the retailer shows | Compare with the "results" figure on each category page |
| Wave completeness | Every URL in the study returns a record before the wave closes | Count failed URLs per wave and rerun them before analysis |
| Comparability | Schema and category mapping stay the same across waves | Validate each wave against the schema and diff the category map |
| Duplicate rate | Each product is counted once across retailers | Match on model number or GTIN and report the unmatched share |
| Delivery date | Each wave lands on the same day every month | Track the finish time of each Batch job against the plan |
| Rerun share | Only a small, stable share of URLs needs a second pass each wave | Count URLs rerun per wave and per retailer, and watch for a retailer whose share climbs |

## Market research web scraping FAQ

### How can I use web scraping for market research?

Use it wherever the answer is already published online. Count products and brands in a category to size it, track prices and promotions over time, read reviews to find unmet needs, and follow forums to see which products people discuss more each month. Each of these starts from a list of URLs, which you collect with Crawl and refresh on a schedule.

### What market research data can you collect from the web?

Product names, brands, prices, list prices, promotions, stock status, category position, specs, ratings, review text and review counts from retailers and marketplaces. From forums and communities you get threads, replies and dates. From company and news sites you get launches and announcements. Anything a visitor can read on a public page can become a field in your dataset.

### Can web data replace surveys and panels?

It answers a different question. Surveys tell you what people say they do and why; web data shows what is actually on sale, at what price, and what buyers wrote after they bought it. A common design uses web data to size a category and pick the questions, then field a smaller survey to explain what the data shows.

### How do I run a recurring market tracker?

Freeze the schema, the category map and the source list before the first wave. Each month, rerun Crawl on the category roots to catch new products, submit the full URL list to Batch, rerun any failed URLs, and only then close the wave. Store every wave with its date so you can rebuild any chart later.

### How do I analyse thousands of reviews and forum posts?

Fetch them with Markdown output to get clean text, then code them. A small team can hand-code a sample of a few hundred to build the codeframe, and an LLM can apply it to the rest. See sentiment analysis data for collecting review and social text at scale.

### Can I collect market data from other countries?

Yes. Use geotargeting to pick the country the request comes from, for example Germany for German retailers or Japan for Japanese marketplaces, so sites return local assortment, prices, currency and language. Run each market as its own wave so the data stays comparable within a country.

### What is category research, and how does web data support it?

Category research maps who sells what in a product category: brands, price tiers, pack sizes, features and how shelf space is shared. Web data gives you that map directly. Crawl each retailer's category pages, extract brand, price and attributes from every product, then count products and median price per brand and tier. Repeating the same crawl each quarter shows which brands are gaining range and where prices are moving.

## Related use cases

- [Competitive intelligence](https://www.zenrows.com/use-cases/competitive-intelligence.md): Pricing, hiring, reviews, launches and partners across your market, structured for battlecards and strategy.
- [Review scraping](https://www.zenrows.com/use-cases/review-scraping.md): Review text, ratings, dates and replies from Amazon, Google, Yelp, Trustpilot and G2, every page.
- [Sentiment analysis data](https://www.zenrows.com/use-cases/sentiment-analysis-data.md): Review text with ratings, forum threads and comments, cleaned and labeled for sentiment and NLP models.
- [Ecommerce web scraping](https://www.zenrows.com/use-cases/ecommerce-web-scraping.md): Product, price, stock, review and seller data from Amazon, Walmart, eBay and Shopify stores, as JSON.
- [Retail data scraping](https://www.zenrows.com/use-cases/retail-data.md): Store locations, store-level prices, promotions and click-and-collect stock from grocery and high-street chains.
- [Web scraping for research](https://www.zenrows.com/use-cases/web-scraping-for-research.md): Article metadata, forum posts, statistics and news text collected for studies, with a reproducible fetch log.

Start free: https://app.zenrows.com/register. Docs: https://docs.zenrows.com