--- title: "Market research web scraping" description: "Market research web scraping for category sizing and consumer studies: products, prices and reviews from Amazon, Walmart, Reddit and niche retailers." canonical: https://www.zenrows.com/use-cases/market-research-web-scraping updated: 2026-10-18 --- # Market research web scraping Market research web scraping collects public web data for market sizing, trend tracking and consumer research: product assortments, prices, reviews and forum discussion across many sites. Zenrows crawls category pages, returns products and reviews as structured JSON or Markdown, and reruns the study as a recurring tracker, so each wave compares cleanly with the last. ## Sample output: one product in a category wave ```json { "study": "home-espresso-machines-uk", "wave": "2026-10", "retailer": "example-retailer.co.uk", "category_path": ["Kitchen", "Coffee", "Espresso machines"], "product": "Example Brand Bean-to-Cup 300", "brand": "Example Brand", "model_number": "EB-BTC300", "price": 449.0, "list_price": 499.0, "currency": "GBP", "on_promotion": true, "rank_in_category": 7, "rating": 4.3, "review_count": 812, "attributes": { "type": "bean-to-cup", "milk_frother": true, "pump_pressure_bar": 15 }, "review_themes": ["easy to clean", "noisy grinder"], "collected_at": "2026-10-16T08:12:00Z" } ``` Illustrative. One schema across retailers lets you add up the market and compare waves. ## What market research data the web holds Web data adds observed behaviour to panels and surveys: what is on the shelf, at what price, and what owners say. Most studies combine several jobs. | Job | Data you collect | Typical sources | |---|---|---| | Market sizing | Product counts, price points and brands per category | Amazon, Walmart, Target, Argos, specialist retailers | | Price and promotion tracking | Price, list price, promotion depth and frequency | Retailer category and product pages | | Consumer sentiment | Reviews, ratings, recurring praise and complaints | Amazon reviews, Best Buy reviews, Trustpilot | | Trend spotting | New entrants, rising products, discussion volume | Reddit, Product Hunt, Kickstarter, niche forums | | Share of shelf | Listings and top positions by brand | Marketplace search and category rankings | | Spec benchmarking | Attributes, specs, bundles, variants | GSMArena, Notebookcheck, manufacturer spec sheets | ## Why market research scraping is hard to repeat A one-off study survives manual fixes. A monthly tracker needs complete, comparable waves, or the trend line shows your scraper's bad days instead of the market. | What breaks | What it looks like | How Zenrows handles it | |---|---|---| | Incomplete waves | A block mid-category fakes an assortment drop | Zenrows' stealth gets through retailer protections; each wave reports failed URLs to rerun | | Category trees change | A split category drops half the products from your list | Crawl the root each wave with include patterns to pick up new subcategories | | A different schema per site | Twelve retailers, twelve labels for price, rating and specs | Extract returns structured fields; map them to one schema once | | Regional markets | Assortment, prices and language vary by country | Geotargeting collects each market from inside it | | Text at volume | Thousands of reviews and threads to code | Markdown gives clean text for a coding frame or LLM classifier | ## How to use web scraping for market research 1. **Scope the category and the sources**: retailers, marketplaces and forums covering the market, and the schema, agreed first. 2. **Crawl the category tree** from each root, with an include pattern for product URLs. 3. **Collect products and reviews**: products with Extract, threads and reviews as Markdown. 4. **Normalise and deduplicate** into your schema, matching products by model number or GTIN. 5. **Rerun as a tracker**: the URL list plus new crawl finds, as a Batch job on the same day monthly. ```bash cURL # Collect product URLs from a category; read them from GET /v1/crawls/{crawl_id} curl -X POST "https://api.zenrows.com/v1/crawls" \ -H "X-API-Key: YOUR_ZENROWS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "CATEGORY_URL", "max_items": 5000, "max_pages": 5000, "include_patterns": ["/product/"], "exclude_patterns": ["/write-a-review"] }' ``` ```python Python # pip install requests import json, requests PRODUCT_URLS = ["PRODUCT_URL_1", "PRODUCT_URL_2"] # from the crawl results with open("wave-2026-10.jsonl", "w") as out: for url in PRODUCT_URLS: params = { "apikey": "YOUR_ZENROWS_API_KEY", "url": url, "mode": "auto", "proxy_country": "gb", "extract": "auto", } r = requests.get("https://api.zenrows.com/v1/", params=params) if r.ok: out.write(json.dumps({"url": url, "data": r.json()}) + "\n") ``` ```bash CLI npx -y @zenrows/cli init zenrows crawl create CATEGORY_URL --include-patterns /product/ --max-pages 500 --follow zenrows crawl results --out product-urls.jsonl # Forum threads and review pages for this wave, returned as Markdown for coding zenrows batch create forum-threads.jsonl --output markdown --wait ``` ```text Claude Code (MCP) # Connect once claude mcp add zenrows -e ZENROWS_API_KEY=YOUR_ZENROWS_API_KEY -- npx -y @zenrows/mcp # Then ask: Use Zenrows to crawl the robot vacuum category on argos.co.uk, then extract brand, model, price, rating and review count from every product page, and give me the results as a CSV with one row per product. ``` ## Which Zenrows product fits | If you need to | Use | Why | |---|---|---| | List every product in a category or retailer | [Crawl](/products/crawl) | Follows categories and pagination, keeps URLs matching your patterns | | Get one product schema across many retailers | [Extract](/products/extract) | Structured JSON without per-site selectors | | Pull one page or forum thread | [Fetch](/products/fetch) | HTML or Markdown, protected retail sites included | | Rerun a 20,000-URL study every month | [Batch](/products/batch) | One job per wave, results by webhook | | Run waves from a script or scheduler | [CLI](/cli) | One command submits the file and waits | ## What success looks like | Measure | What good looks like | How to check it | |---|---|---| | Category coverage | Product counts match each retailer's | Against the "results" figure per category page | | Wave completeness | A record per URL before the wave closes | Failed URLs per wave, rerun before analysis | | Comparability | Same schema and category map every wave | Validate against the schema; diff the category map | | Duplicate rate | Each product counted once | Match on model number or GTIN; report the unmatched share | | Delivery date | Each wave on the same day monthly | Batch job finish time against plan | | Rerun share | A small, stable share needs a second pass | Reruns per wave and retailer; watch for one that climbs | ## Market research web scraping FAQ ### How can I use web scraping for market research? Wherever the answer is already online: count products and brands to size a category, track prices and promotions, read reviews for unmet needs, follow forums for rising products. Each starts from a URL list collected with Crawl and refreshed on a schedule. ### What market research data can you collect from the web? From retailers and marketplaces: names, brands, prices, list prices, promotions, stock, category position, specs, ratings, review text and counts. From forums: threads, replies and dates. From company and news sites: launches and announcements. Anything on a public page can become a field. ### Can web data replace surveys and panels? It answers a different question. Surveys capture what people say they do and why; web data shows what is on sale, at what price, and what buyers wrote. A common design: web data sizes the category and picks the questions, a smaller survey explains the data. ### How do I run a recurring market tracker? Freeze schema, category map and sources before wave one. Monthly, recrawl the roots for new products, run the URL list in Batch, rerun failures, then close the wave. Store each wave dated, so any chart can be rebuilt. ### How do I analyse thousands of reviews and forum posts? Fetch them as Markdown for clean text, hand-code a few hundred to build the codeframe, and let an LLM apply it to the rest. See sentiment analysis data for review and social text at scale. ### Can I collect market data from other countries? Yes. Geotargeting sends the request from the country, Germany for German retailers, Japan for Japanese marketplaces, so sites return local assortment, prices, currency and language. Run each market as its own wave. ### What is category research, and how does web data support it? Mapping who sells what in a category: brands, price tiers, pack sizes, features and shelf share. Web data gives that map directly: crawl each retailer's category, extract brand, price and attributes, then count products and median price per brand and tier. A quarterly rerun shows which brands gain range and where prices move. ## Related use cases - [Competitive intelligence](https://www.zenrows.com/use-cases/competitive-intelligence.md): Pricing, hiring, reviews, launches and partners across your market, structured for battlecards and strategy. - [Review scraping](https://www.zenrows.com/use-cases/review-scraping.md): Review text, ratings, dates and replies from Amazon, Google, Yelp, Trustpilot and G2, every page. - [Sentiment analysis data](https://www.zenrows.com/use-cases/sentiment-analysis-data.md): Review text with ratings, forum threads and comments, cleaned and labeled for sentiment and NLP models. - [Ecommerce web scraping](https://www.zenrows.com/use-cases/ecommerce-web-scraping.md): Product, price, stock, review and seller data from Amazon, Walmart, eBay and Shopify stores, as JSON. - [Retail data scraping](https://www.zenrows.com/use-cases/retail-data.md): Store locations, store-level prices, promotions and click-and-collect stock from grocery and high-street chains. - [Web scraping for research](https://www.zenrows.com/use-cases/web-scraping-for-research.md): Article metadata, forum posts, statistics and news text collected for studies, with a reproducible fetch log. Start free: https://app.zenrows.com/register. Docs: https://docs.zenrows.com ## Structured data