---
title: "Web scraping for research"
description: "Web scraping for research: collect study data from journals, Google Scholar, Wikipedia, forums and government statistics, with a reproducible fetch log."
canonical: https://www.zenrows.com/use-cases/web-scraping-for-research
updated: 2026-10-18
---

# Web scraping for research

Web scraping for research collects the data for a study from public websites, such as article metadata, forum posts, news coverage, government statistics or Wikipedia revisions. Zenrows fetches those pages, including JavaScript-heavy and protected ones, and returns HTML, Markdown or JSON, so you can record exactly what was collected, from where and when, for your methods section.

## Sample output: one record in a study corpus

```json
{
  "study_id": "housing-discourse-2026",
  "record_id": "r-000184",
  "source_type": "public forum",
  "url": "https://forum.example.org/t/rent-increase-notice/48213",
  "collected_at": "2026-10-17T09:41:27Z",
  "request": { "mode": "auto", "response_type": "markdown" },
  "http_status": 200,
  "content_sha256": "3b1f0c9a...",
  "title": "Rent increase notice after lease renewal",
  "published": "2026-09-30",
  "author_id": "u_7f3a (pseudonymized)",
  "word_count": 412,
  "replies": 23,
  "language": "en",
  "inclusion": { "matched_terms": ["rent increase", "lease"], "excluded": false }
}
```

An illustrative record with its provenance attached: the URL, fetch time, request parameters and content hash a reviewer needs to check how it entered the dataset.

## What researchers collect from the web

Research projects scrape for the same reason they run surveys: the data a question needs does not exist yet as a clean dataset. The sources below are the ones that come up most across disciplines.

| Job | Data you collect | Typical sources |
|---|---|---|
| Literature and citation mapping | Title, authors, venue, year, DOI, abstract, citation counts | Google Scholar, arXiv, PubMed, Crossref, publisher pages such as Springer, Wiley and MDPI |
| Social science corpora | Posts, comments, timestamps, thread structure, votes | Reddit, Stack Exchange, public forums, YouTube comments |
| Media framing studies | Headlines, article text, section, publication date, byline | News sites and their archive pages |
| Policy and legislative analysis | Bills, consultation responses, committee reports, agency statistics | parliament.uk, europa.eu, state legislature sites, national statistics offices |
| Economics and price research | Prices, product attributes, availability, sampled on a fixed schedule | Retailer and marketplace pages |
| Knowledge production studies | Article revisions, edit histories, talk pages, contributor counts | Wikipedia, Fandom and other public wikis |

## Why research data collection goes wrong

A scraper that works for a weekend prototype rarely survives peer review. The failures that hurt studies most are the ones that leave silent gaps or cannot be explained afterwards.

| What breaks | What it looks like | How Zenrows handles it |
|---|---|---|
| Collection you cannot reproduce | A reviewer asks how the corpus was built and the only answer is a script that no longer runs | Every request is a URL plus explicit parameters, so the collection log documents how each page was fetched |
| Gaps from blocking | Google Scholar or a publisher starts serving CAPTCHAs halfway through, and the missing pages skew the sample | Zenrows' stealth keeps the run going through CAPTCHAs and challenges, and you retry failed URLs until the sample is complete |
| JavaScript-rendered sources | Comments, citation counts or statistics tables load after the page, so the raw HTML has none of them | Adaptive Stealth Mode escalates to browser rendering when a page needs it and returns the page after it renders |
| Noisy text | Menus, ads and cookie banners end up in the text your coders annotate | Markdown output returns clean text ready for annotation or NLP |
| Incomplete sample frames | The search page shows the first results, and you need every thread in a forum section | Crawl follows pagination and keeps every URL that matches your section pattern |
| Content that varies by country | The same site shows different results to visitors from different countries | Geotargeting fixes the country the request comes from, and you report it in your methods |

## How to use web scraping for research

1. **Write the collection protocol first.** List sources, inclusion rules, date range, fields and request pace, and get ethics approval where your study involves people.
2. **Use official APIs and dumps where they exist.** Wikipedia, arXiv and Crossref publish APIs or full dumps. Scrape what they do not cover.
3. **Build the sample frame with Crawl.** Collect every URL in scope before you fetch content, and save that list as part of the dataset.
4. **Fetch with fixed parameters and log everything.** Keep the URL, fetch time, parameters, status and a content hash for every record, and pace requests per domain.
5. **Archive raw responses.** Store each response as fetched, so you can re-code the corpus later without collecting it again.

```python Python
# Fetch a sample frame politely and log provenance for every record
# pip install requests
import csv, hashlib, time, requests
from datetime import datetime, timezone

PARAMS = {"mode": "auto", "response_type": "markdown"}
urls = [line.strip() for line in open("sample_frame.txt") if line.strip()]

with open("collection_log.csv", "a", newline="") as log:
    writer = csv.writer(log)
    for i, url in enumerate(urls):
        r = requests.get("https://api.zenrows.com/v1/",
                         params={"apikey": "YOUR_ZENROWS_API_KEY", "url": url, **PARAMS})
        digest = hashlib.sha256(r.content).hexdigest()
        with open(f"raw/{i:06d}.md", "wb") as f:
            f.write(r.content)
        writer.writerow([i, url, datetime.now(timezone.utc).isoformat(),
                         r.status_code, digest, PARAMS])
        time.sleep(2)  # pace requests per your protocol
```

```bash cURL
curl -X POST "https://api.zenrows.com/v1/crawls" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "FORUM_SECTION_URL",
    "max_items": 5000,
    "include_patterns": ["/t/"],
    "exclude_patterns": ["/u/", "/login"]
  }'
```

```bash CLI
npx -y @zenrows/cli init
zenrows crawl create FORUM_SECTION_URL --include-patterns /t/ --max-pages 5000 --follow
zenrows crawl results CRAWL_ID --out sample_frame.jsonl
zenrows fetch https://en.wikipedia.org/wiki/Open_science --output markdown
```

```text Claude Code (MCP)
# Connect once
claude mcp add zenrows -e ZENROWS_API_KEY=YOUR_ZENROWS_API_KEY -- npx -y @zenrows/mcp

# Then ask:
Use Zenrows to read the first five Google Scholar results pages for "misinformation" "social media" since 2020 and return a table of title, authors, year, venue and citation count, so I can check the scope before I write my collection protocol.
```

## Which Zenrows product fits

| If you need to | Use | Why |
|---|---|---|
| Collect pages from a fixed URL list | [Fetch](/products/fetch) | One request per page with explicit parameters, as HTML, Markdown or named fields |
| Build the sample frame for a forum, archive or site section | [Crawl](/products/crawl) | Follows links and pagination and returns every URL that matches your patterns |
| Collect tens of thousands of pages for a large corpus | [Batch](/products/batch) | The whole URL list runs as one job, each task keyed with your record ID |
| Scope sources before writing a protocol | [MCP Server](/mcp) | Ask Claude or Cursor to read a handful of pages through Zenrows and summarize what is there |

## What success looks like

| Measure | What good looks like | How to check it |
|---|---|---|
| Sample completeness | Every URL in the sample frame has a stored response or a logged, explained failure | Compare the collection log with the sample frame before analysis starts |
| Reproducibility | Another researcher can rerun the collection from your URL list and parameters | Publish the URL list, code and parameters, and test the rerun on a subset |
| Provenance | Every record carries its URL, fetch time, parameters and content hash | Check the schema of the final dataset for missing provenance fields |
| Data minimization | The dataset holds only the fields your protocol and ethics approval allow | Audit the stored fields against the protocol before sharing |
| Load on sources | Requests per domain stay within the pace your protocol sets | Count requests per domain per minute in the collection log |

## Web scraping for research FAQ

### Is web scraping allowed for academic research?

Many universities and funders accept web data as a research source, and several jurisdictions have text and data mining exceptions for research, such as Article 3 of the EU copyright directive and section 29A of the UK copyright act. Site terms of use and privacy law still apply. Check with your institution's research office or legal counsel before collection starts.

### Do I need ethics approval to scrape data?

If your study collects data about people, such as posts, comments or profiles, most institutions treat it as human subjects research and expect review by an ethics board or IRB. Plan to pseudonymize usernames, collect only the fields you need, store data securely and avoid quoting posts verbatim where a quote could identify the author.

### How do I make a scraped dataset reproducible?

Save the sample frame, the collection code, the request parameters and a log with each record's URL, fetch time, status and content hash. Archive the raw responses. Web pages change, so a rerun will not match byte for byte, but your log lets another researcher see exactly what you collected and repeat the method.

### How fast should I scrape for research?

Slowly enough that the site would not notice a difference in load. Set a pace per domain in your protocol, such as one request every few seconds for small sites, read robots.txt, and schedule large collections over several days. Zenrows handles the access side, and the pace is a research ethics decision that stays yours.

### Can I scrape Google Scholar for a literature review?

Yes. Google Scholar has no public API and starts serving CAPTCHAs after a few automated queries. Fetch results pages through Zenrows' stealth, collect title, authors, venue, year and citation count, then enrich each record with the full metadata from Crossref or the publisher's page using the DOI.

### How do I collect forum data for a study?

Start with the sample frame: crawl the forum section and keep every thread URL that matches its pattern, within your date range. Then fetch each thread as Markdown or as named fields (post text, author handle, timestamp, reply depth) and pseudonymize the handles before analysis. Run a pilot of a few hundred threads first, check that pagination inside long threads is captured, and fix the protocol before the full collection.

### Can I share a scraped dataset with other researchers?

Share the URL list, the code and the collection log openly. Sharing the raw content is a separate question, because the text may be copyrighted or contain personal data. Many projects publish identifiers and derived variables, and give the full text to other researchers under a data use agreement.

### Is web scraping useful for data science projects?

Yes, when the data you need is on web pages rather than in a downloadable file. Request Markdown for text models or name the fields with CSS selectors for tabular work, load the results into pandas, and keep the same provenance log so your features can be traced back to their source pages.

## Related use cases

- [AI training data](https://www.zenrows.com/use-cases/ai-training-data.md): Web corpora for pre-training and fine-tuning, collected as Markdown JSONL with a source URL on every document.
- [Market research web scraping](https://www.zenrows.com/use-cases/market-research-web-scraping.md): Assortments, prices, reviews and forum talk across a whole category, for sizing studies and recurring trackers.
- [Social media scraping](https://www.zenrows.com/use-cases/social-media-scraping.md): Public posts, profiles, comments and engagement from Reddit, YouTube, TikTok and X, as structured JSON.
- [News scraping](https://www.zenrows.com/use-cases/news-scraping.md): Full-text articles, headlines, bylines and dates from publishers, Google News and wires, as clean Markdown or JSON.
- [Sentiment analysis data](https://www.zenrows.com/use-cases/sentiment-analysis-data.md): Review text with ratings, forum threads and comments, cleaned and labeled for sentiment and NLP models.

Start free: https://app.zenrows.com/register. Docs: https://docs.zenrows.com