---
title: "News scraping"
description: "News scraping API for full-text articles, headlines, bylines and dates from publishers, Google News and press wires. Clean Markdown or JSON per article."
canonical: https://www.zenrows.com/use-cases/news-scraping
updated: 2026-10-18
---

# News scraping

News scraping collects articles from publisher websites, Google News and press release wires: headline, byline, publish date, section and full body text. Zenrows fetches each article from protected and JavaScript-heavy news sites and returns it as clean Markdown or structured JSON, so you can build an article dataset for search, NLP or aggregation.

## Sample output: one news article

```json
{
  "url": "https://www.example-news.com/business/2026/10/14/port-strike-ends",
  "canonical_url": "https://www.example-news.com/business/2026/10/14/port-strike-ends",
  "source": "example-news.com",
  "headline": "Port workers end strike after nine-day standoff",
  "subheadline": "Container backlog expected to clear by early November",
  "authors": ["Staff reporter"],
  "section": "Business",
  "tags": ["shipping", "labour", "ports"],
  "language": "en",
  "published_at": "2026-10-14T06:42:00Z",
  "modified_at": "2026-10-14T11:05:00Z",
  "word_count": 812,
  "image": "https://www.example-news.com/img/port-strike.jpg",
  "body_markdown": "Dock workers at the city's two container terminals returned to work on Tuesday...",
  "outbound_links": 7
}
```

An illustrative article record. Extract returns fields like these without selectors, and Markdown output gives you the body text with the menus, ads and related-story widgets removed.

## What news scraping teams collect

A news scraper usually has two halves: something that finds new article URLs, and something that turns each article page into a clean record. What changes between teams is the source list and how far back they go.

| Job | Data you collect | Typical sources |
|---|---|---|
| Article datasets for NLP and search | Full body text, headline, byline, publish date, language | nytimes.com, theguardian.com, elpais.com, lefigaro.fr, ndtv.com |
| News aggregation products | Headline, summary, image, section, canonical URL | National outlets, regional papers, trade press such as techcrunch.com |
| Google News harvesting | Story titles, outlets, timestamps, article links per query | Google News search pages and its public RSS search feeds |
| Press release archives | Company, release title, date, full text, media contacts block, for every release in a period | businesswire.com, accesswire.com, company newsroom archives |
| Financial news for research | Headlines, tickers named, publish time to the minute, full text for event studies | reuters.com, ft.com, businessinsider.com, forbes.com |
| Historical archive backfill | Every article in a date range, with its metadata | Publisher sitemaps, date archives and section pages |

## Why news site scraping breaks

Publishers earn money from page views and subscriptions, so the large ones protect their pages and wrap every article in scripts, ads and consent banners. The article text is often the smallest part of the HTML.

| What breaks | What it looks like | How Zenrows handles it |
|---|---|---|
| Bot protection on large publishers | 403 pages or challenge screens from major news sites after a few hundred requests | Zenrows' stealth gets through Cloudflare, DataDome and similar protections |
| Body loaded by JavaScript | The raw HTML has the headline but an empty article container | Adaptive Stealth Mode escalates to browser rendering and returns the page after the article has rendered |
| Boilerplate in the text | Newsletter prompts, "read more" links and ad copy end up in your corpus | Markdown output returns the readable content, and Extract separates headline, byline, date and body |
| A different template per outlet | Selectors written for one paper return nothing on the next fifty | Extract returns the same article fields across publishers without a selector set per site |
| Regional editions | The same outlet serves different front pages and stories by country | Geotargeting fetches the edition readers see in that country |
| Backfilling years of articles | A list of 500,000 archive URLs takes days through a single script | Batch runs the full list as one job and reports each URL's result |

## How to scrape news articles with Zenrows

1. **Find the article URLs.** Read the publisher's news sitemap or RSS feed, or start a Crawl from a section page with URL patterns to keep that match its article paths.
2. **Fetch each article.** Send the URL to Fetch with Adaptive Stealth Mode, which starts with the lightest setup and escalates to browser rendering and stealth only when an outlet needs it.
3. **Choose the output.** Use Extract for headline, byline, date and body as JSON, or Markdown output when the text is going into an LLM or search index.
4. **Backfill and refresh in bulk.** Submit archive URLs to Batch, then re-fetch recent articles once to pick up corrections and updated timestamps.

```python Python
# pip install requests
import requests

params = {
    "apikey": "YOUR_ZENROWS_API_KEY",
    "url": "ARTICLE_URL",  # one article page
    "mode": "auto",
    "extract": "auto",
}
response = requests.get("https://api.zenrows.com/v1/", params=params)
article = response.json()
print(article)
```

```bash cURL
# Collect article URLs (as Markdown pages) from one publisher section
curl -X POST "https://api.zenrows.com/v1/crawls" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "SECTION_URL",
    "max_items": 1000,
    "max_pages": 1000,
    "include_patterns": ["/2026/"],
    "exclude_patterns": ["/video/", "/live/"],
    "output_format": "markdown"
  }'
```

```bash CLI
npx -y @zenrows/cli init
# Turn one article into clean Markdown, or into article fields as JSON
zenrows fetch ARTICLE_URL --output markdown
zenrows extract ARTICLE_URL --autoparse
# Backfill an archive: archive.jsonl holds one {"url": "..."} per line
zenrows batch create archive.jsonl --output markdown --wait
```

```text Claude Code (MCP)
# Connect once
claude mcp add zenrows -e ZENROWS_API_KEY=YOUR_ZENROWS_API_KEY -- npx -y @zenrows/mcp

# Then ask:
Use Zenrows to read the BBC News technology section, open the ten newest articles,
and return headline, author, publish time and the full body text of each as JSON.
```

## Which Zenrows product fits

| If you need to | Use | Why |
|---|---|---|
| Pull one article or front page on demand | [Fetch](/products/fetch) | One request returns the rendered page from protected news sites |
| Get headline, byline, date and body as JSON | [Extract](/products/extract) | Returns article fields across outlets with no selectors to maintain |
| Collect every article URL in a section or archive | [Crawl](/products/crawl) | Follows links on the publisher's domain and keeps only paths that match your patterns |
| Backfill hundreds of thousands of archive URLs | [Batch](/products/batch) | Runs the whole list as one job, with results by webhook or download |
| Test a source quickly from the terminal | [CLI](/cli) | Fetch an article as Markdown with one command before you write any code |

## What success looks like

| Measure | What good looks like | How to check it |
|---|---|---|
| Body completeness | Article records carry the full body, not the first paragraph or a teaser | Compare word counts against a hand-checked sample from each outlet |
| Clean text | No navigation, ad copy or newsletter prompts inside the body field | Search the corpus for recurring boilerplate strings per source |
| Date accuracy | Publish and update times match what the outlet shows, in one timezone | Spot-check a sample per outlet and store every timestamp in UTC |
| Source coverage | Every outlet on your list returns articles on every run | Track successful articles per source per day and alert on a drop to zero |
| Duplicate rate | One record per story per outlet, even when URLs carry tracking parameters | Count records that share a canonical URL or body hash |
| Backfill throughput | An archive job finishes in the window you planned, with failed URLs retried once | Log job duration and failed URLs per Batch job |

## News scraping FAQ

### Is it legal to scrape news articles?

Reading and analysing publicly available articles is common in research, media analysis and search. Copyright is the main issue: headlines, short snippets and links are treated differently from republishing full articles, which usually needs a licence from the publisher. Respect each site's terms and robots rules, and check your use case with your legal team, especially if you plan to show article text to your own users.

### How do I scrape Google News?

Fetch a Google News search URL, such as news.google.com/search?q=your+topic , with Adaptive Stealth Mode, and read the story titles, outlets, times and links from the result. Google News also publishes RSS search feeds that return the same stories as XML. The links point to the original articles, which you then fetch one by one for the full text.

### Can a news scraper get paywalled articles?

Zenrows fetches what a public visitor sees. On soft-metered sites that is often the full article. Content reserved for paying subscribers belongs to those subscriptions, so get it through a licence, a publisher feed or your own subscription terms rather than a scraper.

### What is the difference between a news scraper API and a news data API?

A news data API sells a fixed, pre-collected index of outlets and fields. A news scraper API such as Zenrows fetches the pages you choose, when you choose, and returns them as Markdown or JSON. Use the scraper approach when you need outlets, regions or fields a fixed index does not cover, or when freshness has to be on your schedule.

### How do I find new articles as they are published?

Read the publisher's news sitemap, which lists articles from the last two days with publish times, or its RSS feed. Compare each read against the URLs already in your dataset and fetch only the rest. If you need alerts on specific companies or keywords rather than a complete article set, see news monitoring .

### Which fields should a news article scraper return?

At minimum: canonical URL, outlet, headline, author, publish time, update time, section and the full body text. Useful extras are the lead image, tags, language and word count. Store times in UTC and keep the canonical URL as the key, so the same story reached through a tracking link or an AMP page does not become a second record.

### Can I scrape news in other languages and countries?

Yes. Fetch returns the page in whatever language the outlet publishes, and Markdown output keeps non-Latin scripts intact. Use geotargeting to pick the country the request comes from when an outlet shows different editions or stories by country.

## Related use cases

- [AI training data](https://www.zenrows.com/use-cases/ai-training-data.md): Web corpora for pre-training and fine-tuning, collected as Markdown JSONL with a source URL on every document.
- [Sentiment analysis data](https://www.zenrows.com/use-cases/sentiment-analysis-data.md): Review text with ratings, forum threads and comments, cleaned and labeled for sentiment and NLP models.
- [Alternative data for investors](https://www.zenrows.com/use-cases/alternative-data.md): Hiring, pricing, app ranks, reviews and store counts per ticker, collected weekly as point-in-time series.
- [Adverse media screening](https://www.zenrows.com/use-cases/adverse-media-screening.md): Negative news on customers, directors and suppliers from global and local sources, with text and screenshots as evidence.

Start free: https://app.zenrows.com/register. Docs: https://docs.zenrows.com