---
title: "LLM web scraping for RAG and agents"
description: "LLM web scraping API that turns live pages into Markdown or JSON for RAG, grounding and agents. Docs, news and product pages, plus an MCP server."
canonical: https://www.zenrows.com/use-cases/llm-web-scraping
updated: 2026-10-18
---

# LLM web scraping for RAG and agents

LLM web scraping turns live web pages into clean Markdown or JSON that a language model can read, for RAG pipelines, grounding and agent tool calls. Zenrows Fetch returns a URL as Markdown in one request, including protected and JavaScript-heavy pages, and the Zenrows MCP server lets Claude, Cursor and other agents make the same call on their own.

## Sample output: one page, ready for a model

```json
{
  "url": "https://docs.example.com/api/rate-limits",
  "fetched_at": "2026-10-17T09:14:03Z",
  "format": "markdown",
  "markdown": "# Rate limits\n\nEach API key can run 50 concurrent requests.\n\n## Retry headers\n\n| Header | Meaning |\n|---|---|\n| Retry-After | Seconds to wait before the next call |\n\nSee [Errors](https://docs.example.com/api/errors).",
  "chunks": [
    { "id": "rate-limits#0", "heading": "Rate limits", "chars": 58 },
    { "id": "rate-limits#1", "heading": "Retry headers", "chars": 132 }
  ],
  "links": [
    "https://docs.example.com/api/errors",
    "https://docs.example.com/api/authentication"
  ]
}
```

An illustrative record: Fetch returns the `markdown` string, and your pipeline adds the URL, timestamp and chunk ids before it embeds or prompts.

## What LLM web scraping is used for

An LLM pipeline reads the web at two moments. Before the question, they index pages into a vector store. During the question, a model or agent reads a page it chose a second ago.

| Job | Data you collect | Typical sources |
|---|---|---|
| URL to Markdown for a prompt | One page's body as Markdown, with headings, tables and links kept | Docs sites, changelogs, pricing pages, GitHub READMEs |
| RAG ingestion | Markdown chunks with source URL and fetch time | Developer docs, regulator guidance, standards bodies, Wikipedia |
| Grounding at query time | The current text of the pages the answer will cite | Reuters, AP News, vendor pricing pages, government sites |
| Agent tool calls | A `scrape` or `extract` result returned in the middle of a task | Whatever URL the agent decides to open |
| Structured context for tools | Named fields as JSON instead of prose | Amazon product pages, company websites, Indeed job posts |
| Index refresh | A new copy of every indexed URL, compared with the last one | The same URL list, re-read daily or weekly |

## Why web data breaks LLM pipelines

Most failures in a RAG or agent stack trace back to the page the model was handed. A blocked request or an empty shell looks like a normal response, so the model answers from memory and nobody notices.

| What breaks | What it looks like | How Zenrows handles it |
|---|---|---|
| Raw HTML in the context | Scripts, styles and class names take most of the token budget | Markdown output returns the page with headings, tables and links kept |
| JavaScript-rendered pages | A docs site or app returns an empty `<div id="root">` | Adaptive Stealth Mode escalates to browser rendering when a page needs it |
| Blocked fetches | A Cloudflare or DataDome challenge comes back instead of the article | Zenrows' stealth gets through Cloudflare, DataDome, Akamai and similar protections |
| Stale index | The assistant quotes last quarter's pricing with confidence | Re-fetch the URL list with Batch on a schedule and re-embed what changed |
| Output a tool cannot parse | The agent gets a 400 KB HTML string and truncates it | Markdown for reading, or Extract for JSON fields |

## How to scrape the web for LLMs

1. **Pick the output per job.** Use Markdown when a model will read the page, and Extract when a tool needs named fields.
2. **Fetch the page.** Call Fetch with Markdown output and Adaptive Stealth Mode, which starts with the lightest setup and escalates to browser rendering and stealth only when a page needs it.
3. **Chunk and keep the source.** Split on headings and store the URL and fetch time with every chunk, so answers can cite where they came from.
4. **Give agents the same tool.** Connect the MCP server once, or run the CLI from a terminal or a coding agent.

```python Python
# pip install requests
import re
from datetime import datetime, timezone
import requests

TARGET_URL = "https://docs.example.com/api/rate-limits"  # any page you want to index

def fetch_markdown(url):
    params = {
        "apikey": "YOUR_ZENROWS_API_KEY",
        "url": url,
        "mode": "auto",
        "response_type": "markdown",
    }
    r = requests.get("https://api.zenrows.com/v1/", params=params, timeout=120)
    r.raise_for_status()
    return r.text

def chunk_by_heading(url, markdown):
    fetched_at = datetime.now(timezone.utc).isoformat()
    parts = re.split(r"\n(?=#{1,3} )", markdown)
    return [{"url": url, "fetched_at": fetched_at, "text": p.strip()}
            for p in parts if p.strip()]

chunks = chunk_by_heading(TARGET_URL, fetch_markdown(TARGET_URL))
print(len(chunks), "chunks ready to embed")
```

```javascript Node.js
// Node 18+. A tool function any agent framework can register.
export async function readPage(url, { fields = false } = {}) {
  const params = new URLSearchParams({
    apikey: process.env.ZENROWS_API_KEY,
    url,
    mode: "auto",
  });
  if (fields) params.set("extract", "auto");
  else params.set("response_type", "markdown");

  const res = await fetch(`https://api.zenrows.com/v1/?${params}`);
  if (!res.ok) throw new Error(`Zenrows ${res.status}: ${await res.text()}`);
  return fields ? res.json() : res.text();
}
```

```bash CLI
npx -y @zenrows/cli init
zenrows fetch https://docs.example.com/api/rate-limits --output markdown
zenrows extract https://docs.example.com/api/rate-limits --autoparse
zenrows batch create docs-urls.jsonl --wait
```

```text Claude Code (MCP)
# Connect once
claude mcp add zenrows -e ZENROWS_API_KEY=YOUR_ZENROWS_API_KEY -- npx -y @zenrows/mcp

# Then ask:
Use Zenrows to read https://docs.example.com/api/rate-limits as Markdown. Answer my
question using only that page, quote the sentence you relied on, and give me the URL.
```

## Which Zenrows product fits

| If you need to | Use | Why |
|---|---|---|
| Turn one URL into Markdown for a prompt | [Fetch](/products/fetch) | Markdown output returns LLM-ready text in one request |
| Hand a tool JSON fields without writing selectors | [Extract](/products/extract) | Returns the page's meaningful data as structured JSON |
| Re-read thousands of indexed URLs on a schedule | [Batch](/products/batch) | One job for the whole list, results by webhook or download |
| Let Claude, Cursor or Codex read pages mid-task | [MCP Server](/mcp) | The agent calls `scrape` and `extract` as standard MCP tools |
| Wire web access into your own agent framework | [Agent Toolkit](/agents) | Skills and plugins that tell the agent which Zenrows call fits each task |

## What success looks like

| Measure | What good looks like | How to check it |
|---|---|---|
| Content completeness | The Markdown holds the text a visitor sees, tables included | Spot-check a sample of pages against the live site each week |
| Tokens per page | Pages arrive as Markdown, a fraction of the raw HTML size | Compare token counts of HTML and Markdown on the same URLs |
| Failed reads | Blocked or empty pages are retried or flagged, never embedded | Log every non-200 and every page under a minimum length |
| Index freshness | Each indexed URL was re-read within your refresh window | Store `fetched_at` per chunk and alert on anything older |
| Citation accuracy | Every answer links a URL whose text contains the quoted claim | Run a string match of quoted claims against the stored chunks |

## LLM web scraping FAQ

### What is LLM web scraping?

It means two related things. Scraping for LLMs prepares web pages as input a model can use: Markdown for reading, JSON for tools, both with a source URL. Scraping with LLMs uses a model to pull fields out of a page instead of CSS selectors. Zenrows covers both: Fetch with Markdown output for the first, and Extract for structured fields without selectors.

### How do I convert a URL to Markdown with an API?

Send a GET request to https://api.zenrows.com/v1/ with your apikey , the url , and Markdown output turned on. The response body is the page as Markdown, with headings, lists, tables and links kept. Add Adaptive Stealth Mode and Zenrows renders the page in a browser or applies its stealth only when the site needs it. From a terminal, zenrows fetch URL --output markdown does the same.

### What is LLM grounding, and where does web data fit?

Grounding means giving the model source text at answer time so it quotes facts instead of recalling them. Web data is usually the freshest source available: a vendor's current pricing page, today's news, the latest version of a regulation. Fetch the pages the answer depends on, pass the Markdown into the prompt, and ask the model to cite the URL for each claim.

### How do AI agents scrape websites through MCP?

Connect the Zenrows MCP server to the agent once. Claude, Cursor, Codex, Windsurf, Gemini CLI and other MCP clients then see tools such as scrape and extract , and call them when a task needs a web page. The remote server lives at https://mcp.zenrows.com/mcp and takes your API key as a Bearer token. Claude can also connect with an OAuth sign-in.

### Should I give the model Markdown or JSON?

Markdown when the model has to read and reason: articles, docs, policies, long product descriptions. JSON when code or a tool consumes the result: prices, ratings, dates, addresses. Many pipelines do both on the same page, Markdown into the vector store and Extract fields into a database.

### Does it work with LangChain, LlamaIndex and agent SDKs?

Yes. Fetch is a plain HTTP API, so any framework can call it from a tool function like the Node.js example above. The Zenrows docs also include integration guides for LangChain, LlamaIndex, CrewAI, Mastra, the OpenAI Agents SDK and the Claude Agent SDK, and any framework that speaks MCP can use the remote server directly.

### How do I keep a RAG index fresh?

Store the fetch time with every chunk, then re-read the indexed URLs on a schedule that matches how fast each source changes: news daily, docs weekly, policies monthly. Send the list as one Batch job, hash each page's Markdown, and re-embed only the pages whose hash changed. Flag any page that comes back empty or much shorter than last time instead of overwriting the old chunks.

## Related use cases

- [AI agent browser](https://www.zenrows.com/use-cases/ai-agent-browser.md): A cloud browser your agent drives over MCP or CDP to log in, click and fill forms.
- [Chatbot training data](https://www.zenrows.com/use-cases/chatbot-training-data.md): Your website, help center and docs as clean Markdown for a chatbot knowledge base, refreshed on schedule.
- [AI training data](https://www.zenrows.com/use-cases/ai-training-data.md): Web corpora for pre-training and fine-tuning, collected as Markdown JSONL with a source URL on every document.

Start free: https://app.zenrows.com/register. Docs: https://docs.zenrows.com