---
title: "AI training data"
description: "Collect AI training data from the web: crawl docs, Q&A sites, news and filings into Markdown JSONL with source URLs for LLM pre-training and fine-tuning."
canonical: https://www.zenrows.com/use-cases/ai-training-data
updated: 2026-10-18
---

# AI training data

AI training data is the text, code or images a model learns from during pre-training or fine-tuning, and much of it comes from public web pages. Zenrows Crawl finds the URLs on each source site and Batch fetches them at volume as clean Markdown, so you build a corpus with a source URL and fetch time on every document.

## Sample output: one corpus record

```json
{
  "id": "sha256:4be1c09a7d3e",
  "url": "https://www.example-standards.org/guides/water-quality-testing",
  "domain": "example-standards.org",
  "fetched_at": "2026-10-17T03:41:56Z",
  "format": "markdown",
  "language": "en",
  "license_note": "CC BY 4.0 per page footer",
  "tokens": 1842,
  "text": "# Water quality testing\n\nSamples should be collected in sterile containers and analysed within 24 hours.\n\n## Field parameters\n\n| Parameter | Method |\n|---|---|\n| pH | Electrode |\n| Turbidity | Nephelometer |"
}
```

An illustrative JSONL line: Zenrows returns the Markdown `text`, and the hash id, language, licence note and token count come from your own pipeline.

## What AI training data teams collect

Training data projects differ from RAG in scale and in what they keep. A RAG index holds the pages a bot needs today; a training corpus holds millions of documents, frozen per version, each with enough provenance to defend later.

| Job | Data you collect | Typical sources |
|---|---|---|
| Domain pre-training corpus | Long-form text in one field or industry | Government publications, standards bodies, university repositories, Wikipedia |
| Instruction and Q&A pairs | Questions with accepted or expert answers | Stack Exchange sites, product FAQ pages, documentation |
| Multilingual corpora | Native-language text per market | National news sites and public broadcasters, read from each country with geotargeting |
| Code and technical text | Docs, READMEs, tutorials, API references | Official language docs, GitHub READMEs, developer blogs |
| Domain fine-tuning sets | Product, legal, medical or financial language | Amazon product pages, court opinions, PubMed abstracts, SEC EDGAR filings |
| Fresh evaluation sets | Pages published after a model's knowledge cutoff | Recent news, new filings, new documentation releases |

## Why AI training data collection stalls

Once a URL list passes a few hundred thousand, collection tends to slow down on throughput first, then on cleaning the text and tracing each record back to its source.

| What breaks | What it looks like | How Zenrows handles it |
|---|---|---|
| Throughput | A homegrown queue of millions of URLs takes weeks and needs constant restarts | Batch runs the whole URL list as one job and reports per-URL status |
| Blocked sources | Q&A sites and publishers answer with DataDome or Cloudflare challenges | Zenrows' stealth gets through Cloudflare, DataDome, Akamai and similar protections |
| Markup in the text | Navigation, scripts and inline styles leak into training documents | Markdown output returns structured text instead of raw HTML |
| Duplicate URLs | The same page fetched under `www`, trailing-slash and tracking variants | Crawl normalizes URLs, strips tracking parameters and drops duplicates before fetching |
| Missing provenance | Nobody can say where a document came from or when | Every Batch result keeps its URL and your `external_id`, so each record traces to a source |
| Regional content | A site serves different text by country | Geotargeting picks the country each request comes from, so you read each market's version of the page |

## How to collect AI training data with Zenrows

1. **Define the sources.** List the domains and paths your model should learn from, and record each source's terms and licence before you collect.
2. **Discover the URLs.** Run Crawl on each source with URL patterns to keep and to skip, and no output format, to get a clean URL list.
3. **Fetch at volume.** Send the URLs to Batch with Markdown output and Adaptive Stealth Mode, which starts with the lightest setup and escalates to browser rendering and stealth only when a page needs it.
4. **Dedupe and package.** Hash each document to drop exact duplicates, run your near-duplicate and quality filters, and write JSONL shards with the URL and fetch time on every record.

```bash cURL
# Discover article URLs on one source (URLs only, no page content)
curl -X POST "https://api.zenrows.com/v1/crawls" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "START_URL",
    "max_items": 100000,
    "max_pages": 100000,
    "include_patterns": ["/guides/", "/articles/"],
    "exclude_patterns": ["/tag/", "/search", "/print/"]
  }'
```

```python Python
# pip install requests
import hashlib, json, time
from datetime import datetime, timezone
import requests

API = "https://async.api.zenrows.com/v1"
HEADERS = {"X-API-Key": "YOUR_ZENROWS_API_KEY"}

job = requests.post(f"{API}/jobs", headers=HEADERS, timeout=60, json={
    "type": "regular",
    "status": "closed",
    "zenrows_params": {"mode": "auto", "response_type": "markdown"},
    "tasks": [{"url": u, "external_id": str(i)} for i, u in enumerate(SOURCE_URLS)],
}).json()
job_id = job["job_id"]

while True:  # wait for the run to finish
    run = requests.get(f"{API}/jobs/{job_id}", headers=HEADERS, timeout=60).json()["latest_run"]
    if run["status"] in ("completed", "failed", "stopped"):
        break
    time.sleep(30)

seen, params = set(), {"status": "successful"}
with open("corpus-000.jsonl", "a") as shard:
    while True:
        page = requests.get(f"{API}/jobs/{job_id}/results", headers=HEADERS,
                            params=params, timeout=60).json()
        for row in page["results"]:
            text = requests.get(row["result_url"], timeout=60).text
            digest = hashlib.sha256(text.encode()).hexdigest()
            if digest in seen:
                continue  # exact duplicate
            seen.add(digest)
            shard.write(json.dumps({"id": digest, "url": row["url"], "text": text,
                "fetched_at": datetime.now(timezone.utc).isoformat()}) + "\n")
        if not page.get("next_cursor"):
            break
        params["cursor"] = page["next_cursor"]
```

```bash CLI
npx -y @zenrows/cli init
zenrows crawl create START_URL --include-patterns /guides/ --include-patterns /articles/ --max-pages 1000 --follow
zenrows crawl results CRAWL_ID --out urls.jsonl
# keep one {"url": "..."} per line for Batch
zenrows batch create urls.jsonl --output markdown --wait
zenrows batch results JOB_ID --out corpus.jsonl
```

```text Claude Code (MCP)
# Connect once
claude mcp add zenrows -e ZENROWS_API_KEY=YOUR_ZENROWS_API_KEY -- npx -y @zenrows/mcp

# Then ask:
Use Zenrows to crawl the /guides/ section of START_URL, fetch every guide as Markdown, and save one JSONL record per page with the URL, title and text.
```

## Which Zenrows product fits

| If you need to | Use | Why |
|---|---|---|
| Build the URL list for a source site | [Crawl](/products/crawl) | Follows links within the domain, keeps matching paths, removes duplicate URLs |
| Fetch millions of documents | [Batch](/products/batch) | Whole URL lists as jobs, Markdown output, results per URL to download |
| Test a new source before a full run | [Fetch](/products/fetch) | One request shows exactly what the corpus will receive for that page |
| Turn pages into fields for fine-tuning pairs | [Extract](/products/extract) | Automatic structured extraction returns JSON, such as question and answer, without selectors |

## What success looks like

| Measure | What good looks like | How to check it |
|---|---|---|
| Corpus size | Token count per source and in total reaches the training plan | Sum token counts per shard and per domain after each run |
| Fetch success per source | Each source delivers usable text for nearly every URL submitted | Read successful and failed counts from each Batch run's stats |
| Duplicate rate | Exact duplicates removed, near-duplicates below your threshold | Report documents dropped at each dedup stage per run |
| Provenance | Every record carries URL, fetch time and source licence note | Validate shards against a schema before they enter training |
| Text quality | Records hold content, not menus, cookie notices or error pages | Sample records per domain and score them, or run a quality classifier |
| Refresh time | A full re-collection of every source finishes inside your training schedule | Compare each Batch run's start and finish times with the date the data is due |

## AI training data FAQ

### What is AI training data?

AI training data is the material a model learns from. For language models that means text and code, used in pre-training (broad corpora) and fine-tuning (smaller, task-shaped sets such as question and answer pairs). Public web pages are one of the largest sources, which is why teams collect them with crawlers and scraping APIs, then clean and filter the result.

### Should I buy AI training data or collect it myself?

Ready-made datasets from AI training data providers are quick to start with, but you get what everyone else gets, on the provider's schedule. Collecting your own lets you choose the sources, the domain mix and the refresh date, and keep a URL trail for every document. Zenrows is the collection layer: you decide what to collect and own the corpus.

### How do I collect LLM training data from websites?

Use Crawl to discover the URLs on each source site, then fetch them in bulk with Batch and Markdown output. Store each document as a JSONL record with its URL and fetch time, then run deduplication, language detection and quality filtering before the data reaches a training run.

### How do I deduplicate scraped training data?

Work in layers. Crawl already drops duplicate URLs and URL variants with tracking parameters. After fetching, hash each document to remove exact copies, as in the Python example. Then run a near-duplicate pass, such as MinHash on shingles, to catch syndicated articles and boilerplate pages that differ by a date or a byline.

### Is it legal to scrape web data for AI training?

It depends on the source, the jurisdiction and the use. Teams typically restrict collection to publicly available pages, check each site's terms and robots rules, honour licences such as Creative Commons attribution, and avoid or redact personal data. Copyright questions around training are still being settled. Ask your legal team before collecting for a specific model.

### What format should web training data be in?

Most pipelines store one document per line in JSONL, with the text in Markdown or plain text and fields for URL, fetch time, language and licence. Markdown keeps headings, lists and tables, which helps chunking and quality scoring; you can strip it to plain text later if your tokenizer prefers.

### How large should an LLM training dataset be?

It depends on the stage. Pre-training corpora are measured in billions of tokens and favour breadth. Fine-tuning sets are far smaller and favour quality: a few thousand clean, consistent examples of the task often teach more than a large noisy set. Start small, hold out an evaluation set from the same sources, and add data where the evaluation shows gaps instead of collecting everything at once.

### Can I collect image data for vision models too?

Yes. The same Crawl and Batch setup collects image URLs, alt text and captions, and Fetch downloads the image files themselves. The computer vision training data page covers that workflow.

## Related use cases

- [Chatbot training data](https://www.zenrows.com/use-cases/chatbot-training-data.md): Your website, help center and docs as clean Markdown for a chatbot knowledge base, refreshed on schedule.
- [LLM web scraping for RAG and agents](https://www.zenrows.com/use-cases/llm-web-scraping.md): Live pages as Markdown or JSON for RAG pipelines, grounding and agent tool calls.
- [Sentiment analysis data](https://www.zenrows.com/use-cases/sentiment-analysis-data.md): Review text with ratings, forum threads and comments, cleaned and labeled for sentiment and NLP models.
- [News scraping](https://www.zenrows.com/use-cases/news-scraping.md): Full-text articles, headlines, bylines and dates from publishers, Google News and wires, as clean Markdown or JSON.
- [Web scraping for research](https://www.zenrows.com/use-cases/web-scraping-for-research.md): Article metadata, forum posts, statistics and news text collected for studies, with a reproducible fetch log.

Start free: https://app.zenrows.com/register. Docs: https://docs.zenrows.com