---
title: "Chatbot training data"
description: "Turn a website, help center and docs into chatbot training data: crawl Zendesk, Intercom or GitBook pages to Markdown, chunk them and refresh weekly."
canonical: https://www.zenrows.com/use-cases/chatbot-training-data
updated: 2026-10-18
---

# Chatbot training data

Chatbot training data is the content a support or sales chatbot answers from: your website, help center, docs and policies, converted to clean text. Zenrows Crawl walks the site from a start URL and returns each kept page as Markdown, and Batch re-reads those URLs on a schedule, so the knowledge base matches what the site says today.

## Sample output: one help center article

```json
{
  "url": "https://help.example.com/articles/return-an-order",
  "content_status": "fetched",
  "content": "# Return an order\n\nYou can return most items within 30 days of delivery from **Orders > Start a return**.\n\n## Items that cannot be returned\n\nGift cards and opened personal care items are final sale.\n\n## Related\n\n- [Track your refund](https://help.example.com/articles/track-refund)",
  "kb": {
    "collection": "returns",
    "locale": "en",
    "chunks": 3,
    "last_crawled": "2026-10-17T06:00:00Z",
    "content_hash": "9f2c41d0"
  }
}
```

An illustrative line from a crawl's results file: Crawl returns `url`, `content_status` and the Markdown `content`, and the `kb` block is metadata your ingestion step adds.

## Website content a chatbot learns from

Most chatbot knowledge bases come from a handful of site sections. The work is choosing which paths belong in the bot, ingesting them cleanly, and keeping them in step with the live site.

| Job | Data you collect | Typical sources |
|---|---|---|
| Support bot from a help center | Article text, steps, troubleshooting tables | Zendesk Guide, Intercom Articles, Freshdesk and Help Scout help centers |
| Docs assistant | Guides, API reference, code samples | Docusaurus, GitBook, ReadMe and Mintlify docs sites |
| Pre-sales bot | Plans, pricing, features, integrations, security pages | The company's own marketing website |
| Store assistant | Product details, shipping, returns, size guides | Shopify and WooCommerce storefronts |
| Multilingual bot | The same articles in every locale | Locale paths such as `/fr/`, `/de/` and `/es/` |
| Chatbot platform onboarding | A new customer's whole site, ingested at signup | Small-business websites, some behind Cloudflare |

## Why website-to-chatbot ingestion goes wrong

A chatbot is only as accurate as the pages it was given. Wrong answers usually trace back to junk pages in the index, a section the crawler never reached, or a policy that changed after the last ingest.

| What breaks | What it looks like | How Zenrows handles it |
|---|---|---|
| Junk pages in the index | Tag archives, search results and login pages crowd out real articles | URL patterns to keep and to skip hold the crawl to the paths you choose |
| Duplicate URLs | One article indexed three times under tracking parameters and `www` variants | Crawl normalizes URLs, strips `utm_*` and similar parameters, and drops duplicates |
| JavaScript-rendered help centers | Articles come back with a title and no body | Collect URLs with Crawl, then read them with Batch in Adaptive Stealth Mode, which escalates to browser rendering when a page needs it |
| Protected sites | A customer's site behind Cloudflare stops the ingest at a challenge page | Zenrows' stealth gets through Cloudflare, AWS WAF and similar protections, so the ingest reaches the article |
| Stale answers | The bot quotes a refund window the site changed last month | Batch re-reads the URL list weekly; re-embed only pages whose hash changed |
| Slow onboarding | New users wait on a site scan and leave | Start the crawl at signup and read results page by page as they arrive |

## How to turn a website into chatbot data

1. **Choose the sections.** Pick a start URL per section and write URL patterns to keep and to skip, for example keep `/articles/` and drop `/search` and `/login`.
2. **Crawl to Markdown.** Start a crawl with Markdown output and a page limit that fits the site size.
3. **Chunk and label.** Split each page on its headings, store the URL with every chunk, and add the section and locale your retrieval filters need.
4. **Refresh on a schedule.** Re-read the kept URLs weekly with Batch, compare content hashes, and re-embed only what changed.

```bash cURL
curl -X POST "https://api.zenrows.com/v1/crawls" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://help.example.com/",
    "max_items": 2000,
    "max_pages": 5000,
    "include_patterns": ["/articles/"],
    "exclude_patterns": ["/search", "/login", "/tags/"],
    "output_format": "markdown"
  }'
# Returns a crawl_id. Check progress with GET /v1/crawls/{crawl_id}
```

```python Python
# pip install requests
import hashlib, json, re
import requests

CRAWL_ID = "YOUR_CRAWL_ID"
r = requests.get(
    f"https://api.zenrows.com/v1/crawls/{CRAWL_ID}/download",
    headers={"X-API-Key": "YOUR_ZENROWS_API_KEY"},
    timeout=300,
)
r.raise_for_status()

chunks = []
for line in r.text.splitlines():
    row = json.loads(line)
    if row.get("content_status") != "fetched":
        continue
    page_hash = hashlib.sha256(row["content"].encode()).hexdigest()[:8]
    for i, part in enumerate(re.split(r"\n(?=## )", row["content"])):
        chunks.append({"id": f'{row["url"]}#{i}', "url": row["url"],
                       "hash": page_hash, "text": part.strip()})

print(len(chunks), "chunks ready for the vector store")
```

```bash CLI
npx -y @zenrows/cli init
zenrows crawl create https://help.example.com/ --include-patterns /articles/ --max-pages 5000 --output-format markdown --follow
zenrows crawl download CRAWL_ID --out help-center.jsonl
# Weekly refresh: jobs.jsonl lists the article URLs the crawl kept
zenrows batch create jobs.jsonl --output markdown --wait
```

```text Claude Code (MCP)
# Connect once
claude mcp add zenrows -e ZENROWS_API_KEY=YOUR_ZENROWS_API_KEY -- npx -y @zenrows/mcp

# Then ask:
Use Zenrows to crawl https://help.example.com/ and keep only pages under /articles/. Return each article as Markdown with its URL and title, and list any article that mentions the refund window.
```

## Which Zenrows product fits

| If you need to | Use | Why |
|---|---|---|
| Ingest a whole site or help center | [Crawl](/products/crawl) | Follows links from a start URL, keeps the paths you choose, returns Markdown |
| Refresh every known article on a schedule | [Batch](/products/batch) | Thousands of URLs as one job, with browser rendering when the site needs it |
| Re-read one page the moment it changes | [Fetch](/products/fetch) | Markdown output returns the updated article in one request |
| Run ingestion from a script or cron job | [CLI](/cli) | `batch create` with `--output markdown` and `--wait` in one line |

## What success looks like

| Measure | What good looks like | How to check it |
|---|---|---|
| Coverage | Every article in the chosen sections is in the knowledge base | Compare kept URLs with the sitemap or the help center's article count |
| Index cleanliness | No search, tag, login or duplicate pages in the index | Sample indexed URLs and count the ones that match exclude rules |
| Freshness | A change on the site reaches the bot within one refresh cycle | Edit a test article and time how long until the bot answers with the change |
| Answer accuracy | The bot answers a fixed question set from current pages | Rerun the set after each refresh and diff the answers |
| Ingest time | A new site is answerable soon after signup | Log time from crawl start to the first and last chunk embedded |
| Refresh run time | The weekly refresh finishes before the bot's content cut-off, even as sites grow | Compare each Batch job's start and finish times per site |

## Chatbot training data FAQ

### How do I train a chatbot on my website data?

Crawl the sections the bot should know, convert each page to Markdown, split the pages into chunks, embed them into a vector store, and let the chatbot retrieve the closest chunks for each question. Zenrows Crawl does the first two steps in one job: give it the start URL and the paths to keep, and it returns every kept page as Markdown with its URL.

### Is this training the model, or retrieval?

For almost every website chatbot it is retrieval. The model is not retrained; it reads your pages at answer time through a RAG pipeline. That is why freshness matters more than volume: when a policy changes, re-reading the page and re-embedding it updates the bot the same day. Fine-tuning is a separate job, covered on the AI training data page.

### Which pages should a chatbot knowledge base include?

Include pages a customer would accept as an answer: help articles, docs, pricing, plans, policies, product pages. Leave out search results, tag and category archives, login and account pages, and paginated listings, which repeat content and confuse retrieval. Express both lists as URL patterns to keep and to skip in the crawl.

### How often should I refresh the chatbot's content?

Match the refresh to how often the site changes. Weekly suits most help centers and docs; daily suits pricing, stock or policy pages that change often. Re-read the URL list with Batch, compare a hash of each page with the last run, and re-embed only the pages that changed.

### Can it ingest help centers built with JavaScript or behind Cloudflare?

Yes. Use Crawl to collect the article URLs, then read them through Batch or Fetch with Adaptive Stealth Mode. It starts with the lightest setup and escalates to browser rendering and stealth only when a page needs it, so the article body is rendered on JavaScript help centers, and Zenrows' stealth gets through Cloudflare, AWS WAF and similar protections. Plain sites stay on the lightest setup without any per-domain configuration.

### Can one chatbot cover several languages?

Yes. Crawl each locale path, such as /fr/ or /de/ , store the locale with every chunk, and filter retrieval by the user's language. Keeping locales as separate collections stops the bot from answering a French question with an English article.

### How should I chunk website pages for a RAG chatbot?

Split on the page's own structure. Markdown keeps headings, so cut each article at its H2 or H3 headings, and split any section that is still long at paragraph breaks. Keep the article title and the heading path with every chunk, so a chunk about refunds still says which product it belongs to. Store the source URL too, so the bot can cite the page and you can re-embed only the chunks from pages that changed.

## Related use cases

- [LLM web scraping for RAG and agents](https://www.zenrows.com/use-cases/llm-web-scraping.md): Live pages as Markdown or JSON for RAG pipelines, grounding and agent tool calls.
- [AI training data](https://www.zenrows.com/use-cases/ai-training-data.md): Web corpora for pre-training and fine-tuning, collected as Markdown JSONL with a source URL on every document.
- [Website content migration](https://www.zenrows.com/use-cases/website-content-migration.md): Every page, title, meta tag, image and URL from a legacy site, exported as Markdown or JSON.

Start free: https://app.zenrows.com/register. Docs: https://docs.zenrows.com