---
title: "Insurance data scraping"
description: "Insurance data scraping for competitor quotes and premiums, policy terms, rate filings and property risk data from comparison sites and insurer pages."
canonical: https://www.zenrows.com/use-cases/insurance-data
updated: 2026-10-18
---

# Insurance data scraping

Insurance data scraping collects the public market data insurers and insurtechs use for pricing and underwriting, such as competitor quotes and premiums, policy terms, rate filings, property risk records and provider networks. Zenrows runs multi-step quote forms in a real browser, fetches protected comparison sites, and returns results as structured JSON tied to your test profiles.

## Sample output: quotes for one test profile

```json
{
  "profile_id": "motor-gb-p017",
  "line": "motor",
  "market": "GB",
  "profile": {
    "driver_age": 34,
    "postcode_area": "LS",
    "vehicle": "2021 hatchback, 1.0 petrol",
    "annual_mileage": 8000,
    "no_claims_years": 5,
    "cover": "comprehensive",
    "voluntary_excess": 250
  },
  "captured_at": "2026-10-16T07:40:00Z",
  "quotes_returned": 38,
  "currency": "GBP",
  "quotes": [
    { "rank": 1, "insurer": "Insurer A", "annual_premium": 612.40, "compulsory_excess": 250 },
    { "rank": 2, "insurer": "Insurer B", "annual_premium": 648.00, "compulsory_excess": 200 },
    { "rank": 3, "insurer": "Insurer C", "annual_premium": 655.75, "compulsory_excess": 250 }
  ]
}
```

An illustrative capture from a comparison site. Every quote is tied to a fixed test profile, so a premium change between runs reflects a move in the insurer's pricing.

## What insurance teams collect from the web

Pricing teams want to know where they rank for a given risk. Underwriting and claims teams want facts about the risk itself. Both come from public pages, but from very different sites.

| Job | Data you collect | Typical sources |
|---|---|---|
| Competitor quote monitoring | Premium, excess, rank and panel size per test profile | Compare the Market, MoneySupermarket, GoCompare, Confused.com, Check24, Verivox, The Zebra |
| Product and policy terms | Cover limits, exclusions, add-ons, product information documents | Insurer product pages, policy wording and IPID documents |
| Rate filing research | Filed rate changes, effective dates, rating factors | SERFF Filing Access, state insurance department sites |
| Property risk enrichment | Flood zone, building age, floor area, permit history | FEMA Flood Map Service Center, county assessor sites, listing portals |
| Replacement cost for claims | Current retail price of claimed items, used vehicle values | Retailer product pages, automotive marketplaces |
| Network and reputation | Provider network listings, complaint data, customer reviews | Insurer provider directories, NAIC complaint data, review sites |

## Why insurance data is hard to collect

A premium does not exist until someone answers the questions. That makes quote collection a browser job, on sites that are built to stop repeated automated submissions.

| What breaks | What it looks like | How Zenrows handles it |
|---|---|---|
| Quotes behind multi-page forms | Six to ten pages of questions before any price appears | Browser runs each step in a real Chromium session over Playwright |
| Comparison sites flag automation | Bot protection such as Akamai or Imperva ends the session halfway through the form, or the results page never loads | Zenrows' stealth gets through Akamai, Imperva and similar protections, with a fresh session per profile |
| Results arrive in waves | The panel fills in over a minute as insurers respond, so an early read misses quotes | Wait until the result count stops growing before reading the page |
| Prices depend on the market | The same product page shows different cover and prices by country | Geotargeting loads the page from the customer's country, as a local customer sees it |
| Terms spread across documents | Exclusions and limits sit in long wording pages and document libraries | Markdown output and the tables output filter return text and tables ready to compare |

## How to collect insurance data with Zenrows

1. **Fix your test profiles.** Agree a set of synthetic risk profiles with your pricing and compliance teams, each with a stable ID.
2. **Run each profile through the quote form.** Drive the form in Browser and wait for the panel to finish loading.
3. **Store every quote with its profile.** Save rank, insurer, premium, excess and capture time against the profile ID.
4. **Fetch the static pages.** Read product pages, rate filing summaries and risk records with Fetch, or as a Batch job when the list is long.
5. **Compare terms with an assistant.** Point an MCP client at two product pages and ask for the differences in cover.

```javascript Node.js
// npm install playwright
import { chromium } from "playwright";

const browser = await chromium.connectOverCDP(
  "wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY"
);
const page = await browser.newPage();
await page.goto(QUOTE_FORM_URL);

// Example selectors: replace them with the form's own
// Step 1: vehicle, then continue through each page of the form
await page.fill("#registration", PROFILE.vehicle_reg);
await page.click("text=Continue");
await page.fill("#postcode", PROFILE.postcode);
await page.selectOption("#mileage", String(PROFILE.annual_mileage));
await page.click("text=Get quotes");

await page.waitForSelector(".quote-row", { timeout: 120_000 });
const quotes = await page.$$eval(".quote-row", (rows) =>
  rows.map((row) => ({
    insurer: row.querySelector(".insurer")?.textContent.trim(),
    premium: row.querySelector(".premium")?.textContent.trim(),
    excess: row.querySelector(".excess")?.textContent.trim(),
  }))
);
console.log(PROFILE.id, quotes);
await browser.close();
```

```python Python
# pip install requests
import requests

params = {
    "apikey": "YOUR_ZENROWS_API_KEY",
    "url": PRODUCT_PAGE_URL,  # a cover comparison or rate table page
    "mode": "auto",  # Adaptive Stealth Mode
    "outputs": "tables",
}
response = requests.get("https://api.zenrows.com/v1/", params=params)
print(response.json())  # every HTML table on the page as JSON
```

```bash CLI
npx -y @zenrows/cli init
# A policy wording page as Markdown, read from the UK
zenrows fetch POLICY_WORDING_URL --proxy-country gb --output markdown
# The tables on a rate filing summary as JSON
zenrows extract FILING_SUMMARY_URL --outputs tables
```

```text Claude Code (MCP)
# Connect once
claude mcp add zenrows -e ZENROWS_API_KEY=YOUR_ZENROWS_API_KEY -- npx -y @zenrows/mcp

# Then ask:
Use Zenrows to read these two UK home insurance product pages and their product information documents, and list the differences in buildings cover limit, contents cover, accidental damage, excess and exclusions.
```

## Which Zenrows product fits

| If you need to | Use | Why |
|---|---|---|
| Get quotes from a multi-step form | [Browser](/products/browser) | A fresh browser session per test profile, scripted page by page with Playwright or Puppeteer |
| Read product pages, filings and risk records | [Fetch](/products/fetch) | Returns protected pages as HTML, Markdown or tables |
| Refresh thousands of property or product pages | [Batch](/products/batch) | Runs every property, filing and product URL as one job, with results by webhook |
| Compare policy wordings with an assistant | [MCP Server](/mcp) | Claude, Cursor and other MCP clients read the pages through Zenrows |

## What success looks like

| Measure | What good looks like | How to check it |
|---|---|---|
| Profile completion | Every test profile reaches a results page on every run | Divide completed profiles by attempted profiles per run |
| Panel completeness | The number of quotes captured matches what a manual run shows | Spot-check a few profiles by hand each month |
| Run window | The full profile set finishes inside your pricing review window, such as overnight | Log start and end time of each run |
| Rank change detection | A competitor's premium or rank move shows up in the next run | Diff each profile's quotes against the previous run |
| Session time per profile | Time from form start to results stays stable as forms change | Log session duration per profile and investigate jumps after a site update |

## Insurance data FAQ

### What insurance data can I collect from the web?

Quotes and premiums from comparison sites and insurer quote forms, product features and policy terms, filed rate changes, property and vehicle facts for underwriting, replacement prices for claimed items, provider network listings, complaint data and customer reviews. Collection covers the public pages any customer can open in a browser.

### How do insurers monitor competitor prices?

The usual method is a fixed set of test profiles, each a realistic but synthetic risk, and run them through comparison sites on a schedule. Because the profiles stay the same, changes in rank and premium between runs reflect competitors' pricing moves. Zenrows Browser runs those forms in real browser sessions.

### Is it legal to scrape insurance quotes?

Comparison sites and insurers set terms for how their quote tools are used, so read them first. Never enter a real person's details: use synthetic profiles agreed with your compliance team. Where pricing data feeds rating decisions, rules on rating factors and fair pricing may apply. Ask your legal team about your specific use.

### Can Zenrows fill multi-step insurance quote forms?

Yes. Browser gives you a real Chromium session over Playwright or Puppeteer, so your script can type, choose from dropdowns, click through each page and wait for the results panel. Use a fresh session per profile so one profile's answers never carry into the next.

### How do I build test profiles for insurance price comparison?

Pick the rating factors that matter for the line, such as age, postcode area, vehicle, mileage and claims history for motor, and vary one or two at a time across a grid. Keep each profile synthetic and give it a stable ID. Twenty to fifty profiles per market usually cover your core segments, and fixed profiles make every run comparable with the last.

### Where can I find insurance rate filings?

In the US, many states publish property and casualty rate filings through SERFF Filing Access, and some through their own insurance department sites. Filing summaries list the product, rate change and effective date. Fetch reads those pages, and the tables output filter returns their tables as JSON.

### Can web data help with underwriting?

Yes, as an enrichment source. Property facts from assessor sites and listings, flood zones from FEMA maps and vehicle details from marketplaces fill gaps in an application. Keep the source URL and capture date with each fact, so underwriters can see where a value came from.

## Related use cases

- [Healthcare provider data](https://www.zenrows.com/use-cases/healthcare-provider-data.md): Clinician and facility records from payer directories and hospital sites, matched on NPI and re-verified on schedule.
- [Automotive data](https://www.zenrows.com/use-cases/automotive-data.md): Vehicle listings, dealer inventory, prices, specs and parts from marketplaces and dealer sites, keyed on VIN.
- [Real estate web scraping](https://www.zenrows.com/use-cases/real-estate-web-scraping.md): Listings, prices, agents and price history from property portals, kept current at portfolio scale.
- [Competitor price scraping](https://www.zenrows.com/use-cases/price-scraping.md): Competitor prices, promotions, shipping and stock per SKU from retailers and marketplaces, refreshed on your schedule.
- [Public records API](https://www.zenrows.com/use-cases/public-records-api.md): Permits, violations, licenses and registry filings from city, county and state portals, returned as JSON on demand.
- [Competitive intelligence](https://www.zenrows.com/use-cases/competitive-intelligence.md): Pricing, hiring, reviews, launches and partners across your market, structured for battlecards and strategy.

Start free: https://app.zenrows.com/register. Docs: https://docs.zenrows.com