---
title: "Web scraping for lead generation"
description: "Web scraping for lead generation: build B2B prospect lists from Clutch, Google Maps, Yelp, BBB, trade show exhibitor lists and association directories."
canonical: https://www.zenrows.com/use-cases/web-scraping-for-lead-generation
updated: 2026-10-18
---

# Web scraping for lead generation

Web scraping for lead generation builds prospect lists by collecting companies from public directories, maps, marketplaces and member lists that match your ideal customer profile. Zenrows crawls directory categories, fetches every profile page through protected access, and returns structured fields you can filter, deduplicate and load into your CRM or outbound tool.

## Sample output: prospects from one directory

```json
{
  "source": "https://www.example-directory.com/agencies/ecommerce-development/germany",
  "collected_at": "2026-10-17",
  "results_on_page": 3,
  "prospects": [
    {
      "company": "Northwind Commerce GmbH",
      "website": "https://northwind-commerce.example",
      "city": "Berlin",
      "country": "DE",
      "employees_band": "50-249",
      "min_project_size": "10,000+",
      "services": ["Shopify development", "Ecommerce strategy"],
      "rating": 4.9,
      "review_count": 31,
      "profile_url": "https://www.example-directory.com/profile/northwind-commerce"
    },
    {
      "company": "Blue Harbor Digital",
      "website": "https://blueharbor.example",
      "city": "Hamburg",
      "country": "DE",
      "employees_band": "10-49",
      "min_project_size": "5,000+",
      "services": ["Magento development"],
      "rating": 4.7,
      "review_count": 12,
      "profile_url": "https://www.example-directory.com/profile/blue-harbor-digital"
    }
  ]
}
```

Illustrative rows from a B2B agency directory category page; each prospect keeps its profile URL so you can re-check it or enrich it later.

## Where lead generation scraping finds prospects

Every lead list starts with a page that already groups the companies you want: a directory category, a map search, an exhibitor list. The job is to read every result, keep the ones that fit your ideal customer profile, and capture enough to route them.

| Job | Data you collect | Typical sources |
|---|---|---|
| Agency and service-provider lists | Company, website, services, team size, minimum project size, location | Clutch, DesignRush, GoodFirms, partner directories of software vendors |
| Local business prospecting | Name, category, address, phone, website, rating | Google Maps, Yelp, Yellow Pages, BBB |
| Software buyer lists | Products in a category, vendor company, review count | G2 and Capterra category pages, app marketplaces |
| Event and association lists | Exhibitors, sponsors, members, booth or member category | Trade show exhibitor pages, industry association member directories |
| Hiring-intent lists | Companies posting roles that signal a need for your product | Indeed, company careers pages, Greenhouse and Lever boards |
| Supplier and manufacturer lists | Company, product lines, country, certifications | IndiaMART, Thomasnet, Europages, chamber of commerce directories |

## Why lead scraping stalls

Directories make money from their listings, so many of them rate-limit, paginate aggressively or protect result pages. Lists also go stale fast, and duplicates pile up when the same company appears in five directories.

| What breaks | What it looks like | How Zenrows handles it |
|---|---|---|
| Protected directories | A directory or review site returns a challenge page after a few dozen requests | Zenrows' stealth gets through [DataDome](/bypass/datadome), Cloudflare and similar protections |
| Deep pagination | A category has 120 result pages and your script stops at page 10 | Crawl follows pagination links and keeps only URLs that match your patterns |
| Results rendered in the browser | Map and marketplace results appear only after JavaScript runs | Adaptive Stealth Mode escalates to browser rendering when a page needs it and returns the results |
| Listing data split across pages | The category page has the name; the website and team size are on the profile | Batch fetches every profile URL as one job and returns the fields per profile |
| Regional results | A directory shows different companies depending on where you browse from | Geotargeting returns the results a local visitor sees |

## How to scrape leads with Zenrows

1. **Pick the list pages.** Find directory categories, map searches or exhibitor lists that already filter by your ideal customer: industry, region, size.
2. **Collect every profile URL.** Run Crawl on the category with a URL pattern to keep for profile pages, so pagination is followed for you.
3. **Read the fields.** Fetch a results page and name the columns with CSS selectors, or send profile URLs to Batch and let Extract return structured JSON.
4. **Filter and deduplicate.** Drop companies outside your profile and merge duplicates on the website domain before you load the list.
5. **Hand off to enrichment.** Pass the domains to your [lead enrichment](/use-cases/lead-enrichment) step and then to your CRM or sequencer.

```bash cURL
curl -X POST "https://api.zenrows.com/v1/crawls" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "DIRECTORY_CATEGORY_URL",
    "max_items": 2000,
    "include_patterns": ["/profile/"],
    "exclude_patterns": ["/reviews/", "/compare/"]
  }'
# Returns a crawl_id. Read the profile URLs with:
# GET https://api.zenrows.com/v1/crawls/{crawl_id}
```

```python Python
# pip install requests
import json
import requests

fields = {
    "company": ".provider-name",
    "website": "a.website-link @href",
    "location": ".provider-location",
    "rating": ".rating-value",
}
response = requests.get("https://api.zenrows.com/v1/", params={
    "apikey": "YOUR_ZENROWS_API_KEY",
    "url": "DIRECTORY_CATEGORY_URL",
    "mode": "auto",
    "css_extractor": json.dumps(fields),
})
print(response.json())  # selectors are examples; match them to the directory's markup
```

```bash CLI
npx -y @zenrows/cli init
zenrows crawl create DIRECTORY_CATEGORY_URL --include-patterns /profile/ --max-pages 2000 --follow
zenrows crawl results CRAWL_ID --out profiles.jsonl
# profiles.jsonl: one {"url": "PROFILE_URL"} per line
zenrows batch create profiles.jsonl --wait
```

```text Claude Code (MCP)
# Connect once
claude mcp add zenrows -e ZENROWS_API_KEY=YOUR_ZENROWS_API_KEY -- npx -y @zenrows/mcp

# Then ask:
Use Zenrows to read the Clutch category page for Magento developers in the UK and return every agency on it as JSON with company name, website, location, team size, minimum project size, rating and profile URL.
```

## Which Zenrows product fits

| If you need to | Use | Why |
|---|---|---|
| Collect every company URL in a directory category | [Crawl](/products/crawl) | Follows pagination and keeps only the profile URLs that match your pattern |
| Pull named columns from a results page | [Fetch](/products/fetch) | Name the fields with CSS selectors and get them as JSON from one request |
| Read profile pages that all look different | [Extract](/products/extract) | Automatic structured extraction returns the data without selectors per directory |
| Fetch 20,000 profiles for a new territory | [Batch](/products/batch) | One job for the whole list, results by webhook or download |
| Run list builds from the terminal or a cron job | [CLI](/cli) | `batch create` takes a JSONL file and waits for the job to finish |

## What success looks like

| Measure | What good looks like | How to check it |
|---|---|---|
| Category coverage | Your list holds every company the directory shows for the category | Compare your count with the result count printed on the category page |
| ICP match rate | Most rows meet your size, industry and region rules before any manual review | Sample 50 rows a run and score them against the profile |
| Duplicate rate | Each company appears once across all sources | Count rows per normalized website domain |
| Website presence | Nearly every row has a working company domain for enrichment | Share of rows with a domain that returns a page |
| Time to a new list | A new vertical or region goes from idea to CRM in a day | Track time from first crawl to import |

## Web scraping for lead generation FAQ

### Is web scraping for lead generation legal?

Collecting public business listings, such as company names, categories, addresses and websites from directories, is common practice. Two things need care: the site's terms of use, and personal data. Names, personal emails and direct phone numbers of individuals fall under laws such as GDPR and CCPA, and outreach is also covered by e-marketing rules in many countries. Ask your legal team before collecting data about people.

### What is a lead scraper?

A lead scraper is a tool or script that reads directory, map or marketplace pages and turns the listed businesses into rows: company, website, location, category and similar fields. Some are point-and-click apps for one site. Zenrows is a lead scraper API: you choose the sources and fields, and it returns the pages from protected sites. For a comparison of tools, see our lead scraper roundup .

### Which websites are best for B2B lead scraping?

The best source is the one that already filters by your customer. Agencies are on Clutch and DesignRush, local service businesses on Google Maps, Yelp and BBB, software vendors on G2 and Capterra category pages, and manufacturers on IndiaMART or Thomasnet. Trade show exhibitor lists and association member directories are often the most targeted lists in a niche.

### Can I scrape Google Maps for leads?

Yes. A Google Maps search for a category and city returns businesses with name, address, phone, website, rating and review count. Render the page in a full browser and scroll the results with page actions before it is returned, or open each place page. For a deeper look at local listings, see local business data .

### How often should I refresh a scraped lead list?

Companies move, rebrand, close and change size, so a list loses accuracy every month it sits unused. Re-crawl the source categories before each campaign, or monthly for lists in active use, and compare against the last run by website domain. New domains are fresh leads; missing ones are candidates to suppress. Keep the date each row was last seen, so sales can tell a current listing from an old one.

### How do I avoid duplicate leads across directories?

Use the company website domain as the key. Normalize it (lowercase, strip www. and trailing paths) and merge rows that share it, keeping the source URL of each. Companies without a website are the hard case: match them on name plus city, then confirm by hand.

### Can I get emails from scraped leads?

Company pages often list a general inbox such as info@ or sales@ , and Fetch's emails output filter returns the addresses a page shows. Treat personal work emails as personal data: check GDPR, CCPA and your local e-marketing rules with your legal team before you collect or email them.

## Related use cases

- [Lead enrichment](https://www.zenrows.com/use-cases/lead-enrichment.md): A domain in, an account record out: pricing, hiring, product and review signals from the company's own web.
- [Firmographic data](https://www.zenrows.com/use-cases/firmographic-data.md): Industry, headcount, HQ, locations and funding per company, built from registries, company sites and directories.
- [Job scraping](https://www.zenrows.com/use-cases/job-scraping.md): Job postings, salary ranges and hiring signals from job boards and ATS career pages, as one feed.
- [Review scraping](https://www.zenrows.com/use-cases/review-scraping.md): Review text, ratings, dates and replies from Amazon, Google, Yelp, Trustpilot and G2, every page.
- [Market research web scraping](https://www.zenrows.com/use-cases/market-research-web-scraping.md): Assortments, prices, reviews and forum talk across a whole category, for sizing studies and recurring trackers.

Start free: https://app.zenrows.com/register. Docs: https://docs.zenrows.com