---
title: "How to Build a Self-Hosted Web Scraper From Scratch"
description: "A Python scraper with headers, retries, rate limiting and Playwright rendering, tested on three protected pages: Requests got 403s, Playwright got a timeout, a challenge and an empty shell, and Zenrows Fetch returned all three."
---
# How to Build a Self-Hosted Web Scraper From Scratch

> A Python scraper with headers, retries, rate limiting and Playwright rendering, tested on three protected pages: Requests got 403s, Playwright got a timeout, a challenge and an empty shell, and Zenrows Fetch returned all three.

Source: https://www.zenrows.com/blog/self-hosted-web-scraper

A working web scraper takes only a couple of lines of Python code, which is why teams keep weighing whether to [build or buy](/compare/build-vs-buy). This tutorial builds a **Python scraper for static and JavaScript-rendered pages, then tests where that approach stops working**. It stops at protected pages: pointed at Dexscreener, Sofascore and G2, the scraper got a 403 from all three with Requests, and a timeout, a challenge page or an empty shell with Playwright. Zenrows Fetch with `mode=auto` returned all three pages in full.

For this tutorial, you will start with Python's Requests and BeautifulSoup for static pages, then add capabilities to handle pages that render JavaScript. By the end, you will have enough to decide whether building makes sense for your use case.

**The short version: to build a web scraper in Python, send requests with browser-like headers, retry failed requests with a backoff, pace your requests with random delays, parse the HTML with BeautifulSoup, and render JavaScript pages with Playwright. That covers open sites. On the three protected pages tested here, only Zenrows Fetch got the content back.** Every script below ran on October 21, 2026, and the outputs are copied from those runs.

## TL;DR

- Build a working self-hosted Python web scraper with Requests and BeautifulSoup, rotating `User-Agent` values, retry logic, and rate limiting included.
- Extend it with Playwright to handle JavaScript-rendered pages.
- Run it against three protected sites (a Cloudflare challenge, a Varnish block followed by a JavaScript challenge, and DataDome) and a documented per-IP rate limit to see where it holds and where it breaks.
- Requests got a 403 from all three protected pages, and Playwright got a timeout, a challenge page and an empty shell. Zenrows Fetch with `mode=auto` returned all three with a 200 and the full page.
- All code is in the [self-hosted-web-scraper repository on GitHub](https://github.com/ZenRows/self-hosted-web-scraper).

## What You Will Build

First, you will learn how to build a static scraper using Requests and BeautifulSoup with real headers, rate limiting, and retry logic. Then you'll add a Playwright layer to render JavaScript-heavy pages. Once done, you will test the scraper against three protected sites.

### Prerequisites

Before you start building, you need the following:

- Python 3.10 or above.
- Basic familiarity with HTTP requests. You don't need any scraping experience, and the [Python web scraping guide](/blog/web-scraping-python) covers the basics if you want a refresher.
- A terminal and text editor. You don't need any API keys to build the scraper. The last section uses a free Zenrows API key.

Create a virtual environment for your dependencies.

```bash
python3 -m venv .venv
source .venv/bin/activate
```

Then install `requests` for the HTTP calls, `beautifulsoup4` for parsing HTML, `lxml` as the parser backend, `playwright` for the browser rendering, and `brotli` for data decompression.

```bash
python3 -m pip install requests beautifulsoup4 lxml playwright brotli
```

Playwright ships its browser binaries separately, so download Chromium.

```bash
python3 -m playwright install chromium
```

With the dependencies installed, you're ready to build your web scraper.

## Building the Scraper for Static Pages

Many sites will send you HTML markup in response to a request. Thus, the first step is to get the HTML before parsing it. You'll build this in five steps, and test it to see if it can **handle headers, rate limiting, and retries**.

For our test, you'll scrape a demo store at [scrapingcourse.com/ecommerce](https://www.scrapingcourse.com/ecommerce/) and extract information like each product's name, price, and URL.

### 1. Send a Request That Looks Like a Browser

Every **HTTP request carries headers**, which the **server reads before deciding what to send back**. One of the most obvious is the `User-Agent`, which tells the server what client is making the request. You can use Python's `requests` library to fetch the page, but by default, its headers identify the client as a script. Browser-like headers may help with simple filters, but they do not reproduce a full browser or reliably bypass bot detection.

However, this is often not enough because a real Chrome request sends more than just the `User-Agent`. Headers like `Accept`, `Accept-Language`, and `Accept-Encoding` need to also be consistent with the browser you're pretending to be. For many scrapers, you also need to rotate between a list of browsers' `User-Agent` values so all requests don't look the same and to increase the chance of getting information from the site.

Create a Python file and write the script below. Every code block in this tutorial goes into one file, `scraper.py`, and each test runs as its own stage, so you can run one step at a time with a command like `python3 scraper.py static`.

```python
import random
import requests

USER_AGENTS = [
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36",
]


def build_headers() -> dict[str, str]:
    return {
        "User-Agent": random.choice(USER_AGENTS),
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.9",
        "Accept-Encoding": "gzip, deflate, br",
        "Connection": "keep-alive",
    }
```

### 2. Add Retries

Your requests can fail or return a [503](/glossary/http-errors/what-is-a-503-error) due to a connection drop halfway through a request or a proxy timeout. This failure is more expected when you're scraping hundreds of pages. To ensure your scraper is reliable, it needs to recover when this happens, rather than restarting the entire run from scratch. You also **need to factor in delays between retries so they don't happen immediately**, and **set a timeout so an unresponsive server doesn't leave your scraper waiting forever**.

```python
# continuing scraper.py
import time

RETRY_STATUS = {429, 500, 502, 503, 504}


def fetch(session: requests.Session, url: str, max_retries: int = 3) -> str | None:
    for attempt in range(max_retries):
        try:
            response = session.get(url, headers=build_headers(), timeout=15)

            if response.status_code == 200:
                return response.text

            if response.status_code not in RETRY_STATUS:
                print(f"{url} returned {response.status_code}, not retrying")
                return None

            time.sleep(2 ** attempt)

        except requests.RequestException as error:
            print(f"{url} failed with {type(error).__name__}")
            time.sleep(2 ** attempt)

    return None
```

Also, while on the topic of retries, not every failure needs a retry. A 404 means the page doesn't exist. So another attempt wouldn't change the result.

### 3. Rate Limit Your Scraper

When trying to scrape hundreds of pages, how often you send your requests becomes as important as what you are sending. For example, **a burst of requests from a single IP address can trigger site-specific rate limits** or bot detection. The threshold depends on the site and request pattern. The solution here would be to **rate-limit your scraper and add some delays between requests**. These delays also need to be random, as pauses by a real browser aren't fixed or uniform. An actual user opens a page and reads or scans before clicking.

```python
# continuing scraper.py

def polite_delay(base: float = 1.5, jitter: float = 1.0) -> None:
    time.sleep(base + random.uniform(0, jitter))
```

### 4. Parse the HTML

Now that you have an HTML string, you need to turn it into something you can search with [CSS selectors](/glossary/web-data-extraction/what-is-a-css-selector). For this, you'll use BeautifulSoup with the `lxml` parser. `lxml` handles malformed HTML better than BeautifulSoup's built-in parser. To find the selectors, open the store in your browser, right-click a product and choose Inspect: each product is an `li.product` element holding a `.product-name`, a `.product-price` and a product link. The script below does that and puts each product into a dataclass, giving your **scraped data a consistent structure that can be updated when the site's markup changes**.

```python
# continuing scraper.py

from bs4 import BeautifulSoup
from dataclasses import dataclass, asdict


@dataclass
class Product:
    name: str
    price: str
    url: str


def parse_products(html: str) -> list[Product]:
    soup = BeautifulSoup(html, "lxml")
    products = []

    for card in soup.select("li.product"):
        name = card.select_one(".product-name")
        price = card.select_one(".product-price")
        link = card.select_one("a.woocommerce-LoopProduct-link")

        if not (name and price and link):
            continue

        products.append(Product(
            name=name.get_text(strip=True),
            price=price.get_text(strip=True),
            url=link["href"],
        ))

    return products
```

### 5. Run It

The last piece of the puzzle would be putting everything together and **allowing your scraper to run across multiple pages**. You'll use a `requests.Session` to fetch each page, and pass the HTML to `parse_products` to extract the products. Once it is done, `asdict` turns each product dataclass into a dictionary and `run_static()` writes the result to a CSV file.

```python
# continuing scraper.py

import csv

BASE_URL = "https://www.scrapingcourse.com/ecommerce/page/{}/"


def scrape(pages: int = 3) -> list[Product]:
    all_products = []

    with requests.Session() as session:
        for page in range(1, pages + 1):
            html = fetch(session, BASE_URL.format(page))

            if html is None:
                continue

            products = parse_products(html)
            print(f"page {page}: {len(products)} products")
            all_products.extend(products)
            polite_delay()

    return all_products


def run_static() -> None:
    products = scrape()

    with open("products.csv", "w", newline="", encoding="utf-8") as file:
        writer = csv.DictWriter(file, fieldnames=["name", "price", "url"])
        writer.writeheader()
        writer.writerows(asdict(p) for p in products)

    print(f"saved {len(products)} products")
```

Finally, add the main block below at the very bottom of `scraper.py`. It runs the stage you name on the command line, and every function you add later in this tutorial goes above it.

```python
# keep this block at the bottom of scraper.py
import sys

if __name__ == "__main__":
    # run the stage named on the command line, for example: python3 scraper.py static
    stage = sys.argv[1] if len(sys.argv) > 1 else "static"
    globals()[f"run_{stage}"]()
```

Run the static stage. Below is the output. You have **a scraper that retries, paces itself, and writes clean, structured data on sites that serve HTML**. Next, let's explore JavaScript-rendering pages.

```
(.venv) user@Mac web_scraper % python3 scraper.py static
page 1: 16 products
page 2: 16 products
page 3: 16 products
saved 48 products
(.venv) user@Mac web_scraper %
```

## Handling JavaScript-Rendered Pages

**Some sites build themselves in the browser**. If you point the current scraper at one, it will report success but return nothing. Add the function below to your `scraper.py` file, above the main block, to test it against a product grid that renders client-side. You can use the [JavaScript rendering demo page](https://www.scrapingcourse.com/javascript-rendering) on the Scrapingcourse site.

```python
# continuing scraper.py, above the main block

def run_dynamic() -> None:
    url = "https://www.scrapingcourse.com/javascript-rendering"

    with requests.Session() as session:
        html = fetch(session, url)

    if html is None:
        raise RuntimeError(f"Could not fetch {url}")

    soup = BeautifulSoup(html, "lxml")

    for card in soup.select(".product-item")[:3]:
        name = card.select_one("[data-testid=product-name]")
        price = card.select_one("[data-testid=product-price]")
        link = card.select_one("a.product-link")
        print(repr(name.get_text(strip=True)), repr(price.get_text(strip=True)), repr(link["href"]))
```

Run `python3 scraper.py dynamic`, and the script will return the output below.

```
(.venv) user@Mac web_scraper % python3 scraper.py dynamic
'' '' ''
'' '' ''
'' '' ''
(.venv) user@Mac web_scraper %
```

This output is returned because the server sends the page structure without any content. Add the line below to the end of `run_dynamic()`, outside the for loop, to see one of the site's card markups.

```python
print(soup.select(".product-item")[0].prettify()[:400])
```

The cards that are shown in the output below show the `href`, `src`, name, and price are empty because JavaScript fills them after the page loads. Hence, our scraper returns empty values even though the parser finds the element. To see what your scraper sees more clearly, you can use View Source or print the response HTML.

```
<div class="product-item flex flex-col items-center rounded-lg" data-content="product-item" data-testid="product-item" itemscope="" itemtype="http://schema.org/Product">
 <a class="product-link" data-content="product-link" data-testid="product-link" href="">
  <img alt="" class="rounded-lg border shadow-md object-cover" data-content="product-image" data-testid="product-image" decoding="async" fetc
```

The workaround is to use a real browser for such requests. Playwright can launch Chromium, run JavaScript, and return the [DOM](/glossary/web-data-extraction/what-is-the-dom) (Document Object Model) after the content has loaded. It will [wait for specific selectors](/glossary/web-data-access/what-are-wait-strategies) to appear so parsing doesn't happen too early, and you don't keep waiting for an event that has already occurred. Close the browser in a `finally` block after each run so failures do not leave Chromium processes running.

```python
# continuing scraper.py, above the main block

from playwright.sync_api import sync_playwright


def fetch_rendered(url: str, wait_for: str, timeout: int = 30000) -> str | None:
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page(user_agent=random.choice(USER_AGENTS))

        try:
            page.goto(url, timeout=timeout)
            page.wait_for_selector(wait_for, timeout=timeout)
            return page.content()
        except Exception as error:
            print(f"{url} failed to render: {type(error).__name__}")
            return None
        finally:
            browser.close()
```

Next, parse the rendered page using the same selectors, dataclass, and parsing attributes.

```python
# continuing scraper.py, above the main block

def parse_rendered(html: str) -> list[Product]:
    soup = BeautifulSoup(html, "lxml")
    products = []

    for card in soup.select(".product-item"):
        name = card.select_one("[data-testid=product-name]")
        price = card.select_one("[data-testid=product-price]")
        link = card.select_one("a.product-link")

        if not (name and price and link):
            continue

        products.append(Product(
            name=name.get_text(strip=True),
            price=price.get_text(strip=True),
            url=link["href"],
        ))

    return products


def run_rendered() -> None:
    url = "https://www.scrapingcourse.com/javascript-rendering"
    html = fetch_rendered(url, wait_for="[data-testid=product-name]:not(:empty)")

    if html:
        for product in parse_rendered(html)[:3]:
            print(product)
```

Run `python3 scraper.py rendered`. Below is the output. Your scraper can now use Requests for static pages and Playwright for dynamic pages to handle JavaScript-rendered pages.

```
(.venv) user@Mac web_scraper % python3 scraper.py rendered
Product(name='Chaz Kangeroo Hoodie', price='$52', url='https://scrapingcourse.com/ecommerce/product/chaz-kangeroo-hoodie')
Product(name='Teton Pullover Hoodie', price='$70', url='https://scrapingcourse.com/ecommerce/product/teton-pullover-hoodie')
Product(name='Bruno Compete Hoodie', price='$63', url='https://scrapingcourse.com/ecommerce/product/bruno-compete-hoodie')
(.venv) user@Mac web_scraper %
```

## Requests and Playwright Both Get Blocked on Protected Pages

**Some sites are considered hard targets because they're protected by bot-protection challenges like Cloudflare or DataDome**. For this section, you'll test the scraper against three of them: a Cloudflare challenge on Dexscreener, a Sofascore page that blocks at its edge cache and then serves a JavaScript challenge, and a DataDome-protected G2 category page. A fourth test then hits a documented rate limit.

```python
# continuing scraper.py, above the main block

TARGETS = [
    "https://dexscreener.com/solana",
    "https://www.sofascore.com/football",
    "https://www.g2.com/categories/crm",
]


def run_protected() -> None:
    with requests.Session() as session:
        for url in TARGETS:
            try:
                r = session.get(url, headers=build_headers(), timeout=15)
            except requests.RequestException as error:
                print(f"\n{url}\nfailed: {type(error).__name__}")
                continue

            print(f"\n{url}")
            print("status:", r.status_code, "| bytes:", len(r.content))
            print("server:", r.headers.get("server"))
            print("cf-mitigated:", r.headers.get("cf-mitigated"))
            print("x-datadome:", r.headers.get("x-datadome"))
            print("cookies:", list(r.cookies.keys()))
            print("title:", BeautifulSoup(r.text, "lxml").title)
```

Run `python3 scraper.py protected`. The responses from the run on October 21, 2026 are below. They may differ by IP, session, time, and target configuration. The exact dependency versions are in the [repository](https://github.com/ZenRows/self-hosted-web-scraper/blob/main/requirements.txt).

```
(.venv) user@Mac web_scraper % python3 scraper.py protected

https://dexscreener.com/solana
status: 403 | bytes: 5796
server: cloudflare
cf-mitigated: challenge
x-datadome: None
cookies: ['__cf_bm']
title: <title>Just a moment...</title>

https://www.sofascore.com/football
status: 403 | bytes: 48
server: Varnish
cf-mitigated: None
x-datadome: None
cookies: []
title: None

https://www.g2.com/categories/crm
status: 403 | bytes: 1707
server: cloudflare
cf-mitigated: None
x-datadome: protected
cookies: ['datadome', '__cf_bm']
title: <title>g2.com</title>
(.venv) user@Mac web_scraper %
```

While they all returned 403 responses, the meaning differs because each request was blocked differently.

- Dexscreener. Here, a Cloudflare challenge was encountered. You can tell from the `cf-mitigated: challenge` along with the "Just a moment..." title that was returned, which is Cloudflare's [JavaScript challenge](/glossary/web-data-access/what-is-a-javascript-challenge) page. The scraper encounters it, parses it, and returns the challenge page as output.
- Sofascore blocked the request at its edge. The response identifies `server: Varnish` and returns 48 bytes, with no cookies, challenge, or useful HTML. In a real browser, as the Playwright test below shows, the same page serves a JavaScript challenge instead.
- G2 is a DataDome layered protected page. The response `x-datadome: protected`, plus the `datadome` and `__cf_bm` cookies, tell the story. In this case, the response itself is only a placeholder page, so the parser has nothing to extract.

The three above all reject you for who you appear to be. Rate limiting is different, as it only shows up at volume. [GitHub's API documents its limit](https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api?apiVersion=2026-03-10) (60 requests an hour for unauthenticated clients), which makes it a clean test, even though it is an API rather than a web page.

```python
# continuing scraper.py, above the main block

RATE_LIMIT_URL = "https://api.github.com/users/Python"


def run_ratelimit() -> None:
    with requests.Session() as session:
        for i in range(1, 71):
            r = session.get(RATE_LIMIT_URL, headers=build_headers(), timeout=15)
            remaining = r.headers.get("x-ratelimit-remaining")

            if i == 1 or i % 10 == 0 or r.status_code != 200:
                print(f"request {i}: {r.status_code} | remaining: {remaining}")

            if r.status_code != 200:
                print("limit:", r.headers.get("x-ratelimit-limit"))
                print("reset:", r.headers.get("x-ratelimit-reset"))
                break
```

Run `python3 scraper.py ratelimit`. The loop leaves out the delay from Step 3, and adding it back would not change the result: the delay spreads requests out to avoid burst detection, while this limit is a quota of 60 requests an hour for each IP address.

```
(.venv) user@Mac web_scraper % python3 scraper.py ratelimit
request 1: 200 | remaining: 58
request 10: 200 | remaining: 49
request 20: 200 | remaining: 39
request 30: 200 | remaining: 29
request 40: 200 | remaining: 19
request 50: 200 | remaining: 9
request 60: 403 | remaining: 0
limit: 60
reset: 1792572851
(.venv) user@Mac web_scraper %
```

Every request succeeded until one didn't, and because of this, there is no challenge to recognize. You would also notice that the `x-ratelimit-remaining` header was there the whole time, counting down from the first request. The reset value is a Unix timestamp value for when the window reopens. Until then, every request from that IP returns the same 403, no matter how carefully you set your headers.

You can try using Playwright to run a real Chromium browser and solve the JavaScript challenges using the script below:

```python
# continuing scraper.py, above the main block

def run_browser() -> None:
    for url in TARGETS:
        html = fetch_rendered(url, wait_for="body")

        if html is None:
            print(f"\n{url}\nno content returned")
            continue

        soup = BeautifulSoup(html, "lxml")
        print(f"\n{url}")
        print("bytes:", len(html))
        print("title:", soup.title)
```

However, this also didn't work, as you can see from the output below. Dexscreener timed out before the page loaded, Sofascore returned its "Verifying you are human" challenge page, and G2 returned a 2,609-byte empty shell. While the scraper works fine with static HTML and JavaScript-rendered pages, it fails on protected sites because they detect it as automated before it can reach the site's content.

```
(.venv) user@Mac web_scraper % python3 scraper.py browser
https://dexscreener.com/solana failed to render: TimeoutError

https://dexscreener.com/solana
no content returned

https://www.sofascore.com/football
bytes: 5563
title: <title>Sofascore</title>

https://www.g2.com/categories/crm
bytes: 2609
title: <title>g2.com</title>
(.venv) user@Mac web_scraper %
```

Here is the complete `scraper.py` with all six stages:

```python
import random
import requests

USER_AGENTS = [
    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36",
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36",
    "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 "
    "(KHTML, like Gecko) Chrome/140.0.0.0 Safari/537.36",
]


def build_headers() -> dict[str, str]:
    return {
        "User-Agent": random.choice(USER_AGENTS),
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
        "Accept-Language": "en-US,en;q=0.9",
        "Accept-Encoding": "gzip, deflate, br",
        "Connection": "keep-alive",
    }


# continuing scraper.py
import time

RETRY_STATUS = {429, 500, 502, 503, 504}


def fetch(session: requests.Session, url: str, max_retries: int = 3) -> str | None:
    for attempt in range(max_retries):
        try:
            response = session.get(url, headers=build_headers(), timeout=15)

            if response.status_code == 200:
                return response.text

            if response.status_code not in RETRY_STATUS:
                print(f"{url} returned {response.status_code}, not retrying")
                return None

            time.sleep(2 ** attempt)

        except requests.RequestException as error:
            print(f"{url} failed with {type(error).__name__}")
            time.sleep(2 ** attempt)

    return None


# continuing scraper.py

def polite_delay(base: float = 1.5, jitter: float = 1.0) -> None:
    time.sleep(base + random.uniform(0, jitter))


# continuing scraper.py

from bs4 import BeautifulSoup
from dataclasses import dataclass, asdict


@dataclass
class Product:
    name: str
    price: str
    url: str


def parse_products(html: str) -> list[Product]:
    soup = BeautifulSoup(html, "lxml")
    products = []

    for card in soup.select("li.product"):
        name = card.select_one(".product-name")
        price = card.select_one(".product-price")
        link = card.select_one("a.woocommerce-LoopProduct-link")

        if not (name and price and link):
            continue

        products.append(Product(
            name=name.get_text(strip=True),
            price=price.get_text(strip=True),
            url=link["href"],
        ))

    return products


# continuing scraper.py

import csv

BASE_URL = "https://www.scrapingcourse.com/ecommerce/page/{}/"


def scrape(pages: int = 3) -> list[Product]:
    all_products = []

    with requests.Session() as session:
        for page in range(1, pages + 1):
            html = fetch(session, BASE_URL.format(page))

            if html is None:
                continue

            products = parse_products(html)
            print(f"page {page}: {len(products)} products")
            all_products.extend(products)
            polite_delay()

    return all_products


def run_static() -> None:
    products = scrape()

    with open("products.csv", "w", newline="", encoding="utf-8") as file:
        writer = csv.DictWriter(file, fieldnames=["name", "price", "url"])
        writer.writeheader()
        writer.writerows(asdict(p) for p in products)

    print(f"saved {len(products)} products")


# continuing scraper.py, above the main block

def run_dynamic() -> None:
    url = "https://www.scrapingcourse.com/javascript-rendering"

    with requests.Session() as session:
        html = fetch(session, url)

    if html is None:
        raise RuntimeError(f"Could not fetch {url}")

    soup = BeautifulSoup(html, "lxml")

    for card in soup.select(".product-item")[:3]:
        name = card.select_one("[data-testid=product-name]")
        price = card.select_one("[data-testid=product-price]")
        link = card.select_one("a.product-link")
        print(repr(name.get_text(strip=True)), repr(price.get_text(strip=True)), repr(link["href"]))


# continuing scraper.py, above the main block

from playwright.sync_api import sync_playwright


def fetch_rendered(url: str, wait_for: str, timeout: int = 30000) -> str | None:
    with sync_playwright() as p:
        browser = p.chromium.launch(headless=True)
        page = browser.new_page(user_agent=random.choice(USER_AGENTS))

        try:
            page.goto(url, timeout=timeout)
            page.wait_for_selector(wait_for, timeout=timeout)
            return page.content()
        except Exception as error:
            print(f"{url} failed to render: {type(error).__name__}")
            return None
        finally:
            browser.close()


# continuing scraper.py, above the main block

def parse_rendered(html: str) -> list[Product]:
    soup = BeautifulSoup(html, "lxml")
    products = []

    for card in soup.select(".product-item"):
        name = card.select_one("[data-testid=product-name]")
        price = card.select_one("[data-testid=product-price]")
        link = card.select_one("a.product-link")

        if not (name and price and link):
            continue

        products.append(Product(
            name=name.get_text(strip=True),
            price=price.get_text(strip=True),
            url=link["href"],
        ))

    return products


def run_rendered() -> None:
    url = "https://www.scrapingcourse.com/javascript-rendering"
    html = fetch_rendered(url, wait_for="[data-testid=product-name]:not(:empty)")

    if html:
        for product in parse_rendered(html)[:3]:
            print(product)


# continuing scraper.py, above the main block

TARGETS = [
    "https://dexscreener.com/solana",
    "https://www.sofascore.com/football",
    "https://www.g2.com/categories/crm",
]


def run_protected() -> None:
    with requests.Session() as session:
        for url in TARGETS:
            try:
                r = session.get(url, headers=build_headers(), timeout=15)
            except requests.RequestException as error:
                print(f"\n{url}\nfailed: {type(error).__name__}")
                continue

            print(f"\n{url}")
            print("status:", r.status_code, "| bytes:", len(r.content))
            print("server:", r.headers.get("server"))
            print("cf-mitigated:", r.headers.get("cf-mitigated"))
            print("x-datadome:", r.headers.get("x-datadome"))
            print("cookies:", list(r.cookies.keys()))
            print("title:", BeautifulSoup(r.text, "lxml").title)


# continuing scraper.py, above the main block

RATE_LIMIT_URL = "https://api.github.com/users/Python"


def run_ratelimit() -> None:
    with requests.Session() as session:
        for i in range(1, 71):
            r = session.get(RATE_LIMIT_URL, headers=build_headers(), timeout=15)
            remaining = r.headers.get("x-ratelimit-remaining")

            if i == 1 or i % 10 == 0 or r.status_code != 200:
                print(f"request {i}: {r.status_code} | remaining: {remaining}")

            if r.status_code != 200:
                print("limit:", r.headers.get("x-ratelimit-limit"))
                print("reset:", r.headers.get("x-ratelimit-reset"))
                break


# continuing scraper.py, above the main block

def run_browser() -> None:
    for url in TARGETS:
        html = fetch_rendered(url, wait_for="body")

        if html is None:
            print(f"\n{url}\nno content returned")
            continue

        soup = BeautifulSoup(html, "lxml")
        print(f"\n{url}")
        print("bytes:", len(html))
        print("title:", soup.title)


# keep this block at the bottom of scraper.py
import sys

if __name__ == "__main__":
    # run the stage named on the command line, for example: python3 scraper.py static
    stage = sys.argv[1] if len(sys.argv) > 1 else "static"
    globals()[f"run_{stage}"]()
```

## Why Protected Pages Break a Self-Hosted Scraper

Each of these blocks has a workaround, but every workaround needs upkeep, as sites continuously adjust their fingerprinting, bot scoring, and challenge mechanisms. **Stealth plugins can patch the browser automation signals** these protected sites detect, residential proxies can provide reputable IPs, and CAPTCHA-solving services can handle some challenges. [Anti-detect browsers](/glossary/web-data-access/what-is-an-anti-detect-browser) such as CloakBrowser also exist for some of these use cases. Each one is another component to maintain whenever a protection changes.

This is also one reason why scraping at scale is an operational problem. When scraping thousands of pages, you need to manage many factors. For example, the Playwright setup you used earlier works for a few pages. At thousands of pages, this one-browser-per-call pattern may become constrained by resources and throughput. Scaling it requires [bounded concurrency](/glossary/web-data-access/what-is-concurrency-in-web-scraping), browser and context lifecycle management, crash recovery, and capacity testing.

**This workflow makes your operation more reactive since you need to identify what changed, understand why the request was blocked, and update the scraper before scraping again**. At a small scale or low request volume, this is manageable. But as the workload grows and becomes more repetitive, the infrastructure and maintenance around the scraper become more of a hassle.

So, should you build your own scraper or use a web-scraping API? **This ultimately depends on whether you want to own the infrastructure**. Building a scraper gives you control over the parsing logic and HTTP client. However, using a web-scraping API shifts your workflow from reactive to proactive. This makes more sense when scraping isn't a core part of your application.

## Zenrows Fetch Returned All Three Protected Pages

As mentioned earlier, **the requests from the automated section failed because they were detected as automated before reaching the site content**. [Zenrows Fetch](/products/fetch) with [`mode=auto`](/blog/adaptive-stealth-mode) selects the access configuration each request needs. You just need to add the Fetch function.

You will need a Zenrows API key to test out the code, so create one on the [signup page](https://app.zenrows.com/register). **The Zenrows free plan includes 5,000 credits every month with no credit card required**, so you can follow this tutorial without starting a paid subscription. Once you have your key, create a `.env` file in the root directory of your project and add your API key as `ZENROWS_API_KEY`. Then install the packages that load the environment variables in your application.

```bash
pip3 install requests python-dotenv
```

Also add `.env` to your `.gitignore` file so your credentials aren't accidentally committed to version control. Then, create a new Python file `zenrows_fetch.py`. Copy and run the script below. **You will test Zenrows against the three protected sites you used earlier**.

```python
import os

import requests
from bs4 import BeautifulSoup
from dotenv import load_dotenv

load_dotenv()

TARGETS = [
    "https://dexscreener.com/solana",
    "https://www.sofascore.com/football",
    "https://www.g2.com/categories/crm",
]


def fetch_zenrows(url: str) -> str | None:
    response = requests.get(
        "https://api.zenrows.com/v1/",
        params={
            "apikey": os.getenv("ZENROWS_API_KEY"),
            "url": url,
            "mode": "auto",
        },
        timeout=180,
    )

    if response.status_code == 200:
        return response.text

    print(f"{url} returned {response.status_code}: {response.text[:200]}")
    return None


for url in TARGETS:
    html = fetch_zenrows(url)

    if html:
        soup = BeautifulSoup(html, "lxml")
        print(f"\n{url}")
        print("bytes:", len(html))
        print("title:", soup.title)
```

The output below is from the run on October 21, 2026. **All three requests returned 200 with the full page**.

```
(.venv) user@Mac web_scraper % python3 zenrows_fetch.py

https://dexscreener.com/solana
bytes: 2132120
title: <title data-rh="true">Solana DEX Screener</title>

https://www.sofascore.com/football
bytes: 183264
title: <title data-next-head="">Football Live Scores, Fixtures &amp; Odds | Sofascore | Championship League, Premier League</title>

https://www.g2.com/categories/crm
bytes: 827211
title: <title>Best CRM Software: User Reviews from October 2026</title>
(.venv) user@Mac web_scraper %
```

Here is how the three approaches compare on the same three pages:

| Target | Protection | Requests | Playwright | Zenrows Fetch (`mode=auto`) |
| --- | --- | --- | --- | --- |
| Dexscreener | Cloudflare challenge | 403, "Just a moment..." page | Timed out | 200, full page (2,132,120 bytes) |
| Sofascore | Varnish edge block, then a JavaScript challenge | 403, 48 bytes | "Verifying you are human" page | 200, full page (183,264 bytes) |
| G2 | DataDome | 403, placeholder page | 2,609-byte empty shell | 200, full page (827,211 bytes) |

![Requests and Playwright get 403s, challenge pages and empty shells on Dexscreener, Sofascore and G2, while Zenrows Fetch with mode=auto returns all three full pages](/blog/_img/self-hosted-web-scraper-results.png)

The self-hosted scraper required you to identify what each page needed and implement a fix. `mode=auto` automatically makes the decision for you per target by **selecting the access configuration each target needs.** It starts with the lowest-cost configuration that works and escalates to a heavier configuration only when a page needs one. You're billed once, for the configuration that succeeds. It also handles IP-based restrictions like the per-IP quota from the GitHub test, without a separate step. **You can also retrieve data with Zenrows Fetch on the free tier, which includes 5,000 credits a month**, and test against your own target. You can find more information in [Zenrows' official documentation](https://docs.zenrows.com/fetch/features/adaptive-stealth-mode).

The same applies at scale. Instead of managing browser lifecycles and concurrency yourself, you can send your whole URL list to [Batch](/products/batch) and collect the results when the job finishes, while Batch handles the queue, retries and concurrency. To see a job run from start to finish, read the [Batch announcement](/blog/introducing-batch). The breakdown of [web scraping costs](/blog/web-scraping-cost) compares what each approach costs to operate. To try a target from your terminal before writing any code, use the [Zenrows CLI](/cli).

In our [benchmark of web scraping APIs](/blog/best-web-scraping-apis-benchmarked), Zenrows had the best overall success rate across four protected targets. However, if your use case is more about real-time data, then this article on [web scraping for real-time data](/blog/best-web-scraping-api-for-real-time-data) is a great place to start.

## Self-Host for Open Pages, Use Zenrows for Protected Ones

In this tutorial, you built a scraper that handles static HTML and JavaScript-rendered pages, and then saw why protected targets change the operating model. Keep self-hosting when target access is open and bounded. **Use a managed layer when protected access and ongoing maintenance are part of the workload you do not want to own**.

[Start building free](https://app.zenrows.com/register) if you intend to scrape data without managing the workload for your workflow.

## FAQ

### How Do You Build a Web Scraper From Scratch?

Fetch the page with an HTTP client such as Python's Requests and browser-like headers, retry failed requests with a backoff, add random delays between requests, and parse the HTML with BeautifulSoup and CSS selectors. Pages that render their content with JavaScript need a browser such as Playwright. For protected pages, send the request through Zenrows Fetch with `mode=auto`, which returned all three protected targets in this tutorial.

### What API Should I Use When a Website Blocks My Scraper?

Start by identifying whether the response is a rate limit, challenge page, or empty shell, then check the site's terms and access policy. For a one-off open target, adjust your client and pacing; the fixes for a [BeautifulSoup 403 error](/blog/beautifulsoup-403) and a [Playwright 403 error](/blog/playwright-403) cover the common cases. For recurring protected access, use a managed API such as Zenrows Fetch, which got through all three protected pages in this tutorial.

### Web Scraping API vs. Building Your Own Scraper: What Makes More Sense?

[Build your own scraper](/glossary/web-data-access/scraping-api-vs-building-your-own) when targets are open, volume is bounded, and parser control matters. Choose an API when protected access, JavaScript rendering, proxy management, retries, and ongoing maintenance would distract from your product development. With Zenrows, you get protected web access without that maintenance, and you're billed only for successful requests.

### Did Zenrows Get Through All Three Protected Sites?

Yes. In this test, Zenrows Fetch returned all three targets with a 200 and the full page, whereas Requests got three 403s and the local Playwright run timed out or returned a challenge page and an empty shell. That is one run of three targets. Across production traffic, Zenrows' published success rate is 99.93 percent.

### How Do You Scrape a Website That Uses Bot Detection Without Writing Custom Bypass Logic?

A managed access service can maintain the retrieval layer, so your parser doesn't need target-specific bypass code. In this tutorial, `mode=auto` selected the access configuration each request needed, and the parsing code stayed the same. Follow each site's access rules before using it in production.

### What Is the Best Web Scraping API for AI Agents?

Choose based on the agent's required output and interaction model. Fetch is a fit when the agent needs current page content from open or protected targets, [Extract](/products/extract) returns structured JSON, and [Browser](/products/browser) handles pages that need clicks and typing. The [Zenrows MCP server](/mcp) gives an agent all of it through plain-English prompts. The guide to the [best web scraping API for AI agents](/blog/best-web-scraping-api-for-ai-agents) compares the options for agents, LLMs, and RAG pipelines.