---
title: "Build a Web Research Agent With Google ADK and Zenrows"
description: "Register Zenrows as a Google ADK function tool in Python, or connect the hosted Zenrows MCP server, and turn a Cloudflare 403 into Markdown your Gemini agent can cite."
---
# Build a Web Research Agent With Google ADK and Zenrows

> Register Zenrows as a Google ADK function tool in Python, or connect the hosted Zenrows MCP server, and turn a Cloudflare 403 into Markdown your Gemini agent can cite.

Source: https://www.zenrows.com/blog/web-research-agent-google-adk-zenrows

A Google ADK agent can read a Cloudflare-protected page as clean Markdown when its fetch tool sends the request through Zenrows. In this tutorial's live test, Python's `requests` library got `403 Forbidden` from the target URL, while Zenrows returned the page heading and verification message, and the agent quoted both in its answer.

When a Google ADK agent uses Python's `requests` library to fetch a Cloudflare-protected page, the server may return `403 Forbidden` and a [Just a moment... challenge page](/glossary/web-data-access/what-is-a-javascript-challenge). ADK receives that response as the function's output, leaving the model with challenge markup instead of the content it needs to research.

Google ADK controls when the registered fetch function runs, while the function determines what content reaches the model. A basic HTTP client may be enough for open websites, but agents that research protected or JavaScript-rendered pages need a retrieval tool that can handle those access conditions. Zenrows provides this web access through a scraping API that can adapt the request configuration to the target.

In this tutorial, you'll register Zenrows as a Google ADK function tool and retrieve content from a Cloudflare-protected test page. You'll also send the same URL through Python's `requests` library and compare its `403 Forbidden` response with the content Zenrows returns. Then you'll connect a second agent to the hosted Zenrows MCP server, the route Zenrows' own [Google ADK integration](/integrations/google-adk) uses.

**The short version: Google ADK (Agent Development Kit) is Google's open-source, code-first framework for building AI agents, and it turns a typed Python function into a tool the model can call. Put a Zenrows Fetch request with `mode=auto` and Markdown output inside that function, and the agent reads protected pages instead of challenge pages. To skip the function code, connect the agent to the Zenrows MCP server through ADK's `MCPToolset`.**

## Why Use Zenrows as the Web Tool

A Google ADK function tool can only return what its HTTP client retrieves. If the client receives `403 Forbidden`, the function may still complete and pass the Cloudflare challenge page back to the model. Unless your code checks the response, the agent may treat that blocked page as source material.

For agentic AI workflows, the [web data infrastructure](/glossary/ai-agents-and-the-web/what-is-web-access-for-ai-agents) has to work on the hardest target in the research set. Successful requests to open pages say little about how the same tool will behave against stricter access controls. Zenrows [Fetch](https://docs.zenrows.com/fetch/) handles this with [Adaptive Stealth Mode](https://docs.zenrows.com/fetch/features/adaptive-stealth-mode): each request starts with the lightest configuration likely to work, escalates only when the page needs it, and bills only for the configuration that succeeds.

Your ADK tool sends each target URL through Zenrows using Adaptive Stealth Mode, which selects the request configuration according to the page's access requirements. It requests the page as Markdown, giving the model [readable content](/glossary/web-data-extraction/what-is-llm-ready-content) without adding an HTML-cleaning step. Since the function already contains these request options, the agent only needs to supply the target URL.

[Extract](https://docs.zenrows.com/extract/setup) also supports automatic structured extraction. Adding `extract=auto` returns page data as JSON without requiring CSS selectors or a predefined schema.

Since this build uses Markdown, you'll now wrap the Zenrows Fetch request in a Python function and register it with Google ADK.

## Setting Up Zenrows as an ADK Tool

### Prerequisites

Before you begin, make sure you have:

- Python 3.10 or later.
- A [Google AI Studio API key](https://aistudio.google.com/app/apikey).
- A [Zenrows API key](https://app.zenrows.com/register). The Zenrows free plan currently includes 5,000 credits per month, no credit card required.

You can find the complete code for this tutorial in the [GitHub repository](https://github.com/ZenRows/google-adk-zenrows-research-agent). Clone or download it, then open its root directory in your terminal.

Create a virtual environment and install the project:

```bash
python -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
```

The requirements file pins the versions this tutorial was tested with, including `google-adk` 2.9.2 with its MCP extra, and installs the project itself.

On Windows PowerShell, replace the activation command with:

```powershell
.venv\Scripts\Activate.ps1
```

### Project Structure

After installation, your project structure should look like this:

```text
google-adk-zenrows-research-agent/
├── scripts/
│   ├── compare_fetch.py
│   └── list_mcp_tools.py
├── tests/
│   ├── test_agent.py
│   ├── test_mcp_agent.py
│   └── test_web.py
├── zenrows_mcp_agent/
│   ├── __init__.py
│   └── agent.py
├── zenrows_research_agent/
│   ├── __init__.py
│   ├── agent.py
│   └── web.py
├── .env.example
├── .gitignore
├── LICENSE
├── pyproject.toml
├── README.md
└── requirements.txt
```

`web.py` contains the HTTP and Zenrows retrieval functions that `agent.py` imports to register `fetch_protected_page` in ADK's `tools` list. The comparison script sends the same URL through both retrieval paths, while the tests validate the agent configuration. The `zenrows_mcp_agent` package holds the MCP version of the agent, covered at the end.

### Configure the API Keys

Copy the included environment template:

```bash
cp .env.example .env
```

On Windows PowerShell, run:

```powershell
Copy-Item .env.example .env
```

Open `.env` and replace the placeholder values:

```bash
GOOGLE_API_KEY=your_google_api_key
ZENROWS_API_KEY=your_zenrows_api_key
```

`GOOGLE_API_KEY` is the key created in Google AI Studio. The project loads both values from `.env`, which is excluded from Git through `.gitignore`.

### Define the Zenrows Fetch Function

Open `zenrows_research_agent/web.py` to review the Zenrows fetch function:

```python
"""Web retrieval tools used by the Google ADK agent."""

import os

import requests

ZENROWS_ENDPOINT = "https://api.zenrows.com/v1/"
DEFAULT_TIMEOUT = 180
MAX_TOOL_CONTENT = 12_000


def fetch_with_http(url: str) -> dict:
    """Fetch a public URL with a standard HTTP request for comparison."""
    try:
        # send the request without a protected-site access service.
        response = requests.get(
            url,
            headers={"User-Agent": "google-adk-research-agent/0.1"},
            timeout=20,
        )
    except requests.RequestException as exc:
        return {"status": "error", "url": url, "error": str(exc)}

    return {
        "status": "success" if response.ok else "blocked",
        "url": url,
        "status_code": response.status_code,
        "content": response.text[:2_000],
    }


def fetch_protected_page(url: str) -> dict:
    """Fetch a protected or JavaScript-rendered page through Zenrows.

    Args:
        url: The complete public URL the research agent needs to read.

    Returns:
        A dictionary containing the request status, source URL, and page content.
    """
    api_key = os.getenv("ZENROWS_API_KEY")
    if not api_key:
        return {
            "status": "error",
            "url": url,
            "error": "ZENROWS_API_KEY is not configured.",
        }

    # let Adaptive Stealth Mode select the request configuration.
    params = {
        "apikey": api_key,
        "url": url,
        "mode": "auto",
        "response_type": "markdown",
        "original_status": "true",
    }

    try:
        # give Adaptive Stealth Mode enough time to retrieve the page.
        response = requests.get(
            ZENROWS_ENDPOINT,
            params=params,
            timeout=DEFAULT_TIMEOUT,
        )
        response.raise_for_status()
    except requests.RequestException as exc:
        # remove the API key from the error text before it reaches the model.
        error = str(exc).replace(api_key, "[REDACTED]")
        return {"status": "error", "url": url, "error": error}

    # do not return a successful result without page content.
    if not response.text.strip():
        return {
            "status": "error",
            "url": url,
            "error": "Zenrows returned an empty response.",
        }

    return {
        "status": "success",
        "url": url,
        "content": response.text[:MAX_TOOL_CONTENT],
    }
```

Setting `mode="auto"` enables Adaptive Stealth Mode, which selects the request configuration for the target, while `response_type="markdown"` returns content the model can read without an additional HTML-cleaning step. The 12,000-character limit prevents a large page from taking over the model's context. The 180-second timeout gives the request enough time to complete on a protected target, and the empty-response check prevents the function from reporting success when Zenrows returns no page content.

`original_status="true"` makes Zenrows return the status code the target page sent, so `raise_for_status()` turns a target's 404 or 500 into an error result instead of passing an error page to the model. When a request fails, `requests` puts the full request URL in its error message, and that URL carries your API key, so the function removes the key before the error text becomes the tool result the model reads.

### Register the Function With Google ADK

Open `zenrows_research_agent/agent.py` to see how the function is registered:

```python
"""Google ADK agent configuration."""

from pathlib import Path

from dotenv import load_dotenv
from google.adk.agents import Agent

from .web import fetch_protected_page

# load the API keys stored in the project environment file.
load_dotenv(Path(__file__).resolve().parents[1] / ".env")

# register the Zenrows retrieval function as an ADK tool.
root_agent = Agent(
    model="gemini-3.6-flash",
    name="web_research_agent",
    description="Researches public web pages that ordinary HTTP tools cannot read.",
    instruction=(
        "You are a web research agent. Use fetch_protected_page whenever a task "
        "requires content from a supplied URL. Base the answer only on returned "
        "page content, mention the source URL, and state clearly when retrieval "
        "fails or the page does not contain the requested information."
    ),
    tools=[fetch_protected_page],
)
```

Adding the Python function to `tools` makes ADK wrap it as a `FunctionTool`. ADK reads the function name, type hint, and docstring to generate the tool schema that Gemini receives. The instruction also prevents the agent from inventing an answer when retrieval fails.

Now, run the agent against the Cloudflare test page to verify that ADK selects `fetch_protected_page` and Zenrows returns the requested content.

## Zenrows Returns the Page Where a Plain Request Gets a 403

For the live test, you'll send your agent to `https://www.scrapingcourse.com/cloudflare-challenge`, where a standard HTTP request returns a Cloudflare challenge instead of the requested content. The prompt asks the agent to identify the page title and quote the visible verification message, which also shows whether access succeeded.

When the prompt includes a target URL, Google ADK selects `fetch_protected_page` and passes the URL to it, allowing the function to retrieve the page through Zenrows before the model writes its answer.

### Compare Standard HTTP With Zenrows

The comparison logic lives in `scripts/compare_fetch.py`, where the same target URL is sent through standard HTTP and Zenrows before both results are printed as JSON.

```python
"""Compare standard HTTP access with Zenrows access."""

import json

from dotenv import load_dotenv

from zenrows_research_agent.web import fetch_protected_page, fetch_with_http

TARGET_URL = "https://www.scrapingcourse.com/cloudflare-challenge"


def summarize(result: dict) -> dict:
    """Reduce a fetch result to fields that are safe to print."""
    return {
        "status": result["status"],
        "url": result["url"],
        "status_code": result.get("status_code"),
        "content_preview": result.get("content", "")[:300],
        "error": result.get("error"),
    }


if __name__ == "__main__":
    # load the Zenrows API key from the project environment.
    load_dotenv()

    # retrieve the same URL through both access methods.
    comparison = {
        "standard_http": summarize(fetch_with_http(TARGET_URL)),
        "zenrows": summarize(fetch_protected_page(TARGET_URL)),
    }

    # print the comparison without exposing the API key.
    print(json.dumps(comparison, indent=2))
```

Run the script from the project root to compare both retrieval paths:

```bash
python scripts/compare_fetch.py
```

The live test produced the following output:

```json
{
  "standard_http": {
    "status": "blocked",
    "url": "https://www.scrapingcourse.com/cloudflare-challenge",
    "status_code": 403,
    "content_preview": "<!DOCTYPE html><html lang=\"en-US\"><head><title>Just a moment...</title><meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\"><meta http-equiv=\"X-UA-Compatible\" content=\"IE=Edge\"><meta name=\"robots\" content=\"noindex,nofollow\"><meta name=\"viewport\" content=\"width=device-width,initial-scal",
    "error": null
  },
  "zenrows": {
    "status": "success",
    "url": "https://www.scrapingcourse.com/cloudflare-challenge",
    "status_code": null,
    "content_preview": "[![](https://www.scrapingcourse.com/assets/images/logo.svg) Scraping Course](http://www.scrapingcourse.com/)\n\n# Cloudflare Challenge\n\n![](https://www.scrapingcourse.com/assets/images/challenge.svg)\n\n## You bypassed the Cloudflare challenge! :D",
    "error": null
  }
}
```

![The same Cloudflare-protected URL fetched with plain requests returns a 403 and the Just a moment challenge page, while the Zenrows fetch function returns the page heading and verification message as Markdown](/blog/_img/google-adk-requests-vs-zenrows-markdown.png)

The standard request received HTTP 403 and the `Just a moment...` challenge page, whereas Zenrows returned the page heading and verification message as Markdown. The Zenrows result shows `status_code` as `null` because `fetch_protected_page` returns the status, URL and content only.

### Run the Research Agent

Start the ADK command-line interface from the project root so it can load `root_agent` from the `zenrows_research_agent` package:

```bash
adk run zenrows_research_agent
```

When the CLI displays the `[user]:` prompt, submit the following research task:

```text
Fetch https://www.scrapingcourse.com/cloudflare-challenge and report the page title, whether the challenge was passed, and the visible verification text. Cite the source URL.
```

The verified run returned:

```text
[web_research_agent]: Based on the content retrieved from [https://www.scrapingcourse.com/cloudflare-challenge](https://www.scrapingcourse.com/cloudflare-challenge):

* **Page Title / Main Heading:** Cloudflare Challenge
* **Challenge Status:** Passed / Bypassed
* **Visible Verification Text:** "You bypassed the Cloudflare challenge! :D"
* **Source URL:** `https://www.scrapingcourse.com/cloudflare-challenge`
```

The returned heading and verification text match the content retrieved through Zenrows, confirming that the model answered from the tool output rather than the Cloudflare challenge page. To test the same agent in a browser instead of the terminal, run `adk web` from the project root and pick `zenrows_research_agent` from the agent list.

## Connect Google ADK to the Zenrows MCP Server

Zenrows also has a [Google ADK integration](https://docs.zenrows.com/integrations/google-adk) that needs no function code. ADK supports MCP natively through `MCPToolset`, so an agent can connect to the hosted [Zenrows MCP server](https://docs.zenrows.com/mcp/overview) at `https://mcp.zenrows.com/mcp` and load its tools on startup: `scrape` for page content, `extract` for structured JSON, Crawl and Batch tools for multi-page jobs, and browser tools for clicks and form fills.

The `zenrows_mcp_agent` package in the repository is that version of the research agent:

```python
"""Google ADK agent that uses the hosted Zenrows MCP server."""

import os
from pathlib import Path

from dotenv import load_dotenv
from google.adk.agents import Agent
from google.adk.tools.mcp_tool import MCPToolset, StreamableHTTPConnectionParams

# load the API keys stored in the project environment file.
load_dotenv(Path(__file__).resolve().parents[1] / ".env")

# connect to the hosted Zenrows MCP server with the API key as a bearer token.
zenrows_toolset = MCPToolset(
    connection_params=StreamableHTTPConnectionParams(
        url="https://mcp.zenrows.com/mcp",
        headers={"Authorization": f"Bearer {os.getenv('ZENROWS_API_KEY', '')}"},
    )
)

# the agent chooses between the Zenrows scrape and browser tools on its own.
root_agent = Agent(
    model="gemini-3.6-flash",
    name="web_research_mcp_agent",
    description="Researches public web pages through the Zenrows MCP server.",
    instruction=(
        "You are a web research agent. Use the Zenrows tools whenever a task "
        "requires content from a supplied URL. Base the answer only on returned "
        "page content, mention the source URL, and state clearly when retrieval "
        "fails or the page does not contain the requested information."
    ),
    tools=[zenrows_toolset],
)
```

`StreamableHTTPConnectionParams` points the toolset at the Zenrows endpoint and sends your API key as a bearer token. ADK reads the tool list from the server, so there's no schema to write or keep in sync.

Check which Zenrows tools ADK loads before you start the agent:

```python
"""Check which Zenrows MCP tools Google ADK loads for the MCP agent."""

import asyncio

from zenrows_mcp_agent.agent import zenrows_toolset

RESEARCH_TOOLS = ["scrape", "extract", "crawl_create", "batch_create", "browser_navigate"]


async def main() -> None:
    # ask the hosted Zenrows MCP server which tools it exposes.
    tools = await zenrows_toolset.get_tools()
    await zenrows_toolset.close()
    names = {tool.name for tool in tools}

    # confirm the tools a research agent relies on are loaded.
    print(f"Loaded {len(names)} Zenrows tools")
    for name in RESEARCH_TOOLS:
        print(f"{name}: {'available' if name in names else 'not loaded'}")


if __name__ == "__main__":
    asyncio.run(main())
```

```bash
python scripts/list_mcp_tools.py
```

ADK prints a few experimental-feature warnings first. The tool check itself returned:

```text
Loaded 62 Zenrows tools
scrape: available
extract: available
crawl_create: available
batch_create: available
browser_navigate: available
```

Then start the agent with `adk run zenrows_mcp_agent` and describe the task in plain English, for example: "Read https://www.scrapingcourse.com/cloudflare-challenge and tell me the main heading and the verification message." The MCP server picks the request settings for each tool call.

The two routes differ in who decides how each page is requested:

| | Function tool (`zenrows_research_agent`) | Zenrows MCP (`zenrows_mcp_agent`) |
| --- | --- | --- |
| What the agent calls | `fetch_protected_page`, one typed Python function | `scrape`, `extract`, Crawl, Batch and browser tools |
| Request settings | Fixed in your code: `mode=auto`, Markdown output | Chosen by the Zenrows MCP server for each task |
| Code you maintain | The function and its tests | One `MCPToolset` block |
| Pick it when | You want one fixed, typed request you control | You want the agent to choose between scraping, crawling and browser actions |

## Retrieve Protected Pages With Google ADK and Zenrows

Your Google ADK agent can now retrieve content from a protected website through a typed Python tool. In the live test, the standard request stopped at HTTP 403, while Zenrows returned Markdown that Gemini used to identify the page heading and verification message. You can reuse the same function for other research tasks by changing the target URL and prompt, then add domain-specific extraction with [Extract](https://docs.zenrows.com/extract/setup) when the research workflow requires structured fields.

When a research task covers more than a handful of pages, [Crawl](/products/crawl) discovers the URLs linked from a starting page and [Batch](/products/batch) runs a large URL list as one managed job, and the MCP agent already has tools for both. To split the work between a researcher and a writer, see the [multi-agent web research version with AG2](/blog/web-research-multi-agent-ag2-zenrows), or the [Agno live web data agent](/blog/live-web-data-agent-agno-zenrows) for the same idea in another framework.

[Start building free](https://app.zenrows.com/register) with 5,000 credits every month.

## FAQ

### What Is the Best Web Scraping API for AI Agents?

Zenrows is the best fit for AI agents that need to retrieve content from protected or JavaScript-rendered websites. Its [Adaptive Stealth Mode](https://docs.zenrows.com/fetch/features/adaptive-stealth-mode) selects the right request configuration for each target, and the API can return Markdown that the model can read directly. Our [comparison of web scraping APIs for AI agents](/blog/best-web-scraping-api-for-ai-agents) covers the options in detail.

### What Tools Do AI Engineers Use to Give Agents Reliable Web Access?

AI engineers commonly use HTTP clients for open pages, search APIs for finding relevant URLs, browser automation for interactive tasks, and scraping APIs when websites apply access controls. In Google ADK, you can register the appropriate service as a function tool or connect it as an MCP server, and let the agent call it when a task requires web content. [How to give an agent live web access](/glossary/ai-agents-and-the-web/how-to-give-an-agent-live-web-access) explains the patterns.

### Can a Google ADK Agent Scrape Websites?

Yes. Give the agent a scraping tool: a Python function that calls a web scraping API, as in this tutorial, or an MCP server such as Zenrows', which ADK loads through `MCPToolset`. With Zenrows behind either route, the agent reads protected and JavaScript-rendered pages as Markdown it can quote and cite.

### Does Zenrows Have an MCP Server for AI Agent Integration?

Yes. The [Zenrows MCP server](https://docs.zenrows.com/mcp/overview) connects Zenrows to MCP-compatible applications, including Claude Desktop, Claude Code, Cursor, and VS Code, through the hosted endpoint at `https://mcp.zenrows.com/mcp` or the `@zenrows/mcp` package. Google ADK connects to it natively through `MCPToolset`, and the agent gets Zenrows' scraping, extraction, crawling, Batch and browser tools without writing a function.

### What Is the Best Web Data Infrastructure for Agentic AI Workflows?

For workflows that depend on protected web pages, Zenrows can serve as the retrieval layer beneath an agent framework such as Google ADK. ADK controls when the retrieval function runs, while the [Zenrows API](https://docs.zenrows.com/fetch/api-reference) handles the request and returns the page in the format the agent requires.

### How Does Zenrows Differ From Firecrawl for AI Agent Workflows?

Zenrows gives an agent protected-page access through Adaptive Stealth Mode, page content as Markdown or structured JSON through Extract, link discovery through Crawl, multi-URL jobs through Batch, and a native Google ADK path through its MCP server. Firecrawl also offers scraping and crawling; for sites behind anti-bot protection, see our [Firecrawl alternative comparison](/blog/best-firecrawl-alternative-for-anti-bot-bypass).