# Build a web-aware agent with OpenAI Agents SDK and Zenrows

> Register Zenrows Fetch as a @function_tool next to OpenAI's built-in WebSearchTool, and let one agent pick discovery or full-page retrieval per turn.

Source: https://www.zenrows.com/blog/web-aware-agent-openai-agents-sdk-zenrows

Large language models reason well. They struggle with questions about information that changes. The OpenAI Agents SDK ships a built-in web search tool that is useful for discovery, but it cannot reach every page, especially protected sites and pages that only fill in after JavaScript runs.

The fix is not to replace the built-in search. It is to give the same agent a second way to reach the web, one built for pages the index never captures. This tutorial defines a `@function_tool` that wraps [Zenrows Fetch](/products/fetch) and registers it alongside `WebSearchTool` in one agent. By the end you have an agent that runs both tools in a single loop and decides which one each turn needs.

All the code is on [GitHub](https://github.com/ZenRows/web-aware-agent-openai-agents-sdk-zenrows).

## Prerequisites

You need two API keys and one SDK.

- Python 3.9 or later.
- An OpenAI API key, from your [OpenAI developer dashboard](https://platform.openai.com/api-keys).
- A Zenrows API key, from your [Zenrows dashboard](https://app.zenrows.com/register).

Install the Agents SDK and `python-dotenv`:

```bash
python3 -m pip install openai-agents python-dotenv
```

Create a `.env` file in your project root with both keys, and add `.env` to `.gitignore` so the credentials never reach version control:

```bash
OPENAI_API_KEY=your_openai_api_key_here
ZENROWS_API_KEY=your_zenrows_api_key_here
```

Then load them in your script:

```python
from dotenv import load_dotenv

load_dotenv()
```

## Set up a baseline agent with the built-in search

`WebSearchTool` is a hosted tool. It runs on OpenAI's infrastructure through the Responses API, so enabling it takes one line and no configuration.

```python
import asyncio
from dotenv import load_dotenv
from agents import Agent, Runner, WebSearchTool

# load your api key from the .env file
load_dotenv()

# the agent starts with only the built-in web search enabled
agent = Agent(
    name="Product Researcher",
    instructions=(
        "You research products and web pages. "
        "Report exactly what you find, and state plainly when "
        "specific information is missing from your results."
    ),
    tools=[WebSearchTool()],
)


async def main():
    # a broad, recency-dependent query
    result = await Runner.run(
        agent,
        "Which laptops did Apple release most recently, and where can I buy them?",
    )
    print(result.final_output)


asyncio.run(main())
```

On a broad question this works. The agent finds relevant pages, pulls indexed content, and answers with source links:

```text
# truncated output
$ python3 agent_one.py
Apple's most recently released laptops, launched in March 2026, include three
distinct models:

- MacBook Neo: a new 13-inch MacBook powered by an A18 Pro iPhone-class chip,
  starting at US $599 (US $499 with education pricing) (en.wikipedia.org).
- MacBook Air with M5 chip: 13-inch from US $1,099, 15-inch from US $1,299
  (apple.com).
- MacBook Pro with M5 Pro or M5 Max chips: 14-inch from about US $1,999,
  16-inch from US $2,999 at Apple's store (en.wikipedia.org).

Where to buy (United States): Apple's online store and retail stores, plus
Amazon, Best Buy, B&H Photo, Walmart and Apple Premium Resellers
(macworld.com, macrumors.com).
```

## Where the built-in web search stops

The [OpenAI Agents SDK](https://developers.openai.com/api/docs/guides/agents) built-in search supports reasoning, tool use, and task completion over indexed public content, with no proxy setup on your side. Full-page retrieval runs into two problems.

### Issue 1: it returns snippets from an index, not the live page

When a site permits the crawler, built-in search returns a snippet of OpenAI's indexed copy. For documentation that is usually fine. For anything where freshness matters, an exchange rate or a stock level, the indexed copy lags the live page. The page can change faster than the index refreshes.

### Issue 2: content that only exists after JavaScript runs

The initial HTML of many sites is a shell. Prices, inventory, and reviews arrive only once the page's JavaScript executes in a browser. A search index typically stores that initial HTML, so the fields you want were never in the indexed copy. Crawling more often does not help.

Product pages, dashboards, and single-page apps fall into this category. It is why asking the built-in search for a live price on a JavaScript-heavy retailer returns a confident answer with the number missing:

```python
import asyncio
from dotenv import load_dotenv
from agents import Agent, Runner, WebSearchTool

load_dotenv()

agent = Agent(
    name="Product Researcher",
    instructions=(
        "You retrieve product details from web pages. "
        "Report exactly what you find, and say so plainly if content is missing."
    ),
    tools=[WebSearchTool()],
)


async def main():
    # a live price and stock check
    result = await Runner.run(
        agent,
        "What is the current price and stock status of "
        "https://www.amazon.com/dp/B0GR1JKMBV/ref=fs_a_mbt2_us1?th=1",
    )
    print(result.final_output)


asyncio.run(main())
```

The script runs without errors. It returns the product title and a clear statement of what it could not get:

```text
# truncated output
$ python3 failure_two.py
I visited the Amazon product page for ASIN B0GR1JKMBV, titled "Apple 2026
MacBook Pro Laptop with Apple M5 Pro chip with 18-core CPU and 20-core GPU...".

- The page displays a prompt: "To see product details, add this item to your
  cart." That message appears several times in place of visible pricing or
  availability information (amazon.com).
- As a result, the current price and stock status are not visible in the
  portion of the page that I can access.

Therefore I could not retrieve the current price or availability.
```

In both cases the agent finds the page and still returns stale or incomplete content. What is missing is a retrieval layer built for protected pages. That is what [Zenrows](/) does: reliable access to anti-bot-defended and JavaScript-rendered targets, at scale.

## Add Zenrows Fetch as a @function_tool

You need one Python function that calls [Zenrows Fetch](https://docs.zenrows.com/fetch/api-reference) and a decorator that exposes it to the agent.

```python
import os
import requests
from agents import function_tool


@function_tool
def fetch_page_content(url: str) -> str:
    """Fetch the full, current content of a specific web page as Markdown.

    Use this when you need the complete content of a known URL rather than
    a search result summary, such as live prices, stock levels, or data
    that loads via JavaScript after the page opens.

    Args:
        url: The full URL of the page to retrieve.
    """
    response = requests.get(
        "https://api.zenrows.com/v1/",
        params={
            "url": url,
            "apikey": os.getenv("ZENROWS_API_KEY"),
            # mode=auto lets Zenrows pick the settings each site needs
            "mode": "auto",
            "response_type": "markdown",
        },
        timeout=90,
    )

    # return the error as a string so the agent can react instead of crashing
    if response.status_code != 200:
        return f"Zenrows returned {response.status_code} for {url}"

    return response.text
```

Four parts make this work:

- **The function.** `fetch_page_content()` takes a URL and returns the page contents. To the agent it is one more tool it can call when it needs something the search did not give it.
- **The request.** Fetch is a single endpoint. `mode=auto` is [Adaptive Stealth Mode](https://docs.zenrows.com/fetch/features/adaptive-stealth-mode): it turns on JavaScript rendering and escalates to premium proxies per request, only when the target needs them, so you are not paying a premium proxy on pages that do not. `response_type=markdown` strips the markup so the model reads clean text.
- **The decorator.** `@function_tool` builds the tool schema from your function signature, so the model knows what the tool accepts and returns.
- **The docstring.** The model reads it every turn to decide whether the tool fits the task. The phrase "rather than a search result summary" is what stops the agent defaulting to built-in search for retrieval work.

## Run both tools in a single agent loop

Put both tools in the same `tools` list. The model then selects between them each turn from their descriptions. To force both into one run, the prompt below gives no URL, so the agent has to discover the page before it can read it.

```python
import asyncio
import os
import requests
from dotenv import load_dotenv
from agents import Agent, Runner, WebSearchTool, function_tool

load_dotenv()


@function_tool
def fetch_page_content(url: str) -> str:
    """Fetch the full, current content of a specific web page as Markdown.

    Use this when you need the complete content of a known URL rather than
    a search result summary, such as live prices, stock levels, or data
    that loads via JavaScript after the page opens.

    Args:
        url: The full URL of the page to retrieve.
    """
    response = requests.get(
        "https://api.zenrows.com/v1/",
        params={
            "url": url,
            "apikey": os.getenv("ZENROWS_API_KEY"),
            "mode": "auto",
            "response_type": "markdown",
        },
        timeout=90,
    )

    if response.status_code != 200:
        return f"Zenrows returned {response.status_code} for {url}"

    return response.text


# the model routes between both tools from their descriptions alone
agent = Agent(
    name="Web-Aware Researcher",
    instructions=(
        "You research products on the web. "
        "Use web search to find the product page URL. "
        "Then call fetch_page_content on that URL and report the price "
        "and stock status from the fetched page content. "
        "State which tool each figure came from."
    ),
    tools=[WebSearchTool(), fetch_page_content],
)


def print_trace(items):
    for item in items:
        raw = getattr(item, "raw_item", None)
        name = getattr(raw, "name", None) or getattr(raw, "type", "")
        print(f"[{item.type}] {name}")

        if item.type == "tool_call_item" and name == "fetch_page_content":
            print("  args:", getattr(raw, "arguments", ""))

        if item.type == "tool_call_output_item":
            out = str(getattr(item, "output", ""))
            print(f"  output: {len(out)} chars")
            print(f"  body: {out[:500]}")


async def main():
    result = await Runner.run(
        agent,
        "Find the PriceOye page for the iPhone 17 Pro "
        "and report its current price and stock status.",
    )

    print_trace(result.new_items)
    print("\n---\n")
    print(result.final_output)


asyncio.run(main())
```

The trace shows the decision happening in two steps. Turn one is a `web_search_call` that names a product but returns no URL, which is the discovery step. Turn two calls `fetch_page_content` with the URL it found, matching that tool's docstring almost word for word.

```text
# truncated output
$ python3 agent_both_tool.py
[tool_call_item] web_search_call
[message_output_item] message
[tool_call_item] fetch_page_content
  args: {"url":"https://priceoye.pk/mobiles/apple/apple-iphone-17-pro"}
[tool_call_output_item]

---

Here are the details for the iPhone 17 Pro from its PriceOye product page:

Current Price: Rs 471,999
Stock Status: Only 1 left in stock

Price and stock status are directly extracted from the fetched page content
(functions.fetch_page_content tool).
```

The final output carries the current price and a stock status of "Only 1 left," both attributed to `fetch_page_content`. The routing behind the whole run lives in the tool descriptions. There are no conditionals and no orchestration code of your own. The agent discovers the URL through search, then reads the live page through Zenrows.

![Chart showing how the web-aware agent routes each turn between WebSearchTool for discovery and the Zenrows fetch_page_content tool for full-page retrieval](/blog/_img/openai-agents-sdk-websearch-vs-zenrows-fetch.png)

For interactive lookups, one fetch per turn is the right shape. For recurring or high-volume retrieval, such as refreshing prices across a full catalog, use [Zenrows Batch](/products/batch) instead. It runs many jobs without you managing concurrency, and the [Batch documentation](https://docs.zenrows.com/batch/introduction) covers submitting your first one.

## When to use each tool

Use `WebSearchTool` when the agent needs to discover something: open-ended research, finding relevant sites and documentation, answering questions from indexed public content, or identifying URLs worth following up.

Use Zenrows as the retrieval layer. It gives the agent dependable access to pages the built-in search cannot reach, including protected targets and JavaScript-rendered sites, and returns content the model can read directly. If you want specific fields rather than a whole page, [Zenrows Extract](https://docs.zenrows.com/extract/introduction) returns structured data, so the model never has to parse Markdown to find a price.

The point is not to pick one. Let the built-in search handle discovery, and call Zenrows whenever the agent needs the full, live contents of a known URL. The same routing idea carries over to multi-agent setups, which is what the guide on [building a web research multi-agent system with AG2 and Zenrows](/blog/web-research-multi-agent-ag2-zenrows) walks through. The wrapper carries over too: [smolagents](/blog/zenrows-smolagents) takes the same function under a `@tool` decorator.

## Conclusion

Your agent now reaches the web two ways and knows when to use each. `WebSearchTool` finds the URL. Zenrows reads the full, live page once it has one, including JavaScript-rendered content and anti-bot-defended targets the index never captures. The model routes between them from the tool descriptions alone, so the agent works from what the page shows now rather than from a stale snippet.

When you need many pages on a schedule rather than one per turn, [Zenrows Batch](/products/batch) runs those jobs without you managing concurrency, so a monitoring or research agent can refresh a whole catalog in one pass.

## FAQ and debugging

### Does Zenrows replace the built-in web search?

No. Both tools run in the same agent and cover different jobs. The built-in search handles discovery and answers questions; Zenrows retrieves full-page content from specific URLs.

### The agent always uses the built-in search and never calls my Zenrows tool

That is the docstring. The model reads it every turn to decide whether the tool applies, so name the conditions explicitly. To confirm which tool it actually called, inspect `result.new_items`, as the `print_trace` helper above does.

### Can I use the Zenrows MCP server instead of the @function_tool approach?

Yes. The [Zenrows MCP server](https://docs.zenrows.com/mcp/overview) connects through the OpenAI Responses API using the MCP tool type, which gives you ready-made tools without writing a wrapper. The Agents SDK also has native MCP support through `HostedMCPTool`, covered in the [OpenAI Agents SDK integration guide](https://docs.zenrows.com/integrations/openai-agents-sdk). Setup lives in the [Zenrows MCP repository](https://github.com/ZenRows/zenrows-mcp).

```python
import os
from openai import OpenAI

ZENROWS_API_KEY = os.environ["ZENROWS_API_KEY"]
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

response = client.responses.create(
    model="gpt-5",
    tools=[
        {
            "type": "mcp",
            "server_label": "zenrows",
            "server_description": (
                "Web data infrastructure for reliable access to live web "
                "content, including dynamic and protected pages."
            ),
            "server_url": "https://mcp.zenrows.com/mcp",
            "authorization": ZENROWS_API_KEY,
            "require_approval": "never",
        }
    ],
    input="Visit https://news.ycombinator.com/ and summarize the three most recent posts.",
)

print(response.output_text)
```

### Which SDK version should I use?

This tutorial was written and tested against `openai-agents` 0.8.4. The registration pattern shown here, a decorated Python function passed in the `tools` list, has been stable across recent releases, but the SDK moves quickly, so check the [release notes](https://openai.github.io/openai-agents-python/) before you start.

The [TypeScript implementation](https://openai.github.io/openai-agents-js/) follows the same approach, using Zod schemas in place of type hints and docstrings.

### Can I use another web data provider?

If you are still evaluating, the [comparison roundup](/blog/best-web-scraping-api) covers how the alternatives differ on reliability and protected access.