Crawl: the most reliable web crawler APISee what’s new
Zenrows
Talk to sales Start free

Deep research agent

A deep research agent answers a question by planning searches, opening and reading many live web pages, and writing a report that cites each source. Zenrows handles the reading: Fetch returns every page as Markdown, protected and JavaScript-heavy sources included, and the MCP server lets Claude or your agent call it, so each claim keeps its URL.

Updated

Sample output: a cited research answer
{
  "question": "How do three outdoor retailers handle returns of used gear?",
  "sources_planned": 12,
  "sources_read": 12,
  "findings": [
    {
      "claim": "Retailer A accepts used items within 365 days for members.",
      "source": "https://www.example-outdoor-a.com/help/returns",
      "quote": "Members may return used gear within one year of purchase.",
      "fetched_at": "2026-10-17T10:02:11Z"
    },
    {
      "claim": "Retailer B gives store credit only for used items.",
      "source": "https://www.example-outdoor-b.com/policies/returns",
      "quote": "Used items are refunded as store credit.",
      "fetched_at": "2026-10-17T10:02:14Z"
    }
  ],
  "unanswered": ["Retailer C's policy page does not cover worn footwear"]
}
Illustrative. Zenrows supplies the page text behind each quote; the agent writes the claims and keeps each URL and fetch time.

What deep research agents investigate

A run loops: plan, find, read, check gaps, read again, write.

The agent can only state what it read.

JobData you collectTypical sources
Vendor and market research Pricing, plans, features, release notes Vendor pricing pages, docs, changelogs, G2 profiles
Company due diligence Leadership, filings, press, litigation Company sites, SEC EDGAR, Companies House, court portals
Regulatory research Rule text, effective dates, guidance EUR-Lex, Federal Register, regulator websites
Literature scans Abstracts, methods, results, citations arXiv, PubMed, Google Scholar, journal pages
Technical investigations Known issues, fixes, version differences GitHub issues, Stack Overflow, official docs
Product comparisons Specs, prices, reviews, return terms Retailer product pages, manufacturer spec sheets

Why research agents come back thin

Vague reports often start upstream: the model read snippets, not pages, or half its sources failed and it filled the gaps.

What it looks like

A two-line excerpt quoted, the table below it missed

How Zenrows handles it

Fetch returns the full page as Markdown

How to build a deep research agent

  1. Plan the question: the model splits it into sub-questions and the sources each needs.
  2. Find candidate URLs with your search tool or a fetched Google results page; keep the ones worth reading.
  3. Read in parallel with Fetch and Markdown output; Adaptive Stealth Mode escalates to rendering and stealth only when a source needs it.
  4. Check coverage, then write: re-read for sub-questions without a source, then write each claim with its URL and quote.
# pip install requests
from concurrent.futures import ThreadPoolExecutor
from datetime import datetime, timezone
import requests

def read_page(url):
    """Agent tool: one page as Markdown, with what it needs to cite it."""
    params = {
        "apikey": "YOUR_ZENROWS_API_KEY",
        "url": url,
        "mode": "auto",  # Adaptive Stealth Mode
        "response_type": "markdown",
    }
    r = requests.get("https://api.zenrows.com/v1/", params=params, timeout=180)
    return {
        "url": url,
        "ok": r.ok,
        "fetched_at": datetime.now(timezone.utc).isoformat(),
        "markdown": r.text if r.ok else "",
    }

def read_many(urls, workers=10):
    """Fan out over the planner's reading list."""
    with ThreadPoolExecutor(max_workers=workers) as pool:
        return list(pool.map(read_page, urls))

sources = read_many(READING_LIST)
evidence = [s for s in sources if s["ok"] and len(s["markdown"]) > 500]
print(f"{len(evidence)} of {len(sources)} sources usable")

What success looks like

MeasureWhat good looks likeHow to check it
Source read rate Nearly every planned URL returns usable text 200s above a minimum text length, per run
Citation support Each quote appears in its cited page Match quotes against stored Markdown
Coverage A primary source per sub-question Map findings to sub-questions
Time to report Done within what users will wait Wall-clock time per run and round
Retry rounds Most runs: one round, at most one retry Rounds and failed sources in agent logs

Deep research agent FAQ

  • An AI agent that does a researcher's legwork on a broad question: it splits it, searches, reads many sources, fills gaps, and writes a report citing each claim. The model plans and writes; a web data API like Zenrows does the reading.

  • A tool-calling model, a way to find URLs, a way to read them. Give the model a read_page tool calling Zenrows Fetch with Markdown output, or connect the Zenrows MCP server for scrape and the other tools with no code. Prompt it to plan, read in parallel, check coverage and cite.

  • Any model with tool calling. The Anthropic and OpenAI APIs call the remote MCP server at https://mcp.zenrows.com/mcp directly; LangChain, LlamaIndex, CrewAI, Mastra and the OpenAI Agents SDK register Fetch as a function tool. The Zenrows docs have a step-by-step guide to a citing research agent.

  • Usually from a search tool. It can also fetch a Google results page through Zenrows, or follow a site's listings, like a regulator's publications index. Reading works the same.

  • Keep URL and fetch time on every page, and have the agent quote the sentence behind each claim. Check each quote against the stored Markdown; unmatched claims get another round or leave the report.

  • Yes, for accounts you are entitled to use: the MCP server's browser_* tools log in, run a search form and read results in one cloud browser session. Without a login, Fetch is faster.

  • A primary source per sub-question and two for key claims: often twenty to fifty pages over a few rounds for a market or policy question. Read more only where sources disagree or a sub-question is empty. Fewer, better sources in full beat snippets from a hundred.

Turn the live web into usable data.