Introducing Batch: send a list, collect the results
Fetching 100,000 URLs is not a bigger version of fetching one. Batch is the queue, the retries and the per-URL accounting, run by us, so you submit a list and pick the results up when it finishes.
Fetching a hundred thousand URLs is not a bigger version of fetching one. It needs a queue. It needs retries that do not re-run what already worked. It needs somewhere to put the results, back-pressure when a target slows down, and some way to find out the run finished without holding a connection open for hours.
Every team that reaches that scale ends up building the same layer, around a synchronous API that was never designed for it. The scraper is rarely the hard part. The machinery around it is.
Today we're launching Batch. Submit your URLs as one job, walk away, and collect the results when it finishes.
What is Batch?
Batch is the orchestration layer over Fetch. You hand it a set of URLs as one job instead of holding a request open per URL.
Three nouns describe the whole model:
- A Job carries your URLs and their scraping options.
- A Run is one execution of that job.
- A Task is one URL, with its own final status, real HTTP code, and either a result or a structured error.
Submit up to 100,000 URLs in a single job. It runs on the same API key as Fetch, so there is nothing new to set up:
import requests
API_KEY = "YOUR_API_KEY"
resp = requests.post(
"https://async.api.zenrows.com/v1/jobs",
headers={"X-API-Key": API_KEY},
json={
"tasks": [
{"url": "https://example.com/product/1", "external_id": "sku-1"},
{"url": "https://example.com/product/2", "external_id": "sku-2"},
]
},
)
job_id = resp.json()["job_id"]
print(job_id)
The call returns as soon as the work is accepted. It does not wait for the scraping to finish, which is the whole point.
How Batch works
Every URL ends with an answer
Each task finishes successful or failed, carries the real HTTP status code, and a failed task carries a structured error with a code and a detail. There is no silent shortfall to reconcile at the end, and no guessing which of your 100,000 URLs never came back.
Pass an external_id on any task and it comes back verbatim on the result row, so joining results to your own records is a dictionary lookup rather than URL string matching:
results = requests.get(
f"https://async.api.zenrows.com/v1/jobs/{job_id}/results",
headers={"X-API-Key": API_KEY},
).json()
for row in results["results"]:
print(row["external_id"], row["status"], row["result_url"])
Each result_url is a presigned link to that page's content, valid for 2 hours. Re-list the results whenever you need a fresh one.
Reruns only touch what failed
When a run finishes with failures, rerun the failed tasks alone. The successful results carry over untouched, and you are not paying to scrape 97,000 pages a second time to recover 3,000. Transient failures are retried inside the run before they ever reach you.
One job, many domains
A job is a list of URLs plus options, so the targets can be mixed. Put twelve different sites in one job and override the scraping parameters on the individual URLs that need different handling. You do not need one pipeline per site shape.
You know the cost before you submit
Estimate a job's cost client side before you send it, then read the actual spend on the run afterwards. Batch adds no charge of its own: you pay for the Fetch calls it runs and nothing on top. Requests that fail aren't charged, except pages that answer 404 or 410, which bill as in Fetch and show as failed tasks; read the run's stats.spend for the real total.
Send the list however it lives
Inline tasks, as above, take up to 100,000 URLs per job. If the list already sits in a spreadsheet, upload it as a CSV instead (up to 50 MB or 100,000 rows) and map its columns to the URL and your own ID. And if the URLs arrive over time, open a job and keep adding tasks to it, up to 10,000 per call, then close it when the last batch is in.
Run it once, or on a schedule
A job can run once, at a set time, at a regular interval, or on chosen days and hours, in your own timezone, so daylight saving never shifts it. Each scheduled fire starts a new run over the same list. Pause the schedule and it waits; resume it and it carries on.
Stay in control of a run
Pause a run and resume it later, or stop it outright when you have what you need. When you no longer want a job stored, delete it, results included.
Get told when it finishes
Instead of polling, give the job a webhook. Batch calls it when the run completes, optionally signed with an HMAC signature so you can verify the request came from us. The webhook tells you the run is done; the results come from the results endpoint.
Choose what comes back
Each task returns HTML, JSON, Markdown, plain text or a PDF, using the same scraping options as Fetch. Add Extract to a job and each page comes back as structured fields instead of raw content. Set your own headers on the requests a job runs, and send a target a POST with a body of up to 16 KiB instead of a GET when the page needs one.
Collect results your way
List a run's results and filter them by status, so the failures are one call away. Download one page through its result link, or export a whole run as a ZIP. Results stay available for 14 days after each task finishes; after that a row shows as expired.
Use Batch from wherever you work
- The dashboard: run a job from the async playground and watch it work, no code needed.
- The API: the REST endpoints in the snippets above, with the same API key as Fetch.
- SDKs: Batch has its own sections in the Python, Node and Go SDKs.
- The CLI: create, watch, estimate and retry jobs from your terminal.
- MCP: the Zenrows MCP server exposes Batch, so an AI agent can submit a job, wait for it, and collect the results itself.
What you can build with Batch
Refresh a whole catalog. Point one job at every product URL you track, run it, and read per-URL status to see exactly which SKUs came back and which need another pass.
Seed a RAG index. Send the whole corpus as one job, collect clean page content when it finishes, and use the per-task accounting to prove your index is complete rather than assuming it.
Retire your queue. If you already run Celery or a thread pool around a scraping API, Batch is that component, managed. The dispatcher, the retry logic and the result store stop being code you maintain.
Enrich a list you were handed. Drop in a list of company URLs, walk away, and come back to structured results with the failures already labelled.
Try it today
Batch is live for every Zenrows account with an API key.
Send your first job with the snippet above, or read the guide for the full API.