delivery
POST hooks.team/results
X-Signature: sha256=...
{
"event": "run.completed",
"job_id": "abc123",
"urls": 4300,
"results_url": "/v1/jobs/abc123/results"
}Batch Beta
Run large web data jobs reliably.
Send thousands. Scale to millions, job by job.
Process up to 100,000 URLs per job, asynchronously, protected and dynamic pages included, with queues, retries, per-job webhooks, and job observability built in.
Hand us a list of websites, including the ones that block ordinary tools. Get clean, structured answers back, ready for your CRM or your dashboard. Thousands per job, more by running more jobs.
What you'd ship without Batch
What your team normally pays for
- Worker pool Spin up, tear down, limit concurrency per domain
- Backoff logic Handle 429s, add jitter, respect crawl delays
- Retry queue Catch transient failures, back off and retry
- Webhook layer Fire on completion, sign payloads, handle drops
- Progress API Track active, delivered, and failed in real time
workers 0 32
Push more. Workers follow.
Send more. Capacity adjusts.
Workers spin up when the queue grows. They idle out when it drains. Add up to 10,000 URLs per call, as many calls as it takes to fill the job's 100,000-URL cap.
We add capacity as your job grows. Results land as we go. No infrastructure to manage.
The box
Everything else comes with it.
Everything else comes with it.
One signed webhook on completion. Retries on transient failure. Failed tasks stay visible. Live status endpoint. No glue code.
One signed signal when it's done. Automatic retries. Transparent failures. Live progress. No code to write.
Scale
100,000 URLs a job. As many as needed.
One list, one job. Add more to scale.
Same workers. Same JSON contract per URL. Scale by running more jobs.
Same setup, every job. Add jobs to go from thousands to millions.
Regular jobs
Or just a list.
Or just a list.
Push a CSV, up to 100,000 rows or 50MB. Workers drain it. Get the JSON back when the last URL is done. Same engine, closed set, no queue to manage.
Hand us a CSV, up to 100,000 rows. We process every row. Your team gets the answers when it's done. Same engine, fixed list, no setup.