--- title: "The Most Reliable Web Crawler API | Zenrows Crawl" description: "Zenrows Crawl is the most reliable web crawler API. Give it any public URL and it follows the links on that site as deep as you choose, then returns every URL and page HTML." --- # The Most Reliable Web Crawler API | Zenrows Crawl > Zenrows Crawl is the most reliable web crawler API. Give it any public URL and it follows the links on that site as deep as you choose, then returns every URL and page HTML. **pages fetched** **URLs kept** **depth** **duplicates removed** northtrail.example ยท a made-up shop New ## The most reliable web crawler API. Give Crawl any public URL. It follows the links on the same site as deep as you choose, fetches each page through Fetch with Adaptive Stealth Mode to get past anti-bot protection, and hands back every URL and every page's HTML. ## It follows every link. Crawl reads every link on the page you give it, keeps the ones on the same site, and opens them for you. ## Depth sets how far it goes. Shallow gives you what the starting page links to. Deeper opens those pages and follows their links too. ## Fetched with Adaptive Stealth Mode. Each page goes through the same engine as Fetch, which picks the right settings page by page to get past anti-bot protection. You decide how far the crawl goes. ## Keep only the URLs you want. Include keeps what matches a pattern, Exclude drops what does not belong, and duplicates are removed on the way. ## Every URL, ready to use. Take the URLs, or the HTML of every page too. Read results while it runs, or download them all as one file. ## What Zenrows Crawl does A web crawler API: give it any public URL, and it returns every page it finds on that site. - You give it Any public URL, how deep to go, and what to keep. - It does Follows the links on that site and fetches each page past anti-bot protection. - You get back Every URL it found, and each page's HTML if you ask. - Where it runs On Zenrows. Nothing to host, scale or maintain. ## The crawler you no longer have to write. ## A single request starts it. Send the starting URL and how deep to go. Crawl runs in the background, and you read the results while it works. ``` curl -X POST "https://api.zenrows.com/v1/crawls" \ -H "X-API-Key: YOUR_ZENROWS_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://www.example-shop.com/", "depth": 1, "max_items": 50, "max_pages": 100, "include_patterns": ["/product/"], "output_format": "html" }' ``` - Link loop: every link opened and followed, as deep as you choose. - Link filter: keep the pages you want, drop the noise. - Deduplication: every URL once, duplicates dropped. - Proxies and browsers: every page fetched by the engine behind Fetch. - Queue and storage: results kept, readable while it runs, downloadable at once. ## Crawl, answered. - What is Zenrows Crawl? Zenrows Crawl is a web crawler API. Give it any public URL, a homepage, a category page or a search page. It opens the page, follows the links on it as deep as you choose, keeps only the URLs you want, drops duplicates, and fetches each page through Fetch with Adaptive Stealth Mode to get past anti-bot protection. You get back every URL it found, and each page's HTML if you want it. You don't write, host or maintain a crawler. - What is a web crawler API? A web crawler API runs the crawl for you on its own infrastructure. You send a starting URL and a few rules, and it opens the pages, follows the links, and returns what it found, so you don't host, scale or unblock a crawler yourself. - How is Crawl different from Fetch? Fetch returns a page you already know. Crawl starts from a page and finds the pages you don't know yet: it follows the links, drops duplicates, and returns every URL it found. Every page Crawl opens is fetched by the same engine as Fetch. - Where can a crawl start? Anywhere public: a homepage, a category page, a search results page or any other URL. The crawl stays on that site, subdomains included, and include and exclude patterns keep only the URLs you want. - What does Crawl return? Every URL it found, by default. Ask for HTML when you start a crawl and you get every page's HTML too. You can read results while the crawl runs, or download them all as one file. - How does Crawl deal with anti-bot protection? Every page Crawl opens goes through Fetch with Adaptive Stealth Mode, which picks the fetch settings page by page and escalates when a page needs it. You don't configure proxies, headless browsers or retries. - Does Crawl return duplicate URLs? No. Crawl drops duplicates as it goes, so each URL appears in the result once. - Is Crawl available now? Yes. Crawl is live, and any active Zenrows API key can use it with no setup. Try it in the playground at app.zenrows.com/crawl, or call it through the API. ## The most reliable web crawler API. Give Crawl any public URL and every page it finds on that site comes back, as URLs and HTML. _Canonical HTML: https://www.zenrows.com/products/crawl_ ## Site Navigation ### Global Hub - [Home](https://www.zenrows.com/index.md) - [Pricing](https://www.zenrows.com/pricing.md) - [Products](https://www.zenrows.com/products.md) - [Agent Toolkit](https://www.zenrows.com/agents.md) - [Compare](https://www.zenrows.com/compare.md) - [Alternatives](https://www.zenrows.com/alternatives.md) - [Blog](https://www.zenrows.com/blog) - [Docs](https://docs.zenrows.com) - [llms.txt](https://www.zenrows.com/llms.txt) - [Agent skill](https://www.zenrows.com/SKILL.md) ### On This Topic - [Products overview](https://www.zenrows.com/products.md) - [Fetch](https://www.zenrows.com/products/fetch.md) - [Browser](https://www.zenrows.com/products/browser-sessions.md) - [Extract](https://www.zenrows.com/products/extract.md) - [Search](https://www.zenrows.com/products/search.md) - [Monitor](https://www.zenrows.com/products/monitor.md) - [Batch](https://www.zenrows.com/products/batch.md) - [API](https://www.zenrows.com/api.md) - [SDK](https://www.zenrows.com/sdk.md) - [MCP Server](https://www.zenrows.com/mcp.md) - [CLI](https://www.zenrows.com/cli.md)