--- title: "Introducing Crawl: the most reliable web crawler API" description: "Zenrows Crawl is the most reliable web crawler API. Give it any public URL and it follows the links on that site as deep as you choose, then hands back every URL it found and, if you want it, every page's HTML." --- # Introducing Crawl: the most reliable web crawler API > Zenrows Crawl is the most reliable web crawler API. Give it any public URL and it follows the links on that site as deep as you choose, then hands back every URL it found and, if you want it, every page's HTML. Source: https://www.zenrows.com/blog/introducing-crawl The web was built to be crawled. The first crawlers were written so people could find things, and every search engine, price tracker, dataset and AI agent since has started the same way: open a page, follow the links, keep going. What changed is how much it costs to do that well. Somebody has to write the link loop, the filter and the deduplication. Somebody has to keep proxies and headless browsers running so the pages don't block you. Somebody has to retry what failed, store what worked, and fix it all when a site changes. Call it the crawler tax. Every team that needs web data pays it before a single record lands in a database. **Today we're launching Crawl: the most reliable web crawler API. Give Crawl any public URL. It follows the links on that site as deep as you choose, fetches each page through Fetch with Adaptive Stealth Mode to get past anti-bot protection, and hands back every URL it found.**
## What Crawl does [Crawl](/products/crawl) is a web crawler API. Give it any public URL: a homepage, a category page or a search page. It opens the page, follows the links on it as deep as you choose, keeps only the URLs you want, and drops duplicates. Each page goes through [Fetch](/products/fetch) with Adaptive Stealth Mode to get past anti-bot protection. You get back every URL it found, and each page's HTML if you want it. You don't write, host or maintain a crawler. You send a request: ```python import requests resp = requests.post( "https://api.zenrows.com/v1/crawls", headers={"X-API-Key": "YOUR_ZENROWS_API_KEY"}, json={ "url": "https://www.scrapingcourse.com/ecommerce/", "depth": 1, "max_items": 50, "max_pages": 100, "include_patterns": ["/product/"], }, ) print(resp.json()["crawl_id"]) ``` `max_items` caps how many URLs the crawl keeps and `max_pages` caps how many pages it opens. Without them a crawl stops at the defaults, so set both, with `max_pages` above `max_items`. The crawl starts at once and runs in the background. You read its results while it works, or download all of them when it's done. ## Every link on the site, as deep as you set Crawl reads every link in the page you give it and keeps the ones on the same site, subdomains included, that match your include and exclude patterns. Then it opens those pages and does the same, as many levels deep as you set. At the first level you get the links on the starting page; one level deeper, it also opens those pages and collects their links. Every page it opens goes through Fetch with Adaptive Stealth Mode, which picks the fetch settings page by page to get past anti-bot protection, so you never decide which pages need a browser or a premium proxy. ## What you get back By default, Crawl returns every URL it found, once each. Set `output_format` to `html` when you start the crawl and every kept page comes back with its HTML too. Results arrive as the crawl runs. Read them page by page with a cursor, or download the whole result as a single NDJSON file, ready for a database, a queue or a training set. You can stop a running crawl at any time and keep what it found. ## What every crawl reports Every crawl tells you what it did, while it runs and after it ends: - the pages it fetched - the pages that failed - the URLs it found - the duplicates it removed - why it stopped: it ran out of links, it reached your limits, or you stopped it ## Built to scale Crawl runs on Zenrows' managed infrastructure, with nothing to provision. You send a request and read the results; there's no crawler fleet to run and nothing to babysit. ## What you can build with Crawl **Find a store's products.** Start from a category page or the homepage, keep the URLs that match `/product/`, and get the product URLs behind it, and their HTML. **Map a site.** Start from a homepage and get the URLs on the site, then keep only the article, listing or documentation pages you need. **Collect pages for your AI.** Give an agent or a RAG pipeline the pages of a site it needs, fetched with Adaptive Stealth Mode, instead of the handful a person found by hand. **Retire your crawler.** If you run Scrapy, a headless browser fleet or a homegrown queue just to find pages, Crawl is that component, managed. ## Try it today Crawl is live, and any active Zenrows API key can use it. Open the [Crawl playground](https://app.zenrows.com/crawl), paste any URL and preview what's behind it, then copy the generated code into your project. [Start crawling free](https://app.zenrows.com/register?primitive=crawl) · [Read the docs](https://docs.zenrows.com/crawl/introduction) · [Crawl product page](/products/crawl) ## Frequently asked questions ### What is Zenrows Crawl? Zenrows Crawl is a web crawler API. Give it any public URL, a homepage, a category page or a search page. It opens the page, follows the links on it as deep as you choose, keeps only the URLs you want, drops duplicates, and fetches each page through Fetch with Adaptive Stealth Mode to get past anti-bot protection. You get back every URL it found, and each page's HTML if you want it. ### Where can a crawl start? Anywhere public: a homepage, a category page, a search results page or any other URL. The crawl stays on that site, subdomains included. ### What does Crawl return? Every URL it found. Start the crawl with `output_format` set to `html` and you get each page's HTML as well. ### How many URLs does a crawl return? As many as you allow. Set `max_items` for the URLs to keep and `max_pages` for the pages to open, with `max_pages` above `max_items`. Without them, a crawl stops at the defaults. ### How do I export the results? Read results while the crawl runs, page by page with a cursor, or download everything as a single NDJSON file with a JSON object per line. ### Where do I start? Open the [Crawl playground](https://app.zenrows.com/crawl) in your Zenrows dashboard, or follow the [Crawl guide](https://docs.zenrows.com/crawl/setup) in the docs.