Crawl: the most reliable web crawler API
Crawl
Give Crawl any public URL and get back every page behind it. It follows the links as deep as you choose, drops duplicates, fetches every page past anti-bot protection, and returns every URL it found, plus each page's HTML if you ask.