What is a seed URL?

A seed URL is a crawl's starting point. The crawler fetches it, extracts its links, and works outward from there. Everything it will ever find has to be reachable by following links from the seeds you gave it.

That constraint is the reason seed selection deserves more thought than it usually gets. A crawler cannot discover what nothing links to.

Why one seed is usually the wrong number

Starting from a site's homepage feels natural and quietly loses coverage. Homepages link to current, featured and popular content. Older articles, deep category pages and anything the site has stopped promoting may be several hops away or effectively unreachable.

Better seed sets combine sources:

• The XML sitemap, if there is one. This is the site telling you what it considers its pages, and it is almost always the highest-value seed source available.
• Category and index pages, which fan out into a section far faster than a homepage does.
• Paginated listings, which reach a whole catalogue directly.
• A known URL list, when you already have one, in which case you may not need discovery at all.

Checking /sitemap.xml and /robots.txt, which often points at the sitemap, takes seconds and frequently removes the need for most of the crawl.

Seeds set the crawl's shape

A seed also implies a boundary. Most crawls restrict themselves to the seed's domain, and without that restriction a crawler follows an outbound link and starts crawling the internet.

Depth is measured from the seed too, so a depth limit of three means something different starting from a homepage than from a category page. Starting deeper reaches more of a section within the same limit.

When you do not need a seed

If you already have the URLs, discovery is a phase you can skip. Scraping ten thousand known product pages is a fetching job rather than a crawling one, and building a crawler for it adds complexity with no benefit.

Separating the two phases is worth doing even when you need both. Discovery produces a URL list; fetching consumes it. The list becomes a checkpoint, so an interrupted run resumes from a known position rather than starting over.

Where Zenrows fits

The seed URL guide covers building a crawler in Python that starts from a seed, discovers internal links and scrapes at scale. Once the URL list exists, Batch runs it as a managed job, which is the natural split between the two phases.