Crawl: the most reliable web crawler APISee what’s new
Zenrows
Talk to sales Start free

What is a sitemap?

A sitemap is a file that lists the URLs a website wants crawlers to find. The common form is an XML sitemap at /sitemap.xml, defined by the sitemaps.org protocol, which every major search engine reads. Each entry gives a page's address and, optionally, when it last changed, so a crawler can discover pages without having to reach them through links.

For anyone collecting web data, a sitemap is the site's own list of what it considers its pages, which makes it one of the best seed sources available when it exists and is kept up to date.

What an XML sitemap looks like

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/products/blue-hoodie</loc>
    <lastmod>2026-09-30</lastmod>
  </url>
  <url>
    <loc>https://example.com/products/grey-hoodie</loc>
    <lastmod>2026-10-02</lastmod>
  </url>
</urlset>

A single sitemap file holds at most 50,000 URLs and 50 MB uncompressed. Larger sites publish a sitemap index, a file that lists other sitemaps, often split by section: products, categories, blog posts, images.

Types of sitemaps

Type Format Used for
XML sitemap XML, sitemaps.org protocol The standard list of page URLs for crawlers
Sitemap index XML Pointing to several sitemap files on large sites
Image, video and news sitemaps XML extensions Media and news content for search features
HTML sitemap A normal web page Human navigation, also crawlable through its links
Text or RSS sitemap A plain list of URLs, or a feed Simple or frequently updated sites

How to find a website's sitemap

  1. Try /sitemap.xml and /sitemap_index.xml at the site root.
  2. Read robots.txt. Sites often declare their sitemap there with a Sitemap: line.
  3. Check the CMS defaults: WordPress serves /wp-sitemap.xml, and many shop platforms publish one per content type.

The blog's guide to finding all URLs on a domain covers sitemaps alongside the other discovery methods.

How much to trust a sitemap

A sitemap is a statement of intent rather than a guarantee. In practice:

  • It can be stale. Sitemaps generated once and never refreshed list deleted pages and miss new ones.
  • It can be partial. Some sites list only their most important pages, or only one section.
  • lastmod is often wrong. Some generators stamp every URL with today's date, which makes the field useless for incremental crawling.
  • priority and changefreq are mostly ignored by search engines, and you can ignore them too.

The reliable approach is to treat the sitemap as a strong starting point and confirm coverage another way, for example by checking listing pages and the item counts they show.

Sitemaps for SEO

For site owners, a sitemap helps search engines find pages that internal links reach poorly, such as new products or deep archive pages. It does not guarantee indexing. List only canonical, indexable URLs that return 200, keep lastmod honest, and submit the sitemap in Google Search Console and Bing Webmaster Tools.

Where Zenrows fits

A sitemap gives you a URL list, and a URL list is a fetching job rather than a crawling one. Batch runs a list of URLs as one managed job, handling concurrency and retries. When you don't have a list yet, Zenrows Crawl starts from any page you give it, follows its links as deep as you choose, and returns the URLs it finds, with each page's HTML if you want it.

Frequently asked questions

What is the difference between an XML sitemap and an HTML sitemap?

An XML sitemap is written for crawlers and lists URLs in a machine-readable format. An HTML sitemap is a page of links written for people, which crawlers can also follow.

Does every website have a sitemap?

No. Most large sites and most CMS-based sites do, but many smaller or custom-built sites do not, and some keep theirs out of public view.

How many URLs can a sitemap contain?

Up to 50,000 URLs and 50 MB uncompressed per file. Larger sites use a sitemap index that points to several files.

Can I scrape a site using only its sitemap?

Often, if the sitemap is complete and current. Check it against the site's listings first, because stale or partial sitemaps are common.

Go deeper on the blog

In the docs

Last updated: Oct 9, 2026

Get reliable web data in minutes.

Free plan, 5,000 credits every month, no credit card required.