Crawl: the most reliable web crawler APISee what’s new
Zenrows
Talk to sales Start free

What is pagination in web scraping?

Pagination is the practice of splitting a long list of results across several pages. A category with two thousand products shows forty per page and gives you a way to reach the rest: page numbers, an offset, a "next" link or a cursor. In web scraping, handling pagination means following those pages until the list ends, so you collect every item instead of only what fits on page one.

It is the most common reason a scraper returns fewer results than the site holds. The first page works, the loop to reach page two is missing or wrong, and the output looks plausible while being a fraction of the data.

Types of pagination

Type What the URL or request looks like How to follow it
Page number ?page=2, /page/2/, /p2.html Increment the number until a page returns no items
Offset ?start=40, ?offset=40&limit=20 Add the page size each time
Next link A <a rel="next"> or "Next" button with a real href Follow the link until there is none
Cursor ?cursor=eyJpZCI6... from the previous response Pass the returned cursor until it comes back empty
Load more / infinite scroll No new URL; JavaScript fetches more items Call the underlying API directly, or drive a browser

Page-number and offset pagination are easy to generate, but they can shift under you: if items are added while you crawl, an item can slide from page 3 to page 4 and get skipped or seen twice. Cursor pagination, common in APIs, avoids that because the cursor marks a stable position in the result set.

How to scrape paginated pages reliably

  • Stop on content, not on a guess. End when a page returns no new items or no next link, rather than assuming a fixed page count.
  • Check the total. Many listings state "2,148 results". Comparing what you collected against that number is the quickest completeness check there is.
  • Deduplicate items. Sponsored and featured items repeat across pages. Deduplication on the item URL keeps the count honest.
  • Cap it. Some sites paginate forever, returning the last page again for any higher number. A page cap stops a spider trap.
  • Keep pagination separate from depth. Following page 2, 3 and 4 of a listing is not going deeper into the site. Counting it as crawl depth makes a depth limit cut listings short.
  • Watch for sort order. Without a stable sort, items can move between pages during a crawl. Sorting by newest or by ID usually helps.

Framework-specific walkthroughs are on the blog for Scrapy, Playwright, Selenium and Python Requests.

Pagination and SEO

The same structure matters to search engines. Google no longer uses rel="next" and rel="prev" as an indexing signal, but paginated pages still need crawlable links so every item is reachable. Pages that only load more items on scroll, with no paginated URLs behind them, can leave products undiscovered by crawlers of every kind.

Covering more pages with Zenrows Crawl

Zenrows Crawl follows the links in each page's HTML as deep as depth allows. To walk more pages of a listing with Crawl, raise depth and use include patterns to keep only the URLs you want, such as /product/. Duplicates are dropped as the crawl runs, and you get back every URL it found, with each page's HTML if you want it. See how Crawl works in the docs.

Frequently asked questions

What is the difference between pagination and infinite scroll?

Pagination gives each set of results its own URL or request. Infinite scroll appends more results to the same page as you scroll, usually by calling an API in the background. Scraping infinite scroll means finding that API or driving a real browser.

How do I know how many pages to scrape?

Do not hardcode it. Follow pages until one returns no new items or no next link, and compare your total with any count the site displays.

What is cursor-based pagination?

Each response includes an opaque token pointing to the next set of results, and you send it back to get the next page. It stays consistent when data changes during the crawl, which is why most APIs use it.

Why does my scraper only get the first page?

Usually because the next pages load through JavaScript or an API call your HTTP client never makes, or because the loop that builds the next URL is missing. Inspect the network requests the page makes when you click "Next".

Go deeper on the blog

In the docs

Last updated: Oct 9, 2026

Get reliable web data in minutes.

Free plan, 5,000 credits every month, no credit card required.