Crawl: the most reliable web crawler APISee what’s new
Zenrows
Talk to sales Start free

What is a spider trap?

A spider trap, also called a crawler trap, is a part of a website that produces an endless or enormous number of URLs. A crawler that wanders in keeps finding "new" pages and never finishes, spending its budget on pages that are empty, duplicated or meaningless. Most traps are accidental side effects of how a site is built; a few are planted on purpose to catch bots.

Search engines care because traps waste crawl budget that should go to real pages. Anyone running their own crawler cares because a trap can turn a ten-minute job into one that never ends.

Common types of spider traps

Trap What it looks like Why it never ends
Infinite calendar /events?month=2026-11, then 2026-12, then 2027-01... There is always a next month
Faceted navigation ?color=red&size=m&brand=x&sort=price Filters multiply into huge numbers of combinations. See faceted navigation
Session IDs in URLs ?sid=8f3a... on every link Each visit generates fresh URLs for the same pages
Relative link loops /a/b/a/b/a/b/page A broken relative link nests the path deeper on every hop
Endless pagination ?page=9999 returns the last page again Any page number returns content, so the crawler keeps counting
Soft 404s Missing pages return 200 with links Error pages generate more links to more error pages
Honeypot links Links hidden with CSS Planted to identify bots that follow invisible links

How to detect a spider trap

  • URL length keeps growing, or the path repeats the same segments.
  • The same content comes back under new URLs. Content hashing exposes it quickly.
  • One host or path pattern dominates the queue. If 90% of pending URLs match /calendar/, something is wrong.
  • Item counts stop rising while pages fetched keep climbing.
  • Parameter values increment without limit.

How to avoid spider traps

  1. Set a crawl depth limit and a maximum page count from the first version.
  2. Normalize URLs and drop session and tracking parameters before the seen-set check.
  3. Use include and exclude patterns so only URL shapes you want, such as /product/, are followed.
  4. Cap pages per path pattern, for example 200 URLs per calendar or filter prefix.
  5. Crawl breadth-first. A trap is an infinitely deep branch, and breadth-first crawling cannot be swallowed by one the way depth-first can.
  6. Respect rel="nofollow" and robots.txt disallow rules, which often fence off exactly these areas.
  7. Skip links that are invisible to users if you parse rendered pages, which avoids most honeypots.

How site owners prevent them

Block infinite spaces in robots.txt, add nofollow to filter and calendar links, use canonical URLs for parameter variants, return a real 404 for pages that do not exist, and keep session IDs in cookies rather than URLs.

How Zenrows Crawl stays out of traps

You bound every Zenrows Crawl job with a depth, a maximum number of pages to fetch and a maximum number of URLs to find, and it stops at whichever comes first. It stays on the start URL's domain, drops duplicates as it goes, and include and exclude patterns keep it on the URLs you want. The parameters are in the Crawl endpoint reference.

Frequently asked questions

What is the difference between a spider trap and a honeypot?

A spider trap is any structure that generates endless URLs, usually by accident. A honeypot is a deliberate trap: a link hidden from people that only bots follow, used to identify and block them.

Are spider traps bad for SEO?

Yes. Search engines spend crawl budget on worthless URLs, which can delay discovery of real pages and fill the index with duplicates.

How do I find spider traps on my own site?

Crawl it with an SEO crawler and look for very long URLs, repeating path segments, large numbers of parameter variants and duplicate titles. Server logs showing Googlebot hitting the same pattern thousands of times are another clue.

Can a crawl depth limit alone prevent spider traps?

It limits the damage but does not prevent waste. Faceted navigation can create thousands of URLs at a shallow depth, so combine depth limits with URL patterns and page caps.

Go deeper on the blog

In the docs

Last updated: Oct 9, 2026

Get reliable web data in minutes.

Free plan, 5,000 credits every month, no credit card required.