Breadth-first or depth-first crawling?
Breadth-first and depth-first are the two basic orders in which a crawler can visit pages. Breadth-first search (BFS) visits every page one link away from the start, then every page two links away, and so on, level by level. Depth-first search (DFS) follows one chain of links as far as it goes, then backtracks and tries the next branch.
They are the same graph algorithms taught in computer science, applied to a website: pages are nodes and links are edges. The only implementation difference is the data structure holding the URLs waiting to be visited, the frontier. BFS uses a queue, first in, first out. DFS uses a stack, last in, first out.
Side by side
| Breadth-first (BFS) | Depth-first (DFS) | |
|---|---|---|
| Frontier | Queue (FIFO) | Stack (LIFO) |
| Visit order | Level by level from the start | One branch to the bottom, then backtrack |
| Pages found first | The closest, usually the most important | Whatever sits at the end of the first branch |
| Memory | Grows with the width of a level | Grows with the depth of a branch |
| Trap risk | Low: a trap only adds pages to one level | High: one infinite branch can absorb the whole crawl |
| Behaviour if stopped early | A useful, even slice near the start | A narrow, lopsided slice |
| Typical use | General and listing crawls, search engines | Exhaustive exploration of small, bounded trees |
Why most crawlers are breadth-first
Websites are shallow and wide. Most important pages sit within a few clicks of the homepage or a category page, and the number of pages grows quickly with each level. BFS reaches the important pages early and spreads requests across sections.
It also fails gracefully. A crawl that hits its page budget or gets interrupted with BFS has covered the nearest levels evenly. The same interruption with DFS leaves you with a deep slice of one branch and nothing from the others.
And it resists spider traps. An infinite calendar or a loop of filter URLs is an infinitely deep branch. DFS walks straight down it. BFS treats each level of the trap as one more level and moves on, especially with a crawl depth limit in place.
When depth-first makes sense
DFS is a reasonable choice when the structure is a bounded tree and you want complete branches one at a time: a documentation site where you want each section finished before the next, a file-tree style directory listing, or a crawl with a strict depth limit where memory matters more than order.
Beyond plain BFS
Production crawlers usually use a priority queue instead of a strict FIFO. Each URL gets a score, based on depth, how many pages link to it, whether it matches the patterns you care about, or how recently it changed, and the highest score is fetched next. A focused crawler is one whose priority function favours pages on a chosen topic. Politeness rules sit on top: per-host queues and a crawl delay stop BFS from hitting one server with every URL of a level at once.
Minimal BFS crawler in Python
from collections import deque
def crawl(seed, get_links, max_depth=2):
seen = {seed}
queue = deque([(seed, 0)])
while queue:
url, depth = queue.popleft() # popleft = BFS; pop() = DFS
if depth == max_depth:
continue
for link in get_links(url):
if link not in seen:
seen.add(link)
queue.append((link, depth + 1))
return seen
Switching popleft() to pop() turns the same code into a depth-first crawler, which is the whole difference in one line.
How Zenrows Crawl counts depth
In Zenrows Crawl, depth is counted in link steps from the start URL. At depth 1 it returns the links on the start page; at depth 2 it also opens those pages and returns their links, and so on as deep as you choose. Include and exclude patterns decide which links are kept and opened, and duplicates are dropped along the way. The Crawl docs explain how a crawl runs.
Frequently asked questions
Is BFS or DFS better for web crawling?
BFS in most cases. It finds important pages sooner, spreads coverage evenly, and is far less likely to get stuck in an infinite branch.
What data structure does BFS use?
A queue. URLs are added to the back and taken from the front, so pages are visited in the order they were discovered.
What data structure does DFS use?
A stack, or recursion, which uses the call stack. The most recently discovered URL is visited next.
Does Google crawl breadth-first?
Google does not crawl in a strict order. It prioritises URLs by importance and freshness, which in practice behaves more like a weighted breadth-first crawl than a depth-first one.
Go deeper on the blog
In the docs
Last updated: Oct 9, 2026