What is JavaScript crawling?
JavaScript crawling is crawling with a crawler that runs each page's JavaScript before reading it. Many modern sites send a nearly empty HTML shell and build the page in the browser, so links, product lists and even the main text only exist after scripts run. A crawler that reads the raw HTML sees none of it; a JavaScript crawler renders the page first, usually in a headless browser, and then extracts links and content from the result.
The term is used in two settings. In SEO, it describes how Google and other search engines render JavaScript sites so they can index them. In web scraping, it describes crawlers that must render pages to find the URLs and data they are after.
HTML crawling vs JavaScript crawling
| HTML crawling | JavaScript crawling | |
|---|---|---|
| What it reads | The server's raw HTML response | The DOM after scripts run |
| Tooling | HTTP client plus parser | Headless browser, or a rendering API |
| Speed and cost per page | Fast and light | Slower and heavier |
| Sees client-side links | No | Yes |
| Sees content loaded by API calls | No | Yes, if it waits for them |
| Typical use | Server-rendered sites, static pages | Single-page apps, React, Vue and Angular sites |
How JavaScript crawling works
- Fetch the URL in a browser engine instead of a plain HTTP client.
- Run scripts and wait until the content you need is present, using wait strategies such as waiting for a selector or for network activity to settle.
- Read the rendered DOM and extract links and data.
- Normalize, deduplicate and enqueue the new links, as any crawler does.
Google does the same in two waves: it crawls the HTML first, then queues the page for rendering when resources allow, which is why content that appears only after rendering can be indexed later than server-rendered content.
Does a site need JavaScript crawling?
- View the page source (not the inspector). If the products or links you need are missing, they are rendered client-side.
- Disable JavaScript in the browser and reload. Whatever disappears needs rendering.
- Look for an empty
<div id="root">or<div id="app">, the signature of a single-page app. - Check the network tab. Data loaded from a JSON API can often be requested directly, which is faster than rendering.
Best practices
- Render only when needed. Rendering costs far more than a plain request, so detect which pages need it.
- Use real links when they exist. Links built from
onclickhandlers without anhrefare invisible to most crawlers, including Google. - Block what you do not need, such as images, fonts and analytics, to speed up rendering.
- Prefer the underlying API when the page fetches its data from one.
- Expect more blocking. Headless browsers carry detectable fingerprints, so rendering at volume usually needs stealth measures too.
How Zenrows Crawl handles rendering
Every page Zenrows Crawl opens is fetched by Fetch, and Zenrows picks the fetch settings page by page, including when a page needs JavaScript rendering and how to get past anti-bot protection. You give it any public URL and get back the URLs it finds, with each page's HTML if you want it, without deciding per page whether to render. The Crawl docs describe how a crawl runs.
Frequently asked questions
Can Google crawl JavaScript?
Yes. Googlebot renders pages with an up-to-date Chromium. Rendering can lag behind the initial crawl, and links that are not real <a href> elements are still not followed.
Is JavaScript crawling slower than HTML crawling?
Yes, by a wide margin per page, because the crawler has to download scripts and run a browser engine. Rendering only the pages that need it keeps the total cost reasonable.
What tools are used for JavaScript crawling?
Headless browsers driven by Playwright, Puppeteer or Selenium, crawling frameworks that integrate them, and scraping APIs that render pages on request.
What is the difference between JavaScript rendering and JavaScript crawling?
Rendering is running scripts to produce the final page. JavaScript crawling is a crawl that renders each page so it can discover the links and content that only exist after rendering.
Go deeper on the blog
In the docs
Last updated: Oct 9, 2026