Crawl: the most reliable web crawler APISee what’s new
Zenrows
Talk to sales Start free

What is JavaScript crawling?

JavaScript crawling is crawling with a crawler that runs each page's JavaScript before reading it. Many modern sites send a nearly empty HTML shell and build the page in the browser, so links, product lists and even the main text only exist after scripts run. A crawler that reads the raw HTML sees none of it; a JavaScript crawler renders the page first, usually in a headless browser, and then extracts links and content from the result.

The term is used in two settings. In SEO, it describes how Google and other search engines render JavaScript sites so they can index them. In web scraping, it describes crawlers that must render pages to find the URLs and data they are after.

HTML crawling vs JavaScript crawling

HTML crawling JavaScript crawling
What it reads The server's raw HTML response The DOM after scripts run
Tooling HTTP client plus parser Headless browser, or a rendering API
Speed and cost per page Fast and light Slower and heavier
Sees client-side links No Yes
Sees content loaded by API calls No Yes, if it waits for them
Typical use Server-rendered sites, static pages Single-page apps, React, Vue and Angular sites

How JavaScript crawling works

  1. Fetch the URL in a browser engine instead of a plain HTTP client.
  2. Run scripts and wait until the content you need is present, using wait strategies such as waiting for a selector or for network activity to settle.
  3. Read the rendered DOM and extract links and data.
  4. Normalize, deduplicate and enqueue the new links, as any crawler does.

Google does the same in two waves: it crawls the HTML first, then queues the page for rendering when resources allow, which is why content that appears only after rendering can be indexed later than server-rendered content.

Does a site need JavaScript crawling?

  • View the page source (not the inspector). If the products or links you need are missing, they are rendered client-side.
  • Disable JavaScript in the browser and reload. Whatever disappears needs rendering.
  • Look for an empty <div id="root"> or <div id="app">, the signature of a single-page app.
  • Check the network tab. Data loaded from a JSON API can often be requested directly, which is faster than rendering.

Best practices

  • Render only when needed. Rendering costs far more than a plain request, so detect which pages need it.
  • Use real links when they exist. Links built from onclick handlers without an href are invisible to most crawlers, including Google.
  • Block what you do not need, such as images, fonts and analytics, to speed up rendering.
  • Prefer the underlying API when the page fetches its data from one.
  • Expect more blocking. Headless browsers carry detectable fingerprints, so rendering at volume usually needs stealth measures too.

How Zenrows Crawl handles rendering

Every page Zenrows Crawl opens is fetched by Fetch, and Zenrows picks the fetch settings page by page, including when a page needs JavaScript rendering and how to get past anti-bot protection. You give it any public URL and get back the URLs it finds, with each page's HTML if you want it, without deciding per page whether to render. The Crawl docs describe how a crawl runs.

Frequently asked questions

Can Google crawl JavaScript?

Yes. Googlebot renders pages with an up-to-date Chromium. Rendering can lag behind the initial crawl, and links that are not real <a href> elements are still not followed.

Is JavaScript crawling slower than HTML crawling?

Yes, by a wide margin per page, because the crawler has to download scripts and run a browser engine. Rendering only the pages that need it keeps the total cost reasonable.

What tools are used for JavaScript crawling?

Headless browsers driven by Playwright, Puppeteer or Selenium, crawling frameworks that integrate them, and scraping APIs that render pages on request.

What is the difference between JavaScript rendering and JavaScript crawling?

Rendering is running scripts to produce the final page. JavaScript crawling is a crawl that renders each page so it can discover the links and content that only exist after rendering.

Go deeper on the blog

In the docs

Last updated: Oct 9, 2026

Get reliable web data in minutes.

Free plan, 5,000 credits every month, no credit card required.