What is an HTML parser?

An HTML parser takes a string of markup and produces a tree of nodes you can navigate and query. Without one you have text; with one you have a document, and you can ask it for the third row of a table or every link inside an element.

Why parsing HTML is harder than it sounds

XML parsers reject malformed input. HTML parsers cannot afford to, because a large share of real pages are malformed and browsers render them anyway. Unclosed tags, elements nested in orders the specification forbids, attributes without quotes, stray characters: a parser that refuses these would fail on much of the web.

So HTML parsers implement error recovery, and the HTML5 specification actually defines that recovery precisely, which is why different compliant parsers produce the same tree from the same broken input. Parsers that predate it, or that take shortcuts, do not.

This has a practical consequence: a selector that works in your browser's developer tools can fail in your scraper, because the browser recovered from a markup error one way and your parser recovered differently, producing a different tree from identical bytes.

The common parsers

BeautifulSoup is worth a note because it is a wrapper rather than a parser. Its html.parser, lxml and html5lib backends differ in speed and in how they recover from malformed markup, so switching backends can change your results on a broken page.

What a parser does not do

It does not run JavaScript. A parser receives whatever HTML arrived in the response, and if a page builds its content in the browser after loading, the parser sees an empty shell. This is the most common reason a selector that works in developer tools returns nothing in a script: the tools show the live DOM after scripts ran, while your parser sees the original source.

The fix is fetching the rendered page rather than a better parser.

Where Zenrows fits

If you would rather not run a parser at all, the CSS Extractor applies your CSS or XPath selectors on the Zenrows side and returns JSON, so the parsing happens with the fetch. Where the content only exists after scripts run, JavaScript rendering returns the rendered DOM, which is what your selectors were written against in the first place.