How do I extract only the main content of a page?

TL;DR

Two approaches, and which one fits depends on how many sites you have. For a known list, a selector per site is exact and cheap. For an open-ended set, a readability algorithm scores blocks by how much text they contain relative to how many links, and keeps the winner. Both leave residue, so strip the usual suspects afterwards regardless.

Why converting to Markdown is not enough

Converting a page to Markdown removes markup. It does not remove the navigation menu, the footer, the cookie banner, the newsletter prompt or the related-article rail, all of which convert into perfectly clean Markdown that is still not the article.

In a retrieval system this is actively harmful. Boilerplate repeats on every page of a site, so every chunk containing it resembles every other, and retrieval starts matching on shared furniture instead of subject matter.

Approach one: a selector per site

If you know your sources, find the container once and use it. article, main, [role="main"] or a site-specific class covers most pages, and many sites are consistent enough that one selector serves the whole domain.

Exact, free to run, and it breaks on redesign like any other selector. Fine for ten sites, unmanageable for a thousand.

Approach two: readability scoring

Readability algorithms, the family descended from Arc90's Readability and used in browser reader modes, work without knowing the site. The scoring is roughly:

• Text length per block, since real content is long and menus are short.
• Link density, since a block that is mostly links is navigation.
• Tag semantics, since article and p count for more than nav and aside.
• Comma and sentence counts, since prose has punctuation that menus do not.

The highest-scoring subtree wins. This works well on articles and documentation, and less well on pages that are legitimately list-shaped, such as product grids or search results, where it may discard the very thing you wanted.

Cleaning up afterwards

Both approaches leave residue worth removing:

• Cookie and consent banners, which often sit inside the main container.
• Inline newsletter and subscription prompts.
• Social sharing rows.
• "Related" and "You may also like" blocks at the end.
• Advertisement placeholders, whether or not they filled.

A short blocklist of selectors and text patterns, applied after extraction, handles most of these and is worth maintaining as you notice new ones.

How to know it worked

Compare extracted length against page length. A ratio that is very low means you probably grabbed the wrong container; very high means you kept the boilerplate. Spot-checking a sample by eye once per source is worth more than any automated check here, because the failure is obvious to a person and subtle to a script.

Where Zenrows fits

Output filters return only chosen data types from a page, such as headings or links, which handles some cases without any content-scoring at all. Markdown response removes the markup layer at fetch time, and the CSS Extractor covers the per-site approach by taking your container selector and returning just what it matched.

Key takeaways

Pick per-site selectors when the source list is known and readability scoring when it is not. Either way, strip banners, prompts and related-article blocks afterwards, and check the ratio of extracted text to page text as a cheap signal that extraction grabbed the right container.