Web data extraction
Once you have the HTML, you still have to find the parts you want inside it. That has traditionally meant selectors written against a specific page structure, which break the moment the site is redesigned. Schema-based and model-driven extraction change the tradeoff by describing the fields you want instead of where they sit. The terms below cover both approaches, the formats involved, and how pages get cleaned up before a model reads them.
How do I extract only the main content of a page?
Use a readability algorithm to score blocks by text density and link ratio, or write a selector per site when you control the list, then strip what survives.
JSON, JSONL, CSV or XML: which output format?
JSON handles nested records, JSONL adds streaming and append-safety for large runs, CSV is flat and universally readable, and XML is mostly encountered rather than chosen.
Structured or unstructured data: what is the difference?
Structured data has a defined schema so a program knows what each value means, while unstructured data carries meaning only in a form a human or a model has to interpret.
What is a CSS selector?
A CSS selector is a short pattern that identifies elements in a page by tag, class, attribute or position, and it is the usual way to tell a scraper which part of the document you want.
What is an HTML parser?
An HTML parser reads markup and builds a tree you can query, handling the broken and malformed HTML that real websites are full of.
What is an XPath selector?
XPath is a query language for navigating a document tree that can move in any direction and match on text content, which lets it express selections CSS cannot.
What is HTML to Markdown conversion?
It converts a page's markup into Markdown, keeping the structure that carries meaning and discarding the rest, which typically cuts the size by an order of magnitude.
What is LLM-ready content?
LLM-ready content is a page reduced to what a model can actually use: main content only, structure preserved, markup and boilerplate removed, small enough to fit in a context window.
What is schema-based extraction?
Schema-based extraction means describing the fields you want and letting the extractor find them, instead of writing selectors that point at specific places in the HTML.
What is the Document Object Model?
The DOM is the live tree a browser builds from a page, which scripts can change, so it often differs from the HTML the server originally sent.
What is web data extraction?
Web data extraction is the step that turns a fetched page into structured records, taking HTML meant for human eyes and producing fields a program can use.
Get reliable web data in minutes.
Free plan, 5,000 credits every month, no credit card required.