What is a 406 error?

A 406 Not Acceptable means the server performed content negotiation and could not produce anything that satisfied the Accept headers on your request. You asked for a format it does not serve, so it refused rather than sending something you said you would not take.

In ordinary web use this is rare. In scraping it turns up far more often than the specification would suggest, and usually for a different reason than the one the code describes.

Why scrapers see it more than browsers do

Content negotiation covers four request headers: Accept for media types, Accept-Language, Accept-Encoding and the largely obsolete Accept-Charset. A real browser sends all of them, with long and specific values. Chrome's Accept header for a page request is a lengthy list with quality weightings.

An HTTP library sends almost nothing. Python Requests defaults to Accept: */*, which technically means "anything is fine" and should never produce a 406. Some WAF rules treat it as a signal anyway, because no browser sends it for a document request.

So a 406 in scraping splits into two cases:

• A genuine mismatch. You requested application/json from an endpoint that only serves HTML, or the reverse. Real, and fixed by asking for what the server actually has.
• A header-shaped block. The server serves HTML perfectly well, and is using the 406 as a polite way to refuse a request whose headers do not look like a browser's. More common than the first case.

Telling them apart

Send a full set of browser headers and try again. If the 406 disappears, it was never about content negotiation. If it persists, look at what you are actually asking for: an Accept header naming a type the endpoint does not produce will keep failing no matter how convincing the rest of the request looks.

It is also worth checking Accept-Encoding. Advertising an encoding you cannot decompress, or omitting it when every browser sends it, is another combination that some rules flag.

Getting the headers right

Copy a complete header set from a real browser rather than assembling one by hand, and keep it internally consistent. An Accept-Language of en-US arriving from an IP in Germany, or a Chrome user agent without any of Chrome's Sec-Fetch-* headers, is the sort of mismatch that turns a 406 into a permanent one.

Where Zenrows fits

Header consistency is handled for you on Zenrows Fetch, which sends a coherent browser header set by default rather than a library's defaults, so the header-shaped variety of 406 does not arise. Where you genuinely need to negotiate a format, custom headers let you set Accept explicitly and the rest stays consistent around it.