How do websites detect web scrapers?

Detection works on contradiction. Every request makes claims, in its user agent, its headers and what its JavaScript reports. Every request also produces evidence it cannot easily control: the shape of its TLS handshake, the timing of its connections, the way its graphics stack renders a canvas. When the claims and the evidence disagree, that gap is the signal.

What gives a scraper away

Before any data is sent. The TLS handshake carries a cipher suite list, extension set and ordering that differ between clients. Python's ssl module and Chrome produce recognisably different handshakes, so a request claiming to be Chrome with an OpenSSL handshake is contradicting itself before the first HTTP byte. HTTP/2 settings frames leak the same way.

In the request headers. Real browsers send many headers in a stable order. HTTP libraries send few, in a different order, and often alphabetise them. Missing Sec-Fetch-* headers on a request claiming to be modern Chrome, or an Accept of */* for a page request, are among the easiest tells to check.

Once the page runs. A client that ignores JavaScript never executes the detection script, and never returning a result is itself an answer. A client that does execute it exposes its fingerprint, including whether navigator.webdriver is set and whether properties have been redefined in ways no real browser does.

Over the session. Requests exactly two seconds apart, a navigation order no person would follow, hitting pages with no referrer chain, or never loading a single image or stylesheet. Real browsing is irregular and wasteful; automation tends to be neither.

The honeypot family

Some detection is a trap rather than a measurement. A link hidden with CSS that no visitor can see, a form field positioned off-screen, a URL disallowed in robots.txt and linked nowhere else. A person never touches them. A crawler that follows every link and fills every field walks straight in, and the site now has certainty rather than a probability.

Reading robots.txt and respecting it is worth doing for its own sake, and it also avoids this category of trap entirely.

Why patching one signal rarely works

Each fix tends to create a new contradiction. Overriding the user agent without changing the TLS handshake makes the mismatch worse than leaving it alone. Adding a fake plugins array leaves traces of the override that detection scripts check for directly. Rotating IPs while keeping one fingerprint produces the improbable pattern of many addresses sharing an identical, unusual machine.

Consistency across every layer at once is what actually passes, which is why this is difficult to maintain by hand as detection evolves.