What is a reverse proxy?

A reverse proxy sits in front of one or more web servers and handles requests on their behalf. Visitors connect to it, believing they are connecting to the site. It decides what to do with each request: serve it from cache, forward it to an application server, or refuse it.

The direction is what the name encodes. A forward proxy works for the client, hiding who is asking. A reverse proxy works for the server, hiding what is answering.

Why it matters when you are scraping

Nearly every site of any size is behind one, and the practical consequence is that the reverse proxy is usually what blocks you, not the website. The application server may never see your request at all.

This explains several things that are otherwise confusing:

• Blocks arrive instantly, with no page-generation delay, because the decision happens at the edge.
• The error page is generic and branded by the proxy vendor rather than by the site.
• Rate limiting applies across the whole site rather than per endpoint, since one component counts every request.
• Response headers name a CDN rather than the site's own stack.

A 502 or 504 is a reverse proxy telling you it could not get a usable answer from the servers behind it, and Cloudflare's 520 and 522 are the same idea with more precision.

What it does besides blocking

Load balancing across application servers, terminating TLS so the servers behind it do not have to, caching static responses, compressing output, and routing paths to different backends. Anti-bot filtering and web application firewall rules run at this layer too, which is why a CDN and a bot-management product are so often the same vendor.

The naming confusion worth clearing up

When a scraping article says "use a proxy", it means a forward proxy: something you route your own requests through to change the address they appear to come from. When infrastructure documentation says "proxy", it usually means a reverse proxy, part of the site's own stack. The two are unrelated in purpose despite sharing a word, and only the first is something you configure.