What is a web application firewall?

A web application firewall inspects HTTP requests before they reach the application behind it and refuses the ones matching patterns it considers dangerous. It runs at the reverse proxy layer, so its decision happens before the site's own code sees anything.

A network firewall filters on addresses and ports. A WAF reads the request itself: the URL, the parameters, the headers and the body.

What it is looking for

The core ruleset targets application attacks rather than automation. SQL injection patterns in query parameters, cross-site scripting payloads, path traversal sequences, command injection, and requests to paths associated with known vulnerabilities. Most deployments start from a public baseline such as the OWASP Core Rule Set and add their own rules on top.

Bot management is frequently sold alongside it by the same vendor, and often shares the same edge, which is why the two get conflated. They are separate functions: a WAF asks whether the request is an attack, while an anti-bot system asks whether the visitor is a person.

Why a scraper trips one

Usually by accident, and the causes are worth knowing because they are easy to fix:

• Punctuation in a URL. A search term containing a quote, a semicolon or -- can match an injection signature. Scraped text used to build the next URL is a common source.
• Unencoded characters in a path segment assembled from page content.
• Unusual header values. Very long headers, unexpected header names, or a cookie jar that has grown large across a crawl.
• Paths that look like probing. Requesting /admin, /.env or /wp-login.php while enumerating a site is indistinguishable from a scanner.
• A malformed request from a hand-built HTTP client that a browser would never produce.

Recognising a WAF block

It tends to look different from a bot block. The refusal is instant, since no application code ran. The response is often a plain 403 with a vendor-branded page and a reference or support ID. Crucially, it reproduces exactly: the same request fails every time, and a slightly different request on the same page succeeds.

That reproducibility is the useful diagnostic. A bot block depends on your identity and varies with IP and fingerprint. A WAF block depends on the request content, so it follows the request rather than the client.

What to do about it

Fix the request rather than the identity. Percent-encode everything going into a URL, particularly values taken from scraped text. Keep headers to a normal browser set and watch cookie size on long crawls. Avoid enumerating administrative paths. If one URL pattern fails consistently while its neighbours work, the pattern is matching a rule, and changing proxies will not help.