Are your web scraping requests getting blocked by Akamai? Akamai's security system is so strong that you can only bypass it with the correct blueprint.
We've looked deep into Akamai's defenses and devised three ways to bypass them. In this article, you'll learn to use these solutions to bypass Akamai and avoid getting blocked during web scraping.
- Method 1: Use ZenRows Web Scraper API.
- Method 2: Use a Fortified Headless Browser.
- Method 3: JavaScript Deobfuscation.
What Is Akamai Bot Manager?
Akamai Bot Manager is an advanced security measure designed to detect, classify, and manage website bot traffic. It maintains a directory of known bots, allowing it to differentiate between legitimate and malicious bots in real-time using an AI-powered detection framework that learns bot-like patterns over time.Â
While analyzing incoming traffic, Akamai returns a trust score based on learned behavioral patterns, HTTP request characteristics, and other historical data. Akamai-protected websites typically use this score to determine how to handle requests. Unfortunately, this configuration often blocks web scrapers, especially if their behavior mimics that of malicious bots.
How Does Akamai WAF Detect Bots?
Akamai's web application firewall (WAF) uses various bot-detection techniques, which makes bypassing it more challenging. Understanding these detection methods will help you to handle the anti-bot efficiently during web scraping.Â
So, before implementing the Akamai bypass methods, let's explain how it detects your web scraper.
1. IP Reputation
One of Akamai's detection methods is detecting suspicious IP addresses. If your IP has been banned in the past due to rate-limiting violations from multiple scraping activities or geo-blocking, Akamai can detect and block you easily.
The bot manager also maintains a negative trust score for certain IP types, particularly shared and datacenter IPs, as regular users don't typically use those. Residential IPs are better than datacenter IPs since they belong to real network users.
When you send a request to an Akamai-protected site, the bot manager scans your IP against a database of disallowed and trusted IP addresses to determine whether your traffic is genuine.
2. TLS Fingerprinting
TLS (Transport Layer Security) fingerprinting is one of the advanced methods Akamai uses to detect secure client-server connections. When a client (like a browser or a scraper) connects to a web server, the communication starts with a TLS handshake.
During this handshake, the client and server exchange information, such as the cipher suite (the encryption methods), TLS version, and extensions. Each client sends a unique set of information during the handshake, forming a profiling method called a JA3 fingerprint. Akamai compares this fingerprint to a database of known, trusted ones and often blocks requests that don't match.
Even though most web scraping tools can create secure TLS connections, other issues like incorrect HTTP headers or unnatural browsing behavior can still cause the bot manager to flag a request.
3. Behavior Analysis
To differentiate between bots and humans, Akamai also analyzes user behavioral patterns, such as navigation, clicking, hovering, scrolling, typing, mouse movement, and more.
The anti-bot can detect if a request is from an automated script if it spots a consistent interaction pattern, such as scrolling the same height repeatedly and rapidly within a short time or clicking the same element multiple times in a row.
Unlike bots, human interactions are typically dynamic and unpredictable, making it easy for Akamai to distinguish bots from real users. For example, users can visit products directly from time to time. But if they never go to a category or search page, it might trigger an alert.  Â
4. JavaScript Challenge and Execution
When a client visits a website protected by Akamai, the system may prompt it to run a client-side JavaScript in the background within a specified time frame. The result of this script helps determine whether the client is legitimate. Akamai may block the request or trigger a CAPTCHA if the script is executed incorrectly or if it returns a suspicious result.
Regular HTTP clients, such as Python's Requests and JavaScript's Axios, typically fail this challenge because they can't execute JavaScript. Even browser automation tools like Puppeteer and Playwright often get detected because their fingerprints are incomplete and unrealistic enough to meet the demands of solving the challenge correctly.Â
That said, Akamai uses other detection mechanisms, such as session monitoring (to track session cookies), honeypots, browser fingerprinting, direct HTTP request header analysis, and more. However, the ones discussed above are the most important.
In the next section, we'll understand the three techniques to bypass Akamai Bot Manager.
Method 1: Use a Web Scraping API for Akamai Bypass
The easiest way to avoid Akamai's security measures while web scraping is to use web scraping APIs. One of the top solutions is the ZenRows scraper API, which provides all the essential toolkits to bypass sophisticated anti-bots like Akamai at scale.Â
The ZenRows scraper API helps you handle all the complexities of fingerprinting evasion, premium proxy rotation, JavaScript execution, actual user spoofing, request header management, anti-bot and CAPTCHA bypass under the hood, allowing you to focus on your data extraction logic.
ZenRows' implementation is straightforward. It only requires sending a single API call with your chosen programming language, making it the easiest option, especially for beginners. Let's see how it works by scraping the full-page HTML of similarweb, an Akamai-protected website.
Sign up for free to open the Request Builder. Paste the target URL in the link box, and activate Premium Proxies and JS Rendering.
Select your programming language (Python, in this case) and choose the API connection mode. Then, copy and paste the generated code into your scraper file.
Here's what the generated Python code looks like:
# pip install requests
import requests
url = "https://www.similarweb.com/"
apikey = "<YOUR_ZENROWS_API_KEY>"
params = {
"url": url,
"apikey": apikey,
"js_render": "true",
"premium_proxy": "true",
}
response = requests.get("https://api.zenrows.com/v1/", params=params)
print(response.text)
The code above outputs the protected site's full-page HTML, as shown:
<html lang="en">
<head>
<!-- ... -->
<title>
Similarweb Digital Intelligence: Unlock Your Digital Growth
</title>
<!-- ... -->
</head>
<body>
<!-- ... -->
</body>
</html>
That was super-easy 🎉! Using a few code lines, you just scraped an Akamai-protected website with the ZenRows scraper API.
While this technique remains the best ready-made solution, there are other methods if you want to do it yourself.
Method 2: Use a Fortified Headless Browser
Browser automation libraries like Puppeteer, Playwright, and Selenium let you perform user interactions, execute JavaScript, and run the browser in headless and GUI modes.
Unfortunately, these standard browser automation tools can't bypass Akamai because they expose bot-like signals, such as the HeadlessChrome
flag in headless mode. They also present browser fingerprinting issues like the presence of a WebDriver, missing plugins, and more.
The good news is that you can fortify them with evasion tweaks to increase the chances of bypassing anti-bot detection. That's where fortified headless browsers come in. They patch botlike properties such as the browser runtime and other automation flags, allowing you to run the browser instance like a regular browser.
Common fortified headless browsers include:
- Puppeteer Stealth: A stealth plugin with anti-bot evasions for Puppeteer.Â
- Playwright Stealth: To bypass anti-bots in Playwright.
- SeleniumBase and Undetected ChromeDriver (UC): To bypass Akamai with Selenium.Â
Let's use SeleniumBase with Undetected ChromeDriver in Python to bypass the previous target website (similarweb).Â
First, install the library using pip
:
pip3 install seleniumbase
The following code uses SeleniumBase with UC to run the browser in headless mode. It then visits the protected website and takes a screenshot:
# pip3 install seleniumbase
from seleniumbase import Driver
# initialize driver with UC mode enabled
driver = Driver(uc=True, headless=True)
# set target URL
url = "https://www.similarweb.com/website/facebook.com/"
# open URL using UC mode with 4 second reconnect time to bypass initial detection
driver.uc_open_with_reconnect(url, reconnect_time=4)
# screenshot the page
page_html = driver.save_screenshot("similarweb.png")
print(page_html)
# close the browser and end the session
driver.quit()
The above returns a screenshot of the protected site's page, indicating that SeleniumBase bypassed Akamai:
Awesome! You bypassed Akamai with SeleniumBase.Â
Want to learn about bypassing Akamai with the other headless browsers? Check out our tutorial on bypassing Akamai with Playwright.Â
You can also read how to avoid Akamai detection in Puppeteer if you're using Puppeteer for scraping.
However, open-source stealth tools like SeleniumBase and Undetected ChromeDriver aren't suitable for large-scale scraping. They rely on browser instances, which results in huge memory overhead.Â
Since they're open-source, they also struggle to keep up with the evolving complexity of anti-bots like Akamai. So, they don't guarantee sustained success.
Method 3: JavaScript Deobfuscation (Advanced)
As we can see in the image below, the Akamai script triggers a POST request with a considerable payload. Understanding this payload is crucial to bypass Akamai Bot Detection. However, it won't be easy.
As we can see in the image below, the Akamai script triggers a POST request with a considerable payload. This payload is obfuscated, and understanding it is crucial to bypass Akamai Bot Detection. However, it won't be easy.
You can download the obfuscated script here. To see it live, visit KICKZ and look for the file on DevTools. You won't understand a thing, don't worry; that's the idea of obfuscation!
First, run the above script on a JavaScript Deobfuscator to convert the weird characters into strings. Then, we need to replace the references to the initial array with those strings.
To complicate things, the Akamai script doesn't declare variables or object keys with a straight name. It uses indirection: referencing an array with the corresponding index.
We haven't found an online tool that nails the replacement process. But you can do the following:
- Cut the _acxj variable from the generated code.
- Create a file and place that variable inside it.
- Then, add the rest of the code to another variable.
- Replace (not perfect) all references to the array; see the code below.
- Review since some of them will fail.
var _acxj = ['csh', 'RealPlayer Version Plugin', 'then' /* ... */];
const code = `var _cf = _cf || [], ...`;
const result = code
.replace(/\[_acxj\[(\d+)\]\]/g, (_, i) => `.${_acxj[i]}`)
.replace(/_acxj\[(\d+)\]/g, (_, i) => JSON.stringify(_acxj[i]));
The code requires some manual adjustments, as the approach above is clumsy. A proper replacement would need more details and exceptions.
We've done that to save you time. Download our final version to see how it looks. The original file changes frequently, and the result might be different now. But it'll help you understand what data and ID Akamai sends and how it sends it to the server.
Akamai's Sensor Data
See the data sent for processing below. Take the items highlighted in red as examples. We can guess where the first two come from: user agent and screen size. The third one looks like a JSON object, but we can't know what it represents just by its keys. But let's find out!
The first key, cpen, is present in the obfuscated script. A quick look at the deobfuscated file will tell us so. Here's the line that references it:
var t = [],
a = window.callPhantom ? 1 : 0;
t.push(',cpen:' + a);
What does it mean? The script checks if callPhantom
exists. A quick search on Google tells us that it's a feature that PhantomJS introduced to communicate with browser contexts. So, sending cpen:1 is probably an alert for Akamai because no legitimate browser implements that function.
You'll see that Akamai keeps sending browser data even in the next lines of the payload. window.opera
, for example, should never be true if the browser isn't Opera. Similarly, mozInnerScreenY
only exists on Firefox browsers. Do you see a pattern? No single data point is a deal breaker (well, maybe the PhantomJS one), but they reveal a lot when analyzed together!
The function called bd
generates all these data points. If we look for its usage, we arrive at a line with many concatenated variables, as shown:
n + "," + o + "," + m + "," + r + "," + c + "," + i + "," + b + "," + bmak.bd().
Believe it or not, but o
is the screen's available height.
How can we know that? Go to the definition of the variable. Control + click or similar on an IDE will take you there.
The definition itself tells us nothing useful: o = -1
. But look at a few lines below:
try {
o = window.screen ? window.screen.availHeight : -1;
} catch (t) {
o = -1;
}
There you have it! You followed what browser/sensor data Akamai sends and how it sends it for backend processing.
We won't cover all the items, but you get the idea. Apply the same process for any data point you're interested in.
However, the most crucial question is: why do this? 🤔
To bypass Akamai's defenses, you must understand how it works by checking what data it uses for detection. With that knowledge, you can find ways to access an Akamai-protected page without blocks.
Mask Your Sensor Data
If all your machines send similar data during web scraping, Akamai might fingerprint them, allowing it to detect and group them. Using the same browser vendor, screen size, processing times, and browser data for all machines reveals a pattern.
You can avoid this pattern by improving the evasions of Puppeteer Stealth. Take a look at the Puppeteer Stealth evasions for inspiration!Â
There are no evasions for availHeight
, so we'll switch to hardwareConcurrency
for simplicity.Â
Let's say all your production machines share the same specs, hardware, software, etc. Their concurrency would be the same, for example, hardwareConcurrency: 4
.
The hardwareConcurrency
is only a fraction of the available evasions. But remember that Akamai Bot Manager processes hundreds of data points. You can make your scraper harder to detect by tweaking some of these evasions.
For example, the code below spoofs different hardware concurrency values for different machines:
// somewhere in your config.
// there should be a helper function called `sample`.
const options = { hardwareConcurrency: sample([4, 6, 8, 16]) };
// the evasion itself.
// proxy navigator.hardwareConcurrency getter and return a custom value.
utils.replaceGetterWithProxy(
Object.getPrototypeOf(navigator),
'hardwareConcurrency',
utils.makeHandler().getterValue(options.hardwareConcurrency)
);
In this case, the proxy is an intermediary for the hardwareConcurrency
function on the object navigator. When called, instead of returning the original, it'll replace it with the one we set in the options. It can be, for example, a random number from a list of typical values.
What do we get with this approach? Akamai would see different random values for hardwareConcurrency
. It's hard to see a pattern if we do it for several parameters.
Isn't this a complicated process for Akamai to run on each visit? The good part for everyone is that Akamai does it only once and sets cookies to avoid running all the processes again.
Cookies to Avoid Continuous Challenges
Why is setting cookies good for you? Once you obtain the cookies, you can use them to bypass Akamai WAF when making subsequent HTTP requests!
For security reasons, we suggest using the same IP to simulate an actual user session.
The standard cookies used by Akamai are _abck
, ak_bmsc
, bm_sv
, and bm_mi
. It's not easy to find what these mean. Thanks to cookie policies, some sites list and explain them.
Note that ak_bmsc
is HTTP-only. That means that you can't access its content from JavaScript. You'll need to check the response headers on the sensor data call. You can check the headers or call document.cookie
on the browser for the others.
That cookie content is critical! The sensor call will allow your first request and generate that cookie for your session. Once obtained, send it every time to avoid new checks.
Conclusion
In this article, you've learned three ways to bypass Akamai and scrape without getting blocked. Although Akamai employs several detection mechanisms, you can avoid them using a scraper API or manual approaches, including fortified headless browsers and JavaScript deobfuscation.
However, the manual methods are complex and often unreliable at scale. The easiest solution to bypass Akamai is to use the ZenRows Scraper API, which provides an all-in-one toolkit for scraping any website without limitations.
Try ZenRows for free today without a credit card!
Frequent Questions
How to Bypass Akamai's Rate Limit?
You can bypass Akamai's rate limit using stealth headless browsers and JavaScript deobfuscation. However, these approaches are unreliable as Akamai learns from its visitors. Web scraping APIs like ZenRows can help you better.
What Does Akamai Do Exactly?
Akamai's primary goal is to protect its users against threats such as DDoS attacks and malicious bot traffic. Its other services include content delivery, cloud security, edge computing, etc.
How Does Akamai Bot Manager Work?
Akamai Bot Manager identifies and blocks malicious bots and works using techniques like IP blocking, browser fingerprinting, behavior analysis, sensor data analysis, CAPTCHAs, etc.
How Can You Tell If a Site Uses Akamai?
You can tell if a site uses Akamai by inspecting its source code and response headers. Look for references such as Akamai or edgekey. You can also send a request using Pragma Headers to see if the URL can be cached on Akamai.
However, not all Akamai use cases have apparent indicators. Some use anonymous or obfuscated scripts that may require reverse engineering.
Is Akamai a Firewall?
Yes, Akamai offers Web Application Firewall (WAF) services, including bot management and protection against DDoS attacks, cross-site scripting (XSS), SQL injection, and more.