How to Bypass Akamai
Web scraping is an old and still common technique to extract data. Akamai Bot Manager, and other vendors, try to mitigate the problems that scraping might cause. Their goal is to stop attacks such as DDoS or fraudulent authentication attempts. Our goal, for educational purposes, is to bypass Akamai.
Scrapers aren't their primary target, but they might get their content blocked anyways. Differentiating good and bad bots isn't an easy task. Let's see what and how they do it and learn how to bypass Akamai Bot Manager!
What Is a Bot Detection Software
Bot detection, also known as Web Application Firewall (WAF) or anti-scraping protection, is a group of techniques to classify and detect bots.
In the past, the measures involved detecting high-load IPs and checking headers. That allowed the defensive system to block most of the scraping traffic.
As those techniques evolved, so did the scraping world. Many scrapers tried to bypass those measures, causing the Bot-Detection industry to improve.
Years later, anti-scraping software started to include passive and active approaches. Passive as in storing IPs and botnets and identifying them for each request. Active as in monitoring the browser and the user activity or feeding browsing history to machine learning programs.
According to an official white paper (page 13), Akamai was already doing Behavioral Analysis in 2016. They've been evolving since then, as do all the Bot-Detection software vendors. They now offer a bot scoring model for the site owner to fine-tune their blocking aggressiveness.
What Does Akamai Bot Manager Do
Akamai Bot Manager's primary goal is to stop the most dangerous, evasive bots before they erode customer trust. That covers a wide variety of actions, web scraping included.
To achieve their goal, they use a mix of actions. They maintain a bot directory, apply AI to detect new kinds of bots, and, since they are present in many massive-traffic sites, they gain collective intelligence.
All this security also applies to more aggressive attacks: DDoS, fake account creation, or replicating sites with phishing intent. Telling apart the purpose is a crucial aspect.
The two critical aspects of bot management are:
- Distinguish humans from bots.
- Differentiate good bots from bad bots.
Not all bots are malicious. Nobody wants to leave Google out, and they crawl and scrape too. So, how do they do it?
How Does Akamai Bot Manager Work
As mentioned above, they use a wide variety of techniques. We'll explain some of them before we dive into the technical implementation. And learn how to bypass them, of course.
These are the moving parts to bypass Akamai Bot Manager:
- Botnets. Maintain historical data on known bots and feed their system the same range of IPs, common mistakes in user-agent naming, or similar patterns. Any of these might give away a botnet. Once identified and recorded, blocking that pattern feels safe. No humans would browse like that.
- IPs. Blocking IPs might sound like the easiest approach. It's more complex. IPs don't change ownership often, but whoever uses them does. And attackers might use IPs that belong to common ISPs, masking their origin and intent. A regular user might get that IP address a week later. By then, it should access the content without blocks.
- CAPTCHAs. The best procedure for telling humans from bots, right? Again, this is a controversial one. By using CAPTCHAs in every session, a site would drive several users away. Nobody wants to solve them for every other request. As a defensive tactic, CAPTCHAs will only appear if the traffic is suspicious.
- Browser and sensor data. Inject a JavaScript file that will watch and run some checks. Then send all the data to the server for processing. We'll see later how it's done and what data to send.
- Browser fingerprinting. As with the botnets, common structures or patterns can give you away. Scrapers might change and mask their requests. But some details (browser or hardware-related) are hard to hide. And Akamai will take advantage of that.
- Behavioral analysis. Compare historical user behavior on the sites. Checks patterns and common actions. For example, users can visit products directly from time to time. But if they never go to a category or search page, it might trigger an alert.
Scraping detection isn't black or white. Akamai Bot Manager combines all the above and several others. Then, based on the site's setup, it'll decide whether to block a user. Bot detection services use a mix of server-side and browser-side detection techniques.
If we want to skip Akamai Bot Manager, we have first to understand how it uses them. Or else face the "Access Denied Page":
We can only guess how Akamai does the server-side detection. But we can take a look at their client side.
It'll get technical from here. Brace yourselves!
Akamai's JavaScript Challenge Explained
As we can see in the image above, the script triggers a POST request with a huge payload. Understanding this payload is crucial if we want to bypass Akamai Bot Detection. It won't be easy.
Deobfuscate The Challenge
You can download the file here. To see it live, visit KICKZ and look for the file on DevTools. You won't understand a thing, don't worry; that's the idea!
First, run the content on a JavaScript Deobfuscator. That will convert the weird characters into strings. Then, we need to replace the references to the initial array with those strings.
They don't declare variables or object keys with a straight name to make things more complicated. They use indirection: referencing that array with the corresponding index.
We haven't found an online tool that nails the replacement process. But you can do the following:
- Cut the
_acxjvariable from the generated code. - Create a file and place that variable.
- Then the rest of the code is in another variable.
- Replace (not perfect) all references to the array; see the code below.
- Review since some of them will fail.
var _acxj = ['csh', 'RealPlayer Version Plugin', 'then', /* ... */];
const code = `var _cf = _cf || [], ...`;
const result = code
.replace(/\[_acxj\[(\d+)\]\]/g, (_, i) => `.${_acxj[i]}`)
.replace(/_acxj\[(\d+)\]/g, (_, i) => JSON.stringify(_acxj[i]));
It'll need some manual adjustments since it's a clumsy attempt. A proper replacement would need more details and exceptions. Download our final version to see how it looks.
To save you time, we've done that. The original file changes frequently. This result might be different now. But it'll help you understand what data and how they send ID to the server.
Akamai's Sensor Data
You can see above the data sent for processing. We can take as examples the items highlighted in red. From its content, we can guess where the first two come from: user agent and screen size. The third one looks like a JSON object, but we can't know what it represents just by its keys. But let's find out!
The first key, cpen, is present in the script. A quick find on the file will tell us so. So, the line that references it.
var t = [],
a = window.callPhantom ? 1 : 0;
t.push(",cpen:" + a);
What does it mean? The script checks if callPhantom exists. A quick search on Google tells us that it's a feature that PhantomJS introduces, which means that sending cpen:1 is probably an alert for Akamai. No legit browser implements that function.
If you check the following lines, you'll see that they keep sending browser data. window.opera, for example, should never be true if the browser isn't Opera. Or mozInnerScreenY only exists on Firefox browsers. Do you see a pattern? No single data point is a deal breaker (well, maybe the PhantomJS one), but they reveal a lot when analyzed together!
The function called bd generates all these data points. If we look for its usage, we arrive at a line with many variables concatenated. n + "," + o + "," + m + "," + r + "," + c + "," + i + "," + b + "," + bmak.bd(). Believe it or not, but o is the screen's available height.
How can we know that? Go to the definition of the variable. Control + click or similar on an IDE will take you there.
The definition itself tells us nothing useful: o = -1. But look at a few lines below:
try {
o = window.screen ? window.screen.availHeight : -1
} catch (t) {
o = -1
}
And there you have it! You followed what and how Akamai sends browser/sensor data for backend processing.
We won't cover all the items, but you get the idea. Apply the same process for any data point you're interested in.
But the most crucial question is: why are we doing this? 🤔
To bypass Akamai's defenses, we must understand how they do it. Then, check what data they use for that. And with that knowledge, find ways to access the page without blocks.
Mask Your Sensor Data
If all your machines have similar data sent, Akamai might fingerprint it. Meaning that they detect and group them. Same browser vendor, screen size, processing times, and browser data. Is there a pattern? Check your data. They're already doing it.
Assuming that, how to avoid it? There are several ways to mask them, such as Puppeteer stealth.
And how do they do it? It's open source, and we can take a look at the evasions!
There are no evasions for availHeight, so we'll switch to hardwareConcurrency. We picked a simple one for simplicity. Most evasions are more complicated.
Let's say that all your production machines are the same. It's usual: same specs, hardware, software, etc. Their concurrency would be the same, for example, "hardwareConcurrency": 4.
It's just a drop in the ocean. But remember that Akamai Bot Manager processes hundreds of data points. We can make it harder for them by switching some.
// Somewhere in your config.
// There should be a helper function called `sample`.
const options = {hardwareConcurrency: sample([4, 6, 8, 16])};
// The evasion itself.
// Proxy navigator.hardwareConcurrency getter and return a custom value.
utils.replaceGetterWithProxy(
Object.getPrototypeOf(navigator),
'hardwareConcurrency',
utils.makeHandler().getterValue(options.hardwareConcurrency)
)
A proxy acts as an intermediary. In this case, for the function hardwareConcurrency on the object navigator. When called, instead of returning the original, it'll replace it with the one we set in the options. It can be, for example, a random number from a list of typical values.
What do we get with this approach? Akamai would see different values for hardwareConcurrency sent randomly. Assuming we do it for several parameters, it's hard to see a pattern.
Isn't this a complicated process for Akamai to run on each visit? The good part for everyone is that they do it only once. Then sets cookies to avoid running all the processes again.
Cookies to Avoid Continuous Challenges
Why is it good for you? Once obtained the cookies, the next requests should go unchecked. It means that those cookies will bypass Akamai WAF!
We suggest using the same IP to simulate an actual user session for security.
The standard cookies used by Akamai are _abck, ak_bmsc, bm_sv, and bm_mi. It's not easy to find information about these. Thanks to cookie policies, some sites list and explain them.
Notice that ak_bmsc is HTTP-only. That means that you can't access its content from JavaScript. You'll need to check the response headers on the sensor data call. You can check the headers or call document.cookie on the browser for the others.
And that cookie content is critical! The sensor call will allow or not your request and generate that cookie for your session. Once obtained, send it every time to avoid new checks.
Wow, that was a lot to take on.
Let's now see how to skip Akamai WAF by applying the points we just learned.
How to Bypass Akamai Bot Manager
Here are some typical practices that will help when scrapping Akamai-protected sites, and we'll bypass all. The threshold is different for them all. Some might be more aggressive. We'll talk about them later.
- Follow robots.txt.
- Good Rotating Proxies.
- Use headless browsers.
- HTTP headers.
- Stealth techniques for headless browsers.
- Don't contradict the JavaScript challenge.
Follow robots.txt
It might sound obvious and simple. And it is. But scraping content blocked by robots.txt might flag your scrapper. And, if Akamai finds a pattern, all your requests might get blocked after some time. Remember that Akamai Bot Manager learns from the visitors and their actions.
Be extremely careful with honeypots. These are pages designed to lure malicious bots, for example, through invisible links that a human would never follow. They can flag you or slow down your process.
Good Rotating Proxies
A good IP address might not bypass Akamai defenses, but a bad IP will automatically be rejected. The same goes for proxies that don't change their IP. After some requests, its Bot Manager will block it. It's just a matter of time.
Good proxies have two properties (non-exclusive):
- Rotating. They change IPs with each request or after a short time.
- Origin. It determines where those IPs come from. Akamai Bot Manager will have a hard time blocking an IP used by an ordinary 4G provider. Hard to tell a bot from a human. Usual examples are data center or residential.
Use Headless Browsers
As we've seen, passing the JavaScript challenge is a must. And you need a browser to pass it. Static scraping (cURL, Axios in JS, or Requests in Python) won't work long-term. Some sites might work a few times but not reliably.
Enter headless browsers, for example, Selenium. They execute a real browser, like Chrome, without the graphical interface. It means that they'll download and run the Akamai challenge.
Combined with good proxies, it'll look like an actual user. Still, bot detection software can tell them apart.
HTTP Headers
Browsers send a set of headers by default, be careful when changing them. If you add the same headers for Chrome and Safari, Akamai can tell that something is wrong.
Change only the least you need. And always check that the whole set makes sense. Let's say you want to add Google as a referrer. You could add a referer: https://www.google.com/.
Simple, right? Well, not so fast. By default, Chrome would send sec-fetch-site: none. But if coming from Google, the browser will add sec-fetch-site: cross-site.
A small detail, yes. But something that a real browser would never fail to do.
Stealth Techniques for Headless Browsers
We already saw how these work above. These practices consist of modifying how the browser responds to specific challenges. Akamai's detection software will query the browser, hardware, and many other things. The stealth mode overwrites some of them to avoid patterns and fingerprinting. It adds variation and masks the browser.
Some of these actions are detectable. You'll have to test and check for the right combination. Each site can have a different configuration.
The good thing is that you already understand how Akamai decides which requests to block. Or at least you know what data they gather for that.
Don't Contradict the JavaScript Challenge
Again, it might sound obvious, but some data points might be derived from another one or duplicated. If you decide to mask one and forget the other, the Akamai bot detection will see it. And take action, or at least mark it internally.
In the sensor data example image, we can see that it sends window size. Most data points are related to the actual screen and the available inner and outer sizes.
Inner, for example, should never be bigger than outer. Random values won't work here. You'd need a set of actual sizes.
The Easiest Way to Bypass Akamai
Sometimes, letting others do the heavy lifting is a good plan. ZenRows is designed to bypass Akamai Bot Manager or any other anti-scraping system. It allows you to scrape content without worrying about skipping Akamai or others on your own. ZenRows offers API and Proxy modes, choosing the best fit for you.
Focus on building your data extraction system. Get the most out of your data. And forget about handling all the complicated parts we talked about.
Conclusion
It's been quite a ride! We know it went deep, but bot detection is a complex topic.
The main points to bypass Akamai bot detection or other defensive systems are:
- Good proxies with fresh and rotating IPs.
- Follow robots.txt.
- Use headless browsers with stealth mode.
- Understand Akamai's challenges so that you can adapt the evasions.
For updated info, you can check their site. It explains, not in depth, what aspects they cover and some general information. Not much, but it can guide you if anything changes.
If you liked it, you might be interested in our article about how to bypass Cloudflare.
Thanks for reading! We hope that you found this guide helpful. You can sign up for free, try ZenRows, and let us know any questions, comments, or suggestions.
Frequent Questions
How to Bypass Akamai’s Rate Limit?
You can bypass Akamai’s rate limit using practices such as rotating proxies, changing HTTP headers, and stealthy headless browsers. However, these approaches are unreliable as Akamai learns from its visitors, therefore web scraping APIs like ZenRows can help you better.
What Does Akamai Do Exactly?
Akamai’s primary goal is to protect its users against threats such as DDoS attacks and malicious bot traffic. Its other services include content delivery, cloud security, edge computing, etc.
How Does Akamai Bot Manager Work?
Akamai Bot Manager identifies and blocks malicious bots, and works using various techniques: like IP blocking, browser fingerprinting, behavior analysis, sensor data analysis, CAPTCHAs, etc.
How Can You Tell If a Site Uses Akamai?
You can tell if a site uses Akamai by inspecting its source code and response headers. Look for references such as Akamai or edgekey. You can also send a request using Pragma Headers to see if the URL can be cached on Akamai.
However, not all Akamai use has apparent indicators. Some use anonymous or obfuscated scripts that may require reverse engineering.
Is Akamai a Firewall?
Yes, Akamai offers Web Application Firewall (WAF) services, including bot management and protection against DDoS attacks, cross-site scripting (XSS), SQL injection, and more.
Did you find the content helpful? Spread the word and share it on Twitter, or LinkedIn.