Axios: Retry Failed Requests [2024]

HTTP requests can temporarily fail, so it's essential to handle those to ensure reliability.

In this tutorial, you'll learn how to implement the Axios retry feature: which failed requests that can be retried, how different retry mechanisms work, as well as the strategies to use with examples.

Introduction: What to Know to Build the Axios Retry Logic

There are common questions that come to mind when considering building an Axios retry logic for your web scraper in NodeJS. Some of them are: Should I always retry failed requests or just in some cases? When should I retry? How many attempts should I make? We'll answer them in this section.

Types of Failed Requests

When a request is not successful, there are two common scenarios: cases that time out with no response from the server and cases that return an error. Let's look at both scenarios to better understand what strategies are available to deal with each one.

Timed out

Issues such as overloaded servers or your internet speed can cause a delayed response from the server. In the event of a timeout, it's a good idea to double-check your internet connection.

If your internet is stable, the issue is likely to lie with the server. To prevent unnecessarily long connection requests, it's a best practice to set a timeout in milliseconds to abort the request using the Axios timeout config options.

// npm install axios
// CommonJS
// const axios = require('axios');
 
// ES6
import axios from 'axios';
 
axios.get('https://httpbin.io/uuid', {
  // If the request takes longer than 15000ms (15 seconds), the request will be aborted.
  timeout: 15000,
})
  .then((response) => console.log(response.data))
  .catch((error) => console.log(error));

To detect a timeout, it's important to spot timeout-related errors (those with the ECONNABORTED code) and implement an Axios retry mechanism either conditionally or with strategies like exponential backoff, which we'll touch on later. Here's how to handle a timeout error in Axios:

// ...
 
axios.get('https://httpbin.io/uuid', {
  // Timeout is set to 10ms (0.01 seconds), the request will be aborted.
  timeout: 10,
})
  .then((response) => console.log(response.data))
  .catch((error) => {
    if (error.code === 'ECONNABORTED') {
      // Timeout Error (You can implement retry here where suitable)
      console.log('Timeout Error: ', error.message);
    }
  });

Returned an Error

A standard error response to a failed request includes a status code and a descriptive error message. The status code indicates what went wrong, while the error message provides more context for why the request failed.

To address this scenario, check if the request displays an error. Then, take a look at the response status code and message to see what went wrong. You can then retry the request with Axios if the error code indicates a temporary server issue.

Error Status Codes

Client-server communication errors have status codes that fall in the 4xx and 5xx ranges. The most common ones you'll want to retry with Axios when scraping include the following ones:

You can check out the MDN docs for more information on HTTP response status codes.

Number of Retries

It's advisable to impose an Axios retry limit to prevent unnecessary performance and blocking issues arising from an infinite number of retries. Also, when determining the optimal number of retries to make for a failed request, take into account factors like the type of request error and response time.

Temporary errors, such as 429 Too Many Requests, should have more retries than others (in this case, with a different IP). While there's no best number of retries, three to five retries are good numbers to use as a starting point.

Delay

Setting delays between requests is one way to prevent websites and APIs from becoming overloaded, which can result in you being rate limited. Let's see the types of delays you can use and how to code them later on.

Fixed or Random Delays

When you want to set a fixed delay between requests, the setTimeout() global method comes in handy. And to make the delay random, you can use the setTimeout() function with the Math.random() static method together.

There's no hard and fast rule for how long the delay should be. However, you can try out a range of reasonable delay values starting at 300ms until you find what's right for your system or application.

Exponential Delays (Backoff Strategy)

The backoff strategy is a common way to set delays between retries that get longer over time instead of being fixed or random. Each request adds an exponential backoff factor to the delay, which is usually more than one. This strategy gives room for temporary issues to resolve themselves while preventing server overload.

The backoff algorithm is given below:

backoff_factor * (2 ** (current_number_of_retries - 1))

For example, here are the delay sequences for backoff factors 2, 3, and 10:

# 2
1, 2, 4, 8, 16, 32, 64, 128
 
# 3
1.5, 3, 12, 24, 48, 96, 192, 384
 
# 10
5, 10, 20, 40, 80, 160, 320, 640

Tutorial: How to Retry Axios with Interceptors

The axios-retry plugin provides an easy way to implement interceptors to retry Axios requests. It makes retrying failed requests easier by providing a pre-configured interceptor that handles retries automatically based on configurable options.

It's an NPM package, so start by installing it with the command below:

npm install axios-retry

1. Retry on Error with Axios

To retry your failed Axios request, first import the default export interceptor function of the axios-retry plugin. Then, pass your Axios instance and an optional object configuration as its first and second arguments, and call it.

// Retry interceptor function
import axiosRetry from 'axios-retry';
// Default axios instance
import axios from 'axios';
 
// Pass the axios instance to the retry function  and call it
axiosRetry(axios, { 
   retries: 3, // Number of retries (Defaults to 3) 
});
 
// Make Axios requests below
axios.get('https://scrapingcourse.com/ecommerce/') // The request is retried if it fails
  .then((response) => {
    console.log('Data: ', response.data);
  }).catch((error) => {
    console.log('Error: ', error);
  });

Note that, by default, axios-retry will only retry failed Axios requests that occur due to a network error or are idempotent (GET, HEAD, OPTIONS, PUT, or DELETE) with a response status code in the 5xx range. This means that requests that return an error code like 400 or 429 won't be retried automatically. 

To get around that, you can use the retryCondition option to conditionally tell axios-retry to retry a request based on the errors we want. Here's an example:

// ...
 
axiosRetry(axios, { 
  retries: 3, // Number of retries
  retryCondition(error) {
    // Conditional check the error status code
    switch (error.response.status) {
      case 404:
      case 429:
        return true; // Retry request with response status code 404 or 429
      default:
        return false; // Do not retry the others
    }
  },
});
 
// ...

2. Implement an Exponential Backoff Strategy with Axios Interceptors

The axios-retry plugin has an exponentialDelay method that you can pass to the retryDelay option to implement an exponential delay. Here's how:

// ...
 
axiosRetry(axios, { 
  retries: 3, // Number of retries
  retryDelay: axiosRetry.exponentialDelay, // Exponential back-off retry delay between requests
});
 
// ...

The delay or backoff factor by default is 100ms, but this can be configured by adding an extra argument to exponentialDelay. For example, a backoff factor of 1000ms:

// ...
 
axiosRetry(axios, { 
  retries: 3, // Number of retries
  retryDelay: (...arg) => axiosRetry.exponentialDelay(...arg, 1000), // Exponential delay with backoff of 1000ms
});
 
// ...

3. Learn the retry-axios Options

As a recap and expansion, these are the options to retry Axios requests:

Avoid Getting Blocked with retry-axios

The biggest issue when crawling is being blocked due to being recognized as a bot. If a website determines that your request is automated, it may block your IP address or take other measures to avoid giving you access to the site.

Let's try scraping a G2.com secure page to demonstrate that:

import axios from 'axios';
 
axios.get('https://www.g2.com/products/notion/reviews')
  .then((response) => console.log('Response: ', response.data))
  .catch((error) => {
    // If error, log the response data and status code
    console.log('Error: ', error.response.data, error.response.status)
  });

If you run the code, you'll see a 403 error message similar to the one shown below:

   
<!-- // ... -->
   <div class="cf-main-wrapper" role="main">
      <div class="cf-header cf-section">
         <div class="cf-error-title">
            <h1>Access denied</h1>
            <span class="cf-code-label">Error code <span>1020</span></span>
         </div>
         <div class="cf-error-description">
            <p>You do not have access to www.g2.com.</p><p>The site owner may have set restrictions that prevent you from accessing the site.</p>
         </div>
      </div>
   </div>
 
<!-- // ... -->
 
403

Web scraping proxies can be useful in situations like the one we just encountered. Unfortunately, proxies aren't enough for many websites since they aren't likely to be able to keep up with ever-changing anti-bot systems.

Web scraping APIs are a dedicated toolkit for avoiding being blocked that is as simple to use as making a single API call. They often have features such as premium proxies, rotating User-Agent, and so on.

ZenRows is a popular and excellent web scraping API that can be integrated with Axios. Start by signing up for ZenRows to get your free API key, and you'll be taken to the Request Builder page.

Paste your target URL, turn on the Anti-bot boost mode, enable premium proxies, and choose Node.js, like in the image below.

On the right of your dashboard, you'll see a ready-to-use scraper code that uses Axios.

// npm install axios
import axios from 'axios';
 
const url = 'https://www.g2.com/products/notion/reviews';
const apikey = '<YOUR_ZENROWS_API_KEY>';
axios({
    url: 'https://api.zenrows.com/v1/',
    method: 'GET',
    params: {
        'url': url,
        'apikey': apikey,
        'js_render': 'true',
        'antibot': 'true',
        'premium_proxy': 'true',
    },
})
    .then(response => console.log(response.data))
    .catch(error => console.log(error));

Note: In the code above, replace <YOUR_ZENROWS_API_KEY> with your API key.

You'll get a response similar to this one: 

Awesome! You were able to scrape the Notion review page using ZenRows.

POST Retry with Axios

Requests made using other HTTP methods, such as POST for creating resources on the server or PUT for making modifications to already-existing ones, can also be retried with the axios-retry plugin. To do that, use your desired HTTP method in your Axios request as in the below code with post, for example, and axios-retry will retry it if it fails:

import axios from 'axios';
import axiosRetry from 'axios-retry';
 
// Configure axios-retry
axiosRetry(axios, { 
  retries: 3, // Number of retries
  retryCondition: () => true, // Retry all errors
});
 
// POST request with payload
axios.post('https://httpbin.io/post', { payload: 'Request One' }) // Will be retried if it fails
  .then((response) => {
    console.log(response.data); //
  }).catch((error) => {
    console.log('Error: ', error);
  });

Conclusion

Building a solid web scraper relies heavily on how you deal with requests that fail. In this guide, we explored why retrying failed requests is important and how to implement it in systems that use Axios for requests.

You know now what it takes to build an Axios retry logic, how to avoid getting blocked and what way to retry Axios requests with different HTTP methods.

As part of your scraping workflow, ZenRows can give you more than one hand. Give it a shot today!

// npm install axios axios-retry
import axios from 'axios';
import axiosRetry from 'axios-retry';
 
// Configure axios-retry with options
axiosRetry(axios, { retries: 3 });
 
// Axios Requests...

// ...
// Configure `axios-retry` with `retries` option
axiosRetry(axios, { retries: 4 }); // Set the number of retries to 4

// ...
// Configure `axios-retry` with `retries` and `retryDelay` options
 
axiosRetry(axios, { 
  retries: 3, 
  retryDelay: (retryCount) => retryCount * 1000, // Increase timeout by 2 seconds per retry
}); 

Did you find the content helpful? Spread the word and share it on Twitter, or LinkedIn. It looks like this tutorial has helped you. Be someone's hero today: Share on Twitter Share on LinkedIn