Axios: Retry Failed Requests [2024]

August 25, 2023 · 8 min read

HTTP requests can temporarily fail, so it's essential to handle those to ensure reliability.

In this tutorial, you'll learn how to implement the Axios retry feature: which failed requests that can be retried, how different retry mechanisms work, as well as the strategies to use with examples.

Introduction: What to Know to Build the Axios Retry Logic

There are common questions that come to mind when considering building an Axios retry logic for your web scraper in NodeJS. Some of them are: Should I always retry failed requests or just in some cases? When should I retry? How many attempts should I make? We'll answer them in this section.

Types of Failed Requests

When a request is not successful, there are two common scenarios: cases that time out with no response from the server and cases that return an error. Let's look at both scenarios to better understand what strategies are available to deal with each one.

Timed out

Issues such as overloaded servers or your internet speed can cause a delayed response from the server. In the event of a timeout, it's a good idea to double-check your internet connection.

If your internet is stable, the issue is likely to lie with the server. To prevent unnecessarily long connection requests, it's a best practice to set a timeout in milliseconds to abort the request using the Axios timeout config options.

program.js
// npm install axios
// CommonJS
// const axios = require('axios');
 
// ES6
import axios from 'axios';
 
axios.get('https://httpbin.io/uuid', {
  // If the request takes longer than 15000ms (15 seconds), the request will be aborted.
  timeout: 15000,
})
  .then((response) => console.log(response.data))
  .catch((error) => console.log(error));

To detect a timeout, it's important to spot timeout-related errors (those with the ECONNABORTED code) and implement an Axios retry mechanism either conditionally or with strategies like exponential backoff, which we'll touch on later. Here's how to handle a timeout error in Axios:

program.js
// ...
 
axios.get('https://httpbin.io/uuid', {
  // Timeout is set to 10ms (0.01 seconds), the request will be aborted.
  timeout: 10,
})
  .then((response) => console.log(response.data))
  .catch((error) => {
    if (error.code === 'ECONNABORTED') {
      // Timeout Error (You can implement retry here where suitable)
      console.log('Timeout Error: ', error.message);
    }
  });

Returned an Error

A standard error response to a failed request includes a status code and a descriptive error message. The status code indicates what went wrong, while the error message provides more context for why the request failed.

To address this scenario, check if the request displays an error. Then, take a look at the response status code and message to see what went wrong. You can then retry the request with Axios if the error code indicates a temporary server issue.

Error Status Codes

Client-server communication errors have status codes that fall in the 4xx and 5xx ranges. The most common ones you'll want to retry with Axios when scraping include the following ones:

Common Error Explanation
403 Forbidden The server understands the request but won't fulfill it because it doesn't have the right permissions or access.
429 Too Many Requests The server has received too many requests from the same IP within a given time frame (rate limiting).
500 Internal Server Error A generic server error occurred, indicating that something went wrong on the server while processing the request.
502 Bad Gateway The server acting as a gateway or proxy received an invalid response from an upstream server.
503 Service Unavailable The server is either overloaded or undergoing maintenance and can't process your request at this time.
504 Gateway Timeout An upstream server didn't respond quickly enough to the gateway or proxy.

You can check out the MDN docs for more information on HTTP response status codes.

Number of Retries

It's advisable to impose an Axios retry limit to prevent unnecessary performance and blocking issues arising from an infinite number of retries. Also, when determining the optimal number of retries to make for a failed request, take into account factors like the type of request error and response time.

Temporary errors, such as 429 Too Many Requests, should have more retries than others (in this case, with a different IP). While there's no best number of retries, three to five retries are good numbers to use as a starting point.

Delay

Setting delays between requests is one way to prevent websites and APIs from becoming overloaded, which can result in you being rate limited. Let's see the types of delays you can use and how to code them later on.

Fixed or Random Delays

When you want to set a fixed delay between requests, the setTimeout() global method comes in handy. And to make the delay random, you can use the setTimeout() function with the Math.random() static method together.

There's no hard and fast rule for how long the delay should be. However, you can try out a range of reasonable delay values starting at 300ms until you find what's right for your system or application.

Exponential Delays (Backoff Strategy)

The backoff strategy is a common way to set delays between retries that get longer over time instead of being fixed or random. Each request adds an exponential backoff factor to the delay, which is usually more than one. This strategy gives room for temporary issues to resolve themselves while preventing server overload.

The backoff algorithm is given below:

Example
backoff_factor * (2 ** (current_number_of_retries - 1))

For example, here are the delay sequences for backoff factors 2, 3, and 10:

Example
# 2
1, 2, 4, 8, 16, 32, 64, 128
 
# 3
1.5, 3, 12, 24, 48, 96, 192, 384
 
# 10
5, 10, 20, 40, 80, 160, 320, 640
Frustrated that your web scrapers are blocked once and again?
ZenRows API handles rotating proxies and headless browsers for you.
Try for FREE

Tutorial: How to Retry Axios with Interceptors

The axios-retry plugin provides an easy way to implement interceptors to retry Axios requests. It makes retrying failed requests easier by providing a pre-configured interceptor that handles retries automatically based on configurable options.

It's an NPM package, so start by installing it with the command below:

Terminal
npm install axios-retry

1. Retry on Error with Axios

To retry your failed Axios request, first import the default export interceptor function of the axios-retry plugin. Then, pass your Axios instance and an optional object configuration as its first and second arguments, and call it.

program.js
// Retry interceptor function
import axiosRetry from 'axios-retry';
// Default axios instance
import axios from 'axios';
 
// Pass the axios instance to the retry function  and call it
axiosRetry(axios, { 
   retries: 3, // Number of retries (Defaults to 3) 
});
 
// Make Axios requests below
axios.get('https://scrapingcourse.com/ecommerce/') // The request is retried if it fails
  .then((response) => {
    console.log('Data: ', response.data);
  }).catch((error) => {
    console.log('Error: ', error);
  });

Note that, by default, axios-retry will only retry failed Axios requests that occur due to a network error or are idempotent (GET, HEAD, OPTIONS, PUT, or DELETE) with a response status code in the 5xx range. This means that requests that return an error code like 400 or 429 won't be retried automatically. 

To get around that, you can use the retryCondition option to conditionally tell axios-retry to retry a request based on the errors we want. Here's an example:

program.js
// ...
 
axiosRetry(axios, { 
  retries: 3, // Number of retries
  retryCondition(error) {
    // Conditional check the error status code
    switch (error.response.status) {
      case 404:
      case 429:
        return true; // Retry request with response status code 404 or 429
      default:
        return false; // Do not retry the others
    }
  },
});
 
// ...

2. Implement an Exponential Backoff Strategy with Axios Interceptors

The axios-retry plugin has an exponentialDelay method that you can pass to the retryDelay option to implement an exponential delay. Here's how:

program.js
// ...
 
axiosRetry(axios, { 
  retries: 3, // Number of retries
  retryDelay: axiosRetry.exponentialDelay, // Exponential back-off retry delay between requests
});
 
// ...

The delay or backoff factor by default is 100ms, but this can be configured by adding an extra argument to exponentialDelay. For example, a backoff factor of 1000ms:

program.js
// ...
 
axiosRetry(axios, { 
  retries: 3, // Number of retries
  retryDelay: (...arg) => axiosRetry.exponentialDelay(...arg, 1000), // Exponential delay with backoff of 1000ms
});
 
// ...

3. Learn the retry-axios Options

As a recap and expansion, these are the options to retry Axios requests:

Name Description Type Default
retries The number of retries before failing. 1 means one retry after the initial failure. Number 3
retryCondition A function that determines whether a request should be retried. By default, if a request is idempotent and the error code is in the 5xx range, it'll automatically be retried. Function isNetworkOrIdempotentRequestError
shouldResetTimeout Sets whether the timeout should start over between retries. Boolean false
retryDelay A function that can be used to modify the delay between retried requests in milliseconds. By default, there's no delay between retries. You can also pass the exponentialDelay method to implement an exponential backoff strategy. Function function noDelay (retryCount, error, requestConfig) { return 0; }
onRetry A function that is invoked before each retry. There won't be any default actions if this option is not set. Function function onRetry(retryCount, error, requestConfig) { return; }

Avoid Getting Blocked with retry-axios

The biggest issue when crawling is being blocked due to being recognized as a bot. If a website determines that your request is automated, it may block your IP address or take other measures to avoid giving you access to the site.

Let's try scraping a G2.com secure page to demonstrate that:

program.js
import axios from 'axios';
 
axios.get('https://www.g2.com/products/notion/reviews')
  .then((response) => console.log('Response: ', response.data))
  .catch((error) => {
    // If error, log the response data and status code
    console.log('Error: ', error.response.data, error.response.status)
  });

If you run the code, you'll see a 403 error message similar to the one shown below:

Output
   
<!-- // ... -->
   <div class="cf-main-wrapper" role="main">
      <div class="cf-header cf-section">
         <div class="cf-error-title">
            <h1>Access denied</h1>
            <span class="cf-code-label">Error code <span>1020</span></span>
         </div>
         <div class="cf-error-description">
            <p>You do not have access to www.g2.com.</p><p>The site owner may have set restrictions that prevent you from accessing the site.</p>
         </div>
      </div>
   </div>
 
<!-- // ... -->
 
403

Web scraping proxies can be useful in situations like the one we just encountered. Unfortunately, proxies aren't enough for many websites since they aren't likely to be able to keep up with ever-changing anti-bot systems.

Web scraping APIs are a dedicated toolkit for avoiding being blocked that is as simple to use as making a single API call. They often have features such as premium proxies, rotating User-Agent, and so on.

ZenRows is a popular and excellent web scraping API that can be integrated with Axios. Start by signing up for ZenRows to get your free API key, and you'll be taken to the Request Builder page.

Paste your target URL, enable Premium Proxies, turn on the JS Rendering boost mode. Then, choose Node.js and click on the Proxy tab.

building a scraper with zenrows
Click to open the image in full screen

On the right of your dashboard, you'll see a ready-to-use scraper code that uses Axios.

program.js
// npm install axios
import axios from 'axios';
 
const url = 'https://www.g2.com/products/notion/reviews';
const apikey = '<YOUR_ZENROWS_API_KEY>';
axios({
    url: 'https://api.zenrows.com/v1/',
    method: 'GET',
    params: {
        'url': url,
        'apikey': apikey,
        'js_render': 'true',
        'premium_proxy': 'true',
    },
})
    .then(response => console.log(response.data))
    .catch(error => console.log(error));

Note: In the code above, replace <YOUR_ZENROWS_API_KEY> with your API key.

You'll get a response similar to this one: 

Notion Review Output
Click to open the image in full screen

Awesome! You were able to scrape the Notion review page using ZenRows.

POST Retry with Axios

Requests made using other HTTP methods, such as POST for creating resources on the server or PUT for making modifications to already-existing ones, can also be retried with the axios-retry plugin. To do that, use your desired HTTP method in your Axios request as in the below code with post, for example, and axios-retry will retry it if it fails:

program.js
import axios from 'axios';
import axiosRetry from 'axios-retry';
 
// Configure axios-retry
axiosRetry(axios, { 
  retries: 3, // Number of retries
  retryCondition: () => true, // Retry all errors
});
 
// POST request with payload
axios.post('https://httpbin.io/post', { payload: 'Request One' }) // Will be retried if it fails
  .then((response) => {
    console.log(response.data); //
  }).catch((error) => {
    console.log('Error: ', error);
  });

Conclusion

Building a solid web scraper relies heavily on how you deal with requests that fail. In this guide, we explored why retrying failed requests is important and how to implement it in systems that use Axios for requests.

You know now what it takes to build an Axios retry logic, how to avoid getting blocked and what way to retry Axios requests with different HTTP methods.

As part of your scraping workflow, ZenRows can give you more than one hand. Give it a shot today!

Frequent Questions

Does Axios Have a Retry?

Axios doesn't have any default retry features built in, but axios-retry and other third-party libraries can be used to add a retry logic to your Axios requests.

How Do I Retry a Failed Axios Request?

To retry a failed Axios request, you can import the axios-retry library and configure it with the Axios instance you're using in your application.

// npm install axios axios-retry
import axios from 'axios';
import axiosRetry from 'axios-retry';
 
// Configure axios-retry with options
axiosRetry(axios, { retries: 3 });
 
// Axios Requests...

How Do I Set Retries in Axios?

To set retries in Axios, you need to configure the retry behavior using the axiosRetry function by specifying the retry option. Here's how to do that:

// ...
// Configure `axios-retry` with `retries` option
axiosRetry(axios, { retries: 4 }); // Set the number of retries to 4

How Do I Increase Timeout in Axios?

To increase the timeout in Axios, you can provide the retryDelay option with a custom delay function that includes an increased timeout. For example, if you want to increase the timeout by 1 second (1000 ms) per retry, this is the process:

// ...
// Configure `axios-retry` with `retries` and `retryDelay` options
 
axiosRetry(axios, { 
  retries: 3, 
  retryDelay: (retryCount) => retryCount * 1000, // Increase timeout by 2 seconds per retry
}); 

Ready to get started?

Up to 1,000 URLs for free are waiting for you