Retry Failed Requests

100% success rate is hardly achievable: sites break down, HTTP connections get interrupted, and many other factors. Most of the time, the first attempt will succeed, but we show here how to implement auto-retry policies for those that don't.

ZenRows won't charge you for failed requests. That means that, for example, 500 credits will be charged for 500 URLs even though 5 of them must be retried. Those would be 505 requests total, but ZenRows will only charge the successful ones.

We offer solutions in Python and JavaScript below. If you need them in a different language or stack, please contact us and we'll gladly help you.

ZenRows Python SDK and JavaScript SDK offer retries out-of-the-box by passing a retries parameter in the constructor.

Python with Requests

For the code to work, you will need python3 installed. Some systems have it pre-installed. After that, install all the necessary libraries by running pip install.

pip install requests

We will also be using Retry from urllib3 and HTTPAdapter from requests. There is no need to install those.

We will create a requests session instead of calling get directly for reusability reasons. We do it once, and all the requests sent with the session will share the defined retries.

import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

apikey = "YOUR_KEY"
urls = [
	""  # ... your URLs here
]
zenrows_api_base = "https://api.zenrows.com/v1/"

requests_session = requests.Session()
retries = Retry(
	total=3,
	backoff_factor=1,
	status_forcelist=[429, 500, 502, 503, 504]
)
requests_session.mount("https://", HTTPAdapter(max_retries=retries))

for url in urls:
	try:
		response = requests_session.get(zenrows_api_base, params={
			"apikey": apikey,
			"url": url,
		})

		print(response.text)  # process response
	except Exception as e:
		print(e)  # will print "Max retries exceeded"

The example uses a list of URLs processed sequentially for simplicity, but you can run them concurrently with a maximum.

JavaScript with axios-retry

For the code to work, you will need Node (or nvm) and npm installed. Some systems have it pre-installed. After that, install all the necessary libraries by running npm install.

npm install axios axios-retry

Instead of calling ZenRows API directly with axios, we will wrap it in axiosRetry and set our configuration. It will handle the retries for all axios calls from that point on. It also offers the option to create an axios client and use the wrapper on that.

const axios = require("axios");
const axiosRetry = require("axios-retry");

const apikey = "YOUR_KEY";
const zenrowsApiBase = "https://api.zenrows.com/v1/";
const urls = [
	// ... your URLs here
];

axiosRetry(axios, {
	retries: 3,
	retryDelay: axiosRetry.exponentialDelay,
	retryCondition: (error) => {
		if (error.response && error.response.status === 429) {
			return true; // example for custom retry condition
		}

		// fallback to default condition
		return axiosRetry.isNetworkOrIdempotentRequestError(error);
	},
});

(async () => {
	for (const url of urls) {
		try {
			const response = await axios.get(zenrowsApiBase, {
				params: { apikey, url },
			});
			console.log(response.data); // process response
		} catch (error) {
			console.log(error);
		}
	}
})();

The example uses a list of URLs processed sequentially for simplicity, but you can run them concurrently with a maximum.