Retry Failed Requests
100% success rate is hardly achievable: sites break down, HTTP connections get interrupted, and many other factors. Most of the time, the first attempt will succeed, but we show here how to implement auto-retry policies for those that don't.
ZenRows won't charge you for failed requests. That means that, for example, 500 credits will be charged for 500 URLs even though 5 of them must be retried. Those would be 505 requests total, but ZenRows will only charge the successful ones.
We offer solutions in Python and JavaScript below. If you need them in a different language or stack, please contact us and we'll gladly help you.
ZenRows Python SDK
and JavaScript SDK
offer retries out-of-the-box by passing a retries
parameter in the constructor.
Python with Requests
For the code to work, you will need python3 installed.
Some systems have it pre-installed.
After that, install all the necessary libraries by running pip install
.
pip install requests
We will also be using Retry
from urllib3
and HTTPAdapter
from requests
.
There is no need to install those.
We will create a requests session instead of calling get
directly for reusability reasons.
We do it once, and all the requests sent with the session will share the defined retries.
Retry
parameters to your needs and check the docs since many more might be helpful for your case.
We included the ones that we consider indispensable.
total
sets the number of allowed retries.backoff_factor
will apply an exponential time factor between attempts:{backoff_factor} * (2 ** ({number of total retries} - 1))
. It will be[1, 2, 4]
for 1 second.status_forcelist
is a list of status codes that will force a retry.
import requests
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
apikey = "YOUR_KEY"
urls = [
"" # ... your URLs here
]
zenrows_api_base = "https://api.zenrows.com/v1/"
requests_session = requests.Session()
retries = Retry(
total=3,
backoff_factor=1,
status_forcelist=[429, 500, 502, 503, 504]
)
requests_session.mount("https://", HTTPAdapter(max_retries=retries))
for url in urls:
try:
response = requests_session.get(zenrows_api_base, params={
"apikey": apikey,
"url": url,
})
print(response.text) # process response
except Exception as e:
print(e) # will print "Max retries exceeded"
The example uses a list of URLs processed sequentially for simplicity, but you can run them concurrently with a maximum.
JavaScript with axios-retry
For the code to work, you will need
Node
(or nvm)
and npm installed.
Some systems have it pre-installed.
After that, install all the necessary libraries by running npm install
.
npm install axios axios-retry
Instead of calling ZenRows API directly with axios
,
we will wrap it in axiosRetry
and set our configuration.
It will handle the retries for all axios
calls from that point on.
It also offers the option to create an axios client
and use the wrapper on that.
axiosRetry
parameters to your needs. We used a configuration that will work most of the time for various use cases.
retries
sets the number of allowed retries.retryDelay
will apply an exponential time factor between attempts plus a 0-20% random delay margin.retryCondition
is a function to determine if the error can be retried. We will use the default one and an extra check for 429 (Too Many Requests) errors in our example. We did it manually since only 5xx errors will be retried by default.
const axios = require("axios");
const axiosRetry = require("axios-retry");
const apikey = "YOUR_KEY";
const zenrowsApiBase = "https://api.zenrows.com/v1/";
const urls = [
// ... your URLs here
];
axiosRetry(axios, {
retries: 3,
retryDelay: axiosRetry.exponentialDelay,
retryCondition: (error) => {
if (error.response && error.response.status === 429) {
return true; // example for custom retry condition
}
// fallback to default condition
return axiosRetry.isNetworkOrIdempotentRequestError(error);
},
});
(async () => {
for (const url of urls) {
try {
const response = await axios.get(zenrowsApiBase, {
params: { apikey, url },
});
console.log(response.data); // process response
} catch (error) {
console.log(error);
}
}
})();
The example uses a list of URLs processed sequentially for simplicity, but you can run them concurrently with a maximum.