Master Parallel Requests in Python Step-by-Step

Most Python applications rely on external data or need to communicate with other servers via HTTP. The problem is that HTTP calls are slow and can easily slow down your application. Avoid that with Python parallel requests!

In this guide, you'll see how to use threads to make more than one network request simultaneously in Python. The performance benefits will be simply amazing.

Let's dive in!

What are Parallel Requests in Python?

Parallel requests in Python involve executing multiple HTTP calls at the same time. The idea is to start several threads and make each of them run network requests in parallel. By harnessing multicore processors, you can achieve better performance and network usage.

Some use cases for parallel requests in web development include:

1. Retrieving data concurrently on page load via AJAX calls to reduce loading times.
2. Calling various microservices simultaneously in a backend API implementation, optimizing system efficiency.

In the context of web scraping, making parallel network requests is crucial. Since web servers tend to be slow, scraping one page at a time is too inefficient. By distributing requests over various threads, you can reduce idle time and save a lot of time.

Parallelization plays a key role in large-scale data extraction projects. That's particularly true when doing web crawling, which involves a lot of HTTP calls. Find out more in our guide on how to build a Python web crawler.

How To Implement Parallel Requests

Using a ThreadPoolExecutor worker is the easiest way to make Python parallel requests. ThreadPoolExecutor is a Python class from the standard concurrent.futures library designed for managing a pool of threads.

Its goal is to enable the parallel execution of tasks in a multithreaded environment. ThreadPoolExecutor proves especially valuable in I/O-bound scenarios, such as network requests. The reason is that it can harness parallelism without requiring separate processes.

The class offers an intuitive API for parallelizing tasks without the complexity of managing threads. The main methods offered by a ThreadPoolExecutor worker are:

• submit(task_func): Submits the task function passed as a parameter and returns a Future object representing the result. Call submit() on that object to block the execution until the task function has terminated.
• map(task_func, iterable): Applies the task function to each item in the iterable argument and returns an iterator over the results.

Using them to make parallel requests helps you save a lot of time. Assume the average request takes around 250ms and you want to make 8. Performing them sequentially will mean having to wait for 2000ms (250ms*8).

Instead, execute them in parallel on 8 threads managed by a ThreadPoolExecutor worker. The same operation will take around ~250ms, plus minor overhead for thread handling. That's an almost x8 time save!

Now, follow the steps below and learn how to build a Python requests parallel script!

Step 1: Install the Necessary Libraries

To build a parallel requests Python script, we'll use these two packages:

• Requests: A popular Python HTTP client that makes it easier to perform web requests.
• Beautiful Soup: A complete and easy-to-use HTML parser to extract data from HTML documents.

We'll rely on them to execute the following operations in parallel:

1. Make GET HTTP calls to a site with Requests.
2. Retrieve the HTML documents associated with a few web pages.
3. Parse their content with Beautiful Soup.

Install the required libraries with the command below:

pip install requests beautifulsoup4

Then, import them by adding the following two lines on top of your Python script:

import requests
from bs4 import BeautifulSoup

Awesome, you're ready to make Python parallel requests!

Step 2: Write Your First Parallel Request Code

First, define the task function ThreadPoolExecutor will execute in parallel. This takes a URL as a parameter, uses it to make an HTTP request, and extracts the title from the retrieved HTML page:

def parse_page(url):
    # perform the HTTP request to the specified URL
    response = requests.get(url)

    # parse the HTML content returned by server
    soup = BeautifulSoup(response.text, "html.parser")

    # extract the title from the page and print it
    title_element = soup.find('title')
    title = title_element.text
    print(title) 

Next, define the list of URLs to apply the task function on:

urls = [
    'https://scrapeme.live/shop/page/1/',
    'https://scrapeme.live/shop/page/2/',
    'https://scrapeme.live/shop/page/3/',
    'https://scrapeme.live/shop/page/4/',
    'https://scrapeme.live/shop/page/5/',
    'https://scrapeme.live/shop/page/6/',
    'https://scrapeme.live/shop/page/7/',
    'https://scrapeme.live/shop/page/8/'
]

Initialize a ThreadPoolExecutor object and use it to execute parse_page() in parallel on 4 URLs at a time:

MAX_THREADS = 4
with concurrent.futures.ThreadPoolExecutor(max_workers=MAX_THREADS) as executor:
    executor.map(parse_page, urls)

Note: ThreadPoolExecutor comes from the concurrent.futures package of the Python Standard Library.

Thus, add the following import statement to your script:

from concurrent.futures import ThreadPoolExecutor

Put it all together, and you'll get:

import requests
from bs4 import BeautifulSoup
from concurrent.futures import ThreadPoolExecutor

def parse_page(url):
    # perform the HTTP request to the specified URL
    response = requests.get(url)

    # parse the HTML content returned by server
    soup = BeautifulSoup(response.text, "html.parser")

    # extract the title from the page and print it
    title_element = soup.find('title')
    title = title_element.text
    print(title)

## list of URLs to parse concurrently
urls = [
    'https://scrapeme.live/shop/page/1/',
    'https://scrapeme.live/shop/page/2/',
    'https://scrapeme.live/shop/page/3/',
    'https://scrapeme.live/shop/page/4/',
    'https://scrapeme.live/shop/page/5/',
    'https://scrapeme.live/shop/page/6/',
    'https://scrapeme.live/shop/page/7/',
    'https://scrapeme.live/shop/page/8/'
]

# max number of threads to use
MAX_THREADS = 4

# initialize ThreadPoolExecutor and use it to call parse_page() in parallel
with ThreadPoolExecutor(max_workers=MAX_THREADS) as executor:
    executor.map(parse_page, urls)

Launch the program, and it'll print something like:

Products – Page 4 – ScrapeMe
Products – Page 2 – ScrapeMe
Products – Page 7 – ScrapeMe
Products – Page 6 – ScrapeMe
Products – Page 5 – ScrapeMe
Products – Page 3 – ScrapeMe
Products – Page 8 – ScrapeMe
Products – ScrapeMe

The print order will change at each run. That means the script executes Python parallel requests as expected. Specifically, it makes as many parallel parse_page() calls as the number set in MAX_THREADS. Perfect!

Step 3: Handle Responses and Exceptions

The current script makes requests in parallel but has a major issue. If a parse_page() fails, the entire process fails. To avoid that, wrap the code inside the function with a try ... except block:

def parse_page(url):
    try:
        # ...
    except Exception as e:
        print('Request failed due to error:', e)

Another limitation is that the task function only prints the extracted data in the terminal. What if you wanted to store that information in a Python data structure for later use? To do so, you can make parse_page() return the scraped data as follows:

def parse_page(url):
    try:
        # perform the HTTP request to the specified URL
        response = requests.get(url)

        # parse the HTML content returned by server
        soup = BeautifulSoup(response.text, "html.parser")

        # extract the title from the page
        title_element = soup.find('title')
        title = title_element.text

        # return the scraped data in a dictionary
        return {'title': title}
    except Exception as e:
        print('Request failed due to error:', e)

Now, initialize a Python list using the values returned by the parse_page() calls. As ThreadPoolExecutor.map() returns an iterator, you can convert it to a list by wrapping it with list():

with ThreadPoolExecutor(max_workers=MAX_THREADS) as executor:
    titles = list(executor.map(parse_page, urls))

The new parallel requests Python logic will be:

import requests
from bs4 import BeautifulSoup
from concurrent.futures import ThreadPoolExecutor

def parse_page(url):
    try:
        # perform the HTTP request to the specified URL
        response = requests.get(url)

        # parse the HTML content returned by server
        soup = BeautifulSoup(response.text, "html.parser")

        # extract the title from the page
        title_element = soup.find('title')
        title = title_element.text

        # return the scraped data in a dictionary
        return {'title': title}
    except Exception as e:
        print('Request failed due to error:', e)


## list of URLs to parse concurrently
urls = [
    'https://scrapeme.live/shop/page/1/',
    'https://scrapeme.live/shop/page/2/',
    'https://scrapeme.live/shop/page/3/',
    'https://scrapeme.live/shop/page/4/',
    'https://scrapeme.live/shop/page/5/',
    'https://scrapeme.live/shop/page/6/',
    'https://scrapeme.live/shop/page/7/',
    'https://scrapeme.live/shop/page/8/'
]

# max number of threads to use
MAX_THREADS = 4

# initialize ThreadPoolExecutor and use it to call parse_page() in parallel
with ThreadPoolExecutor(max_workers=MAX_THREADS) as executor:
    titles = list(executor.map(parse_page, urls))

print(titles)

Execute it, and it'll always print:

[{'title': 'Products – ScrapeMe'}, {'title': 'Products – Page 2 – ScrapeMe'}, {'title': 'Products – Page 3 – ScrapeMe'}, {'title': 'Products – Page 4 – ScrapeMe'}, {'title': 'Products – Page 5 – ScrapeMe'}, {'title': 'Products – Page 6 – ScrapeMe'}, {'title': 'Products – Page 7 – ScrapeMe'}, {'title': 'Products – Page 8 – ScrapeMe'}]

Notice that the resulting list contains elements in the same order as the URL list. The reason is that ThreadPoolExecutor.map() always returns the results in order. That occurs even if it evaluates the task function calls out of order.

Congrats! You just learned how to use Python to make multiple requests in parallel.

Optimizing Performance and Scaling

It's time to explore some tips and tricks to take your Python parallel requests logic to the next level.

Performance Optimization

Performance mainly depends on the number of threads used in ThreadPoolExecutor. Opening too few limits the parallelization potential of the script. In contrast, using too many slows down the script. That's because creating and managing threads introduces a significant time overhead.

Since finding the right number for optimal performance isn't always easy, you should do some experiments. Wrap the Python requests parallel logic as follows to track the execution time:

start_time = time.time()

# parallel ThreadPoolExecutor logic...

end_time = time.time()
elapsed_time = end_time - start_time
print(f"Elapsed time: {elapsed_time:.3f}s")

Don't forget to add the following import:

import time

Your script will now print the elapsed time as below:

Elapsed time: 1.475s

If urls doesn't contain too many elements, you could use the length of the list as the number of threads:

MAX_THREADS = len(urls)

However, len(urls) could be bigger than the number of logical CPUs available you can get with os.cpu_count(). Thus, the best approach is to take the minimum between len(urls) and os.cpu_count():

MAX_THREADS = min(os.cpu_count(), len(urls))

Perfect! ThreadPoolExecutor will now always use a reasonable amount of threads!

Efficient Scaling

With large-scale processes, finding the right batch size becomes even more essential. In such a situation, using all cores available may not always be the best approach. The reasons are:

• Resource saturation: Using all cores in parallel can easily saturate the resources available, leaving no room for other processes to run on the same server.
• Parallelization overhead: Large batches can be slow due to contention for shared resources, such as disk I/O, network bandwidth, or database connections. This usually causes bottlenecks and degrades overall performance.
• Slow error handling: When a request fails, you may need to repeat the entire batch. But the larger the chunk, the longer the execution time. So, large batches are inefficient when dealing with error-prone requests.

This is why small chunks are better for scaling parallel Python requests. The result will be a faster, more responsive process to achieve the goal without saturating all resources.

Pro tip: In large-scale operations, keep track of CPU, memory, and network usage to avoid overloading the system. Integrate an application monitoring tool into your script to streamline the tracking task.

Advanced Techniques

Here, we'll dig into two advanced approaches for making your parallel logic more effective and efficient.

Rate Limiting Handling

Sending too many requests to the same server in a short time may trigger rate limiting measures. A rate limiter defines the maximum number of requests you can send in a particular time window. If you exceed that, the server will start returning 429 Too Many Requests error responses.

You can find out the limitations defined by the server for incoming requests by:

• Inspecting the robots.txt file.
• Verifying if the server responses contain the RateLimit-X header fields.
• Making a certain number of parallel requests in a specified time frame until you get a 429 error.

Once find the right number, make sure to wait for a while before making new Python parallel requests. For example, assume you need to wait for 30 seconds after 10 requests. You can achieve that with a global counter as below:

request_counter = 0

def parse_page(url):
    global request_counter

    # make the request....

    # increment the request counter
    request_counter += 1
    # if it is the 10th request, wait 30 seconds
    if request_counter % 10 == 0:
        time.sleep(30)

    # return the scraped data... 

Great! Rate limiting will no longer be a problem.

Asynchronous Programming

Asynchronous programming relies on parallel tasks instead of threads as in ThreadPoolExecutor. In other words, a task can start without waiting for a thread to complete its operation and become free. That implies reduced idle time and better performance.

Thanks to the asyncio module in Python, a single thread can manage many concurrent tasks. This approach leads to improved overall performance and a more efficient network usage. Also, it reduces the overhead coming from creating and managing several threads.

Thus, the Python parallel requests asyncio approach can be better than ThreadPoolExecutor. That's especially true when threads don't occupy all resources allocated to them.

Common Issues with Parallel Requests for Web Scraping

The biggest challenge when making parallel requests in Python is getting blocked. The more requests you make, the higher the chance that anti-bot solutions will detect and block you as a bot.

An effective way to fool these systems is to randomize your requests as much as possible. Set headers with rotating real-world values and add random delays. Yet, that may not be enough. Your exit IP will always be the same; that detail can easily expose you!

You could use a proxy with Python requests service to avoid that. However, it may not work against advanced solutions like Cloudflare. Bypassing a WAF is always a hard challenge, as it'll try to stop you with CAPTCHAS and JavaScript challenges.

The solution? An all-in-one scraping solution with a powerful AI-based anti-bot toolkit like ZenRows!

ZenRows offers a scraping API you can call in parallel for maximum efficiency. By default, each request will have a new User-Agent and exit IP. What if the request fails? Don't worry, ZenRows will not charge you and will automatically try again for you.

Other Tools to Make Parallel Requests in Python

ThreadPoolExecutor isn't the only tool available to build a Python requests parallel script. Quite the opposite, there are many other packages for performing concurrent HTTP requests. Some integrate with the standard API's methods, while others directly make parallel requests.

The most popular tools to make Python parallel requests are:

• asyncio: A package from the Python Standard Library to write concurrent code using the async/await syntax. Ideal for I/O-bound tasks.
• multiprocessing: A Python vanilla library for executing tasks on multiple cores. Ideal for CPU-bound tasks.
• aiohttp: Asynchronous Python HTTP client built on top of asyncio. It makes it easy to make HTTP calls in parallel.
• twisted: An event-based framework for achieving several tasks in Python web applications.
• tornado`: A Python web framework and asynchronous networking library based on non-blocking network I/O.
• httpx: A fully-featured HTTP client that provides both a synchronous and asynchronous API.

Check out the table below to better understand what these tools have to offer and where they shine:

Conclusion

In this tutorial, you understood what Python parallel requests are. You started with the fundamentals and explored advanced techniques like batching. You're now a Python requests parallel expert.

Now you know:

• What it means to make parallel requests in Python.
• How to perform concurrent requests with ThreadPoolExecutor.
• How to optimize simultaneous HTTP calls.
• What the best Python tools to make network requests in parallel are.

All that remains is to put into practice everything you have learned here! Unfortunately, many websites use anti-bot measures to block your simultaneous requests. Bypass them all with ZenRows, a web scraping API with an unstoppable anti-scraping bypass toolkit, IP rotation, and built-in parallelization capabilities. Try ZenRows for free!

Did you find the content helpful? Spread the word and share it on Twitter, or LinkedIn. It looks like this tutorial has helped you. Be someone's hero today: Share on Twitter Share on LinkedIn