How to Use a Proxy With Python Requests in 2024

Are you getting blocked while web scraping? There's one easy solution you can try: use proxies with Python Requests to hide your IP and bypass restrictions. A proxy server can increase your chances of accessing the data you need.

In this tutorial, you'll learn how to do it step by step. Let's roll!

How to Use Proxies With Python Requests

In this section, you'll learn how to perform basic Python requests with a proxy, where to get proxies from, how to authenticate, and some other day-to-day mechanisms.

1. Perform Requests

Before we begin, make sure you have Python and the Requests library installed. And since following this tutorial will be easier if you know the fundamentals of web scraping with Python, feel free to check our guide.

To use proxies with Python Requests, start by importing the HTTP client library:

import requests

Then, get some valid proxies from Free Proxy List and define a dictionary with the proxy URLs associated with the HTTP and HTTPS protocols:

proxies = {
   'http': 'http://103.167.135.111:80',
   'https': 'http://116.98.229.237:10003'
}

requests will perform HTTP requests over the http proxy and handle HTTPS traffic over the https one.

As you can see above, we used this syntax:

<PROXY_PROTOCOL>://<PROXY_IP_ADDRESS>:<PROXY_PORT>

Now, perform an HTTP request with Python requests through the proxy server:

# target website
url = 'https://httpbin.io/ip'

# making an HTTP GET request through a proxy
response = requests.get(url, proxies=proxies)

The full code of what your basic Python requests proxy script will look like this:

import requests

proxies = {
   'http': 'http://103.167.135.111:80',
   'https': 'http://116.98.229.237:10003'
}

url = 'https://httpbin.io/ip'
response = requests.get(url, proxies=proxies)
print(response)

Verify it works. You'll get the following response:

<Response [200]>

That means the target server website responded with an HTTP 200 status code. In other words, the HTTP request was successful! 🥳

Please note that requests supports only HTTP and HTTPS proxies. If you must route HTTP, HTTPS, FTP, or other traffic, you'll need a SOCKS proxy. The library doesn't support it natively, but you can install the socks extension.

pip3 install requests[socks]

Then, you can specify a SOCKS proxy use.

import requests

proxies = {
    'http': 'socks5://<PROXY_IP_ADDRESS>:<PROXY_PORT>',
    'https': 'socks5://<PROXY_IP_ADDRESS>:<PROXY_PORT>'
}

url = 'https://httpbin.io/ip'
response = requests.get(url, proxies=proxies)
print(response)

2. Print the Response

HTTPBin, our target page, returns the IP of the caller in JSON format, so retrieve the response to the request with the json() method.

print(response.json())

In case of a non-JSON response, use this other method:

print(response.text)

Your Python Requests proxy script should look as follows:

import requests

proxies = {
   'http': 'http://103.167.135.111:80',
   'https': 'http://116.98.229.237:10003'
}

url = 'https://httpbin.io/ip'
response = requests.get(url, proxies=proxies)
print(response.json())

Run it, and you'll get an output similar to this:

{'origin': '116.98.229.237'}

The origin field contains the IP of the proxy, not yours. That confirms requests made the HTTP request over a proxy.

3. Requests Methods

The Python Requests library provides methods that correspond to different HTTP methods. Let's explore the most common ones and their uses.

GET

The GET method is used to retrieve data from a server. It sends a request to a specified URL and returns the server's response. Here's how to use it:

response = requests.get('https://httpbin.io/ip')

POST

The POST method is used to send data to a server to create or update a resource. Here's an example:

response = requests.post('https://httpbin.io/anything', data={"key1": "a", "key2": "b"})

Other Methods

While GET and POST are the most commonly used, Requests support several other HTTP methods that are used less frequently. Here's a table of these additional methods:

4. Proxy Authentication With Python Requests: Username & Password

Some proxy servers are protected by authentication for security reasons so that only users with credentials can access them. That usually happens with premium proxies or commercial solutions.

Follow this syntax to specify a username and password in the URL of an authenticated proxy:

<PROXY_PROTOCOL>://<YOUR_USERNAME>:<YOUR_PASSWORD>@<PROXY_IP_ADDRESS>:<PROXY_PORT>

See an example:

# ...

proxies = {
  'http': 'http://fgrlkbxt:[email protected]:7492',
  'https': 'https://fgrlkbxt:[email protected]:6286'
}

# ...

Error 407: Proxy Authentication Required

The HTTP status error code 407: Proxy Authentication Required occurs when making a request through a proxy server that requires authentication. This error indicates that the user didn't provide valid credentials.

To fix it, ensure the proxy URL contains the correct username and password. Find out more about the several types of authentications supported.

5. Proxy Session Using Python Requests

When making many requests through a proxy server, you may need a session. A Session object can reuse the same TCP connection for several requests, which saves time and improves performance compared to making single requests.

A session retains information across requests, including cookies, authentication, and connection pooling. This persistence can be particularly useful when you need to maintain a logged-in state, handle complex authentication flows, or optimize performance for multiple requests to the same host.

For instance, when scraping a website that requires login, a session can help you stay authenticated throughout your scraping process without repeatedly sending login credentials.

Use a proxy session in Python Requests as shown here:

import requests

# initialize a session
session = requests.Session()

# set the proxies in the session object
session.proxies = {
   'http': 'http://103.167.135.111:80',
   'https': 'http://116.98.229.237:10003'
}

url = 'https://httpbin.io/ip'

# perform an HTTP GET request over the session
response = session.get(url)

6. Environment Variable for a Python Requests Proxy

You can DRY up some code if your Python script uses the same proxies for each request. By default, Requests relies on the HTTP proxy configuration defined by these environment variables:

• HTTP_PROXY: It corresponds to the http key of the proxies dictionary.
• HTTPS_PROXY: It corresponds to the https key of the proxies dictionary.

Open the terminal and set the two environment variables this way:

export HTTP_PROXY="http://103.167.135.111:80"
export HTTPS_PROXY="http://116.98.229.237:10003"

Then, remove the proxy logic from your script, and you'll get to this:

import requests

url = 'https://httpbin.io/ip'
response = requests.get(url)

Great! You now know the basics of proxies in Python with Requests! Let's see them in action in some more advanced scenarios.

Use a Rotating Proxy With Python Requests

When your script sends numerous requests in rapid succession, websites may flag this behavior as suspicious and block your IP address. A rotating proxy strategy can help prevent this. The concept is simple: you switch to a new proxy after a set time interval or number of requests, making each request appear to come from a different user.

Let's explore how to implement an effective proxy rotator in Python using the Requests library.

How to Rotate IPs With Requests? Free Solution

Websites can detect and block requests from specific IP addresses, potentially halting your scraping operation. By rotating IPs, your scraper becomes much more resilient, as each request appears to come from a different user, making detection and blocking more difficult.

Let's implement a simple proxy rotator using free proxies. We'll start by importing the necessary libraries: requests for making HTTP requests and random for selecting proxies randomly.

import requests
import random

Next, define a list of free proxies using the Free Proxy List website. In a real-world scenario, you'd want to replace these with actual working proxies.

# ...

PROXY_LIST = [
    "http://203.24.108.161:80",
    "http://80.48.119.28:8080",
    "http://203.30.189.47:80",
]

Create a function called get_random_proxy(). This function uses Python's random.choice() to randomly select and return a proxy from the list.

# ...

def get_random_proxy():
    return random.choice(PROXY_LIST)

proxy = get_random_proxy()

Now comes the crucial part: making a request using the selected proxy. Use requests.get() to send a GET request to the target URL, passing the selected proxy in the proxies parameter. This tells the Requests library to route the request through the chosen proxy.

# ...

url = "https://httpbin.io/ip"
response = requests.get(url, proxies={"http": proxy, "https": proxy})
print(response.text)

Merge the above snippets. Here's how your final proxy rotator code should look like:

import requests
import random

# list of free proxies (replace with actual working proxies)
PROXY_LIST = [
    "http://203.24.108.161:80",
    "http://80.48.119.28:8080",
    "http://203.30.189.47:80",
]

# randomly select a proxy from the list
def get_random_proxy():
    return random.choice(PROXY_LIST)

# get a random proxy
proxy = get_random_proxy()

# target url to check our IP
url = "https://httpbin.io/ip"

# make a request using the selected proxy
response = requests.get(url, proxies={"http": proxy, "https": proxy})

# print the response, which should show the proxy's IP
print(response.text)

Here's the result of running the above code three times:

# request 1
{
  "origin": "203.30.189.47"
}

# request 2
{
  "origin": "80.48.119.28"
}

# request 3
{
  "origin": "203.24.108.161"
}

By running this script multiple times, you'll observe different IP addresses being used, demonstrating the IP rotation in action.

However, it's important to note that free proxies often have limitations. They can be unreliable, slow, or may not work with more sophisticated websites. For serious scraping tasks, consider using premium proxy services that offer better reliability, speed, and features like automatic rotation and geolocation targeting.

Ignore SSL Certificate

By default, Requests verifies SSL certificates on HTTPS requests. Certification verification can lead to SSLError errors when dealing with proxies.

To avoid those errors, disable SSL verification with verify=False:

# ...
response = requests.request(
    http_method, 
    url, 
    proxies=proxies, 
    timeout=5
    # disable SSL certificate verification
    verify=False
)

Premium Proxy to Avoid Getting Blocked

While free proxies can be useful for basic tasks, they often fail with serious web scraping projects. Using premium proxies that offer auto-rotation, residential IPs, and geolocation features significantly increases your scraping success rate.

One example of a premium proxy service that offers these features is ZenRows. ZenRows provides a proxy rotator solution that automatically handles IP rotation, offers residential IPs, and includes advanced anti-bot bypassing techniques.

To get started, sign up for ZenRows.

Go to the ZenRows Proxy Generator page. Your premium residential proxy will be generated automatically. Customize it according to your requirements.

Once you're done, copy the generated proxy and modify your code as follows:

import requests

proxy = 'http://<ZENROWS_PROXY_USERNAME>:<ZENROWS_PROXY_PASSWORD>@superproxy.zenrows.com:1337'
proxies = { 
    'http': proxy, 
    'https': proxy
}

url = 'https://httpbin.io/ip'
response = requests.get(url, proxies=proxies, verify=False)
print(response.text)

Note that verify=False is mandatory when using premium proxies in ZenRows.

Run the code. Here's an example of what the output would look like:

{
  "origin": "185.220.101.34"
}

This output shows that your request was successfully routed through one of ZenRows' premium proxies.

Congrats! Your premium proxy with Python Requests script is ready!

Conclusion

This step-by-step tutorial covered the most important lessons about proxies with Requests in Python. You started from the basic setup and have become a proxy master!

Now you know:

• What a web proxy is and why free proxies aren't reliable.
• The basics of using a proxy with Requests in Python.
• How to implement a rotating proxy.
• How to use a premium proxy.

Proxies help you bypass anti-bot systems. Yet, some proxies are more reliable than others, and ZenRows offers the best on the market. With it, you'll get access to a reliable rotating proxy system.