What Is "Please Verify You Are a Human" from PerimeterX
The "Please verify you are a human" message means the website owner wants you to confirm you're a real person, not a bot. This action prevents the access of malicious automated systems. Unfortunately, it may also appear when scraping a website using tools like Selenium, Puppeteer, or Playwright.
To verify, you'll be given a task that's easy for humans but hard for bots, such as solving a visual puzzle, answering a question, or performing a specific action. Bypassing PerimeterX (known as HUMAN nowadays) typically requires the "Press & Hold" action, and the screen will look similar to the one below:
How Do I Avoid "Please Verify You Are a Human" from PerimeterX
You can avoid this challenge by relying on the following techniques:
A. Basic Solution: Press and Hold with a Headless Browser
A headless browser for scraping lacks the graphical user interface (GUI) but allows automated interaction with web pages, such as submitting forms.
Since the PerimeterX error is typically seen during the initial request to the website, a headless browser such as Selenium can mimic a press-and-hold action. Let's look at a generic implementation.
First, set up your Selenium driver in headless mode and navigate to your preferred URL. The press-and-hold button will most likely be injected programmatically into an
iframe element, which we have to ensure is available before selecting the button. Then, use Selenium's Action API to simulate press-and-hold actions for about ten seconds.
from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.support.wait import WebDriverWait import selenium.webdriver.support.expected_conditions as EC from selenium.webdriver.common.action_chains import ActionChains import time # create ChromeOptions object options = webdriver.ChromeOptions() options.add_argument("--headless") # Headless mode # create a new Chrome webdriver instance, passing in the options object driver = webdriver.Chrome(options=options) driver.get("your_url") # Switch to iframe that directly houses pX "Press & Hold" button WebDriverWait(driver, timeout=300).until(EC.frame_to_be_available_and_switch_to_it("iframe_name_or_id")) # Get button element btn = driver.find_element(By.XPATH, "//xpath_to_button") # Initialize for low-level interactions action = ActionChains(driver) action.click_and_hold(btn) # Initiate click and hold action on button action.perform() # Keep holding for 10s time.sleep(10) # Release button action.release(btn) # ...continue scraping driver.quit()
Great! Did it work in your case? If not, keep reading.
In some scenarios, you may be unable to select the press-and-hold button. In such cases, PerimeterX injects the button into a random
iframe amongst other
Fortunately, there's a way out! Even though the button is in a closed shadow DOM, that doesn't stop it from being focused on. So, we can leverage keystrokes to trigger a browser focus on the button using the Tab key and simulate pressing it with the Enter key.
Let's test this out on Fiverr, as it implements its press-and-hold button that way. First, import the necessary dependencies and set up your Selenium driver:
from selenium import webdriver from selenium.webdriver.common.by import By from selenium.webdriver.common.action_chains import ActionChains from selenium.webdriver.common.keys import Keys import time driver = webdriver.Chrome() driver.get('https://www.fiverr.com')
Fiverr has a list of categories on its Homepage that link to their respective category pages. Assuming we want to scrape the content of the first category page, we can select the first category link and mimic a click on it.
# ... try: link = driver.find_element(By.CSS_SELECTOR, 'ul.categories-list > li:first-child > a') except: link = None if link: link.click() # Triggers a navigation # Implicitly wait for 10 seconds (for navigation to finish) driver.implicitly_wait(time_to_wait=10)
Because clicking on the link triggers navigation, we've got to implicitly wait for some time, 30 seconds in this case. During the navigation, Fiverr will likely redirect us to the challenge page. A div with an
px-captcha wraps the button and shadow DOM, so its availability after navigation means we've been sent to the challenge page.
If we're on the challenge page, we can send the Tab and Enter strokes like this:
# ... btn_container = driver.find_element(By.ID, 'px-captcha') if (btn_container): # we've been blocked so run bypass logic print('PRESS & HOLD container found') driver.implicitly_wait(10) ActionChains(driver).key_down(Keys.ENTER)\ .pause(5)\ .send_keys(Keys.TAB)\ .pause(5)\ .key_down(Keys.ENTER)\ .pause(10)\ .key_up(Keys.ENTER)\ .perform() else: print('PRESS & HOLD container not found') time.sleep(100) # ...continue scraping driver.quit()
See below it worked!
While that does help with the press-and-hold problems, we may still be detected as bots by PerimeterX. To avoid that, we can use supplementary libraries, which leads us to the next solution.
B. Pro Solution: Anti-bot Bypass Tools
Because PerimeterX uses a combination of fingerprinting and behavior analysis, the "Please verify that you are human" challenge could come up at any time when scraping. However, we could strengthen our headless browser by fixing the known vulnerabilities. For example, extensions like the Undetected ChromeDriver for Selenium or the Puppeteer Stealth plugin will decrease the likelihood of anti-bot software identifying the browser.
Keep in mind the security level for PerimeterX differs from site to site. Namely, the press-and-hold button may be difficult to select, or you may face additional protection from PerimeterX alternatives. As a result, keeping up with bypass strategies can take time and effort.
Therefore, your best bet is to employ a premium solution like ZenRows to handle the complexity of getting blocked while web scraping and bypass the challenge. ZenRows includes premium proxies and everything needed to get around PerimeterX, which saves you hundreds of development hours. In addition, it's cheaper than implementing all the tools for bypassing individually. Sign up to get your 1,000 free API credits now.