How to Scrape With a Headless Browser in PHP

With dynamic and interactive websites dominating the World Wide Web, you'll need more than traditional scraping methods to scrape your desired data. PHP headless browsers provide an effective way to boost your data extraction in PHP.

However, selecting the right headless browser for your project is crucial for a positive outcome. In this tutorial, we'll cover what to look for in a headless browser and how to scrape data using a PHP headless browser.

How to Choose the Right PHP Headless Browser

Headless browser options for PHP are limited compared to other languages like Python.

However, even in this small pool, before you get to evaluating your options, it's important to define what to look out for by answering the question below:

What are your project needs?

For web scraping of any scale, you'll want to consider factors such as performance, ease of use, memory efficiency, resource intensity, browser support, and detection by protected websites.

With that in mind, below is an overview of the best PHP headless browsers, highlighting their strengths and weaknesses in the aforementioned areas.

ZenRows

ZenRows is an all-in-one web scraping API that provides everything you need to scrape without getting blocked, including a built-in headless browser.

As anti-bot systems evolve in their ability to identify and block bots, ZenRows offers up-to-date headless browser features that make it easy to mimic natural user behavior and go undetected.

Using ZenRows, you can emulate human-like page navigation, including varying scroll speeds, realistic mouse movements, and basically any page interaction necessary to achieve your desired results.

Unlike most headless browsers, ZenRows isn't resource-intensive. It is fast, easy to use, memory-efficient, and natively supports PHP. All this and more makes it the ultimate PHP web scraping solution.

Selenium

Although Selenium doesn't natively support PHP, the most widely used pairing for PHP web scraping is the Headless Chrome, controlled by the Selenium WebDriver.

How and why is that?

Firstly, there are popular third-party libraries and bindings, such as the PHP-WebDriver, that enable you to leverage Selenium WebDriver's functionality in PHP.

Secondly, Selenium is one of the most robust headless browsers with a large community of consistent users. This makes it easy to get started, as resources are readily available in developer forums and the like.

However, Selenium can get resource-intensive, especially when running multiple instances in parallel. While its automation properties allow you to simulate user actions (clicking, scrolling, etc.), anti-bot solutions can easily detect them.

Panther

Panther is a relatively new PHP headless browser built on top of Symfony's BrowserKit and DomCrawler APIs.

While this tool was developed to enhance Symfony's functional testing, providing a native way to test web applications in real browsers, it can also be used for web scraping.

Panther's ability to render JavaScript and interact with web page elements enables you to scrape dynamic content, access data that requires user interaction, and more. Since it uses the same API as the popular Goutte library, it can execute web scraping scenarios written for Goutte.

Even though Panther runs the Chrome browser by default, it allows you to control any browser that supports the WebDriver protocol.

What's more, it's easy to use and requires even less setup than Selenium. You only need your local Chrome and a Panther installation.

For more information, check out our Panther web scraping guide.

How to Use a Headless Browser With Selenium in PHP

Since Selenium is the most popular PHP headless browser among the options above, we'll walk you through, step by step, how to use the Selenium headless browser in PHP.

But before we dive in, let's understand Selenium headless browsers better.

What Headless Browsers Are Available in Selenium?

Selenium supports three main browsers: Chrome, Edge, and Firefox.

• Headless Chrome: Chrome is the most popular browser choice for Selenium-based automation due to its widespread adoption and testing compatibility.
• Headless Firefox: Firefox offers headless browsing support via the Gecko rendering engine.
• Headless Edge: Edge is a modern browser powered by the Chromium engine, which gives it the same headless capabilities as Chrome.

Now that that's out of the way, here's how you can set up Selenium in PHP.

Step 1: Set up a Headless Browser With Selenium PHP

Navigate to a directory where you'd like to store your code and initialize a PHP project by running the following command.

composer init

This command creates a composer.json file and prompts you to provide some project details. Follow the instructions and answer the questions to get started.

curl -sS https://getcomposer.org/installer | php

Next, you need a PHP port to the Selenium WebDriver. For this tutorial, we'll use the popular Facebook's php-webdriver.

Add the php-webdriver library to your project's dependencies.

composer require php-webdriver/webdriver

This command puts the package in the vendor folder, allowing you to connect to Selenium using your PHP script.

You'll also need a server that'll listen to these library commands and execute them in the corresponding browser.

Unlike other languages, for example, Python and Java, which can directly interact with the WebDriver, PHP needs this server to bridge that gap.

There are various options for this, the most common of which is the Selenium standalone server. Alternatively, you can send the PHP script to the respective browser driver (Chromedriver, Geckodriver, etc.).

For this tutorial, we'll follow the browser driver (Chromedriver) approach for simplicity.

Download the appropriate Chromedriver binary for your Chrome version and add it to your system PATH.

After adding Chromedriver to your system PATH, run the command below to send the PHP scripts directly to the browser driver (Chromedriver):

chromedriver --port=4444

You’ll get an output indicating that the server is running on port 4444.

# ... omitted for brevity ... #
ChromeDriver was started successfully on port 4444.

That's it! You're all set up.

Create a PHP file (scraper.php) and get ready to write some code.

Step 2: Open a Web Page With Selenium

Start by importing the necessary classes: ChromeOptions, DesiredCapabilities, and RemoteWebDriver.

<?php

use Facebook\WebDriver\Remote\RemoteWebDriver;
use Facebook\WebDriver\Chrome\ChromeOptions;
use Facebook\WebDriver\Remote\DesiredCapabilities;

// load Composer dependencies
require 'vendor/autoload.php';

You need ChromeOptions to set the browser options. The DesiredCapabilities class allows you to send the options to Selenium for the browser.

Lastly, the RemoteWebDriver is the main class that creates the connection variable you'll use to control the Chrome browser.

Next, let's define some settings. This includes the server URL or host and the desired capabilities (the properties of the browser you want to control).

// ...

// chromeDriver URL when started with --port=4444
$serverUrl = 'http://localhost:4444';


// initialize a desired capabilities instance
$desiredCapabilities = DesiredCapabilities::chrome();


// disable accepting SSL certificates
$desiredCapabilities->setCapability('acceptSslCerts', false);

// initialize a ChromeOptions object
$ChromeOptions = new ChromeOptions();
$desiredCapabilities->setCapability(ChromeOptions::CAPABILITY, $ChromeOptions);

// start the chromedriver using the settings above
$driver = RemoteWebDriver::create($serverUrl, $desiredCapabilities);

Now, let's navigate to a website in headed (GUI) mode and take a screenshot so you can see how Selenium works. We'll use the ScrapingCourse e-commerce test site as the target URL.

//...

// navigate to the target website 
$driver->get('https://www.scrapingcourse.com/ecommerce/'); 

// wait a few seconds for the page to load completely
sleep(5);

// take a screenshot 
$driver->takeScreenshot('screenshot.png'); 

// close the session 
$driver->quit(); 

echo "Screenshot taken and saved as 'screenshot.png'.\n";

Combine the code snippets above to get the following code.

<?php

use Facebook\WebDriver\Remote\RemoteWebDriver;
use Facebook\WebDriver\Chrome\ChromeOptions;
use Facebook\WebDriver\Remote\DesiredCapabilities;

// load Composer dependencies
require 'vendor/autoload.php';

// chromeDriver URL when started with --port=4444
$serverUrl = 'http://localhost:4444';

// initialize a desired capabilities instance
$desiredCapabilities = DesiredCapabilities::chrome();

// disable accepting SSL certificates
$desiredCapabilities->setCapability('acceptSslCerts', false);

// initialize a ChromeOptions object
$ChromeOptions = new ChromeOptions();
$desiredCapabilities->setCapability(ChromeOptions::CAPABILITY, $ChromeOptions);

// start the chromedriver using the settings above
$driver = RemoteWebDriver::create($serverUrl, $desiredCapabilities);

// navigate to a website
$driver->get('https://www.scrapingcourse.com/ecommerce/');

// wait a few seconds for the page to load completely
sleep(5);

// take a screenshot
$driver->takeScreenshot('screenshot.png');

// close the session
$driver->quit();

echo "Screenshot taken and saved as 'screenshot.png'.\n";

Without setting any Chrome options, Selenium launches the browser in headed (GUI) mode, navigates to the target website, and takes the screenshot below.

Step 3: Switch Selenium to Headless Mode

Now that you've seen how Selenium automates the browser in real-time, let's switch to headless mode to save resources and increase efficiency.

Enabling headless mode in Chrome is quite straightforward. You only need to add the --headless argument to the ChromeOptions object.

// ...

// initialize a ChromeOptions object
$ChromeOptions = new ChromeOptions();

// enable headless mode
$ChromeOptions->addArguments(['-headless']);

//...

Modify the previous code with this code snippet, but this time, let's retrieve and print the page title instead of a screenshot.

<?php

use Facebook\WebDriver\Remote\RemoteWebDriver;
use Facebook\WebDriver\Chrome\ChromeOptions;
use Facebook\WebDriver\Remote\DesiredCapabilities;

// load Composer dependencies
require 'vendor/autoload.php';

// chromeDriver URL when started with --port=4444
$serverUrl = 'http://localhost:4444';

// initialize a desired capabilities instance
$desiredCapabilities = DesiredCapabilities::chrome();

// disable accepting SSL certificates
$desiredCapabilities->setCapability('acceptSslCerts', false);

// initialize a ChromeOptions object
$ChromeOptions = new ChromeOptions();

// enable headless mode
$ChromeOptions->addArguments(['-headless']);


// initialize a desired capabilities instance
$desiredCapabilities->setCapability(ChromeOptions::CAPABILITY, $ChromeOptions);

// start the chromedriver using the settings above
$driver = RemoteWebDriver::create($serverUrl, $desiredCapabilities);

// navigate to a website 
$driver->get('https://www.scrapingcourse.com/ecommerce/'); 

// wait a few seconds for the page to load completely
sleep(5);

// print the page title to verify it loaded
echo "Page title: " . $driver->getTitle() . "\n";

// close the session 
$driver->quit();

This code launches the Chrome browser in headless mode, navigates to the target website, and prints the page title, as seen below.

Page title: Ecommerce Test Site to Learn Web Scraping - ScrapingCourse.com

Awesome!

Let's take things a step further by instructing Selenium to extract data while running in headless mode.

Step 4: Scrape Data With Selenium Headless Browser in PHP

In this section, you'll learn how to scrape data using Selenium PHP. We'll navigate to the target website, as in the previous steps, locate product data (name, price, and image URL), and extract them.

Selenium offers different ways to locate HTML elements in the DOM, including CSS selectors and XPath. In most cases, CSS selectors are the most reliable options; we'll use them in this tutorial.

Start by identifying the right selectors for the desired data points.

To do that, inspect the target page in a browser, right-click on a product card, and select Inspect. This will open the Developer Tools window, as shown in the image below.

You'll notice that each product is a list item with the class product. The following HTML elements within the list items represent each data point.

• Product name: <h2> with class product-name.
• Product price: span element with class, product-price.
• Product image: <img> tag with class product-image.

Using this information, locate all product items, loop through them, and extract their product name, price, and image URL.

<?php

// import the required classes
use Facebook\WebDriver\WebDriverBy;

// ...

// find all product items
$products = $driver->findElements(WebDriverBy::className('product'));

// loop through each product and extract product name, price, and image URL
foreach ($products as $product) {
    $productName = $product->findElement(WebDriverBy::className('product-name'))->getText();
    $productPrice = $product->findElement(WebDriverBy::className('product-price'))->getText();
    $productImage = $product->findElement(WebDriverBy::className('product-image'))->getAttribute('src');
   
    // print the extracted data
    echo "Product Name: $productName\n";
    echo "Product Price: $productPrice\n";
    echo "Product Image URL: $productImage\n";
    echo "\n";
}

Great!

Now, modify the previous script with this snippet to get the following complete code:

<?php


use Facebook\WebDriver\Remote\RemoteWebDriver;
use Facebook\WebDriver\Chrome\ChromeOptions;
use Facebook\WebDriver\Remote\DesiredCapabilities;
use Facebook\WebDriver\WebDriverBy;


// load Composer dependencies
require 'vendor/autoload.php';


// chromeDriver URL when started with --port=4444
$serverUrl = 'http://localhost:4444';


// initialize a desired capabilities instance
$desiredCapabilities = DesiredCapabilities::chrome();


// disable accepting SSL certificates
$desiredCapabilities->setCapability('acceptSslCerts', false);


// initialize a ChromeOptions object
$ChromeOptions = new ChromeOptions();


// enable headless mode
$ChromeOptions->addArguments(['-headless']);


// initialize a desired capabilities instance
$desiredCapabilities->setCapability(ChromeOptions::CAPABILITY, $ChromeOptions);


// start the Chrome browser using the settings above
$driver = RemoteWebDriver::create($serverUrl, $desiredCapabilities);


// navigate to a website
$driver->get('https://www.scrapingcourse.com/ecommerce/');


// wait a few seconds for the page to load completely
sleep(5);


// find all product items
$products = $driver->findElements(WebDriverBy::className('product'));


// loop through each product and extract product name, price, and image URL
foreach ($products as $product) {
    $productName = $product->findElement(WebDriverBy::className('product-name'))->getText();
    $productPrice = $product->findElement(WebDriverBy::className('product-price'))->getText();
    $productImage = $product->findElement(WebDriverBy::className('product-image'))->getAttribute('src');
   
    // print the extracted data
    echo "Product Name: $productName\n";
    echo "Product Price: $productPrice\n";
    echo "Product Image URL: $productImage\n";
    echo "\n";
}


// close the session
$driver->quit();

This script locates each product item on the target page and extracts its name, price, and image URL.

Here's what your console would look like:

Product Name: Abominable Hoodie
Product Price: $69.00
Product Image URL: https://www.scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/mh09-blue_main.jpg

Product Name: Adrienne Trek Jacket
Product Price: $57.00
Product Image URL: https://www.scrapingcourse.com/ecommerce/wp-content/uploads/2024/03/wj08-gray_main.jpg

// ... omitted for brevity ... //

Congratulations! You now know how to scrape with a PHP headless browser.

However, there's more.

With most websites employing solutions to mitigate bot traffic, you must overcome anti-bot challenges to take advantage of your newly acquired skill.

Here's how you can achieve that. 

Best Selenium Alternative in PHP to Avoid Getting Blocked

Selenium is a powerful automation tool and web scraping solution. Yet, it struggles against anti-bot solutions.

This is because it exhibits characteristics that indicate automated browsing, such as defined and consistent mouse movement and the presence of specific WebDriver flags that websites use to identify and block automated access.

While you can configure the WebDriver using web scraping proxies and other measures to hide the said automation properties, these methods can get tedious and most likely won't work against complex anti-bot solutions.

To scrape without getting blocked, consider the ZenRows' Universal Scraper API, the best solution for scalable web scraping in PHP.

With features such as advanced anti-bot bypass out of the box, geo-located requests, fingerprinting evasion, user agent spoofing, request header management, and more, ZenRows can handle any anti-bot solution for you.

Let's see ZenRows in action against an Antibot Challenge page.

To follow along with this example, sign up for free. You'll then be directed to the Request Builder page.

Input your target URL and activate Premium Proxies and the JS Rendering boost mode.

Next, select the PHP language and choose the API option.

Copy the generated code on the right to your editor for testing. Your code should look like this:

<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://api.zenrows.com/v1/?apikey=<YOUR_ZENROWS_API_KEY>&url=https%3A%2F%2Fwww.scrapingcourse.com%2Fantibot-challenge&js_render=true&premium_proxy=true');
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$response = curl_exec($ch);
echo $response . PHP_EOL;
curl_close($ch);
?>

This code bypasses the anti-bot challenge and retrieves the HTML.

<html lang="en">
<head>
    <!-- ... -->
    <title>Antibot Challenge - ScrapingCourse.com</title>
    <!-- ... -->
</head>
<body>
    <!-- ... -->
    <h2>
        You bypassed the Antibot challenge! :D
    </h2>
    <!-- other content omitted for brevity -->
</body>
</html>

Congratulations! You're now equipped for PHP web scraping at any scale.

Conclusion

If you made it this far, you now know how to scrape using a PHP headless browser. From selecting the best fit for your project to avoiding detection when web scraping, here's a quick recap of your progress.

You've learned how to:

• Set up a headless browser with Selenium PHP.
• Run Selenium in headless mode.
• Scrape data using Selenium in PHP.

Just remember that these steps only become helpful if you can overcome anti-bot challenges. While Selenium is a robust automation tool, it is often limited by these restrictions.

To scrape any website without getting blocked, consider ZenRows, an easy-to-integrate and scalable solution. Sign up now to try ZenRows for free!