Your cursor movements are actually exposing your web scraper as a bot. Web scraping tools like Puppeteer make predictable mouse movements that are easily detected by anti-bots. But Ghost Cursor can mitigate this with realistic human interactions.
In this tutorial, you'll learn to simulate natural human interactions and improve stealth with Ghost Cursor and Puppeteer.
What Is Ghost Cursor?
Ghost Cursor is a Node.js library for generating realistic, human-like cursor movements when scraping with a browser automation tool like Puppeteer. Unlike static, robotic mouse jumps, it replicates natural cursor paths, moving the cursor in a slow, curved, slightly unpredictable path just like a human.
Keep in mind that Ghost Cursor isn't another stealth library. That said, its ability to simulate human cursor navigation makes your scraper less detectable by anti-bot mechanisms that flag bot-like mouse behaviors.
You'll learn to use this tool with Puppeteer in the next section.
How to Set Up and Use Ghost Cursor
In this part, you'll learn to generate human-like cursor movements with Ghost Cursor when scraping with Puppeteer. We'll use the E-commerce Challenge page as the target site, simulating some movement and click events.
Step 1: Requirements and Installation
The first step is to install the Ghost Cursor and Puppeteer libraries using npm:
npm install ghost-cursor puppeteer
Ghost Cursor also works with Puppeteer Extra. You can pair them to boost stealth.
Step 2: Creating and Using the Cursor
Ghost Cursor injects cursor events directly into Puppeteer's page instance. Let's see an example that clicks the next page button on the target site.
First, import the Puppeteer library and the createCursor method from ghost-cursor. Create a scraper function that accepts a URL parameter and starts a new page instance from Puppeteer's browser instance. Create a cursor and connect it to the page instance. Visit the target page and simulate a click event with ghost-cursor:
// npm install ghost-cursor puppeteer
const puppeteer = require('puppeteer');
const { createCursor } = require('ghost-cursor');
const scraper = async (url) => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
// create a cursor and connect it to the page instance
const cursor = createCursor(page);
// open the target page
await page.goto(url);
// simulate a click event with ghost-cursor
const selector = '.next'
await page.waitForSelector(selector);
await cursor.click(selector);
// get the current page title
const title = await page.title();
console.log(title);
await browser.close();
};
scraper('https://www.scrapingcourse.com/ecommerce/');
The above code returns the next page title, as shown, proving the click event simulation went smoothly:
Ecommerce Test Site to Learn Web Scraping - Page 2 - ScrapingCourse.com
Let's test the simulation visually by introducing the installMousehelper method, a ghost-cursor function for monitoring cursor movement. This time, run Puppeteer in non-headless mode for debugging purposes.
Update the previous code with the following changes:
// npm install ghost-cursor puppeteer
const puppeteer = require('puppeteer');
const { createCursor } = require('ghost-cursor');
const scraper = async (url) => {
const browser = await puppeteer.launch();
const page = await browser.newPage({ headless: true });
// monitor cursor movement visually
await installMouseHelper(page);
// create a cursor and connect it to the page instance
const cursor = createCursor(page);
// open the target page
await page.goto(url);
// simulate a click event with ghost-cursor
const selector = '.next';
await page.waitForSelector(selector);
await cursor.click(selector);
// get the current page title
const title = await page.title();
console.log(title);
await browser.close();
};
scraper('https://www.scrapingcourse.com/ecommerce/');
In the demo result below, notice how the cursor moves from the top-left to the next button in the demo result below. The movement is also a bit less random:
The linear-like movement of the cursor is still susceptible to anti-bot detection because it's slightly predictable. You can improve it by setting performRandomMoves to true and changing the vector coordinate. Note that these options are anonymous parameters for the createCursor method. You can implement them as shown:
// ...
const scraper = async (url) => {
// ...
// create a cursor and connect it to the page instance
const cursor = createCursor(page, { x: 114.78, y: 97.52 }, true);
// ...
};
You can also modify the click action to be more human by adding random click delays. The click action accepts the following options:
**hesitate**: Pauses for a specified period (milliseconds) before clicking an element.**waitForClick**: Holds the click for some time in milliseconds.**moveDelay**: Specifies how long the cursor should wait (milliseconds) before acting on the target element.- **
randomizeMoveDelay**: Accepts Boolean options to randomize delay between 0 and themoveDelayvalue. **button**: Accepts aleftorrightoption to specify the click type (left or right-click).**clickCount**: The number of clicks on an element. Its default value is 1.
The code below adds delays to the click event:
// ...
const scraper = async (url) => {
// ...
await cursor.click(selector, {
hesitate: 1000, // waits 1 second before clicking
waitForClick: 200, // holds the click for 200ms
moveDelay: 3000, // delay mouse movement after clicking
randomizeMoveDelay: true, // randomized move delay from 0 to the value of moveDelay
});
// ...
};
Update the previous code with these changes, and you'll get the following complete code:
// npm install ghost-cursor puppeteer
const puppeteer = require('puppeteer');
const { createCursor, installMouseHelper } = require('ghost-cursor');
const scraper = async (url) => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
// monitor cursor movement visually
await installMouseHelper(page);
// create a cursor and connect it to the page instance
const cursor = createCursor(page, { x: 114.78, y: 97.52 }, true);
// open the target page
await page.goto(url);
// simulate a click event with ghost-cursor
const selector = '.next';
await page.waitForSelector(selector);
await cursor.click(selector, {
hesitate: 1000, // waits 1 second before clicking
waitForClick: 200, // holds the click for 200ms
moveDelay: 3000, // delay mouse movement after clicking
randomizeMoveDelay: true, // randomized move delay from 0 to the value of moveDelay
});
// get the current page title
const title = await page.title();
console.log('Page Title:', title);
await browser.close();
};
scraper('https://www.scrapingcourse.com/ecommerce/');
Let's run the above script in non-headless mode to view the cursor simulation:
const scraper = async (url) => {
const browser = await puppeteer.launch({ headless: false });
// ...
};
The cursor moves more randomly this time, using the specified delays and coordinates. See the demo below:
That's great! You just simulated human behavior with the Ghost Cursor.
Other Cursor Events
In addition to `click`, `ghost-cursor` supports other cursor actions. Here are the major ones with their simulation options.
move
This event moves the cursor to a specified selector. In addition to moveDelay and randomizedDelay, which perform similar functions as those in the click event, the move event accepts the following options:
**paddingPercentage**: Sets the cursor's padding position (in percentage) within an element.**destination**: The cursor's destination relative to the element's top-left corner based on a specified coordinate.**maxTries**: The maximum number of times to hover over the target element. It defaults to 0.**moveSpeed**: Sets the cursor speed. This is randomized by default.**overshootThreshold**: Defines how far the cursor can move (in pixels) beyond the intended target before correcting its position.
You can add these options like so:
await cursor.move(
selector,
{ x: 278, y: 300 },
{
paddingPercentage: 80,
moveSpeed: 40,
maxTries: 5,
overshootThreshold: 20,
}
);
moveTo
This method moves the cursor to a defined position based on specified coordinates. It also accepts the moveSpeed, moveDelay, and randomizedDelay options:
wait cursor.moveTo(
{ x: 278, y: 300 },
{
moveSpeed: 40,
moveDelay: 1000,
randomizeMoveDelay: 2000,
}
);
scrollIntoView
This cursor event scrolls an element into view if it's not already. It accepts the following options:
**scrollSpeed**: Specifies the scroll speed. The value varies between 0 and 100.**scrollDelay**: Pauses the scroll action for a specified period.**inViewportMargin**: The margin size around the target element. It determines whether the element is in view and ready for interaction.
await cursor.scrollIntoView(selector, {
scrollSpeed: 50,
scrollDelay: 2000,
inViewportMargin: 20,
});
scrollTo
This method scrolls the page to the top, bottom, left, right, or to a specified coordinate:
await cursor.scrollTo('bottom');
scroll
This parameter scrolls the page to a given coordinate. It also accepts the scrollDelay and scrollSpeed options:
await cursor.scroll(
{ x: 200.78, y: 300.25 },
{ scrollDelay: 2000, scrollSpeed: 10 }
);
Tips for Using Ghost Cursor
The following tips can help mitigate abrupt execution failures while using Ghost Cursor:
- Issue 1: Missing element
- Solution 1: Wait for the page to load completely before starting the cursor simulation.
- Solution 2: Ensure the target element is present and available for interaction before implementing a
clickormovecursor event.
- Issue 2: Incomplete data after scrolling
- Solution: If dealing with infinite scrolling or a "load more" button, use scroll delays and pause after each scroll action to allow elements to load.
- Issue 3: Immature execution context termination
- Solution: Only close the browser after completing all scraping actions and logic.
You now know how to simulate realistic human interactions with the ghost-cursor. However, the tool has some limitations that are worth knowing.
Common Limitations of Ghost Cursor
Despite its ability to heavily humanize mouse movements, the Ghost Cursor library still has the following limitations:
- Inability to bypass anti-bot mechanisms: Ghost Cursor only increases your chance of bypassing anti-bots by a small fraction since it's not a stealth plugin. It doesn't stop your scraper from getting blocked by CAPTCHAs and other anti-bot measures.
- Limited to behavioral factors: The tool only focuses on optimizing the cursor movement to be as human as possible. It doesn't patch other bot-like patterns. Anti-bots often test other parameters, including browser fingerprints, request headers, IP addresses, and more, making Ghost Cursor vulnerable to these detection techniques.
- Shadow element limitation: While Ghost Cursor can move to any position within the DOM, it can't access the position of a shadow element, even with the correct coordinates. This further makes it less effective for solving CAPTCHAs or checkbox challenges, which are typically buried within a shadow DOM.
For instance, Ghost Cursor with Puppeteer gets blocked when scraping a protected site like the Anti-bot Challenge page.
Try it out with the following code. The code simulates mouse movement to the CAPTCHA position and tries to click it. It then takes a screenshot of the page to view the cursor position:
// npm install ghost-cursor puppeteer
const puppeteer = require('puppeteer');
const { createCursor, installMouseHelper } = require('ghost-cursor');
const scraper = async (url) => {
const browser = await puppeteer.launch();
const page = await browser.newPage();
// monitor cursor movement visually
await installMouseHelper(page);
// create a cursor and connect it to the page instance
const cursor = createCursor(page);
// open the target page
await page.goto(url);
await new Promise((resolve) => setTimeout(resolve, 10000));
// move to the CAPTCHA coordinate
await cursor.moveTo(
{ x: 68, y: 270 },
{
moveSpeed: 30,
moveDelay: 1000,
}
);
await new Promise((resolve) => setTimeout(resolve, 4000));
// try to click the CAPTCHA checkbox
await cursor.click();
// take a screenshot of the page
await page.screencast({ path: 'screenshot.png' });
await browser.close();
};
scraper('https://www.scrapingcourse.com/antibot-challenge');
The above code gets blocked, as shown in the screenshot below. The cursor gets stuck above the CAPTCHA box, showing that the Ghost Cursor library can't access the checkbox position:
Unfortunately, Ghost Cursor fails to bypass advanced anti-bot measures. Even if it manages to click the CAPTCHA checkbox, it won't work because the anti-bot can still detect obvious bot signals from Puppeteer.
Let's see a lasting solution in the next section.
Avoid Getting Blocked
Although Ghost Cursor is an excellent tool for simulating human interactions, anti-bots look beyond behavioral patterns to detect bots. This makes Ghost Cursor insufficient against advanced anti-bot measures. Pairing it with stealth tools like Puppeteer Extra also doesn't work, as the plugin leaks many bot-like signals.
The best way to bypass any anti-bot measure at scale with minimal effort is to use a web scraping solution, such as the ZenRows Universal Scraper API. With a single click, ZenRows automatically adds the required human touch to your scraping requests and handles all anti-bot measures under the hood.
ZenRows routes your requests through premium rotating proxies and has headless browser capabilities to interact with dynamic elements, making it a suitable replacement for Puppeteer.
Let's see how ZenRows' Universal Scraper API works by scraping the previous Antibot Challenge page that blocked your ghost-cursor scraper.
Sign up and head on to the Request Builder. Paste the target URL in the URL field and activate the Premium Proxies and JS Rendering options.
Select Node.js as your programming language and choose the API connection mode. Copy the generated code and paste it into your scraper.
The generated JavaScript code should look like this:
// npm install axios
const axios = require('axios');
const url = 'https://www.scrapingcourse.com/antibot-challenge';
const apikey = '<YOUR_ZENROWS_API_KEY>';
axios({
url: 'https://api.zenrows.com/v1/',
method: 'GET',
params: {
url: url,
apikey: apikey,
js_render: 'true',
premium_proxy: 'true',
},
})
.then((response) => console.log(response.data))
.catch((error) => console.log(error));
The above code bypasses the anti-bot challenge and outputs the protected site's full-page HTML, as shown:
<html lang="en">
<head>
<!-- ... -->
<title>Antibot Challenge - ScrapingCourse.com</title>
<!-- ... -->
</head>
<body>
<!-- ... -->
<h2>
You bypassed the Antibot challenge! :D
</h2>
<!-- other content omitted for brevity -->
</body>
</html>
Congratulations! 🎉 You just used ZenRows to bypass an advanced anti-bot protection. Say goodbye to getting blocked during data collection!
Conclusion
You've learned how to simulate human interactions using Ghost Cursor with Puppeteer. Ghost Cursor undoubtedly makes browser automation more realistic and can increase the chances of bypassing interaction-based detection.
However, Ghost Cursor becomes ineffective when it comes to modern, advanced anti-bot measures. Large-scale, real-world projects require an all-in-one scraping solution like ZenRows to bypass complex anti-bot measures. With ZenRows, you can scrape any website confidently without limitations.