---
title: "Social media scraping"
description: "Social media scraping for public posts, profiles, comments and engagement from Reddit, YouTube, TikTok and X. Structured JSON from public pages, at volume."
canonical: https://www.zenrows.com/use-cases/social-media-scraping
updated: 2026-10-18
---

# Social media scraping

Social media scraping collects public posts, profiles, comments, hashtags and engagement counts from platforms such as Reddit, YouTube, TikTok, Instagram and X. Zenrows fetches the public pages a logged-out visitor can see, renders their JavaScript, and returns them as HTML, Markdown or structured JSON, ready for listening, creator research and trend tracking.

## Sample output: one public video post

```json
{
  "platform": "tiktok",
  "url": "https://www.tiktok.com/@example_account/video/7425000000000000000",
  "account": {
    "handle": "@example_account",
    "followers": 184300,
    "verified": false
  },
  "caption": "Three ways to fold a fitted sheet, ranked",
  "hashtags": ["laundryhacks", "homeorganization"],
  "posted_at": "2026-10-09T17:20:00Z",
  "duration_seconds": 41,
  "views": 1250000,
  "likes": 96400,
  "comments": 1830,
  "shares": 5210,
  "sound": "original sound",
  "collected_at": "2026-10-12T09:00:00Z"
}
```

An illustrative public post as JSON. The page shows counts like "1.2M"; store them as numbers with the time you collected them, because engagement keeps moving after the post goes up.

## What teams collect from social media

Most social data projects read a handful of public page types: a profile, a post, its comment thread, and a search or hashtag result page. The job decides which of those you fetch and how often you come back.

| Job | Data you collect | Typical sources |
|---|---|---|
| Social listening | Posts and comments that mention a brand, product or topic | Reddit search and threads, YouTube comments, TikTok search, public forums |
| Influencer research | Public profile stats, recent posts, engagement per post | TikTok, YouTube and Instagram public profiles, Twitch channels |
| Trend and hashtag tracking | Hashtag volume, top posts, sounds and formats over time | TikTok hashtag pages, YouTube search, X public posts |
| Comment mining for product feedback | Comment text, replies, likes, timestamps under relevant posts | YouTube review videos, Reddit product communities |
| Competitor content benchmarking | Posting cadence, formats, engagement on competitor accounts | Public brand pages on TikTok, YouTube, Instagram and X |
| Community and creator platforms | Posts, subscriber counts, comment threads | substack.com, fandom.com, wordpress.com blogs, threads.com |

## Why social media scraping fails

Social platforms assemble each page in the browser from a script bundle. Counts change by the minute, and the platforms rate-limit repeated requests from one IP.

| What breaks | What it looks like | How Zenrows handles it |
|---|---|---|
| Rate limits and bot checks | Requests from one IP start returning errors, empty pages or challenge screens on Reddit and TikTok | Zenrows' stealth and retries get through the platforms' rate limits and bot checks |
| JavaScript-built pages | The HTML has a script bundle and no post text or counts | Adaptive Stealth Mode escalates to browser rendering and returns the page once the post has loaded |
| Infinite scroll and "load more" | You get the first ten comments of a thread with thousands | Page actions scroll and click before the page is returned; Browser runs longer scroll sessions in Playwright |
| Region-specific feeds | Trending and search pages differ by country | Geotargeting fetches the page as a visitor in that market sees it |
| Frequent redesigns | Class names change and selector-based parsers return blanks | Extract returns structured fields from the page without selectors to maintain |
| Volume | Re-checking 50,000 posts for updated engagement every day | Batch submits the whole URL list as one job, with webhooks when it finishes |

## How to scrape social media data with Zenrows

1. **List the public pages.** Decide which profiles, posts, searches or hashtags you need. Use only pages a logged-out visitor can open.
2. **Fetch with Adaptive Stealth Mode.** Send each URL to Fetch, which escalates to browser rendering and stealth only when a platform needs it.
3. **Load the full thread when it matters.** For long comment sections, add page actions that scroll before the page is returned, or open a Browser session and scroll in Playwright.
4. **Structure and store.** Use Extract or your own CSS selectors for fields, convert counts to numbers, and keep a collection timestamp on every record.
5. **Refresh on a schedule.** Re-fetch recent posts with Batch to build an engagement curve for each post.

```python Python
# pip install requests
import requests

params = {
    "apikey": "YOUR_ZENROWS_API_KEY",
    "url": VIDEO_URL,  # a public video or post page
    "mode": "auto",  # Adaptive Stealth Mode
    "extract": "auto",
}
response = requests.get("https://api.zenrows.com/v1/", params=params)
print(response.json())
```

```javascript Node.js
// npm install playwright
import { chromium } from "playwright";

const THREAD_URL = "https://www.reddit.com/r/example/comments/abc123/";  // placeholder
const COMMENT_SELECTOR = "[data-testid='comment']";  // example selector

const browser = await chromium.connectOverCDP(
  "wss://browser.zenrows.com?apikey=YOUR_ZENROWS_API_KEY"
);
const page = await browser.newPage();
await page.goto(THREAD_URL);

// Scroll until the thread stops growing, then read every comment
for (let i = 0; i < 20; i++) {
  await page.mouse.wheel(0, 3000);
  await page.waitForTimeout(1200);
}
const comments = await page.$$eval(COMMENT_SELECTOR, (nodes) =>
  nodes.map((n) => n.innerText)
);
console.log(comments.length, "comments");
await browser.close();
```

```bash CLI
npx -y @zenrows/cli init
# A public video page as structured JSON
zenrows extract VIDEO_URL --autoparse
# Daily engagement refresh: one {"url": "..."} per tracked post
zenrows batch create posts.jsonl --wait
zenrows batch results JOB_ID --out engagement.jsonl
```

```text Claude Code (MCP)
# Connect once
claude mcp add zenrows -e ZENROWS_API_KEY=YOUR_ZENROWS_API_KEY -- npx -y @zenrows/mcp

# Then ask:
Use Zenrows to open the public YouTube channel pages of these four brands and list each channel's last ten videos with title, upload date, views and comment count.
```

## Which Zenrows product fits

| If you need to | Use | Why |
|---|---|---|
| Fetch a public post, profile or search page | [Fetch](/products/fetch) | Renders JavaScript and handles platform rate limits in one request |
| Get post and profile fields without writing selectors | [Extract](/products/extract) | Automatic structured extraction returns JSON that adapts when the layout changes |
| Scroll a long comment thread or click through tabs | [Browser](/products/browser) | A real browser over CDP, driven by your Playwright or Puppeteer code |
| Refresh engagement on thousands of posts daily | [Batch](/products/batch) | One job for the full URL list, results by webhook or download |
| Let an analyst pull public social data in a chat | [MCP Server](/mcp) | Claude, Cursor and other MCP clients call Zenrows as a tool |

## What success looks like

| Measure | What good looks like | How to check it |
|---|---|---|
| Field completeness | Text, author handle, timestamp and engagement counts present on nearly every post | Report empty fields per platform after each run |
| Thread depth | Comment collection reaches the end of the thread, or a depth you set | Compare comments stored with the count the post displays |
| Engagement history | Each tracked post has readings at fixed intervals, not one snapshot | Count readings per post in the first week after posting |
| Platform coverage | Every platform in scope returns data on every run | Track successful requests per platform per day |
| Public-only scope | No record comes from a login-gated or private page | Audit source URLs against your list of allowed page types |
| Refresh run time | The daily engagement refresh finishes inside its window as the post list grows | Log start and end time of each Batch job |

## Social media scraping FAQ

### What is social media scraping?

Social media scraping is the automated collection of public content from social platforms: posts, profiles, comments, hashtags and engagement counts. Teams use it for social listening, creator research, trend tracking and building datasets for sentiment models. It reads the same public pages a logged-out visitor sees and turns them into structured records.

### Is it legal to scrape social media data?

It depends on what you collect, where, and how you use it. Collecting publicly visible content is widely practised, but platform terms apply, and posts and profiles often contain personal data covered by laws such as GDPR and CCPA. Collect only what your purpose needs, avoid sensitive categories, and pseudonymise handles where you can. This is not legal advice: confirm your use case with your legal team.

### What about private accounts or content behind a login?

Zenrows is built for public data: the profiles, posts and threads a logged-out visitor can open. For data about your own accounts, such as ad metrics or page insights, the platform's official API with your credentials is the right source, and you can join it with the public data you collect.

### Which platforms can I collect from?

Any platform whose content is visible to a logged-out visitor. Common targets are Reddit threads and search, YouTube videos, channels and comments, TikTok videos, profiles and hashtag pages, public X posts, and blogs and communities such as Substack and Fandom. How much a platform shows without login varies, so test your exact page types before you scale up.

### Should I use a social media scraping API or the official platform API?

Official APIs suit data about your own accounts and give stable fields, but they cap volume, cost money on several platforms, and often exclude other accounts' public content. A scraping API reads public pages across platforms in one format. A common setup uses both, with the official API for owned channels and scraping for the public conversation.

### How do I scrape social media data without coding?

Connect the Zenrows MCP server to Claude, Cursor or another MCP client and ask in plain English, for example for the last ten videos on a set of YouTube channels with their views. The agent fetches the public pages and returns a table. For repeat jobs, the CLI runs the same extraction from a terminal or a cron schedule with one command.

### Can I get historical posts?

You can collect what the platform still shows publicly, such as a profile's older videos or a thread's full comments. Engagement history is different: a page only shows today's counts, so start collecting early and store a reading on each visit.

## Related use cases

- [Sentiment analysis data](https://www.zenrows.com/use-cases/sentiment-analysis-data.md): Review text with ratings, forum threads and comments, cleaned and labeled for sentiment and NLP models.
- [Brand monitoring](https://www.zenrows.com/use-cases/brand-monitoring.md): Every mention of your brand in news, forums, reviews and search, as clean text for sentiment and alerts.
- [Review scraping](https://www.zenrows.com/use-cases/review-scraping.md): Review text, ratings, dates and replies from Amazon, Google, Yelp, Trustpilot and G2, every page.
- [Competitor ad intelligence](https://www.zenrows.com/use-cases/competitor-ads.md): Creatives, copy and landing pages from ad libraries and paid search results, tracked by advertiser and country.

Start free: https://app.zenrows.com/register. Docs: https://docs.zenrows.com