Crawl: the most reliable web crawler APISee what’s new
Zenrows
Talk to sales Start free

Google Scholar scraper

The Zenrows Google Scholar scraper turns any Scholar search URL into JSON: paper title, authors, publication, year, publisher, snippet, cited-by count, number of versions, Scholar ID and full-text PDF link for every result, plus related searches and the total result count. Fetch in Adaptive Stealth Mode opens the page and Extract returns the fields. Run it from an API call, the CLI or an AI agent.

Site
scholar.google.com
Returns
Papers with authors, venues, years, citation counts and PDF links
Runs on
Fetch, Extract, Batch, Crawl
Updated
{
  "search_query": { "terms": "large language models" },
  "result_summary": { "total_results_approximate": 6650000, "response_time_seconds": 0.11 },
  "articles": [
    {
      "title": "A survey on evaluation of large language models",
      "authors": ["Y Chang", "X Wang", "J Wang", "Y Wu", "L Yang"],
      "publication_name": "ACM transactions on …",
      "publication_year": 2024,
      "publisher_or_host": "dl.acm.org",
      "cited_by_count": 6954,
      "versions_count": 11,
      "scholar_id": "o93zfHYlUTIJ",
      "article_url": "https://dl.acm.org/doi/abs/10.1145/3641289",
      "fulltext_format": "PDF",
      "fulltext_url": "https://dl.acm.org/doi/pdf/10.1145/3641289"
    },
    {
      "title": "Large language models: A survey",
      "authors": ["S Minaee", "T Mikolov", "N Nikzad", "M Chenaghlu"],
      "publication_name": "arXiv preprint arXiv …",
      "publication_year": 2024,
      "publisher_or_host": "arxiv.org",
      "cited_by_count": 3613,
      "versions_count": 7,
      "scholar_id": "msVvatpT9iUJ",
      "article_url": "https://arxiv.org/abs/2402.06196",
      "fulltext_format": "PDF",
      "fulltext_url": "https://arxiv.org/pdf/2402.06196"
    }
  ]
}

Request and output of a live call to a Google Scholar search on Oct 20, 2026, trimmed to two of the ten articles returned. A field is returned when the result shows it.

What data can you scrape from Google Scholar?

Every Scholar result is a paper record: who wrote it, where and when it appeared, how often it is cited and where to read it.

Extract returns each one in articles, in ranking order, with the search context around it.

FieldWhat it holdsExample
articles[].title Paper title A survey on evaluation of large language models
articles[].authors[] Author names as Scholar abbreviates them ["Y Chang", "X Wang", …]
articles[].publication_name Journal, conference or preprint server ACM transactions on …
articles[].publication_year Year of publication 2024
articles[].publisher_or_host Publisher or the site hosting the paper dl.acm.org
articles[].snippet The text excerpt Scholar shows for the query … 3.1 Natural Language Processing Tasks …
articles[].cited_by_count "Cited by" count 6954
articles[].versions_count Number of versions Scholar grouped together 11
articles[].scholar_id Scholar's ID for the result, stable between runs o93zfHYlUTIJ
articles[].article_url Link to the paper's landing page https://dl.acm.org/doi/abs/10.1145/3641289
articles[].fulltext_url Direct full-text link, when Scholar shows one https://dl.acm.org/doi/pdf/10.1145/3641289
articles[].fulltext_format Format of that full text PDF, HTML
articles[].all_versions_url Scholar link to every version of the paper /scholar?cluster=3625720365842685347…
articles[].related_articles_url Scholar's "Related articles" link /scholar?q=related:o93zfHYlUTIJ…
result_summary.total_results_approximate Scholar's "About N results" count 6650000
related_searches[] Suggested queries with their Scholar URLs { "query_text": "prompt large language models" }

How to scrape Google Scholar with Zenrows

  1. Create a free Zenrows account and copy your API key from the dashboard.
  2. Build the Scholar URL. https://scholar.google.com/scholar?q= plus your terms. Add as_ylo=2023 to start from a year, as_yhi to end at one, and start=10 for the second page of ten.
  3. Add hl=en for consistent labels. Scholar's interface language follows the visitor, and an English interface keeps venue names and links uniform across runs.
  4. Send it with the default configuration: Adaptive Stealth Mode to open the page and Extract to return the fields. The sample above ran without a country.
  5. Run topics and pages in Batch, with an ID per query and page, when one search returns what you want.
# pip install requests
import requests

params = {
    "apikey": "YOUR_ZENROWS_API_KEY",
    "url": "https://scholar.google.com/scholar?q=large+language+models",
    "mode": "auto",      # Adaptive Stealth Mode
    "extract": "auto",   # structured JSON
}
response = requests.get("https://api.zenrows.com/v1/", params=params, timeout=180)
response.raise_for_status()

parsed = response.json()["parsed"]
print(parsed["result_summary"]["total_results_approximate"], "results")
for paper in parsed["articles"]:
    print(paper["publication_year"], paper["cited_by_count"], paper["title"])

Input

The scraper reads one Scholar results URL per request.

Terms, year range and page number are all URL parameters, so a literature sweep is a list of URLs sent with the settings below.

ParameterTypeDefaultDescription
url string required A Scholar results URL, such as https://scholar.google.com/scholar?q=large+language+models. Add as_ylo/as_yhi for years, start for pages, cites= for papers citing one result
mode string auto Adaptive Stealth Mode: starts with the lightest setup and adds rendering and protected-site handling only when Scholar needs them
extract string auto Returns articles, related_searches, result_summary and search_query as JSON in parsed
extract_fields string all fields Comma-separated paths to keep, such as articles.title,articles.cited_by_count,articles.scholar_id
proxy_country string any Two-letter country code. Rarely needed for Scholar; useful when a journal link or interface language depends on location
wait_for string none A CSS selector to wait for before the page is read
session_id integer none Keep the same session for up to 10 minutes while you page through one query
Batch input
{
  "type": "regular",
  "status": "closed",
  "zenrows_params": {
    "mode": "auto",
    "extract": "auto"
  },
  "tasks": [
    { "url": "https://scholar.google.com/scholar?q=large+language+models", "external_id": "llm-p1" },
    { "url": "https://scholar.google.com/scholar?q=large+language+models&start=10", "external_id": "llm-p2" },
    { "url": "https://scholar.google.com/scholar?q=retrieval+augmented+generation&as_ylo=2024", "external_id": "rag-2024" }
  ]
}

Output

The response holds html, the Scholar page as served, and parsed, the structured results. Scholar links inside the results (all versions, related articles, related searches) are relative paths; prefix them with https://scholar.google.com to request them. Trimmed from the live response:

Scholar truncates long venue names and author lists with …; article_url leads to the full record on the publisher's site. For a reading list, pd.json_normalize(parsed["articles"]) gives one paper per row, ready for CSV, a reference manager import or a database.

Search results
{
  "search_query": { "terms": "large language models" },
  "result_summary": {
    "response_time_seconds": 0.11,
    "total_results_approximate": 6650000
  },
  "articles": [
    {
      "title": "A comprehensive overview of large language models",
      "authors": ["H Naveed", "AU Khan", "S Qiu", "M Saqib", "S Anwar"],
      "publication_name": "ACM Transactions on …",
      "publication_year": 2025,
      "publisher_or_host": "dl.acm.org",
      "snippet": "… Large Language Models (LLMs) have recently demonstrated remarkable capabilities in \nnatural language processing tasks and beyond…",
      "cited_by_count": 3843,
      "versions_count": 10,
      "scholar_id": "GWi_IEWnRVoJ",
      "article_url": "https://dl.acm.org/doi/abs/10.1145/3744746",
      "fulltext_format": "PDF",
      "fulltext_url": "https://dl.acm.org/doi/pdf/10.1145/3744746",
      "all_versions_url": "/scholar?cluster=6504789152150546457&hl=en&as_sdt=0,33",
      "related_articles_url": "/scholar?q=related:GWi_IEWnRVoJ:scholar.google.com/&scioq=large+language+models&hl=en&as_sdt=0,33"
    }
  ],
  "related_searches": [
    {
      "query_text": "emergent large language models",
      "search_url": "/scholar?hl=en&as_sdt=0,33&qsp=1&q=emergent+large+language+models&qst=ib"
    },
    {
      "query_text": "prompt large language models",
      "search_url": "/scholar?hl=en&as_sdt=0,33&qsp=3&q=prompt+large+language+models&qst=ib"
    }
  ]
}

Connect the MCP server (Claude Code)

claude mcp add zenrows -e ZENROWS_API_KEY=YOUR_ZENROWS_API_KEY -- npx -y @zenrows/mcp

MCP config (Cursor, Claude Desktop, Windsurf)

{
  "mcpServers": {
    "zenrows": {
      "command": "npx",
      "args": ["-y", "@zenrows/mcp"],
      "env": { "ZENROWS_API_KEY": "YOUR_ZENROWS_API_KEY" }
    }
  }
}

Or run the CLI

npx -y @zenrows/cli init
zenrows extract "https://scholar.google.com/scholar?q=large+language+models"

Then ask your agent

Use Zenrows to search Google Scholar for "retrieval augmented generation"
from 2024 onward. Read the first three pages, list the ten most cited
papers with year, venue and PDF link, and summarize the three newest.

Scrape Google Scholar from your coding agent

Claude Code, Cursor, Codex and any MCP client can search Google Scholar and read papers as JSON from a prompt.

The agent runs the same Fetch and Extract request as the code above, so it can rank papers by citations, open the PDF links and summarize them in one session. Connect the Zenrows MCP server once, or install the CLI and let the agent call it in the terminal.

Scrape Google Scholar at scale with Batch and Crawl

Batch runs a literature sweep as one job. Each task is one query and page (start=0, 10, 20), the ID names both, and results come back tagged so you can merge pages and drop duplicates by scholar_id. Rerun it monthly to watch citation counts grow.

Crawl takes over once you have the papers. Scholar points to publishers and repositories; start a crawl on a repository or lab publication list you found there, keep the links that lead to papers (on arXiv, /abs/), and collect the abstracts behind every result.

Scheduled jobs run the same Batch job on a timetable, on Zenrows' side: every few hours, once a day, at set times on chosen days, or once at a future date. Each run keeps its own results, a webhook can tell your app when a run finishes, and you can pause or resume the schedule at any time, with no cron server to keep alive.

curl -X POST "https://async.api.zenrows.com/v1/jobs" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "regular",
    "status": "closed",
    "zenrows_params": { "mode": "auto", "extract": "auto" },
    "tasks": [
      { "url": "https://scholar.google.com/scholar?q=large+language+models", "external_id": "llm-p1" },
      { "url": "https://scholar.google.com/scholar?q=large+language+models&start=10", "external_id": "llm-p2" }
    ]
  }'

# Read the results when the job finishes
curl "https://async.api.zenrows.com/v1/jobs/JOB_ID/results?status=successful" \
  -H "X-API-Key: YOUR_ZENROWS_API_KEY"

What teams do with Google Scholar data

Use caseData they pullWho runs it
Literature reviews Titles, authors, years, venues and PDF links for every query in a review protocol Researchers, PhD students and research librarians
Citation tracking Cited-by counts per paper, month over month Labs, universities and research offices
Research trend analysis Result counts and top papers per topic and year range Analysts, funders and science journalists
Expert and partner discovery The authors behind the most cited work on a topic R&D, biotech and pharma teams
Training and RAG corpora Paper metadata plus full-text links to fetch AI and data teams
Competitive R&D intelligence Papers by a company's researchers and where they publish Corporate strategy and IP teams

Tips for scraping Google Scholar

  • Page with start. Scholar shows ten results per page; start=10 is page two. Each page is its own URL, so put them all in one Batch job.
  • Narrow by year in the URL. as_ylo=2024 keeps papers from 2024 on, as_yhi=2022 stops at 2022. Year slices also help with broad topics, since each slice brings different papers forward.
  • Follow "Cited by" with cites=. The citing papers of a result are a Scholar results page of their own; request it like any search and read articles again.
  • Deduplicate on scholar_id. The same paper can rank for several queries; its Scholar ID stays the same, while titles can differ slightly between versions.
  • Prefix relative links. all_versions_url, related_articles_url and related_searches[].search_url start with /scholar; add https://scholar.google.com before requesting them.
  • Use phrase quotes for exact terms. q=%22large+language+models%22 matches the exact phrase, which cuts noise on common words.

Frequently asked questions

  • Scholar's results pages, with titles, authors, venues and citation counts, are public and shown to anyone who searches, and collecting bibliographic metadata for research is common. Author names are personal data under laws such as the GDPR, so use them for the research purpose you collect them for. Check Google's terms and the publishers' terms for full texts, and ask your lawyer if you are unsure.

  • No. Google has never published an API for Scholar. Open alternatives such as Crossref, OpenAlex and Semantic Scholar offer APIs with their own coverage, but Scholar's ranking, cited-by counts and version grouping are only on the Scholar pages, which is why researchers use a scraper for them.

  • Search results (scholar.google.com/scholar?q=…), including year-filtered searches, later pages, "Cited by" lists (cites=), all-versions lists (cluster=) and related-article lists. All of them are Scholar results pages, built like the search itself.

  • Yes. Every result carries cited_by_count, and versions_count for the number of grouped versions. Run the same queries on a schedule to track how citations grow.

  • Scholar links a full text when one is available, and the scraper returns it in fulltext_url with its format. Request that URL with Fetch to get the file or page, and respect the publisher's terms for full texts.

  • Send a GET request to https://api.zenrows.com/v1/ with your API key, the Scholar URL, Adaptive Stealth Mode and Extract turned on, as in the Python example above. response.json()["parsed"]["articles"] is a list of paper dictionaries. The zenrows package on PyPI wraps the same request.

  • Each page returns ten results. Send every page and every year slice as its own URL in a Batch job, which runs them in parallel and keeps your ID on each result. One Batch job accepts more than 10,000 URLs, inline or as a CSV file.

  • Yes. Connect the Zenrows MCP server to Claude, Cursor, Codex or any MCP client, or give your agent the Zenrows CLI. Ask for papers on a topic and the agent runs Fetch and Extract, reads the JSON and can open the full texts.

  • Yes. Create a scheduled Batch job with your Google Scholar URLs and the same settings as a single request, and Zenrows runs it on its side: every few hours, once a day, at set times on chosen days, or once at a future date. Each run keeps its own results, a webhook can notify your app when a run finishes, and the schedule can be paused or resumed at any time.

  • Zenrows works with n8n, Make, Zapier, Clay, Dify, LangChain, LlamaIndex, CrewAI and Mastra, plus the Python, Node.js and Go SDKs and any HTTP client. LlamaIndex and LangChain suit research assistants that read the papers they find. See the integrations page for the full list.

  • JSON by default, with the papers in parsed and the original page in html. You can also ask for Markdown or plain text, or convert the JSON to CSV for a spreadsheet or reference manager.

Turn the live web into usable data.

Start building free 5,000 credits/month, always