Google Scholar scraper
The Zenrows Google Scholar scraper turns any Scholar search URL into JSON: paper title, authors, publication, year, publisher, snippet, cited-by count, number of versions, Scholar ID and full-text PDF link for every result, plus related searches and the total result count. Fetch in Adaptive Stealth Mode opens the page and Extract returns the fields. Run it from an API call, the CLI or an AI agent.
- Site
scholar.google.com- Returns
- Papers with authors, venues, years, citation counts and PDF links
- Updated
{
"search_query": { "terms": "large language models" },
"result_summary": { "total_results_approximate": 6650000, "response_time_seconds": 0.11 },
"articles": [
{
"title": "A survey on evaluation of large language models",
"authors": ["Y Chang", "X Wang", "J Wang", "Y Wu", "L Yang"],
"publication_name": "ACM transactions on …",
"publication_year": 2024,
"publisher_or_host": "dl.acm.org",
"cited_by_count": 6954,
"versions_count": 11,
"scholar_id": "o93zfHYlUTIJ",
"article_url": "https://dl.acm.org/doi/abs/10.1145/3641289",
"fulltext_format": "PDF",
"fulltext_url": "https://dl.acm.org/doi/pdf/10.1145/3641289"
},
{
"title": "Large language models: A survey",
"authors": ["S Minaee", "T Mikolov", "N Nikzad", "M Chenaghlu"],
"publication_name": "arXiv preprint arXiv …",
"publication_year": 2024,
"publisher_or_host": "arxiv.org",
"cited_by_count": 3613,
"versions_count": 7,
"scholar_id": "msVvatpT9iUJ",
"article_url": "https://arxiv.org/abs/2402.06196",
"fulltext_format": "PDF",
"fulltext_url": "https://arxiv.org/pdf/2402.06196"
}
]
}{
"url": "https://scholar.google.com/scholar?q=large+language+models",
"mode": "auto",
"extract": "auto"
}Request and output of a live call to a Google Scholar search on Oct 20, 2026, trimmed to two of the ten articles returned. A field is returned when the result shows it.
What data can you scrape from Google Scholar?
Every Scholar result is a paper record: who wrote it, where and when it appeared, how often it is cited and where to read it.
Extract returns each one in articles, in ranking order, with the search context around it.
| Field | What it holds | Example |
|---|---|---|
articles[]. |
Paper title | A survey on evaluation of large language models |
articles[]. |
Author names as Scholar abbreviates them | ["Y Chang", "X Wang", …] |
articles[]. |
Journal, conference or preprint server | ACM transactions on … |
articles[]. |
Year of publication | 2024 |
articles[]. |
Publisher or the site hosting the paper | dl.acm.org |
articles[]. |
The text excerpt Scholar shows for the query | … 3.1 Natural Language Processing Tasks … |
articles[]. |
"Cited by" count | 6954 |
articles[]. |
Number of versions Scholar grouped together | 11 |
articles[]. |
Scholar's ID for the result, stable between runs | o93zfHYlUTIJ |
articles[]. |
Link to the paper's landing page | https://dl.acm.org/doi/abs/10.1145/3641289 |
articles[]. |
Direct full-text link, when Scholar shows one | https://dl.acm.org/doi/pdf/10.1145/3641289 |
articles[]. |
Format of that full text | PDF, HTML |
articles[]. |
Scholar link to every version of the paper | /scholar?cluster=3625720365842685347… |
articles[]. |
Scholar's "Related articles" link | /scholar?q=related:o93zfHYlUTIJ… |
result_ |
Scholar's "About N results" count | 6650000 |
related_ |
Suggested queries with their Scholar URLs | { "query_text": "prompt large language models" } |
How to scrape Google Scholar with Zenrows
- Create a free Zenrows account and copy your API key from the dashboard.
- Build the Scholar URL.
https://scholar.google.com/scholar?q=plus your terms. Addas_ylo=2023to start from a year,as_yhito end at one, andstart=10for the second page of ten. - Add
hl=enfor consistent labels. Scholar's interface language follows the visitor, and an English interface keeps venue names and links uniform across runs. - Send it with the default configuration: Adaptive Stealth Mode to open the page and Extract to return the fields. The sample above ran without a country.
- Run topics and pages in Batch, with an ID per query and page, when one search returns what you want.
# pip install requests
import requests
params = {
"apikey": "YOUR_ZENROWS_API_KEY",
"url": "https://scholar.google.com/scholar?q=large+language+models",
"mode": "auto", # Adaptive Stealth Mode
"extract": "auto", # structured JSON
}
response = requests.get("https://api.zenrows.com/v1/", params=params, timeout=180)
response.raise_for_status()
parsed = response.json()["parsed"]
print(parsed["result_summary"]["total_results_approximate"], "results")
for paper in parsed["articles"]:
print(paper["publication_year"], paper["cited_by_count"], paper["title"])// Node 18+ (built-in fetch)
const params = new URLSearchParams({
apikey: "YOUR_ZENROWS_API_KEY",
url: "https://scholar.google.com/scholar?q=large+language+models",
mode: "auto",
extract: "auto",
});
const response = await fetch(`https://api.zenrows.com/v1/?${params}`, {
signal: AbortSignal.timeout(180_000),
});
if (!response.ok) throw new Error(`${response.status}: ${await response.text()}`);
const { parsed } = await response.json();
for (const a of parsed.articles) console.log(a.publication_year, a.cited_by_count, a.title);curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fscholar.google.com%2Fscholar%3Fq%3Dlarge%2Blanguage%2Bmodels&mode=auto&extract=auto"
# Only titles, years, citation counts and PDF links
curl "https://api.zenrows.com/v1/?apikey=YOUR_ZENROWS_API_KEY&url=https%3A%2F%2Fscholar.google.com%2Fscholar%3Fq%3Dlarge%2Blanguage%2Bmodels&mode=auto&extract=auto&extract_fields=articles.title%2Carticles.publication_year%2Carticles.cited_by_count%2Carticles.fulltext_url"# pip install zenrows
from zenrows import ZenRowsClient
client = ZenRowsClient("YOUR_ZENROWS_API_KEY")
response = client.get(
"https://scholar.google.com/scholar?q=large+language+models&as_ylo=2025",
params={"mode": "auto", "extract": "auto"},
)
for paper in response.json()["parsed"]["articles"]:
if paper.get("fulltext_format") == "PDF":
print(paper["title"], paper["fulltext_url"])Input
The scraper reads one Scholar results URL per request.
Terms, year range and page number are all URL parameters, so a literature sweep is a list of URLs sent with the settings below.
| Parameter | Type | Default | Description |
|---|---|---|---|
url |
string | required | A Scholar results URL, such as https://scholar.google.com/scholar?q=large+language+models. Add as_ylo/as_yhi for years, start for pages, cites= for papers citing one result |
mode |
string | auto |
Adaptive Stealth Mode: starts with the lightest setup and adds rendering and protected-site handling only when Scholar needs them |
extract |
string | auto |
Returns articles, related_searches, result_summary and search_query as JSON in parsed |
extract_ |
string | all fields | Comma-separated paths to keep, such as articles.title,articles.cited_by_count,articles.scholar_id |
proxy_ |
string | any | Two-letter country code. Rarely needed for Scholar; useful when a journal link or interface language depends on location |
wait_ |
string | none | A CSS selector to wait for before the page is read |
session_ |
integer | none | Keep the same session for up to 10 minutes while you page through one query |
{
"type": "regular",
"status": "closed",
"zenrows_params": {
"mode": "auto",
"extract": "auto"
},
"tasks": [
{ "url": "https://scholar.google.com/scholar?q=large+language+models", "external_id": "llm-p1" },
{ "url": "https://scholar.google.com/scholar?q=large+language+models&start=10", "external_id": "llm-p2" },
{ "url": "https://scholar.google.com/scholar?q=retrieval+augmented+generation&as_ylo=2024", "external_id": "rag-2024" }
]
}
Output
The response holds html, the Scholar page as served, and parsed, the structured results. Scholar links inside the results (all versions, related articles, related searches) are relative paths; prefix them with https://scholar.google.com to request them. Trimmed from the live response:
Scholar truncates long venue names and author lists with …; article_url leads to the full record on the publisher's site. For a reading list, pd.json_normalize(parsed["articles"]) gives one paper per row, ready for CSV, a reference manager import or a database.
{
"search_query": { "terms": "large language models" },
"result_summary": {
"response_time_seconds": 0.11,
"total_results_approximate": 6650000
},
"articles": [
{
"title": "A comprehensive overview of large language models",
"authors": ["H Naveed", "AU Khan", "S Qiu", "M Saqib", "S Anwar"],
"publication_name": "ACM Transactions on …",
"publication_year": 2025,
"publisher_or_host": "dl.acm.org",
"snippet": "… Large Language Models (LLMs) have recently demonstrated remarkable capabilities in \nnatural language processing tasks and beyond…",
"cited_by_count": 3843,
"versions_count": 10,
"scholar_id": "GWi_IEWnRVoJ",
"article_url": "https://dl.acm.org/doi/abs/10.1145/3744746",
"fulltext_format": "PDF",
"fulltext_url": "https://dl.acm.org/doi/pdf/10.1145/3744746",
"all_versions_url": "/scholar?cluster=6504789152150546457&hl=en&as_sdt=0,33",
"related_articles_url": "/scholar?q=related:GWi_IEWnRVoJ:scholar.google.com/&scioq=large+language+models&hl=en&as_sdt=0,33"
}
],
"related_searches": [
{
"query_text": "emergent large language models",
"search_url": "/scholar?hl=en&as_sdt=0,33&qsp=1&q=emergent+large+language+models&qst=ib"
},
{
"query_text": "prompt large language models",
"search_url": "/scholar?hl=en&as_sdt=0,33&qsp=3&q=prompt+large+language+models&qst=ib"
}
]
}
Connect the MCP server (Claude Code)
claude mcp add zenrows -e ZENROWS_API_KEY=YOUR_ZENROWS_API_KEY -- npx -y @zenrows/mcp
MCP config (Cursor, Claude Desktop, Windsurf)
{
"mcpServers": {
"zenrows": {
"command": "npx",
"args": ["-y", "@zenrows/mcp"],
"env": { "ZENROWS_API_KEY": "YOUR_ZENROWS_API_KEY" }
}
}
}
Or run the CLI
npx -y @zenrows/cli init
zenrows extract "https://scholar.google.com/scholar?q=large+language+models"
Then ask your agent
Use Zenrows to search Google Scholar for "retrieval augmented generation"
from 2024 onward. Read the first three pages, list the ten most cited
papers with year, venue and PDF link, and summarize the three newest.
Scrape Google Scholar from your coding agent
Claude Code, Cursor, Codex and any MCP client can search Google Scholar and read papers as JSON from a prompt.
The agent runs the same Fetch and Extract request as the code above, so it can rank papers by citations, open the PDF links and summarize them in one session. Connect the Zenrows MCP server once, or install the CLI and let the agent call it in the terminal.
Scrape Google Scholar at scale with Batch and Crawl
Batch runs a literature sweep as one job. Each task is one query and page (start=0, 10, 20), the ID names both, and results come back tagged so you can merge pages and drop duplicates by scholar_id. Rerun it monthly to watch citation counts grow.
Crawl takes over once you have the papers. Scholar points to publishers and repositories; start a crawl on a repository or lab publication list you found there, keep the links that lead to papers (on arXiv, /abs/), and collect the abstracts behind every result.
Scheduled jobs run the same Batch job on a timetable, on Zenrows' side: every few hours, once a day, at set times on chosen days, or once at a future date. Each run keeps its own results, a webhook can tell your app when a run finishes, and you can pause or resume the schedule at any time, with no cron server to keep alive.
curl -X POST "https://async.api.zenrows.com/v1/jobs" \
-H "X-API-Key: YOUR_ZENROWS_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"type": "regular",
"status": "closed",
"zenrows_params": { "mode": "auto", "extract": "auto" },
"tasks": [
{ "url": "https://scholar.google.com/scholar?q=large+language+models", "external_id": "llm-p1" },
{ "url": "https://scholar.google.com/scholar?q=large+language+models&start=10", "external_id": "llm-p2" }
]
}'
# Read the results when the job finishes
curl "https://async.api.zenrows.com/v1/jobs/JOB_ID/results?status=successful" \
-H "X-API-Key: YOUR_ZENROWS_API_KEY"# queries.jsonl: one {"url": "...", "external_id": "...", "extract": "auto"} per line
zenrows batch create queries.jsonl --wait
zenrows batch results <job-id> --out scholar-papers.jsonlcurl -X POST "https://api.zenrows.com/v1/crawls" \
-H "X-API-Key: YOUR_ZENROWS_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://arxiv.org/list/cs.CL/recent",
"depth": 1,
"max_pages": 100,
"include_patterns": ["/abs/"]
}'# Runs every day on Zenrows' side; each run keeps its own results
curl -X POST "https://async.api.zenrows.com/v1/jobs" \
-H "X-API-Key: YOUR_ZENROWS_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"type": "scheduled",
"status": "closed",
"schedule": {
"rate": {
"every": 1,
"unit": "day"
}
},
"zenrows_params": {
"mode": "auto",
"extract": "auto"
},
"tasks": [
{
"url": "https://scholar.google.com/scholar?q=large+language+models",
"external_id": "google-scholar"
}
]
}'
# Or at set times: "schedule": { "calendar": { "times_of_day": ["09:00"],
# "cadence": { "weekly": { "days": ["mon", "thu"] } } }, "timezone": "Europe/Madrid" }What teams do with Google Scholar data
| Use case | Data they pull | Who runs it |
|---|---|---|
| Literature reviews | Titles, authors, years, venues and PDF links for every query in a review protocol | Researchers, PhD students and research librarians |
| Citation tracking | Cited-by counts per paper, month over month | Labs, universities and research offices |
| Research trend analysis | Result counts and top papers per topic and year range | Analysts, funders and science journalists |
| Expert and partner discovery | The authors behind the most cited work on a topic | R&D, biotech and pharma teams |
| Training and RAG corpora | Paper metadata plus full-text links to fetch | AI and data teams |
| Competitive R&D intelligence | Papers by a company's researchers and where they publish | Corporate strategy and IP teams |
Tips for scraping Google Scholar
- Page with
start. Scholar shows ten results per page;start=10is page two. Each page is its own URL, so put them all in one Batch job. - Narrow by year in the URL.
as_ylo=2024keeps papers from 2024 on,as_yhi=2022stops at 2022. Year slices also help with broad topics, since each slice brings different papers forward. - Follow "Cited by" with
cites=. The citing papers of a result are a Scholar results page of their own; request it like any search and readarticlesagain. - Deduplicate on
scholar_id. The same paper can rank for several queries; its Scholar ID stays the same, while titles can differ slightly between versions. - Prefix relative links.
all_versions_url,related_articles_urlandrelated_searches[].search_urlstart with/scholar; addhttps://scholar.google.combefore requesting them. - Use phrase quotes for exact terms.
q=%22large+language+models%22matches the exact phrase, which cuts noise on common words.
More Google Scholar and academic & research scrapers
-
scholar.google.com
Google Scholar author profile scraper
Author profiles with affiliation, research interests, h-index, i10-index and listed publications.
-
scholar.google.com
Google Scholar citations scraper
Papers citing a given article, with titles, authors, years and their own citation counts.
-
wikipedia.org
Wikipedia scraper
Encyclopedia articles with summaries, infobox facts, categories, references and last edit dates.
-
arxiv.org
ArXiv scraper
Preprints with titles, authors, abstracts, subject categories and PDF links.
-
books.google.com
Google Books scraper
Book listings with authors, publishers, publication dates, ISBNs and descriptions.
-
wiley.com
Wiley scraper
Journal articles and books with authors, abstracts, journals and DOIs.
- bmj.com BMJ scraper Medical research papers and clinical articles with authors, abstracts and DOIs.
-
springer.com
Springer scraper
Journal articles and book chapters with authors, abstracts and DOIs.
- tandfonline.com Taylor & Francis Online scraper Journal articles with authors, abstracts, publication dates and DOIs.
Frequently asked questions
-
Scholar's results pages, with titles, authors, venues and citation counts, are public and shown to anyone who searches, and collecting bibliographic metadata for research is common. Author names are personal data under laws such as the GDPR, so use them for the research purpose you collect them for. Check Google's terms and the publishers' terms for full texts, and ask your lawyer if you are unsure.
-
No. Google has never published an API for Scholar. Open alternatives such as Crossref, OpenAlex and Semantic Scholar offer APIs with their own coverage, but Scholar's ranking, cited-by counts and version grouping are only on the Scholar pages, which is why researchers use a scraper for them.
-
Search results (
scholar.google.com/scholar?q=…), including year-filtered searches, later pages, "Cited by" lists (cites=), all-versions lists (cluster=) and related-article lists. All of them are Scholar results pages, built like the search itself. -
Yes. Every result carries
cited_by_count, andversions_countfor the number of grouped versions. Run the same queries on a schedule to track how citations grow. -
Scholar links a full text when one is available, and the scraper returns it in
fulltext_urlwith its format. Request that URL with Fetch to get the file or page, and respect the publisher's terms for full texts. -
Send a GET request to
https://api.zenrows.com/v1/with your API key, the Scholar URL, Adaptive Stealth Mode and Extract turned on, as in the Python example above.response.json()["parsed"]["articles"]is a list of paper dictionaries. Thezenrowspackage on PyPI wraps the same request. -
Each page returns ten results. Send every page and every year slice as its own URL in a Batch job, which runs them in parallel and keeps your ID on each result. One Batch job accepts more than 10,000 URLs, inline or as a CSV file.
-
Yes. Connect the Zenrows MCP server to Claude, Cursor, Codex or any MCP client, or give your agent the Zenrows CLI. Ask for papers on a topic and the agent runs Fetch and Extract, reads the JSON and can open the full texts.
-
Yes. Create a scheduled Batch job with your Google Scholar URLs and the same settings as a single request, and Zenrows runs it on its side: every few hours, once a day, at set times on chosen days, or once at a future date. Each run keeps its own results, a webhook can notify your app when a run finishes, and the schedule can be paused or resumed at any time.
-
Zenrows works with n8n, Make, Zapier, Clay, Dify, LangChain, LlamaIndex, CrewAI and Mastra, plus the Python, Node.js and Go SDKs and any HTTP client. LlamaIndex and LangChain suit research assistants that read the papers they find. See the integrations page for the full list.
-
JSON by default, with the papers in
parsedand the original page inhtml. You can also ask for Markdown or plain text, or convert the JSON to CSV for a spreadsheet or reference manager.