Web data access
Every web data project starts with the same problem: turning a URL into content you can actually use. That means choosing between an HTTP request and a real browser, getting served at all when a site would rather not, discovering URLs you do not have yet, and keeping the whole thing running when you move from ten pages to ten million. The terms below cover how pages get fetched, why sites refuse them, the proxy infrastructure that changes that answer, how crawlers find URLs in the first place, and what breaks when volume goes up.
Fetching
Should I use a scraping API or build my own? Build your own when targets are unprotected and stable, and buy when you are spending engineering time on proxies, browsers and anti-bot handling instead of on the data. What does pay per successful request mean? It means you are only billed when a request returns the page, so blocked and failed attempts cost nothing, which matters because failure rates on protected sites are not small. What is a web scraping API? A web scraping API is a service you send a URL to and get page content back from, with proxies, browsers and anti-bot handling managed on its side rather than yours. What is JavaScript rendering in web scraping? JavaScript rendering means running a page in a real browser so scripts execute and build the content, rather than reading only the HTML the server first returned.Browsers
What are wait strategies in browser automation? Wait strategies decide when a script proceeds: sleeping for a fixed time, or waiting for a condition such as an element appearing, which is the difference between flaky automation and reliable automation. What is a headless browser? A headless browser is a full browser running without a visible window, driven by code, which lets a script load pages and interact with them exactly as a person's browser would. What is browser automation? Browser automation is controlling a real browser from code, issuing clicks, keystrokes and navigations so a script can work through a page the way a person would. What is browser session management? Session management is keeping the state that makes a sequence of requests look like one continuous visit: cookies, storage, and the same IP address throughout.Getting blocked
How do websites detect web scrapers? Websites compare what your request claims to be against what it demonstrably is, and flag the mismatches, from the TLS handshake up to the rhythm of your browsing. How do you bypass CAPTCHAs when scraping? Stop the challenge from being issued, by making the request look like an ordinary visitor across IP, TLS, headers and fingerprint, rather than trying to answer a challenge after it appears. What is a CAPTCHA solver? A CAPTCHA solver is a service that receives a challenge a site has already issued and returns an answer token, either from a model or from a person working through a queue. What is a JavaScript challenge? A JavaScript challenge is an interstitial page that runs code in your browser and only serves the real content once that code returns a valid result, which filters out clients that do not execute JavaScript. What is a web application firewall? A WAF inspects HTTP traffic before it reaches an application and blocks requests matching known attack patterns, which is why scrapers sometimes get refused by a rule that was never aimed at them. What is an anti-bot system? An anti-bot system is a layer in front of a website that scores each request on network, browser and behavioural signals, then serves, challenges or blocks it based on that score. What is an anti-detect browser? An anti-detect browser is a modified browser that presents a controllable, internally consistent fingerprint, so each profile it runs looks like a separate ordinary machine. What is browser fingerprinting? Browser fingerprinting identifies a visitor by combining dozens of properties their browser reveals, such as fonts, screen size, graphics rendering and timezone, into a signature that stays stable without needing a cookie. What is protected web access? Protected web access means reaching pages that sit behind anti-bot protection, where an ordinary HTTP request gets a block page instead of the content. What is TLS fingerprinting? TLS fingerprinting identifies your client from the exact contents and ordering of its encryption handshake, which happens before any HTTP header is sent and cannot be changed by setting a user agent.Proxies
Residential or datacenter proxy: which do I need? A residential proxy routes requests through an IP assigned by a home internet provider, so it looks like an ordinary visitor. A datacenter proxy uses a server IP, which is faster and far cheaper but easy for a site to recognise and block. What is a proxy in web scraping? A proxy is an intermediary server that forwards your request so the target site sees the proxy's IP address instead of yours, which is how scrapers spread traffic across many identities. What is a reverse proxy? A reverse proxy sits in front of a web server and answers on its behalf, which is why it is usually the thing deciding whether your scraper gets the page. What is a rotating proxy? A rotating proxy assigns a different IP address to each request or after a set interval, so a site sees your traffic as many visitors rather than one. Why do free proxy lists fail for scraping? Free proxies are mostly dead, already blocked, or misconfigured, and the working minority are shared with enough other traffic that the sites you care about have blocked them already.Crawling
How does a web crawler work? A crawler keeps a queue of URLs, fetches one at a time, extracts the links it finds, adds the new ones back to the queue, and repeats until the queue empties or a limit is reached. What is a seed URL? A seed URL is where a crawl starts, and the choice matters because everything the crawler ever reaches has to be discoverable by following links from it. What is crawl budget? Crawl budget is how many pages a crawler will fetch from a site in a given period, set by what the site can bear and by what the crawler thinks is worth its time. What is URL normalization? URL normalization rewrites URLs into one canonical form so that addresses pointing at the same page collapse into a single entry instead of being crawled several times.Running at volume
What is batch web scraping? Batch scraping submits a list of URLs as a single job that runs asynchronously, so retries, pacing and result collection are handled by the job rather than by a loop you supervise. What is concurrency in web scraping? Concurrency is how many requests are in flight at once, and it is the main lever on how fast a scrape finishes, up to the point where pushing it harder gets you blocked. What is the best way to scrape a large website? Split the job into URL discovery and content fetching, run the fetching as managed batches rather than a live loop, and decide per URL whether it needs a browser, because rendering everything is usually what makes a large crawl unaffordable.Get reliable web data in minutes.
Free plan, 5,000 credits every month, no credit card required.