Ever wondered why your real estate scraper keeps getting blocked or coming back with half-empty rows? Modern property sites lean on JavaScript, spread key details across several pages, and filter traffic that doesn't look like a real browser. On the surface, your requests may even succeed, but the data you care about never fully arrives.
This guide explains why that happens and how to approach web scraping real estate data to get consistently structured results.
What Makes Web Scraping Real Estate Data Challenging
When you’re looking to scrape real estate data, you’re usually interested in a fixed set of fields. For example: price, address, beds, baths, and the listing URL.
The real work is dealing with summary pages versus full listing pages. Search results often show only a short snapshot, while full descriptions, status, and extra details live on separate property pages. Some values appear only after JavaScript rendering and background APIs, so a plain HTML response can look fine but still hide the numbers you’re trying to collect.
On top of that, platforms rely on aggressive anti-bot defenses. They use bot checks, IP filtering, throttling, pagination limits, and region rules. Your scraper might work for a few pages, then suddenly start returning missing listings or 403 errors, even though the same URLs still open normally in your browser.
Building A Simple Scraper With Requests And BeautifulSoup
To see how a real estate site behaves when you scrape real estate data without any scraping infrastructure, start with a Requests and BeautifulSoup scraper. It sends one HTTP request to a search results page, tries to read prices and addresses straight from the HTML, and writes whatever it finds to a CSV file.
Inspecting The Target Page
The first step is to look at the page you want to scrape and see how it exposes the data. In this guide, you’ll use Zillow’s homes-for-sale view in San Francisco as an example search results page.
Open that page in Chrome, right-click one of the property cards, and choose Inspect. In DevTools, you’ll see li elements with data-testid="property-card".
Inside each card, there’s: an address under a[data-test="property-card-link"] > address, a price in span[data-test="property-card-price"], and a link to the detail page in a[data-test="property-card-title-link"]. These are the same pieces your script will try to read from the HTML response.
Installing Dependencies
You only need three Python modules for this first pass.
requestslibrary: Sends the HTTP request and give you the response body.beautifulsoup4: Parses the HTML so you can search for tags and attributes.csvmodule: To write the data you extract into a CSV file.
Install the external libraries with:
pip3 install requests beautifulsoup4
The csv module is a Python standard file, so you don't need to install it.
Writing The Basic Scraper
Start by importing the modules, setting the URL, and adding headers so the request looks like it’s coming from a real browser.
import requests
from bs4 import BeautifulSoup
import csv
url = "https://www.zillow.com/homes/for_sale/San-Francisco_rb/"
# set headers to mimic a real browser request
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/142.0.0.0 Safari/537.36",
"Accept-Language": "en-US,en;q=0.9",
"Referer": "https://www.google.com/"
}
Next, send the request and print the status code so you can see if the site responded.
# ...
# fetch the page
response = requests.get(url, headers=headers)
print("Status code:", response.status_code)
If the response is successful, parse the HTML, look for the same li[data-testid="property-card"] nodes, and treat each one as a separate listing. For every card, your scraper then walks the elements inside it to pull out the address text, the price, and the link to the property page, turning those pieces into a small record you can store.
That record will later become a single row in your CSV file, so each visible card in the UI has a matching entry in your data. If the response isn’t successful, skip parsing altogether, log the status code, and treat it as a blocked or failed request instead of trying to scrape a broken page.
# ...
if response.status_code == 200:
soup = BeautifulSoup(response.content, 'html.parser')
results = []
# find all list items containing property cards
property_items = soup.find_all("li", {"data-testid": "property-card"})
for item in property_items:
try:
# extract address using selector: a[data-test='property-card-link'] > address
address_link = item.find("a", {"data-test": "property-card-link"})
address_elem = address_link.find("address") if address_link else None
address = address_elem.get_text(strip=True) if address_elem else ""
# extract price using selector: span[data-test='property-card-price']
price_elem = item.find("span", {"data-test": "property-card-price"})
price = price_elem.get_text(strip=True) if price_elem else ""
# extract url from: a[data-test='property-card-title-link'] @href
title_elem = item.find("a", {"data-test": "property-card-title-link"})
url_link = title_elem.get("href", "") if title_elem else ""
full_url = f"https://www.zillow.com{url_link}" if url_link and not url_link.startswith("http") else url_link
# only add if we found at least address or price
if address or price:
results.append({
"address": address,
"price": price,
"link": full_url,
})
except Exception as e:
print(f"error parsing card: {e}")
Finally, if you collected any rows, save them to a CSV file so you can inspect the output.
# ...
# save to CSV
if results:
with open("zillow_results.csv", "w", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=["address", "price", "link"])
writer.writeheader()
for row in results:
writer.writerow(row)
print(f"\nscraped {len(results)} listings and saved to 'zillow_results.csv'")
else:
print("\nno listings found.")
else:
print(f"failed to fetch page. Status code: {response.status_code}")
The above code writes every result into zillow_results.csv with one row per property and columns for address, price, and link. The DictWriter call maps each dictionary key to the right column, then writerow appends each listing. If the results are empty, skip the file creation and print an error message.
Here is the full simple scraper code.
import requests
from bs4 import BeautifulSoup
import csv
url = "https://www.zillow.com/homes/for_sale/San-Francisco_rb/"
# set headers to mimic a real browser request
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
"Accept-Language": "en-US,en;q=0.9",
"Referer": "https://www.google.com/"
}
# fetch the page
response = requests.get(url, headers=headers)
print("Status code:", response.status_code)
if response.status_code == 200:
soup = BeautifulSoup(response.content, 'html.parser')
results = []
# find all list items containing property cards
property_items = soup.find_all("li", {"data-testid": "property-card"})
for item in property_items:
try:
# extract address using selector: a[data-test='property-card-link'] > address
address_link = item.find("a", {"data-test": "property-card-link"})
address_elem = address_link.find("address") if address_link else None
address = address_elem.get_text(strip=True) if address_elem else ""
# extract price using selector: span[data-test='property-card-price']
price_elem = item.find("span", {"data-test": "property-card-price"})
price = price_elem.get_text(strip=True) if price_elem else ""
# extract url from: a[data-test='property-card-title-link'] @href
title_elem = item.find("a", {"data-test": "property-card-title-link"})
url_link = title_elem.get("href", "") if title_elem else ""
full_url = f"https://www.zillow.com{url_link}" if url_link and not url_link.startswith("http") else url_link
# only add if we found at least address or price
if address or price:
results.append({
"address": address,
"price": price,
"link": full_url,
})
except Exception as e:
print(f"error parsing card: {e}")
# save to CSV
if results:
with open("zillow_results.csv", "w", newline="", encoding="utf-8") as f:
writer = csv.DictWriter(f, fieldnames=["address", "price", "link"])
writer.writeheader()
for row in results:
writer.writerow(row)
print(f"\nscraped {len(results)} listings and saved to 'zillow_results.csv'")
else:
print("\nno listings found.")
else:
print(f"failed to fetch page. Status code: {response.status_code}")
When you run the script, you'll see:
Status code: 403
failed to fetch page. Status code: 403
A 403 status code means forbidden. The server got your request but refused to return the page. In this case, your script was treated as unwanted traffic and blocked. Zillow uses strict anti-bot protection, so even full browser automation hits the same wall. At this point, tweaking headers or selectors isn’t enough. You need a Zillow scraping setup that’s built to bypass anti-bot mechanisms like ZenRows.
Extracting Real Estate Data With ZenRows
The basic scraper couldn’t even test the CSS selectors before getting blocked. Instead of piling on more headers, delays, or browser automation, this is where you let ZenRows handle blocking, JavaScript, and proxies while you focus on the data you actually need.
ZenRows' Universal Scraper API loads real estate pages like a browser, handles anti-bot checks and IP rotation, and returns structured JSON for each listing that you can write straight to a CSV or a database.
Step 1: Set Up the Request in the ZenRows Request Builder
Sign up for ZenRows if you haven't already. Then open the Universal Scraper API Request Builder in the dashboard. In the URL field, paste the same Zillow search link you used for the basic scraper.
Enable JavaScript Rendering and Premium Proxies. In the code panel, pick Python and switch to the API tab.
The request builder will generate a ready-to-use Python code you can use to scrape the target URL.
# pip install requests
import requests
url = 'https://www.zillow.com/homes/for_sale/San-Francisco_rb/'
apikey = "<YOUR_ZENROWS_API_KEY>"
params = {
'url': url,
'apikey': apikey,
'js_render': 'true',
'premium_proxy': 'true',
}
response = requests.get('https://api.zenrows.com/v1/', params=params)
print(response.text)
When you run this code, you'll get the data in HTML format:
<!-- [Prior HTML omitted for brevity] -->
"sale","countryCurrency":"$","price":"$795,000","unformattedPrice":795000,"address":"515 Head St, San Francisco, CA 94132","addressStreet":"515 Head St","addressCity":"San Francisco","addressState":"CA","addressZipcode":"94132","isUndisclosedAddress":false,"beds":3,"baths":2,"area":1220,"latLong":
<!-- [Rest of HTML omitted for brevity] -->
This confirms the page now loads through the API instead of returning a 403, but it doesn’t yet give you the clean data. Instead of maintaining your own BeautifulSoup parsing code, pass CSS selectors to ZenRows via the css_extractor parameter. ZenRows will then parse the rendered HTML and return structured JSON for each property.
Step 2: Importing Dependencies And Defining Constants
Start by importing csv and json modules. Then define the scraper configurations.
import requests
import json
import csv
# target URL and api credentials
API_ENDPOINT = "https://api.zenrows.com/v1/"
url = "https://www.zillow.com/homes/for_sale/San-Francisco_rb/"
apikey = "<YOUR_ZENROWS_API_KEY>"
# number of detail fields per property: beds, baths, sqft
details_per_property = 3
details_per_property encodes one assumption about the page: each property card has three bullets in the details list (beds, baths, square footage). ZenRows will return all of those bullets in a single flat list, and this number tells your script how to group them back into per-listing chunks.
Step 3: Adding CSS Selectors to the Request
Next, wrap the Request Builder code in a reusable function. Add a css_extractor dictionary so ZenRows knows which fields to pull from the rendered page, and switch from response.text to response.json() to receive structured data.
# ...
def scrape_with_zenrows(url, apikey):
"""scrape property data from zenrows api using css selectors."""
print("starting zenrows scrape...")
# define css selectors for extracting property fields
css_extractor = {
"address": "a[data-test='property-card-link'] > address",
"price": "span[data-test='property-card-price']",
"details": "ul[data-testid='property-card-details'] li",
"url": "a[data-test='property-card-title-link'] @href"
}
# build api request with js rendering and premium proxy
params = {
'url': url,
'apikey': apikey,
'js_render': 'true',
'premium_proxy': 'true',
'css_extractor': json.dumps(css_extractor)
}
print(f"sending request to zenrows api for {url}")
response = requests.get(API_ENDPOINT, params=params)
data = response.json()
print(f"received {len(data.get('address', []))} properties from api")
return data
The extractor uses the same selectors you inspected earlier. ZenRows renders the page with JavaScript, applies those selectors on its side, and returns lists of addresses, prices, details, and URLs instead of a single HTML string.
Step 4: Writing The ZenRows Response To CSV
Loop through the properties, slice out the right details for each one, and write everything to a CSV file.
# ...
def save_to_csv(data, output_file, details_per_property):
"""write scraped data to csv with address, price, beds, baths, sqft, url."""
length = len(data['address'])
print(f"writing {length} properties to csv...")
with open(output_file, mode='w', newline='', encoding='utf-8') as csv_file:
fieldnames = ['address', 'price', 'beds', 'baths', 'sqft', 'url']
writer = csv.DictWriter(csv_file, fieldnames=fieldnames)
writer.writeheader()
for i in range(length):
# calculate start index for this property's detail fields
detail_start = i * details_per_property
# extract beds, baths, sqft from flattened details array
beds = data['details'][detail_start] if detail_start < len(data['details']) else ''
baths = data['details'][detail_start + 1] if detail_start + 1 < len(data['details']) else ''
sqft = data['details'][detail_start + 2] if detail_start + 2 < len(data['details']) else ''
writer.writerow({
'address': data['address'][i],
'price': data['price'][i] if i < len(data['price']) else '',
'beds': beds,
'baths': baths,
'sqft': sqft,
'url': data['url'][i] if i < len(data['url']) else ''
})
print(f"successfully wrote all properties to {output_file}")
ZenRows returns all detail bullets in one flat list. So the code uses the `detail_start` index to jump to the first bullet for a given property and the next two entries as baths and square footage. This keeps the CSV consistent even when a listing is missing one of those values.
Step 5: Running the Scraper and Exporting the CSV
Finally, add an entry point that calls both functions and ties the flow together.
# ...
def main():
"""orchestrate scraping and csv export."""
print("starting property scraper...")
data = scrape_with_zenrows(url, apikey)
save_to_csv(data, 'zillow_properties.csv', details_per_property)
print("all done!")
if __name__ == '__main__':
main()
Here’s the full ZenRows real estate scraper in one place:
import requests
import json
import csv
# target url and api credentials
API_ENDPOINT = "https://api.zenrows.com/v1/"
url = "https://www.zillow.com/homes/for_sale/San-Francisco_rb/"
apikey = "<YOUR_ZENROWS_API_KEY>"
# number of detail fields per property: beds, baths, sqft
details_per_property = 3
def scrape_with_zenrows(url, apikey):
"""scrape property data from zenrows api using css selectors."""
print("starting zenrows scrape...")
# define css selectors for extracting property fields
css_extractor = {
"address": "a[data-test='property-card-link'] > address",
"price": "span[data-test='property-card-price']",
"details": "ul[data-testid='property-card-details'] li",
"url": "a[data-test='property-card-title-link'] @href"
}
# build api request with js rendering and premium proxy
params = {
'url': url,
'apikey': apikey,
'js_render': 'true',
'premium_proxy': 'true',
'css_extractor': json.dumps(css_extractor)
}
print(f"sending request to zenrows api for {url}")
response = requests.get(API_ENDPOINT, params=params)
data = response.json()
print(f"received {len(data.get('address', []))} properties from api")
return data
def save_to_csv(data, output_file, details_per_property):
"""write scraped data to csv with address, price, beds, baths, sqft, url."""
length = len(data['address'])
print(f"writing {length} properties to csv...")
with open(output_file, mode='w', newline='', encoding='utf-8') as csv_file:
fieldnames = ['address', 'price', 'beds', 'baths', 'sqft', 'url']
writer = csv.DictWriter(csv_file, fieldnames=fieldnames)
writer.writeheader()
for i in range(length):
# calculate start index for this property's detail fields
detail_start = i * details_per_property
# extract beds, baths, sqft from flattened details array
beds = data['details'][detail_start] if detail_start < len(data['details']) else ''
baths = data['details'][detail_start + 1] if detail_start + 1 < len(data['details']) else ''
sqft = data['details'][detail_start + 2] if detail_start + 2 < len(data['details']) else ''
writer.writerow({
'address': data['address'][i],
'price': data['price'][i] if i < len(data['price']) else '',
'beds': beds,
'baths': baths,
'sqft': sqft,
'url': data['url'][i] if i < len(data['url']) else ''
})
print(f"successfully wrote all properties to {output_file}")
def main():
"""orchestrate scraping and csv export."""
print("starting property scraper...")
data = scrape_with_zenrows(url, apikey)
save_to_csv(data, 'zillow_properties.csv', details_per_property)
print("all done!")
if __name__ == '__main__':
main()
The output of running Zenrow's scraper is as follows:
starting property scraper...
starting zenrows scrape...
sending request to zenrows api for https://www.zillow.com/homes/for_sale/San-Francisco_rb/
received 9 properties from api
writing 9 properties to csv...
successfully wrote all properties to zillow_properties.csv
all done!
The scraper extracted all the 9 properties from Zillow's first page.
Congratulations! You just bypassed Zillow’s anti-bot defenses and scraped real estate data without getting blocked.
Getting Full Property Details With ZenRows
The search page gives you enough to build a quick CSV, but it only exposes summary data. The full property pages usually include a longer description, extra status fields, etc. Like when you open a property on Zillow, you discover more details.
To reach those, reuse the search scrape to get listing URLs, then call ZenRows again for each property page. Go on and extract the property description from each property you scraped earlier.
The main changes to the earlier script are:
- Keep the list-page scrape, but store everything in memory instead of writing directly to CSV.
- Turn relative links into full URLs so ZenRows can open each property page.
- Add a second ZenRows call inside the loop that extracts the long description (and any other detail fields you care about).
- Write a new CSV that includes both summary fields and the full description.
Here’s a complete script that does that.
import requests
import json
import csv
from urllib.parse import urljoin
# api configuration
API_ENDPOINT = "https://api.zenrows.com/v1/"
apikey = "<YOUR_ZENROWS_API_KEY>"
LIST_URL = "https://www.zillow.com/homes/for_sale/San-Francisco_rb/"
# number of detail fields per property: beds, baths, sqft
details_per_property = 3
def scrape_list_and_details(list_url, apikey):
"""scrape list page and detail pages for each property."""
print("starting list page scrape...")
# scrape list page for addresses, prices, urls
list_css_extractor = {
"address": "a[data-test='property-card-link'] > address",
"title": "a[data-test='property-card-title-link']",
"price": "span[data-test='property-card-price']",
"details": "ul[data-testid='property-card-details'] li",
"url": "a[data-test='property-card-title-link'] @href",
}
list_params = {
"url": list_url,
"apikey": apikey,
"js_render": "true",
"premium_proxy": "true",
"css_extractor": json.dumps(list_css_extractor),
}
print(f"sending request to zenrows api for {list_url}")
list_resp = requests.get(API_ENDPOINT, params=list_params)
list_data = list_resp.json()
addresses = list_data.get("address", [])
prices = list_data.get("price", [])
details_flat = list_data.get("details", [])
urls = list_data.get("url", [])
length = len(addresses)
print(f"received {length} properties from list page")
# scrape detail page for each property
properties = []
for i in range(length):
detail_start = i * details_per_property
beds = details_flat[detail_start] if detail_start < len(details_flat) else ""
baths = details_flat[detail_start + 1] if detail_start + 1 < len(details_flat) else ""
sqft = details_flat[detail_start + 2] if detail_start + 2 < len(details_flat) else ""
rel_url = urls[i] if i < len(urls) else ""
full_url = urljoin("https://www.zillow.com", rel_url)
row = {
"address": addresses[i],
"price": prices[i] if i < len(prices) else "",
"beds": beds,
"baths": baths,
"sqft": sqft,
"url": full_url,
"description": "",
}
# scrape detail page if url exists
if full_url:
detail_css_extractor = {
"description": "[data-testid='description']",
}
detail_params = {
"url": full_url,
"apikey": apikey,
"js_render": "true",
"premium_proxy": "true",
"css_extractor": json.dumps(detail_css_extractor),
}
try:
print(f"scraping detail page {i+1}/{length}")
detail_resp = requests.get(API_ENDPOINT, params=detail_params, timeout=90)
detail_data = detail_resp.json()
# handle zenrows array/string variations
description = detail_data.get("description", "")
if isinstance(description, list):
row["description"] = " ".join([d.strip() for d in description if d.strip()])
else:
row["description"] = description
except Exception as e:
print(f"error fetching details for {full_url}: {e}")
properties.append(row)
return properties
def save_to_csv(properties, output_file):
"""write property data to csv file."""
print(f"writing {len(properties)} properties to csv...")
with open(output_file, mode="w", newline="", encoding="utf-8") as f:
fieldnames = ["address", "price", "beds", "baths", "sqft", "url", "description"]
writer = csv.DictWriter(f, fieldnames=fieldnames)
writer.writeheader()
for row in properties:
writer.writerow(row)
print(f"successfully wrote all properties to {output_file}")
def main():
"""orchestrate scraping and csv export."""
print("starting property scraper...")
properties = scrape_list_and_details(LIST_URL, apikey)
save_to_csv(properties, "zillow_properties_full.csv")
print("all done!")
if __name__ == "__main__":
main()
This script keeps the list-page scrape from the previous section, but instead of writing straight to CSV, it builds a list of property dictionaries. For each one, it calls ZenRows again on the full property URL, extracts the long description, and stores it next to the summary fields. The final CSV now includes address, price, beds, baths, size, URL, and description.
When you run the code, you'll see the following output:
starting property scraper...
starting list page scrape...
sending request to zenrows api for https://www.zillow.com/homes/for_sale/San-Francisco_rb/
received 9 properties from list page
scraping detail page 1/9
scraping detail page 2/9
<!-- omitted for brevity -->
scraping detail page 9/9
writing 9 properties to csv...
successfully wrote all properties to zillow_properties_full.csv
all done!
The saved CSV looks like this:
Congratulations! 🎉 You just scraped live real estate listings from Zillow using Python and ZenRows. You’re now ready to pull reliable property data into your workflows at scale.
Scaling Your Real Estate Data Workflow
So far, you’ve scraped a single Zillow page and enriched its listings. In practice, you need more pages, more cities, and more sites.
Handling pagination for a Zillow scraper
If you want your Zillow scraper to cover a whole market, it can’t stop at the first page. Zillow spreads results across multiple pages, so you either build paginated URLs or follow the next page link in the HTML. For each page, you send the URL to ZenRows, collect the listings, and append them to the same list or table. By the end, you’ve scraped Zillow across all pages in that search and aggregated everything into one dataset instead of a single screen of results.
Reusing the pattern for Realtor, Redfin, Airbnb, and idealista
The same pattern turns into a realtor scraper, Redfin scraper, Airbnb scraper, or an idealista scraper with only selector changes. You can scrape Realtor, Redfin, Airbnb, Idealista, and other real-estate listings by swapping the search URL and CSS selectors while keeping the ZenRows call and output schema identical. Because ZenRows supports concurrent requests, you can run these scrapers in parallel across sites or regions and merge the results into one table keyed by site and listing URL. Cross-site aggregation then becomes a matter of configuration and scheduling, not extra real estate data scraping logic.
Scheduling scrapes and aggregating over time
Once pagination and multi-site support are in place, scaling is mostly about timing and storage. A cron job or workflow tool runs your site-specific scrapers on a daily or hourly schedule and appends new rows to a database.
Using Your Real Estate Dataset With An AI Agent
Once you have structured data from Zillow, Realtor, Redfin, Craigslist, and other sources, you can use it for more than dashboards. You can create an AI helper that filters properties based on user criteria, compares areas by metrics like median price or days on market, and summarizes what’s happening in a specific slice of the market ("three-bedroom homes under $800k in this ZIP over the last month").
ZenRows supports this workflow directly through its official LangChain integration through the langchain-zenrows package. It includes a ZenRowsUniversalScraper tool that lets an agent call ZenRows directly, scrape fresh listing data on demand, and combine it with your stored dataset.
Conclusion
By now, you’ve seen how quickly a simple real estate scraper can run into 403s and missing data, and how ZenRows' real estate scraper turns the same pages into structured property records you can actually use.
Now you know why real estate data is scattered across search pages, JavaScript calls, and individual property views. You understand where basic real estate data scrapers fail and how ZenRows fills those gaps. With ZenRows, you can extract complete property details and unify them in one dataset ready for your dashboards, workflows, or AI tools. Instead of fighting anti-bot defenses and constant layout changes, you focus on building value from your data.
Try ZenRows for free and start pulling real estate data from Zillow, Realtor, Redfin, and Craigslist today.