Website content migration
Website content migration moves every page of an existing site into a new CMS: body copy, headings, metadata, images and the URL structure your redirects depend on. Zenrows Crawl walks the old site and returns each page as Markdown, and Fetch pulls each page's metadata as JSON, giving you an import file instead of weeks of copy and paste.
Updated
{
"old_url": "https://www.example.com/services/commercial-roofing.aspx",
"new_path": "/services/commercial-roofing",
"status": 200,
"title": "Commercial roofing services | Example Co",
"meta_description": "Flat and low-slope roof installation and repair for warehouses and offices.",
"canonical": "https://www.example.com/services/commercial-roofing.aspx",
"h1": "Commercial roofing",
"headings": ["Flat roof systems", "Repairs and inspections", "Service area"],
"body_markdown": "## Flat roof systems\n\nWe install TPO and EPDM membranes on...",
"word_count": 812,
"images": [
{ "src": "/media/roof-tpo-install.jpg", "alt": "TPO membrane being installed" }
],
"internal_links": ["/services/roof-repair.aspx", "/contact.aspx"],
"template": "service-page",
"crawled_at": "2026-10-17T10:03:51Z"
}
new_path and template when mapping; the rest comes from the old site.What a migration needs from the old site
Migrations fail on unlisted pages: start with a full inventory, end by checking every old URL landed.
| Job | Data you collect | Typical sources |
|---|---|---|
| Content inventory | Every URL, status, template, word count | A full crawl, compared with sitemap.xml |
| Body content export | Main copy as Markdown with headings, lists, tables, links | WordPress, Drupal, Sitecore, Adobe Experience Manager, custom ASP.NET or PHP |
| SEO metadata capture | Title, meta description, canonical, H1, hreflang, robots | The <head> of every page |
| Redirect mapping | Old URL, new URL, status | Crawl inventory joined with the new URL plan |
| Media and file inventory | Image URLs, alt text, PDFs and downloads | img and a tags on every page |
| Post-launch verification | New URLs, statuses, titles, word counts against the old site | A second crawl after cut-over |
Why manual content migration breaks
Hand-copying works for twenty pages; past a few hundred, pages vanish, metadata drops and old URLs 404 after launch.
Tables full of shortcodes and layout markup, or no export
Crawl reads rendered pages: the content visitors see
Landing pages outside sitemap and menu, linked only from old posts
Crawl follows every internal link; diff its list against the sitemap
Tabs, accordions, single-page app routes empty over plain HTTP
Adaptive Stealth Mode renders when a page needs it
Navigation, sidebar and footer repeated per page
A CSS selector like article or #main returns only the content
Titles, descriptions, canonicals vanish; search traffic drops
CSS-selected head fields as JSON per page, ready to map
Cloudflare or a WAF challenges the crawler
Zenrows' stealth gets through Cloudflare and similar, or allowlist the crawl
How to run a website content migration
- Inventory with Crawl: start at the homepage with no output format to list every URL, and compare with the sitemap.
- Export content as Markdown: Crawl again with Markdown output, or fetch each URL with a content-container selector.
- Capture metadata as JSON: CSS selectors for title, description, canonical, H1 and images, one row per page.
- Map and import: give each old URL a new path and template, convert the Markdown, and build the redirect map from the same table.
- Verify after launch: crawl the new site, join inventories, fix old URLs with no 301 and pages whose word count dropped.
curl -X POST "https://api.zenrows.com/v1/crawls" \
-H "X-API-Key: YOUR_ZENROWS_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://www.example.com/",
"max_items": 5000,
"exclude_patterns": ["/search", "/cart", "/wp-admin"],
"output_format": "markdown"
}'# Capture SEO metadata for every URL in the inventory
# pip install requests
import csv, json, requests
EXTRACTOR = {
"title": "title",
"description": "meta[name=description] @content",
"canonical": "link[rel=canonical] @href",
"h1": "h1",
"images": "img @src",
}
urls = [line.strip() for line in open("inventory.txt") if line.strip()]
with open("metadata.csv", "w", newline="") as out:
writer = csv.writer(out)
writer.writerow(["url", "title", "description", "canonical", "h1", "images"])
for url in urls:
data = requests.get("https://api.zenrows.com/v1/", params={
"apikey": "YOUR_ZENROWS_API_KEY",
"url": url,
"mode": "auto",
"css_extractor": json.dumps(EXTRACTOR),
}).json()
writer.writerow([url] + [json.dumps(data.get(k)) for k in EXTRACTOR])npx -y @zenrows/cli init
zenrows crawl create https://www.example.com/ --max-pages 5000 --output-format markdown --follow
zenrows crawl results CRAWL_ID --out site-export.jsonl
zenrows fetch https://www.example.com/services/commercial-roofing.aspx --output markdown# Connect once
claude mcp add zenrows -e ZENROWS_API_KEY=YOUR_ZENROWS_API_KEY -- npx -y @zenrows/mcp
# Then ask:
Use Zenrows to crawl www.example.com, then give me a content inventory as a table with each URL, its title, meta description, canonical, H1 and word count, and flag pages that are missing a meta description.Which Zenrows product fits
- List every URL on the old site CrawlFollows internal links; returns the full inventory
- Export every page body as Markdown CrawlClean Markdown per page in one job
- Capture titles, descriptions and canonicals per page FetchCSS-selected head fields as JSON
- Get fields from inconsistent templates ExtractNo per-template selectors
- Spot-check pages from the terminal during mapping CLIA page as Markdown in one command, to compare with the new CMS
What success looks like
| Measure | What good looks like | How to check it |
|---|---|---|
| URL parity | Each old 200 URL has a new page, a 301 or a deliberate 410 | Join old and new inventories; list unmatched old URLs |
| Redirect accuracy | Each 301 reaches the closest page in one hop | Fetch old URLs after launch; record final URL and hops |
| Content fidelity | New word counts within tolerance of old | Compare per page pair; review the largest drops |
| Metadata retention | Titles, descriptions, canonicals on every page | Re-run the extractor; count empty fields |
| Broken links | No internal link to a 404 | Crawl the new site for internal links with errors |
Website content migration FAQ
-
Moving a site's pages, posts, metadata, images and files into a new CMS or platform, alongside design and development; it decides whether the site launches complete. Deliverables: inventory, importable export, redirect map.
-
Crawl the old site, export pages as Markdown or JSON, map old URLs to new paths and templates, and import via the CMS's API or tool. Build redirects from the same table; crawl the new site after launch to confirm nothing is missing.
-
Capture title, meta description, canonical, H1 and hreflang before moving anything and carry them over. Keep URLs where you can, 301 changed ones to the closest page, point internal links at final URLs, and compare crawls after launch.
-
Yes. Crawl reads rendered pages, with no database access or export feature needed: the route for proprietary CMSs, agency sites you lost admin access to, and static sites whose sources are gone.
-
Combine the sitemap.xml, a full link-following crawl, and URLs with analytics traffic over the past year. Pages in analytics but not the crawl are often orphaned landing pages needing a decision before launch.
-
Before launch: URL inventory, content export, metadata, image and file list, mapping table, redirect map. At launch: redirects live, sitemap submitted, internal links final. After: crawl the new site against the old inventory, check 404s and redirect chains, review pages whose word count or metadata changed.
-
Collect image URLs with alt text, and linked PDFs or downloads, during the crawl; load them into the new media library and rewrite the Markdown references. Alt text moves with each reference.
-
Yes. Crawl with Markdown output for the whole site, or Fetch for single pages. Markdown keeps headings, lists, links and tables, and most headless CMSs and static site generators import it directly.