Distributed Web Crawling Made Easy: System and Architecture
September 9, 2022
· 16 min read
Table of contents
- Prerequisites
- Intro to Celery and Redis
- Simple Celery Task
- Distributed Crawling from Task
- Redis for Tracking URLs
- Separate Responsibilities
- Allow Parser Customization
- Custom Parser
- Get HTML: Headless Browsers
- Avoid Detection with Headers and Proxies
- Bringing It All Together
- Points Still Missing
- Conclusion
Share