Python网络爬虫技术与应用 (邓维；李贝；汤小洋 主编；康毅滨；林海玉；刘燕秋；林建雄；刘庆胜；钟晓颖 副主编) (Z-Library)

【One-Line Pitch】 A hands-on, textbook-style guide to Python web scraping that moves from HTTP basics and parsing libraries (requests, regex, XPath, BeautifulSoup) to full framework mastery with PySpider and Scrapy, including distributed crawling, deployment, and anti-crawling countermeasures. Best for students, instructors, and self-taught developers who want a structured, project-driven path from "first request" to "production-grade crawler." 【Book Arc】 - **Opening (~0%–11%)**: Introduces web crawler fundamentals — the basic flow (send request → get response → parse → save), anti-crawling mechanisms (IP frequency limits, captchas), and countermeasures like time delays and proxy pools. Also surveys Python web frameworks (Django, Flask, CherryPy, Pyramid) as context for later deployment topics. - **Early (~11%–33%)**: Covers the core toolchain: environment setup (Python, pip, setuptools), the requests library (sessions, cookies, authentication), regular expressions (match, search, patterns), and Excel read/write with openpyxl and pandas. Ends with a practical news-site scraping case using regex. - **Early-to-Middle (~33%–56%)**: Dives into parsing and browser automation: XPath and lxml, BeautifulSoup (Tag objects, nested selection), Chrome DevTools analysis, Selenium for dynamic pages, Ajax request simulation, and captcha handling. Includes a full novel-scraping project and a simulated-login case. - **Middle (~56%–67%)**: Shifts to data storage and framework foundations: MySQL and MongoDB integration via Python, then an overview of PySpider and Scrapy architectures, including message queues (RabbitMQ), threading basics, and the scheduler-fetcher-processor pipeline. - **Late (~67%–89%)**: Focuses on Scrapy in depth — project structure, Item pipelines, Redis-based distributed crawling (scrapy-redis), data export (JSON/CSV/XML), and deployment tools (Scrapyd, SpiderKeeper, Supervisor). Includes a distributed real-estate scraping system design using Redis, MongoDB, Django, and Docker. - **Ending (~89%–100%)**: Presents advanced实战 cases: waterfall (infinite scroll) scraping, image captcha recognition via grayscale/binarization, sliding captchas, Weibo data extraction with PySpider, RabbitMQ wildcard routing patterns, HTTPS capture with Fiddler, and WeChat public-account article scraping with PyMySQL. 【Key Takeaways】 - **Anti-crawling is a cat-and-mouse game with practical fixes** (Opening): IP frequency limits trigger captchas; solutions include `time.sleep(3)` delays and rotating proxy IPs via `requests` `proxies` parameter. Expect to revisit this throughout real projects. - **The requests library is the workhorse for HTTP interactions** (Early): Sessions maintain cookies across requests (proving HTTP is stateless), and headers with proper User-Agent strings are essential for mimicking real browsers. Master `Session()`, `get()`, and cookie handling early. - **Regex is powerful but brittle; use it for targeted extraction** (Early): `re.match` anchors at string start, and patterns like `^The` show precise control. For messy HTML, regex works but pairing it with structural parsers is more robust. - **XPath and lxml give precise, path-based HTML navigation** (Early): Expressions like `//li[last()-1]/a/text()` extract specific nodes cleanly. This is the backbone for Scrapy selectors later, so invest time here. - **BeautifulSoup is forgiving for imperfect HTML** (Early): It handles unclosed tags gracefully, and nested selection (e.g., `soup.head.title`) returns Tag objects that chain naturally. Great for quick, exploratory parsing. - **Selenium bridges the gap for JavaScript-rendered pages** (Middle): Automating a real browser (e.g., ChromeDriver) handles dynamic content and simulated logins, but it's slower — use it only when requests + parsing can't get the job done. - **Scrapy scales from single-site crawlers to distributed systems** (Late): Items define data models, pipelines handle storage, and scrapy-redis enables shared queues across machines. Deployment via Scrapyd and monitoring via SpiderKeeper turn prototypes into operations-ready tools. - **Captcha handling ranges from simple preprocessing to ML** (Ending): Grayscale + binarization (threshold ~140) cleans image captchas for OCR (pytesseract); sliding captchas require coordinate tweaking loops. The book notes that large-scale recognition may need training on many samples. 【Reading Tips】 - **Skim Chapter 1's framework survey** (Django, Flask, etc.) — it's context, not core. Focus instead on the crawler flow diagram and anti-crawling section, which set up everything later. - **Deep-read the parsing chapters (XPath, BeautifulSoup, Selenium)** — these are the skills you'll use daily. Code along with the examples; the nested-selection and XPath syntax patterns recur in every Scrapy project. - **Treat the novel-scraping and news-scraping cases as templates** — they show the full loop (URL discovery → request → parse → save) with error handling and encoding fixes (UTF-8 vs GBK). Reuse these patterns for your own projects. - **For the Scrapy chapters, focus on the architecture first** (Scheduler/Fetcher/Processor, Item/Pipeline), then the Redis integration — distributed crawling is the book's most advanced and valuable topic. The deployment tools (Scrapyd, SpiderKeeper) are worth skimming unless you're running production crawlers. - **The final chapter is a grab-bag of实战** — pick cases matching your needs (waterfall scraping, captchas, WeChat articles). The Fiddler HTTPS capture section is niche but useful for debugging encrypted traffic. 【Coverage Limits】 Excerpts do not cover the book's full table of contents, exercise answers, or supplementary teaching materials (PPT, videos, exam papers) mentioned in the blurb. Some code fragments are incomplete due to excerpt sampling; refer to the original text for full listings.