Python网络数据采集 (Ryan Mitchell, 陶俊杰, 陈小莉)（Z-Library）

【One-Line Pitch】 A practical, hands-on guide to collecting data from the modern web with Python—from fetching a single page to crawling whole sites, calling APIs, and storing what you gather. Best for developers and researchers who already know some programming and want a working toolkit for real web data. 【Book Arc】 - **Opening (~0%–10%)**: Frames web scraping as a broad, cross-disciplinary skill and sets up the core loop—request HTML, parse it, store the target data, then move to the next page. - **Early (~10%–30%)**: Builds the fundamentals: requesting pages with Python's standard library, parsing messy HTML with BeautifulSoup, handling errors and redirects, and following links across sites (including a Wikipedia "six degrees" project and an intro to Scrapy). - **Early–Middle (~30%–40%)**: Shifts to structured sources—calling web APIs, working with JSON/XML, and handling authentication keys for services like Twitter and Google. - **Middle (~40%–55%)**: Covers storage and document formats: writing to CSV, using MySQL with proper Unicode handling, and extracting text from PDFs and Word files. - **Middle (~55%–70%)**: Turns to data quality—cleaning dirty data at the source, normalizing text, and using n-grams to make scraped content analyzable. - **Late–Ending (~70%–100%)**: Moves into harder territory: reading and processing images, dealing with JavaScript-rendered pages, and legal/ethical considerations for running crawlers. (Excerpts do not cover the exact chapter boundaries here.) 【Key Takeaways】 - **The scraping loop is the foundation** (Opening): fetch → parse → store → repeat. Roughly 90% of real projects reuse only these basics, so master them before chasing advanced tools. - **Robust error handling separates working crawlers from broken ones** (Early): anticipate missing pages, dead servers, and malformed data up front, or your crawler dies the moment you stop watching it. - **BeautifulSoup plus regular expressions gives flexible targeting** (Early): navigating the HTML tree by parent, sibling, and child tags—and filtering with regex—lets you locate data even when page layouts shift. - **APIs and scraping are complementary, not rivals** (Early–Middle): both send HTTP requests and return data; combining them (e.g., pairing Wikipedia edit history with an IP-geolocation API) produces richer results than either alone. - **Storage design should follow the data's structure** (Middle): modeling pages and links as separate tables with IDs and timestamps makes large crawls queryable and analyzable later. - **Unicode and encoding are persistent pain points** (Middle): configure databases and text handling for full Unicode early, or non-English content will break your pipeline. - **Dirty data must be cleaned at the source** (Middle): inconsistent casing, punctuation, and spelling are the norm; normalization and n-grams turn raw text into something analyzable. - **Not all content is plain HTML** (Late): PDFs, Word documents, images, and JavaScript-rendered pages each need their own extraction approach. 【Reading Tips】 - Deep-read the first part (requesting, parsing, error handling)—these chapters are used daily and repay careful practice. - Skim the API and storage chapters if you already know HTTP and SQL; focus instead on the Unicode and schema-design warnings. - Treat the data-cleaning chapter as a reference you'll return to; the n-gram material is more useful once you have real messy data. - Don't skip the legal/ethical appendix material—crawler etiquette and compliance matter as much as the code. - Code examples target Python 3.x; note that some tools mentioned (e.g., Scrapy) required Python 2.7 at the time of writing. 【Coverage Limits】 This guide is synthesized from stratified excerpts covering roughly the first half of the book in detail; later chapters on images, JavaScript rendering, and legal issues are only lightly represented, so specifics there are not fully covered.