Data Pipelines with Apache Airflow (Bas P. Harenslak, Julian Rutger de Ruiter)（Z-Library）

【One-Line Pitch】 A practical, example-driven guide to building, scheduling, and operating data pipelines with Apache Airflow, written for engineers who already know Python and want to move from ad-hoc scripts to reliable, production-grade orchestration. Best suited to data engineers, DevOps practitioners, ML engineers, and sysadmins who need to automate moving and transforming data across a stack. 【Book Arc】 - **Opening (~0%–10%)**: Frames the problem — pipelines that stall, block, or rely on brittle homegrown glue — and introduces Airflow as a single orchestration platform, using a weather/sales forecasting scenario to show how tasks and dependencies form a graph. - **Early (~10%–30%)**: Covers the anatomy of a DAG: defining tasks with operators (Bash, Python), wiring dependencies, reading the UI, and scheduling runs at intervals. Introduces incremental processing, backfilling historical data, and best practices like atomicity and idempotency. - **Early–Middle (~30%–45%)**: Deepens task design with the task context and Jinja templating, passing runtime variables into operators, and connecting to external systems such as Postgres via managed connections. - **Middle (~45%–55%)**: Moves into dependency patterns and data sharing — fan-in/fan-out, branching, and XComs for passing values between tasks — while warning about the trade-offs of hiding logic inside tasks. - **Middle–Late (~55%–75%)**: Tackles triggering and coordination problems, including sensors, polling, and the sensor deadlock that arises when too many tasks block waiting on external conditions; introduces poke vs. reschedule modes as a remedy. - **Late–Ending (~75%–100%)**: Shifts toward production concerns — testing, custom components, containers, and deploying Airflow in real environments. The excerpts do not cover the final chapters in detail. 【Key Takeaways】 - **Airflow replaces pipeline sprawl with one orchestration layer** (Opening): instead of stitching together cron jobs, scripts, and bespoke glue, you define workflows as DAGs and let Airflow handle scheduling, dependencies, and monitoring. - **A DAG is tasks plus dependencies, and that graph determines parallelism** (Early): independent branches (e.g., fetching weather vs. sales data) can run concurrently, so modeling dependencies correctly directly affects runtime and resource use. - **Atomicity and idempotency are the two properties that make tasks safe to rerun** (Early): splitting tightly coupled operations into separate tasks can backfire when they share a strong dependency; the goal is coherent units of work that produce the same result on repeat. - **Scheduling and backfilling turn a pipeline into a time-aware system** (Early): execution dates and intervals let you process data incrementally and reprocess history, which is essential for analytics and ML feature generation. - **The task context and templating connect static code to runtime values** (Early–Middle): Jinja templates and explicit function arguments (like `execution_date`) make DAGs dynamic without hardcoding dates or paths. - **Connections centralize credentials so operators stay clean** (Middle): storing connection details in Airflow lets operators like PostgresOperator handle setup and teardown under the hood. - **XComs are for small handoffs, not bulk data** (Middle): pushing and pulling values like a `model_id` between tasks works well, but the book frames XComs as lightweight coordination rather than a data transport mechanism. - **Sensors can deadlock a system if left unchecked** (Middle–Late): polling tasks accumulate and consume concurrency slots; switching from `poke` to `reschedule` mode is the key mitigation. 【Reading Tips】 - Read the opening chapters closely if you are new to Airflow — the DAG anatomy and scheduling material is foundational and everything later builds on it. - Treat the dependency and XCom chapters as design guidance, not just API reference; the warnings about branching and hidden conditions are the most valuable part. - Skim the operator-specific sections if you already know your stack, but do not skip the sensor deadlock discussion — it is a production failure mode worth understanding before you hit it. - Use the umbrella forecasting scenario as a running case study; tracing it across chapters makes the abstract concepts concrete. - If you are preparing for production deployment, prioritize the late-stage material on testing, custom components, and containers, and expect to supplement it with current Airflow documentation since the excerpts reflect an earlier version. 【Coverage Limits】 This guide is based on stratified excerpts covering roughly the first half to two-thirds of the book; the later production, testing, and deployment chapters are only partially represented, so specifics on those topics are not fully covered here.