Data Pipelines with Apache Airflow Orchestration for data and AI, 2nd ed. (Julian de Ruiter, Ismael Cabral etc.)（Z-Library）

# Reading Guide: Data Pipelines with Apache Airflow ## 【One-Line Pitch】 The definitive, hands-on guide to Apache Airflow 3 for data engineers—covering everything from writing your first DAG to orchestrating GenAI pipelines in production. Whether you're new to workflow orchestration or upgrading from Airflow 2.x, this book takes you from novice to expert through real-world projects and battle-tested best practices. ## 【Book Arc】 - **Opening (~0%–18%)**: Introduces Airflow 3's role in the data ecosystem, explains why pipeline graphs beat sequential scripts, and walks through the anatomy of a DAG—tasks, operators, scheduling basics, and failure handling. Sets up local environments with Python and Docker. - **Early (~18%–36%)**: Dives deep into scheduling—time-based schedules with cron expressions and timetables, data intervals for incremental processing, backfilling strategies, and the new asset-aware (event-based) scheduling using the Dataset API. Covers templating with Jinja and the Airflow context, plus task dependencies including branching, conditional tasks, trigger rules, and XComs. - **Middle (~36%–55%)**: Moves beyond basics—triggering workflows via sensors, REST API, and messages; integrating with external systems like Postgres and S3; building custom hooks, operators, sensors, and deferrable operators; and a full chapter on testing strategies from unit tests to complete DAG validation. - **Middle (~55%–64%)**: Tackles container-native execution with DockerOperator and KubernetesPodOperator, then shifts to best practices—clean DAG design, task factories, task groups, dynamic task mapping, idempotency, data efficiency, and concurrency management with pools. - **Late (~64%–91%)**: Two complete projects: a NYC traffic analysis pipeline (extracting, transforming, and structuring real-world data) and a GenAI project building a RAG system for family recipes with vector databases and LLM fine-tuning. Then moves to production—executor selection, scheduler tuning, logging, monitoring with Prometheus/Grafana, and alerting. - **Ending (~91%–100%)**: Covers security (RBAC, encryption, LDAP, HTTPS, secrets management) and deployment options including managed services (Astronomer, Cloud Composer, MWAA) and Kubernetes with the official Helm chart. ## 【Key Takeaways】 - **Airflow 3 is a major evolution, not a minor update** (Early): New UI, DAG versioning, improved backfills, and asset-aware scheduling fundamentally change how you design pipelines. The book is fully revised for these changes, making it essential reading even for experienced Airflow 2.x users. - **Asset-aware scheduling solves time-based scaling problems** (Early): The Dataset API lets you trigger workflows based on data events rather than fixed schedules, enabling more responsive and efficient pipelines. This is a paradigm shift worth mastering. - **Data intervals are the key to incremental processing** (Early): Understanding how Airflow's data intervals work enables you to design pipelines that process only new data, avoiding costly full refreshes. Combined with backfilling, this is the foundation of efficient data engineering. - **The Taskflow API simplifies Python task chaining** (Early): Using `@task` decorators eliminates boilerplate and makes dependencies explicit through function calls. The book clearly explains when to use it and when traditional operators are better. - **Deferrable operators dramatically reduce resource usage** (Middle): By running tasks asynchronously with the triggerer, you can hold thousands of sensors open without consuming worker slots—a critical optimization for production. - **Testing is non-negotiable for reliable pipelines** (Middle): From DAG integrity checks to unit tests and full DAG execution with `dag.test()`, the book provides a complete testing strategy that integrates with CI/CD pipelines. - **Containers solve the dependency nightmare** (Middle): Running tasks in Docker or Kubernetes isolates dependencies and ensures consistency between development and production. The DockerOperator and KubernetesPodOperator are essential tools for modern Airflow deployments. - **GenAI orchestration is now a first-class Airflow use case** (Late): The RAG project shows how to orchestrate LLM fine-tuning, vector database operations, and retrieval pipelines—demonstrating Airflow's evolution beyond traditional ETL. ## 【Reading Tips】 - **Skim the first chapter if you're experienced**: Chapter 1 covers basic concepts you may already know. Start with Chapter 2 (DAG anatomy) if you're comfortable with workflow orchestration concepts. - **Deep-read Chapters 3–4 on scheduling**: This is where Airflow 3 differs most from previous versions. Understanding data intervals and asset-aware scheduling is critical for designing efficient pipelines. - **Follow along with the code**: The book includes runnable examples (setup in Appendix A). Actually executing the NYC and GenAI projects will cement your understanding far better than reading alone. - **Pay special attention to Chapter 9 (custom components)**: Building custom hooks, operators, and deferrable operators is what separates Airflow users from Airflow power users. The Movielens example is an excellent template. - **Use Part 4 as a reference**: Chapters 15–17 on production operations, security, and deployment are best read when you're actually preparing to deploy. Bookmark them for when you need them. ## 【Coverage Limits】 This guide covers all 12 sampled chunks spanning the full book structure, including the table of contents, foreword, and preface. Detailed technical content from individual chapters is summarized from the table of contents and chapter descriptions; the excerpts do not include full chapter text. ##