Microsoft Fabric Data Factory Playbook - 1 Edition - Design, orchestrate, and scale reliable data pipelines with CICD,… (Mark Kromer)（Z-Library）

# Microsoft Fabric Data Factory Playbook ## 【One-Line Pitch】 A practical guide for data engineers and architects to design, orchestrate, and scale production-ready data pipelines in Microsoft Fabric, covering both low-code pipelines and code-first Apache Airflow approaches. If you're migrating from Azure Data Factory or starting fresh with Fabric, this book gives you the patterns and mental models to build reliable data integration solutions. ## 【Book Arc】 - **Opening (~0%–10%)**: Introduces Microsoft Fabric as the evolution of Azure data services into a unified SaaS platform, explaining how Data Factory fits into the broader ecosystem and why the medallion lakehouse architecture (Bronze/Silver/Gold) serves as the book's running example built on NYC taxi trip data. - **Early (~12%–27%)**: Covers building dynamic Bronze ingestion pipelines with Copilot assistance, then transitions to Apache Airflow Jobs—covering the managed Airflow environment, the MSFabricRunJobOperator, writing ingestion DAGs, debugging, and scheduling both pipelines and DAGs. - **Early (~29%–34%)**: Explores the connector ecosystem—cloud, on-premises, and SaaS sources—including Azure services, SharePoint, Dataverse, on-premises data gateway configuration, file formats (Parquet, Delta, CSV, JSON, Binary), and credential management with Key Vault and workspace identity. - **Middle (~39%–51%)**: Delves into control flow and data processing patterns—Switch, ForEach, If Condition, parallelism tuning, advanced expressions, variables/parameters, error handling with retry policies and notifications—plus a side-by-side comparison of ForEach in Pipelines versus Airflow DAGs. - **Middle (~56%–61%)**: Covers data transformation at scale using Dataflows Gen2, PySpark Notebooks, and dbt Jobs for building the Silver layer, including incremental refresh setup, filtering bad data, joining lookup tables, and wiring transformations into pipelines and Airflow DAGs. - **Late (~66%)**: Explains workspace setup, the Data Factory UI tour, the Monitoring Hub for observability, and a detailed comparison between Fabric Data Factory and Azure Data Factory—what carried over and what changed. ## 【Key Takeaways】 - **OneLake is the unifying storage layer** (Middle): Data written by any Fabric engine—Dataflow Gen2, Notebook, or pipeline—is immediately readable by Warehouse SQL queries and Power BI without data movement. This eliminates the copy-based integration patterns of traditional architectures and reduces cost and latency. - **Low-code pipelines are the right default choice** (Middle): The visual pipeline canvas with activities like ForEach, If Condition, Switch, and Until handles the majority of data integration scenarios. Teams should start here before considering code-first alternatives. - **Apache Airflow Jobs provide code-first orchestration** (Middle): For teams comfortable with Python, Fabric's managed Airflow environment offers DAG-based workflows with a rich provider ecosystem and powerful scheduling expressions—without the infrastructure burden of self-hosted Airflow. - **The medallion architecture gives you a reliable audit trail** (Middle): Bronze preserves raw data as-is, Silver applies cleansing and conforming, and Gold serves analytics-ready aggregates. This layering enables reprocessing when downstream logic changes and provides a trustworthy foundation for decision-makers. - **Workspace identity enables zero-credential connections** (Early): Fabric's workspace identity approach reduces credential management overhead, while Azure Key Vault integration provides a secure fallback for external systems. This is a significant shift from ADF's connection management model. - **The Monitoring Hub is your primary observability point** (Late): It centralizes pipeline runs, dataflow refreshes, and notebook runs with status, duration, and error details—but note that Airflow Job tasks require the separate Airflow monitoring UI. - **Fabric Data Factory is not Azure Data Factory** (Late): While the core mental model carries over, the Power BI-based UI, workspace-centric structure (no ARM templates or resource groups), and native SaaS experience require unlearning ADF habits to avoid architectural mistakes. ## 【Reading Tips】 - **Skim Chapter 1 if you're experienced with Fabric**: The foundation and workspace setup sections (roughly 39%–61%) are essential for beginners but can be skipped if you already have a Fabric-enabled workspace and understand the medallion pattern. - **Deep-read the orchestration comparison sections**: The side-by-side comparisons of Pipelines versus Airflow DAGs (around 29%–34%) are where the book's real value lies—this is the decision framework you'll use on every project. - **Follow the NYC taxi example hands-on**: The running example is deliberately simple and publicly available, making it ideal for actually building along. Don't just read—create the Lakehouses, pipelines, and DAGs as you go. - **Pay special attention to error handling patterns**: The retry policies, on-failure paths, and notification activities (around 29%) are what separate production-ready pipelines from demo scripts. These patterns are directly transferable to your own projects. - **Bookmark the Monitoring Hub section**: Even though it's covered in depth in Chapter 8, knowing where to find run history and error details early will save you debugging time throughout the book. ## 【Coverage Limits】 The excerpts focus primarily on Chapters 1–5, covering foundations, orchestration, connectors, control flow, and transformation tools. Later chapters on advanced monitoring, optimization, and CI/CD are referenced but not detailed in the available material. ##