Page
1
(This page has no text content)
Page
2
Fundamentals of Microsoft Fabric Designing End-to-End Analytics Solutions Nikola Ilic and Ben Weissman
Page
3
Fundamentals of Microsoft Fabric by Nikola Ilic and Ben Weissman Copyright © 2025 Solisyon GmbH and Nikola Ilic. All rights reserved. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800- 998-9938 or corporate@oreilly.com. Acquisitions Editor: Michelle Smith Development Editor: Shira Evans Production Editor: Ashley Stussy Copyeditor: Penelope Perkins Proofreader: Paula L. Fleming Indexer: Ellen Troutman-Zaig Cover Designer: Susan Thompson Cover Illustrator: José Marzan Jr. Interior Designer: David Futato Interior Illustrator: Kate Dullea July 2025: First Edition
Page
4
Revision History for the First Edition 2025-06-25: First Release See http://oreilly.com/catalog/errata.csp? isbn=9781098172923 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Fundamentals of Microsoft Fabric, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the authors and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. 978-1-098-17292-3 [LSI]
Page
5
Foreword In this new era of generative AI, every organization is racing to figure out how to make its teams more productive, to empower its customers, and to reinvent business processes. Data is the fuel that powers AI—now, more than ever, it is critical to ensure that organizational data is accessible, structured, and ready for AI. This is where Microsoft Fabric comes in, making it easier than ever for data teams to land their data in one place, collaborate across this data, and get it in a state where it can be served to the business so that they can make better decisions. The vision of Microsoft Fabric revolves around three main pillars. Fabric is an end-to-end data platform. Data is a team sport. To get a solution up and running takes collaboration among many people with different skills. Before Fabric, these solutions required numerous products to get the job done. This often meant moving data from one product to another; manually integrating these offerings; and managing governance, cost, and usage independently. With Microsoft Fabric, a data team can leverage everything —data integration, data engineering, data warehousing, data science, real-time analytics, and business intelligence—in a shared workspace, with unified development processes and easy adoption. Because Fabric is built as a SaaS platform, it is easy for users to get started, for teams to collaborate in the product, and for administrators to centrally manage security and governance. Copilot is built into all of the authoring experiences in Fabric, making it easy for users to adopt new skills and build their data solutions faster than ever. Fabric is built upon an open, data lake–based architecture.
Page
6
This all starts with OneLake. The goal of OneLake is to make sharing data as easy as sharing files in Microsoft Office. Regardless of where the data is stored—whether directly in OneLake; in an existing data lake in Azure, Amazon Web Services, or Google’s Cloud Platform; or in an operational database from your business apps—OneLake allows all data practitioners to work seamlessly with the data they have access to Fabric has adopted open standards—both Delta and Iceberg formats—allowing customers to land the data once it’s in OneLake and leverage that data in any of the engines in Fabric or in external products that have also adopted these formats. Fabric enables every business user with AI-enabled insights. It is critical that organizations not only have a strong data platform but also make sure business users have access to the relevant data when and where they need it in order to make better decisions. Power BI, as a core part of Fabric, enables business users to easily explore and visualize data. Deep integration across Microsoft Office applications, including Excel, Teams, and PowerPoint, allows business users to access the data in the context of the tools they already know and love. New Copilot-driven data exploration experiences will further enable the business to ask and answer their own questions. There is so much opportunity for anyone getting started with Microsoft Fabric to learn new tools, increase efficiency in their data stack, and ready their business for the era of generative AI. If you’re reading this book, I am guessing you’re excited to get started. This book not only provides a high-level overview of what Microsoft Fabric is
Page
7
but also delves deeply into how to use the product and provides guidance on implementation considerations. As you dive into this book and the product itself, know that Fabric is changing constantly. The product team strives to listen and learn from users, shipping new features, enhancements, and bug fixes weekly. Another resource, aside from this book, is the Fabric user community. You can visit the Fabric Community to join, ask questions, attend a user group meeting, and learn about the latest feature releases. The product team is listening and can’t wait to hear your feedback. Kim Manis Corporate Vice President of Product Microsoft Fabric and Power BI New York, March 2025
Page
8
Preface Every great collaboration starts with a moment of serendipity. Ours began when Nikola, eager to share his insights with the data community, stepped onto the stage at an event organized by Ben. That first encounter—one between a speaker and an organizer—sparked conversations that went far beyond the session itself. Over time, we discovered that while our perspectives and strengths on data and analytics were different, they were also complementary. When Microsoft introduced Fabric, we immediately recognized an opportunity. This new, unified platform brought together the best of data engineering, data science, business intelligence, and real-time analytics. More importantly, it merged the very domains that defined our respective expertise. Fundamentals of Microsoft Fabric is the product of that collaboration. This book is designed to be a practical, hands-on guide for data professionals, engineers, analysts, and business users who want to understand and leverage Microsoft Fabric effectively. Whether you’re coming from the world of SQL-based data warehousing, Power BI modeling, or real-time analytics, this book will help you navigate Fabric’s architecture, capabilities, and use cases. Our goal is to demystify Fabric: you’ll find explanations, best practices, and real-world examples that bridge the gap between theory and application. While Fabric is still evolving, the fundamentals remain the same, and this book serves as a reliable starting point on your journey.
Page
9
We are excited to share our combined knowledge and passion with you. Writing this book has been an adventure, and we hope reading it will be just as rewarding and fun. Welcome to Fundamentals of Microsoft Fabric—let’s build something great together! Who Should Read This Book This book is designed for professionals working with Microsoft Fabric, including data engineers, analytics engineers, data analysts, data scientists, and data architects. Microsoft Fabric is a persona-oriented SaaS (software as a service) solution, and this book follows a similar structure, allowing readers to consume deep dives based on their specific roles. Rather than covering fundamental concepts in depth (such as “What is a data warehouse?”), this book assumes readers already have a solid background in their respective fields. It focuses on the specifics of Fabric, making it ideal for those looking to deepen their understanding of Microsoft Fabric and its ecosystem. Navigating This Book This book is organized into three parts. In Part I, “The Foundation of Fabric”, we talk about what Fabric is in general, what its components are, and how to get started. We also introduce OneLake, the key feature that every feature and experience in Fabric is built on. In Part II, “The Features: A Deep Dive”, we walk through all the experiences (which is what features are called in
Page
10
Fabric) and explain their use cases and how they work. We provide step-by-step guidance and real-world examples. Finally, in Part III, “Putting Fabric into Production”, we make sure you understand not only the individual bits and pieces of Fabric but also what else is needed to put it into production as your analytics platform. Here we answer questions on pricing and security, and we hope to clear up any confusion on when to use what, given that many challenges can be approached from multiple angles or using multiple experiences in Fabric. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords. TIP This element signifies a tip or suggestion.
Page
11
NOTE This element signifies a general note. WARNING This element indicates a warning or caution. O’Reilly Online Learning NOTE For more than 40 years, O’Reilly Media has provided technology and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in- depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit https://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc.
Page
12
1005 Gravenstein Highway North Sebastopol, CA 95472 800-889-8969 (in the United States or Canada) 707-827-7019 (international or local) 707-829-0104 (fax) support@oreilly.com https://oreilly.com/about/contact.html We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/fundamentals-of-microsoft- fabric-1e. For news and information about our books and courses, visit https://oreilly.com. Find us on LinkedIn: https://linkedin.com/company/oreilly- media. Watch us on YouTube: https://youtube.com/oreillymedia. Acknowledgments We want to start by thanking everyone who helped us make this book a reality. Our friends at the Microsoft Product Group (special shoutout to Kim Manis for her foreword!), our amazing tech reviewers Meagan Longoria, Emilie Rønning, Shabnam Watson, Olivier van Steenlandt, and Wolfgang Strasser, as well as our fabulous team at O’Reilly, Michelle Smith, Ashley Stussy, Penelope Perkins, Shira
Page
13
Evans, Paula L. Fleming, and Ellen Troutman-Zaig. We couldn’t have done this without you, and we appreciate you all so much! And speaking of those that we couldn’t have done this without and who are always supporting us: thank you to our families for always supporting us in our endeavors! We love you!
Page
14
Part I. The Foundation of Fabric In this section, we’ll cover the core aspects of Microsoft Fabric, starting with what it is, how it works, and why it plays a major role in modern data management. You will learn about its key components, how they fit together, and how to get started with the platform. Each part of Fabric has a specific purpose, and understanding how they connect will help you make the most of its capabilities. One of the most important components of Fabric is OneLake. This is not just another storage option—it is the backbone of the entire platform. Every tool, feature, and experience within Fabric relies on OneLake for storing, managing, and organizing data. Because of this, knowing how OneLake functions is essential. It provides a unified way to work with data, making it easier to access and use across different workloads. By the time you finish this section, you will have a clear understanding of what Fabric is, what its main components do, and why OneLake is so important. This knowledge will set the stage for working with Fabric effectively and unlocking its full potential.
Page
15
Chapter 1. What Is Microsoft Fabric? Let’s start with one important question, What is Microsoft Fabric? Or, more specifically, Why do we need such an offering? In short, Fabric is the answer to a (if not the) challenge of our times as data professionals. The Why and the What of Microsoft Fabric All organizations have data: it surfaces in different formats, is constantly growing, and needs to be consumed by users to drive decisions and generate insights. Those insights need to run through different workloads, from real-time queries to analytical workloads, reporting, or AI-driven applications, as you see in Figure 1-1. Figure 1-1. Requirements across an organization
Page
16
Those users or personas usually have vastly different skill sets when it comes to working with data. They may not have any coding skills at all, or they may be using different languages like Python on a Spark cluster or SQL across multiple teams in an organization, as illustrated in Figure 1-2. Figure 1-2. Skill sets across an organization This means that to cope with new challenges and tasks, like responding to unprecedented amounts of data in real time in our data-driven world, users need to produce solutions in different ways. Organizations need to allow different users to use different tools that match their skills and needs, as well as organize their data in a way that works for them. This can be raw source data, a data lake, a data warehouse, or a data mart. Microsoft Fabric—which we’ll also refer to as “Fabric”— aims to solve all these problems in a single offering!
Page
17
The Big Picture Fabric addresses each of these complexities, challenges, and needs with a single solution, as you see in Figure 1-3. Figure 1-3. Fabric overview Fabric includes features, services, and tools for every need along the way, from ETL (extract, transform, load; data movement and transformation) to building a data lake or warehouse, performing real-time analytics, and developing visualizations. Fabric even provides artificial intelligence and copilots along the way to support your entire journey. Each of these features within Fabric is called an experience. While we will talk about the concepts, components, and offerings in Fabric in more detail in Chapter 4, let us give you a 30,000-foot overview of what they are and how they contribute to the Fabric ecosystem. Workspaces and Domains Your main organizational units in Fabric are called workspaces, and your workspaces can be grouped into domains.
Page
18
Domains and workspaces are the overarching organizational concepts for both your data in OneLake and your lakehouses, notebooks, warehouses, pipelines, and all your other items. You work with data in workspaces and can tag a workspace or multiple workspaces to a domain. Your workspaces are also what will be connected to your CI/CD (continuous integration, continuous deployment) repository. All Fabric workspaces support Power BI premium features. We’ll talk more about the details of workspaces and domains in Chapter 3. OneLake Fabric uses one shared storage layer called OneLake as its foundation to house all your data. OneLake is often referred to as “the OneDrive for data.” Figure 1-4 shows the general overview of OneLake, including its main components and APIs. OneLake uses the Azure Data Lake Storage (ADLS) Gen2 (Microsoft’s data lake solution for big data analytics) technology to manage all your data and items across your domains and workspaces. In addition to storing data, which can be ingested in a multitude of ways (more on that in Chapter 3), it comes with a data virtualization concept called shortcuts, which allow you to access in real time data that is sitting in other storage accounts like an AWS S3 bucket or Dataverse. Data in OneLake is natively stored in Delta format, an open source data storage format that builds on Parquet, adding ACID transactions (atomicity, consistency, isolation, and durability—properties that ensure reliable and consistent database transactions even in the event of failures),
Page
19
schema evolution, and time travel capabilities to ensure reliability, consistency, and efficient data management across analytical workloads. Additionally, OneLake supports access to storage accounts that use the Apache Iceberg format, an open source table format. Figure 1-4. Overview of OneLake All Fabric engines can read and write Delta; since other third-party tools can read the data with APIs, this means your data is not vendor locked. Data Factory Microsoft Fabric’s Data Factory is designed to offer cloud- scale data movement and transformation services, simplifying the management of complex data integration and ETL processes. It provides a user-friendly, robust, and enterprise-grade experience for data management. As the successor to Azure Data Factory, this service has evolved to incorporate cloud-scale capabilities, addressing the most
Page
20
intricate ETL challenges. Data Factory delivers a modern data integration experience, enabling users to ingest and transform data from various sources, including databases, data warehouses, lakehouses, and real-time data streams. Data Engineering Data Engineering in Microsoft Fabric is a comprehensive suite tailored for data engineers to efficiently manage and transform large volumes of data using Apache Spark (plus other engines for Python and T-SQL notebooks, for example), facilitating the creation of lakehouse architectures. A lakehouse integrates the scalable storage of a data lake with the management capabilities of a data warehouse, providing a unified platform for data ingestion, transformation, and sharing. This architecture supports SQL queries, analytics, machine learning, and other advanced techniques on both structured and unstructured data. Microsoft Fabric enables users to create and manage lakehouses, design data movement pipelines, and utilize Spark job definitions for batch or streaming jobs. Additionally, notebooks are available for writing code related to data ingestion, preparation, and transformation. One key aspect about notebooks is that you can use multiple languages not only across notebooks but also within the same notebook, allowing developers to use the language they are most comfortable with. Data Warehouse Data Warehouse in Microsoft Fabric streamlines data insights while ensuring robust data security and governance through T-SQL constructs. It utilizes a distributed engine to deliver high performance and scalability. Data stored in Parquet format supports ACID