Data Governance with Unity Catalog on Databricks Implement Data and AI Governance with Databricks Data Intelligence Platform (Kiran Sreekumar, Karthik Subbarao) (z-library.sk, 1lib.sk, z-lib.sk)

📄 File Format: PDF

💾 File Size: 13.9 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

Sreekum a r & Sub b a ra o D a ta G overnance w ith U nity C a talog on D a ta b ricks Kiran Sreekumar & Karthik Subbarao Foreword by Matei Zaharia Data Governance with Unity Catalog on Databricks Implement Data and AI Governance with Databricks Data Intelligence Platform

📄 Page 2

9 7 8 1 0 9 8 1 7 9 6 3 2 5 6 9 9 9 ISBN: 978-1-098-17963-2 US $69.99 CAN $87.99 DATA Kiran Sreekumar is a data architect with deep expertise in engineering and data and AI governance. Based in the United Kingdom, he has more than 15 years of experience in data and AI across industries and has built pioneering data products since the early days of Hadoop. Karthik Subbarao is a specialist in data + AI platform and governance at Databricks, where he helps organizations develop secure, governed data and AI architectures at scale. He has more than a decade of experience designing and implementing solutions in the realms of big data and AI. Karthik is based in Germany. Organizations collecting and using personal data must now heed a growing body of regulations, and the penalties for noncompliance are stiff. The ubiquity of the cloud and the advent of generative AI have only made it more crucial to govern data appropriately. Thousands of companies have turned to Databricks Unity Catalog to simplify data governance and manage their data and AI assets more effectively. This practical guide helps you do the same. Databricks data specialists Kiran Sreekumar and Karthik Subbarao dive deep into Unity Catalog and share the best practices that enable data practitioners to build and serve their data and AI assets at scale. Data product owners, data engineers, AI/ML engineers, and data executives will examine various facets of data governance—including data sharing, auditing, access controls, and automation—as they discover how to establish a robust data governance framework that complies with regulations. • Explore data governance fundamentals and understand how they relate to Unity Catalog • Utilize Unity Catalog to unify data and AI governance • Access data efficiently for analytics • Implement different data protection mechanisms • Securely share data and AI assets internally and externally with Delta Sharing “The transformative approach of Unity Catalog meets the unrivaled experience of Databricks specialists Kiran and Karthik. Together, they reveal the future of data governance.” Lars George, lead product specialist, Databricks “The authors succeed in delivering substantive technical content while engaging the reader through a compelling, real-world narrative style that is both refreshing and instructive.” Arup Nanda, managing director, head of data and AI, JPMorganChase Data Governance with Unity Catalog on Databricks Sreekum a r & Sub b a ra o

📄 Page 3

Praise for Data Governance with Unity Catalog on Databricks Featuring up-to-date Unity Catalog capabilities and a fresh retail case study with Nexa Boutique, this book makes data governance on Databricks approachable. The authors methodically guide you step by step, providing clarity and practical insights you can apply immediately. —Ashok Singamaneni, principal software engineer, Nike With vivid examples and lucid explanations, this book is like a roadmap for implementing modern data governance in the age of the lakehouse. —Tristen Wentling, lead solutions architect, Databricks Governance in modern data ecosystems is often shrouded in complexity. Kiran and Karthik cut through the fog, offering actionable insights and engineering wisdom that make Unity Catalog truly accessible to every practitioner. —Lars George, lead product specialist, Databricks Presenting a technical topic as critical and complex as data management is a formidable challenge—particularly when viewed through the lens of a single data platform technology that is rapidly transforming the industry, all while maintaining a balanced perspective. In this book, the authors succeed in delivering substantive technical content while engaging the reader through a compelling, real-world narrative style that is both refreshing and instructive. —Arup Nanda, managing director, head of Data and AI, JPMorganChase

📄 Page 4

(This page has no text content)

📄 Page 5

Kiran Sreekumar and Karthik Subbarao Foreword by Matei Zaharia Data Governance with Unity Catalog on Databricks Implement Data and AI Governance with Databricks Data Intelligence Platform

📄 Page 6

978-1-098-17963-2 [LSI] Data Governance with Unity Catalog on Databricks by Kiran Sreekumar and Karthik Subbarao Copyright © 2025 Kiran Sreekumar and Karthikeya Sampa Subbarao. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 141 Stony Circle, Suite 195, Santa Rosa, CA 95401. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Aaron Black Development Editor: Corbin Collins Production Editor: Elizabeth Faerm Copyeditor: Stephanie English Proofreader: Andrea Schein Indexer: Sue Klefstad Cover Designer: Susan Brown Cover Illustrator: Monica Kamsvaag Interior Designer: David Futato Interior Illustrator: Kate Dullea September 2025: First Edition Revision History for the First Edition 2025-09-11: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781098179632 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Data Governance with Unity Catalog on Databricks, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the authors and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. This book is not authored by or on behalf of Databricks. The views and perspectives expressed herein are those of the authors and may or may not reflect the official position of Databricks. While every effort has been made to ensure the accuracy of the information contained in this book, dis‐ crepancies may exist. In the event of any discrepancies between the content of this book and Databricks’ official public documentation, the official documentation shall take precedence and be regarded as the authoritative source.

📄 Page 7

Table of Contents Foreword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii Prologue: Governance by Choice. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi 1. The Modern Governance Stack. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Introducing Data Governance 1 Benefits of Effective Data Governance 2 The Lifecycle of Data 4 Governing the Ungoverned 6 Complying with Increasing Regulations 7 The Dawn of the Lakehouse 8 Deriving Value from Data 8 Data Warehouses and Data Lakes 11 The Lakehouse Paradigm 16 Databricks Unity Catalog: Enabling Unified Governance 22 Introducing Unity Catalog 22 Databricks Platform Architecture 26 Data Sharing and Collaboration 28 Summary 29 2. Unity Catalog Under the Hood. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 The Governance Story So Far 32 Hive Metastore as the Default Catalog 33 The Dilemma of Governance in Hive Metastore 36 Unity Catalog Architecture 45 v

📄 Page 8

Centralized Governance with Unity Catalog 46 The Governance Model of Unity Catalog 52 Decoupled Storage Credentials 53 External Location for Cloud Object Storage 54 Compute Modes in Unity Catalog 57 Data Management Features 63 Catalog as the Namespace 64 Data Isolation at Catalog and Schema Level 65 Catalog to Workspace Binding 65 Summary 67 3. Identity Management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 Databricks Constructs 71 Cloud-Specific Details 72 Access to Databricks and Beyond 75 Databricks Securables 76 Databricks Identities 77 Databricks Identity Types 77 Predefined Admin Roles and Responsibilities 78 Interfaces to Access the Platform 80 Databricks UI 80 Databricks REST API 81 Identity Provisioning 82 Syncing Identities from Identity Provider to Databricks Account 82 Automatic Identity Management with Microsoft Entra ID 84 Databricks Workspace Assignment 85 User Access Provisioning and De-provisioning 87 Single Sign-On 88 Programmatic Authentication Methods 90 Cloud-Specific Authentication: Azure Databricks 92 Cloud-Specific Authentication: Databricks on GCP 93 OAuth Token Federation 93 Identity Best Practices 95 Summary 96 4. Unity Catalog and Compute. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 Implementing Governance: A Two-Part Problem 98 Classic Compute in Databricks 105 Standard or Shared Access 105 Dedicated or Single User Access 113 Assigned to Group Cluster 119 vi | Table of Contents

📄 Page 9

Going Serverless with Databricks 121 Serverless Generic Compute 123 Serverless Data Warehouse 125 Serverless Model Serving 127 Serverless Databricks Apps 129 Serverless Lakeflow Declarative Pipelines 130 Summary 131 5. Access Controls and Permissions Model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 Access Management 133 Access Controls 136 Workspace Access Controls 136 Unity Catalog Access Controls 138 Permissions Model 141 Access Controls on Nontabular Data 143 Managed and Unmanaged Datasets 145 Advanced Access Controls 149 Data Governance Models 159 Centralized Data Governance 159 Distributed Data Governance 160 Federated Data Governance 161 Data Storage and Distribution 162 Catalog Layout and Nomenclature 163 Data Sharing and Distribution 169 Bringing It All Together 174 Summary 178 6. Governing AI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 What Is AI Governance? 180 AI Model Lifecycle 181 Model Training 182 Model Serving 183 Governing AI Systems on Databricks 184 MLOps 186 Large Language Models 189 Mosaic AI Gateway 192 Components of an AI System 192 Implementing an AI System 194 Summary 197 Table of Contents | vii

📄 Page 10

7. Observability and Discoverability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199 Unity Catalog System Tables 200 Architecture 203 Audit Observability 205 Lineage Observability 207 Cost Observability 208 Compute Observability 210 Jobs Observability 212 Marketplace Observability 214 Model Serving Observability 215 Query History and Storage Observability 216 Observability Assistant 217 Data Quality in Databricks 220 Lakehouse Monitoring 223 The Profiles 224 The Baseline Table 225 The Monitoring Artifacts 226 Data Quality Monitoring 228 Discoverability in Unity Catalog 228 Asset Description 229 Tagging 231 AI-Powered Search 232 BROWSE Privilege 233 Insights and Popularity 235 Lineage 236 Lakehouse Federation 237 Enterprise Catalogs 238 Certification and Deprecation 238 Summary 240 8. Data Sharing and Collaboration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 Databricks Data Access Patterns 244 Data Sharing and Publishing with Delta Sharing 246 Data Governance Beyond Metastores 248 Why Delta Sharing? 249 D2D Sharing Under the Hood 250 Ownership and Privileges 252 Catalog Layout 253 Challenges 256 Internal and External Sharing 262 Data Mesh with Delta Sharing 263 viii | Table of Contents

📄 Page 11

External Sharing 264 Databricks Marketplace and Clean Rooms 265 Summary 266 9. Open Access. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267 Managed Versus External Table 269 Why Use External Tables? 270 Data Independence 271 Managed Tables for the Win 272 Open Source Unity Catalog 275 External Access 277 Unity and Iceberg REST Catalog 278 Credential Vending 281 Catalog Interoperability 283 Summary 283 10. Being Compliant with Regulatory Standards. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285 GDPR Compliance 288 The Platform Decision 290 Simplifying the Compliance Journey on Databricks 291 Treating Data and AI Assets as Products 291 Detecting and Securing Sensitive Data 294 Architecture Best Practices for Handling Sensitive Data 297 Summary 300 11. Accelerating Unity Catalog Adoption. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301 Automatic Enablement of Unity Catalog 302 Default Metastore 305 Default Catalog 306 Default Schema 306 Migrating from HMS to Unity Catalog 308 Upgrade Wizard 308 UCX 309 HMS Federation 318 Supported HMS Variants 319 How to Federate HMS 320 Summary 324 12. The Future of Unity Catalog. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325 Advanced Data Governance 326 Catering to Business Users 327 Table of Contents | ix

📄 Page 12

Unity Catalog Metrics 328 Business User-Friendly Interface 329 Doubling Down on Openness and Interoperability 330 Summary 331 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333 x | Table of Contents

📄 Page 13

Foreword The ability to harness, secure, and govern data and AI assets is not just a technical requirement—it is a strategic imperative. As organizations accelerate their adoption of analytics and AI, the complexity and scale of data ecosystems have grown expo‐ nentially. The challenge is clear: how do we democratize access to data and AI while maintaining robust governance, compliance, and security? We developed Unity Catalog at Databricks in response to this challenge, as the industry’s first unified governance system for data and AI. In 2020, when we began working on the project, we were seeing more and more customers bogged down by the complexity of governing high-quality datasets than by the data analytics tasks themselves. Moreover, these same customers were starting to use unstructured data and AI in addition to tabular data, and these required a completely different governance infrastructure. At this point, we had a choice: patch on individual governance features to our platform or design a unified governance system across the whole lifecycle of data, from unstructured files to AI models. We chose the latter, as we believed that solving this problem the “right” way would greatly simplify life for our customers. This bet paid off, with the vast majority of our workloads now running on Unity Catalog and customers reporting significant improvements from the simplicity of unified governance. Databricks has always advocated for open source software and open formats. Unity Catalog is no different. Our vision is for Unity Catalog to be the most open and interoperable catalog for data and AI. On top of the catalog, we have also pioneered open interfaces for data sharing across companies by creating the Delta Sharing protocol, which is now one of the largest ecosystems for data delivery in the industry. xi

📄 Page 14

This book is a guide to understanding, implementing, and mastering the basics of data governance with Databricks Unity Catalog. Whether you are modernizing legacy systems, scaling data operations, or building a foundation for AI, the principles and best practices outlined here will help you create a unified, secure, and future-ready data platform. You will discover how you can leverage Unity Catalog to streamline governance, enhance transparency, improve data quality, and unlock the full potential of your data estates. Embark on the next stage of data governance, where Unity Catalog centralizes clarity, control, and collaboration within your core data strategy. — Matei Zaharia, CTO and cofounder at Databricks xii | Foreword

📄 Page 15

Preface Welcome to Data Governance with Unity Catalog on Databricks. In the current transi‐ tion from a decade dominated by the evolution of file formats, the next era is being shaped by the prominence of catalogs. Since its introduction in 2021, Unity Catalog has emerged as the foundational component of the Databricks Data Intelligence Platform. The open sourcing of Unity Catalog has unlocked new possibilities for governance innovation, overcoming traditional tool limitations. As the industry continues its shift toward open and interoperable technologies, Unity Catalog stands out as an open and extensible catalog. Its open API and integration with leading file formats and applications in the data and AI landscape enable unpar‐ alleled flexibility. Building on this foundation, with native support for Delta and Iceberg REST catalog and tables, views, cloud storage files, AI models, and functions, Unity Catalog is a true multimodal catalog that provides a unified data-management experience. Why We Wrote This Book and Why Now When we first pitched the idea of this book to O’Reilly, Unity Catalog was a propriet‐ ary component of the Databricks Platform. Over the course of the book’s writing, Unity Catalog became open source and incorporated numerous enhancements and new features. Despite the architecture having undergone significant changes and evo‐ lution over time, the fundamental principles and governance implementation have remained relatively consistent. Having worked closely with Unity Catalog for over three years, we recognized the importance of solidly understanding its core concepts. We determined that the time was ripe to document these foundational elements, as well as the features and functionalities that are currently publicly available. xiii

📄 Page 16

Who This Book Is For Our book on governance with Unity Catalog is primarily aimed at the following people: Data architects and executives Decision makers responsible for shaping the data and AI strategy for entire organizations or business units will get a lot out of this book. It provides a comprehensive understanding of the industry direction on open, centralized multimodal catalogs and interoperability, enabling them to make informed stra‐ tegic decisions that drive business success. Data application teams This book is also for professionals who design and implement ETL pipelines, automate data processing, and make data accessible to various stakeholders, including data engineers, analysts, and scientists. It provides practical guidance on leveraging Unity Catalog to improve data management and governance, ensure data quality, and enable seamless data sharing and collaboration. Cloud platform engineers Experts responsible for providing a platform for data teams and enforcing secu‐ rity and governance standards will benefit from this book. It provides in-depth guidance on leveraging Unity Catalog to build a robust, secure, and compliant data platform that supports their organization’s data-driven initiatives. Databricks consultants For seasoned professionals with experience using the Databricks Platform, this book provides expert guidance on Unity Catalog and its applications, enabling them to enhance their consulting services and deliver high-value solutions to their clients. Machine learning (ML)/AI engineers As the practitioners responsible for designing, developing, and deploying AI/ML models and functions, ML/AI engineers should read this book to learn how Unity Catalog can help them secure their AI/ML assets and ensure the integrity of their models and data. Additionally, the following group may also benefit from this book, although to a lesser extent: Data stewards and chief data officers (CDOs) While the book focuses on the practical aspects of Unity Catalog and related features, the concepts and approaches discussed will help data stewards and CDOs understand the overall governance framework and how Unity Catalog can support their organization’s compliance with regulatory standards. xiv | Preface

📄 Page 17

How This Book Is Organized To get the most out of this book, we recommend starting with Prologue: Governance by Choice, where we introduce Nexa Boutique. This fictional organization serves as the backdrop for our exploration of Unity Catalog. Although our stories and experi‐ ences about Nexa Boutique are made up, they are informed by technical accuracy and designed to illustrate key concepts and best practices. By following Nexa Boutique’s journey, you’ll better understand how Unity Catalog can be applied in a real-world setting. The chapters provide a comprehensive guide to data governance, covering the essen‐ tial concepts, infrastructure, and implementation considerations. They also examine the features and capabilities of Unity Catalog in depth, to provide a thorough under‐ standing of its role in supporting effective data governance: Chapter 1, “The Modern Governance Stack” We explain the fundamentals of data governance, lakehouse, and Databricks architecture, and briefly introduce Unity Catalog. Chapter 2, “Unity Catalog Under the Hood” The chapter discusses the history of governance in HMS and how it led to the creation of Unity Catalog and takes a deep dive into its architecture. The chapter ends with Unity Catalog’s governance model and data-management features. Chapter 3, “ Identity Management” Identity management is a critical foundation for data and AI governance, and we explore it in detail, starting with cloud-specific considerations and progressing to identity provisioning, single sign-on, and best practices. Chapter 4, “Unity Catalog and Compute” Platform teams will find this section particularly relevant as we explore the changes to compute infrastructure in Unity Catalog. It covers standard, dedica‐ ted, and serverless access modes and their respective features and limitations. Chapter 5, “Access Controls and Permissions Model” We explore Unity Catalog access controls in depth, examining the data gover‐ nance models and the range of functionalities available. We dive deep into catalog layout and permissions modelling. Chapter 6, “Governing AI” This chapter explores the new paradigm of AI governance practices in Unity Catalog, covering the lifecycle of AI model governance and the components and implementation of AI systems. Preface | xv

📄 Page 18

Chapter 7, “Observability and Discoverability” This section is particularly relevant for DevOps and FinOps teams, as we explore the critical aspects of data observability and discoverability in a governance platform and how Unity Catalog supports these capabilities. Chapter 8, “Data Sharing and Collaboration” This chapter explores how to share data and AI assets within and outside your organization to enable collaboration. It covers different patterns and best practi‐ ces with a focus on cross-metastore governance. Chapter 9, “Open Access” Open access to data from external tools and engines is crucial when choosing a catalog. This section explores the various integration options for open data access and the open source Unity Catalog. Chapter 10, “Being Compliant with Regulatory Standards” This chapter discusses how to comply with regulatory requirements, including General Data Protection Regulation (GDPR) and architectural best practices for handling sensitive data. Chapter 11, “Accelerating Unity Catalog Adoption” Having learned how to use Unity Catalog, you can now accelerate its adoption in your organization. This chapter covers automatic Unity Catalog enablement, migration, and federation from your Hive metastore to Unity Catalog. Chapter 12, “The Future of Unity Catalog” We conclude this book with a forward-looking perspective on the direction of Unity Catalog, its future, and what’s coming next on its roadmap. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords. xvi | Preface

📄 Page 19

Constant width bold Shows commands or other text that should be typed literally by the user. Constant width italic Shows text that should be replaced with user-supplied values or by values deter‐ mined by context. This element signifies a tip or suggestion. This element signifies a general note. This element indicates a warning or caution. O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit https://oreilly.com. Preface | xvii

📄 Page 20

How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 141 Stony Circle, Suite 195 Santa Rosa, CA 95401 800-889-8969 (in the United States or Canada) 707-827-7019 (international or local) 707-829-0104 (fax) support@oreilly.com https://oreilly.com/about/contact.html We have a web page for this book, where we list errata and any additional infor‐ mation. You can access this page at https://oreil.ly/data-governance-with-unity-catalog- on-databricks. For news and information about our books and courses, visit https://oreilly.com. Find us on LinkedIn: https://linkedin.com/company/oreilly-media. Watch us on YouTube: https://youtube.com/oreillymedia. Acknowledgments This book wouldn’t have been possible without the tremendous work done by the Databricks engineering and product teams under the leadership of Matei Zaharia in building Unity Catalog for the Databricks Platform and eventually making it open source for the community. We would like to extend our heartfelt thanks to Lars George, who provided invaluable guidance and offered insightful comments and suggestions throughout the writing of this book. Special thanks to fellow reviewers Andrew Weaver, Arup Nanda, Ashok Singamaneni, Bennie Haelen, Siddharth Bhai, and Tristen Wentling for their detailed reviews and suggestions, which helped us refine the book. A very special thanks to Bernhard Walter, whose knowledge and ideas have influenced the content of this book. We are thankful to the team at O’Reilly, especially Aaron Black and Corbin Collins, for supporting our proposal to publish this book and partnering with us to make it a reality. xviii | Preface