Data Governance with Unity Catalog on Databricks Implement Data and AI Governance with Databricks Data Intelligence Platform (Kiran Sreekumar, Karthik Subbarao)（Z-Library）

【One-Line Pitch】 A practical field guide to governing data and AI assets on Databricks Unity Catalog, showing how to unify access control, auditing, sharing, and protection across clouds and formats. Best for data engineers, AI/ML engineers, data product owners, and data executives who already work in the Databricks ecosystem and need a governance framework that scales. 【Book Arc】 - **Opening (~0%–10%)**: Sets the regulatory and architectural context — why governance matters now, how the cloud and generative AI raise the stakes, and how Unity Catalog fits into the Databricks Data Intelligence Platform. - **Early (~10%–30%)**: Introduces the modern governance stack through a running fictional case (Nexa), the data lifecycle, and the limitations of Hive metastore that motivated Unity Catalog's unified, open design. - **Early–Middle (~30%–45%)**: Goes under the hood of Unity Catalog architecture — metastore, catalogs, schemas, external locations, storage credentials, and the permission model that separates file-level from table-level access. - **Middle (~45%–60%)**: Covers identity management and the "three As" of IAM — authentication, authorization, auditing — including Databricks identity types, IdP sync, SSO, and programmatic authentication across AWS, Azure, and GCP. - **Late (~60%–85%)**: Moves into compute governance and data protection mechanisms — classic compute access modes, fine-grained access controls, overfetching risks, and the Lakeguard serverless filtering layer. - **Ending (~85%–100%)**: Extends governance beyond the organization through Delta Sharing for internal and external data and AI asset sharing, plus auditing and automation practices. 【Key Takeaways】 - **Unity Catalog is more than a catalog — it is an operational governance layer** (Early): It is a required component of the Databricks Platform, not an optional add-on, and centralizes metadata, access control, and audit logs. - **Unified governance spans clouds, formats, and asset types** (Early): One governance model covers Delta, Iceberg, Hudi, CSV, JSON, and Parquet across AWS, Azure, and GCP, which matters for multicloud and mixed-format estates. - **Openness reduces lock-in and enables interoperability** (Early): Open source licensing, credential vending for external engines, and catalog federation (e.g., with AWS Glue) let Unity Catalog govern assets accessed outside Databricks. - **The metastore is the highest-level abstraction** (Middle): One metastore per cloud region, attachable to multiple workspaces, solves Hive metastore's single-workspace limitation and enables metadata sharing and data democratization. - **Table-level privileges override file-level permissions** (Middle): READ FILES alone is insufficient once a table is registered against an external location; SELECT is required, and this behavior is uniform across S3, GCS, and ADLS. - **Identity management is the foundation of governance** (Middle): Authentication, authorization, and auditing map to real-world flows; IdP integration (Entra ID, Okta, etc.) and SSO are prerequisites for consistent access control. - **Fine-grained access controls introduce overfetching risk** (Late): Elevated compute access can expose data users should not see; Unity Catalog mitigates this by requiring underlying table access and, with Lakeguard, filtering at a serverless control-plane layer. - **Delta Sharing extends governance across organizational boundaries** (Ending): Secure internal and external sharing of data and AI assets is a core governance capability, not an afterthought. 【Reading Tips】 - **Deep-read the architecture and permission chapters** (roughly the middle third): The metastore model, external locations, and table-vs-file privilege distinction are the conceptual core; skimming them will make later chapters confusing. - **Skim the identity chapter if you already manage IdPs**: The hotel analogy and IdP sync mechanics are useful for newcomers, but experienced platform admins can move quickly to the programmatic authentication and SSO sections. - **Pay attention to the Nexa case study**: It threads through the book and turns abstract governance concepts into concrete migration and scaling decisions; use it as a mental model for your own rollout. - **Treat the compute and FGAC chapters as implementation checklists**: The overfetching discussion and Lakeguard mitigation are easy to underestimate; note the constraints before designing views or sharing patterns. - **Read the Delta Sharing material last but do not skip it**: It is where governance meets cross-organization collaboration, and it ties together the access-control concepts from earlier chapters. 【Coverage Limits】 The excerpts cover the book's framing, architecture, identity, compute, and sharing themes, but do not include detailed chapter-level content on auditing automation, specific regulatory compliance mappings, or hands-on Delta Sharing configuration steps. This guide reflects the sampled material only.