ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale (Bennie Haelen)（Z-Library）

📄 File Format: PDF

💾 File Size: 7.6 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

M L a nd G enera tive A I in the D a ta La kehouse Bennie Haelen ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale

📄 Page 2

ISBN: 978-1-098-17849-9 US $79.99 CAN $99.99 DATA Your organization’s data lives in a lakehouse. Your AI ambitions demand models that understand your business, not just the internet. This book bridges that gap, showing you how to build scalable, production-ready ML and generative AI solutions using Databricks and the full power of data lakehouse architecture. Author Bennie Haelen draws on deep enterprise experience across healthcare, energy, and finance to guide you through deploying, tuning, and governing models on Databricks. You’ll learn to combine traditional ML with LLMs, control costs using open source models, and apply best practices that hold up in production. • Build, deploy, and monitor ML and GenAI models on Databricks lakehouse architecture • Extract deeper actionable insights from your business data using large language models • Combine traditional ML and GenAI models for customized, scalable enterprise solutions • Control costs with open source models while maintaining performance and efficiency • Apply governance best practices using MLflow and Unity Catalog within Databricks ML and Generative AI in the Data Lakehouse “Bennie Haelen is a rare soul who can clearly explain complex topics. This book will help you understand how AI and LLMs work so that you can build a better end-to-end ML or AI system.” Cathy Snell, solutions architect, Databricks “This must-read book focuses on how Databricks can be used to design, develop, and deploy various forms of artificial intelligence. Bennie Haelen’s practical advice on solving real-world problems will set up data scientists for success.” John Miner, senior data architect, Insight Enterprises, and Microsoft data platform MVP Bennie Haelen is a principal data and AI architect at Insight, a Microsoft and Databricks partner. He specializes in data warehousing, machine learning, and generative AI across cloud platforms, with project experience in healthcare, energy, and finance. He is also the coauthor of Delta Lake: Up and Running (O’Reilly).

📄 Page 3

Bennie Haelen ML and Generative AI in the Data Lakehouse Building and Deploying AI Applications at Scale

📄 Page 4

978-1-098-17849-9 [LSI] ML and Generative AI in the Data Lakehouse by Bennie Haelen Copyright © 2026 Bennie Haelen. All rights reserved. Published by O’Reilly Media, Inc., 141 Stony Circle, Suite 195, Santa Rosa, CA 95401. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (https://oreilly.com). For more information, contact our corporate/institu‐ tional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Andy Kwan Development Editor: Jeff Bleiel Production Editor: Clare Laylock Copyeditor: Stephanie English Proofreader: Tim Stewart Indexer: Sue Klefstad Cover Designer: Susan Brown Cover Illustrator: José Marzan Jr. Interior Designer: David Futato Interior Illustrator: Kate Dullea June 2026: First Edition Revision History for the First Edition 2026-06-12: First Release See https://oreilly.com/catalog/errata.csp?isbn=9781098178499 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. ML and Generative AI in the Data Lakehouse, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the author and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

📄 Page 5

Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi 1. An Overview of Machine Learning, AI, and GenAI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 What Is AI? 2 The Evolution of AI 2 The Subdomains of ML 8 Supervised Learning 9 Unsupervised Learning 11 Semi-Supervised Learning 13 Reinforcement Learning 14 An Overview of GenAI 17 Types of Generative Models 17 GenAI Applications and Use Cases 19 Responsible AI 20 Data Privacy 20 Accountability 20 Transparency 21 Explainability 21 Privacy Risks 21 Societal Impact 21 Content Integrity and Accuracy 22 Summary 22 2. The Databricks Data Intelligence Platform. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Why the Databricks Platform? 25 Unified Lakehouse Architecture 26 Collaborative Tools and Managed Infrastructure 26 Comprehensive ML Ecosystem 26 iii

📄 Page 6

Training, Inference, and the GenAI Paradigm Shift 26 Enterprise Security and Governance 27 Databricks Platform Overview 28 Data Storage and Reliability 29 Data Governance and Security 29 Data Engineering and Pipelines 30 Analytics and Business Intelligence 30 ML and AI 30 The Databricks Runtime 31 Optimized Spark Engine 31 Integrated ML Libraries 32 Language Support 32 Advanced Analytics and AI Capabilities 33 Runtime Summary 33 Clusters, Workspaces, and Notebooks 33 Understanding Databricks Clusters 33 Navigating Databricks Workspaces 36 Leveraging Databricks Notebooks 37 Integrating Clusters, Workspaces, and Notebooks 38 Setting Up an ML Environment 39 Understanding the Databricks ML Workspace 39 Prerequisites 40 Creating an Account and a Workspace 40 Setting Up Clusters for ML 43 Runtime Selection 47 Driver and Worker Nodes 51 Extending the ML Runtime with Additional Libraries 55 Summary 57 3. An Introduction to ML on Databricks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 The End-to-End ML Environment 59 Data Preparation 60 Model Training 63 Model Creation and Validation 63 Model Productionalization 64 Managing the ML Lifecycle with MLflow 64 Challenge 1: Experiment Tracking and Reproducibility 64 Challenge 2: Model Versioning and Lineage 65 Challenge 3: Collaboration Across Teams 66 Challenge 4: Model Packaging and Deployment 67 MLflow in the MLOps Landscape 68 MLflow Components 70 iv | Table of Contents

📄 Page 7

Example Workflow 75 Summary 76 4. End-to-End ML with MLflow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 ML in Context 79 Matching ML Approaches to Business Problems 80 Focus on Implementation, Not Model Selection 81 What You Will Build 83 Exploratory Data Analysis 84 Loading the Dataset 85 Creating a Volume 85 Uploading the Dataset to a Volume 86 Exploring the Dataset 87 Feature Engineering 95 Handling Missing Values 95 Data Cleansing 96 Saving to Unity Catalog 97 Model Training 98 Creating an MLflow Experiment 98 Model Building 98 Model Training with MLflow Tracking 102 Model Predictions 109 Preparing the Test Data 109 Loading the Model from the Unity Catalog Registry 109 Loading as a Native scikit-learn Model 110 Inspecting Feature Importances 111 Loading the Model as a Spark UDF 111 Loading the Model as a Generic PyFunc Model 113 When to Use Each Loading Method 113 Model Deployment 114 Verifying the Registered Model 114 Reviewing Model Metadata and Lineage 115 Managing the Model Lifecycle with Aliases 116 Deploying to a Model Serving Endpoint 118 Querying the Endpoint 120 Updating the Endpoint with a New Version 121 Model Inferencing 122 Preparing the Test Data 123 Retrieving Workspace Credentials 123 Defining the Scoring Function 124 Loading the Champion Model for Local Comparison 124 Calling the Endpoint and Comparing Predictions 125 Table of Contents | v

📄 Page 8

Verifying Prediction Consistency 125 Evaluating Accuracy and Prediction Distribution 126 Summary 127 5. Feature Engineering in the Unity Catalog. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 The Role of Feature Engineering in ML 129 Delta Tables: The Foundation of Feature Engineering 130 Feature Tables: The Core of the Feature Store 131 Integration with Unity Catalog 131 Feature Engineering in the Lakehouse 132 Offline and Online Stores 133 Feature Consistency and Reusability 134 ML Workflow with the Feature Store 134 Step 1: Creating and Managing Feature Tables 134 Step 2: Training with Features from the Feature Store 135 Step 3: Real-Time Inference 137 Example: LendingClub Feature Store 138 Introduction to the LendingClub Dataset 138 Approach 139 Step 1: Loading and Preprocessing the LendingClub Dataset 139 Step 2: Build Your Feature Table 140 Step 3: Building the Training Dataset Using FeatureEngineeringClient 144 Step 4: Building the Model with Features from the Feature Table 146 Step 5: Model Visualizations 148 Step 6: Logging and Registering the Model in Unity Catalog 154 Summary 157 6. ML at Scale. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 The Role of Spark in ML 160 Scaling ML with Spark 161 Distributed Data Processing 162 pandas API on Spark 164 Using Spark MLlib 165 Distributed Deep Learning: TorchDistributor, DeepSpeed, and Ray 165 Parallelized Model Training 167 Spark Structured Streaming for Real-Time ML 167 Hyperparameter Tuning with Optuna, Hyperopt, and Ray Tune 168 The Role of Delta Lake in Scalable ML 168 ACID Transactions and Schema Enforcement 169 Time Travel and Data Versioning 169 Efficient Incremental Data Processing 170 Synergy with Spark and Databricks 170 vi | Table of Contents

📄 Page 9

Platform Capabilities for ML at Scale 171 The Shift Away from AutoML 172 The Broader Mosaic AI Direction 174 Accelerating ML Workloads with Photon 175 MLflow Integration at Scale 176 Serverless Compute for ML Workloads 177 Unity Catalog Integration for ML at Scale 178 Cost Management at Scale 178 ML at Scale Use Case 179 Step 1: Data Loading and Initial Type Conversions 181 Step 2: Data Preprocessing 185 Step 3: Feature Engineering 186 Step 4: Feature Transformations and Vectorization 187 Step 5: Efficient Storage with Delta Lake 189 Step 6: Model Training at Scale 190 Step 7: Hyperparameter Tuning with TrainValidationSplit 192 Step 8: Distributed Hyperparameter Tuning with Optuna 194 Step 9: Log and Register the Best Model in Unity Catalog 198 Use Case Results 201 Summary 202 7. GenAI in the Lakehouse: Foundations and Architecture. . . . . . . . . . . . . . . . . . . . . . . . . 203 From Predictive to Generative Intelligence 204 GenAI Architecture 206 The Mathematics of Scale: Why Bigger Models Work Better 211 Compute-Optimal Training: The Chinchilla Insight 213 Practical Implications for the Lakehouse 215 The Transformer Architecture 216 Why Transformers Succeeded: Parallelism at Scale 216 Three Transformer Variants 217 Core Components of a Transformer Block 217 Autoregressive Generation: From Parallel Training to Sequential Output 219 Implications for the Lakehouse 221 Model Serving Optimization in Databricks 223 Unity Catalog Integration: Governance for Foundation Models 224 Infrastructure Trade-Offs and RAG 225 Tokenization, Embeddings, and Context Windows 225 Tokenization: Breaking Text into Learnable Units 227 Embeddings: From Tokens to Geometry 227 Context Windows: The Memory Bottleneck 229 From Probabilities to Text: Decoding Strategies 229 Foundation Models as the New Transfer Learning Paradigm 230 Table of Contents | vii

📄 Page 10

Emergent Capabilities at Scale 231 From Foundation Models to Production Systems 232 Foundation Models in the Lakehouse 234 GenAI in the Lakehouse 234 Summary 236 8. GenAI in a Databricks Lakehouse Environment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 Prompt Engineering 238 Understanding Prompts and Prompt Engineering 238 Anatomy of a High-Quality Prompt 238 Prompt Engineering Techniques 239 Practical Tips for Effective Prompting 242 The AI Playground in Databricks 243 Exploring the Playground 243 Model Exploration and Prompt Design 244 Retrieval-Augmented Generation 246 RAG Definition 246 How Do Language Models Learn Knowledge? 247 Factual Recall with Context 249 Vector Stores 255 Databricks AI Vector Store 257 Practical RAG Implementation 259 Preparing Data for RAG 259 Executing Vector Retrieval via Databricks SQL or PySpark 273 Summary 287 9. AI Agents in the Lakehouse. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289 Fundamentals of AI Agents 289 What Is an AI Agent? 290 Benefits of Agents 290 Choosing the Right Execution Model 290 Core Components of a Lakehouse Agent 292 A Lakehouse Agent Reference Architecture 293 Agent Lifecycle 296 Memory Approaches for Lakehouse Agents 297 Agent Memory Shapes 297 Memory Design Patterns 298 Agent Patterns 299 Reflection 299 Tool Use 300 ReAct 302 Planning 303 viii | Table of Contents

📄 Page 11

Multiagent Collaboration 304 Bringing the Patterns Together 305 Failure Modes and Controls for Lakehouse Agents 306 Evidence-Based Diagnostic Analytics with ReAct 308 The Scenario: Asking the Agent “Why?” 309 Overview of the Lakehouse Architecture 309 ReAct Logic and a Custom-Built Toolkit 310 Code Walkthrough 312 ReAct Agent Summary 326 Summary 326 10. The Model Context Protocol. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329 What Is MCP? 330 LLM Tool Standardization with MCP 332 REST Versus MCP: Static Contracts Versus Dynamic Discovery 335 The REST Model: Static Contracts 335 The MCP Model: Dynamic Discovery 335 Key Differences 336 MCP Architecture 337 Core Components 338 The Three Primary Server Primitives 338 Why This Architecture Matters in the Lakehouse 340 Deployment Modes and Runtime Behavior 340 Local Mode 340 Remote Mode 341 Scalability and Flexibility 342 Client/Server Communication 343 Phase 1: Initialization and Capability Negotiation 343 Phase 2: Discovery and Invocation 343 Example: Weather API Service 344 Progress Notifications and Long-Running Operations 344 Why the Lifecycle Matters 345 Databricks Unity Catalog MCP Server 345 Key Capabilities 346 Server Architecture and Initialization 348 Run It Now! 348 Main Solution Components 352 Project Structure 356 Design Principles 358 Implementation Walkthrough 358 Testing and Interacting with the MCP Server 362 Summary 371 Table of Contents | ix

📄 Page 12

11. Agent-to-Agent Communication and the DSPy Framework. . . . . . . . . . . . . . . . . . . . . . 373 A2A Protocol 375 Overview and Foundation 376 Core Architecture 377 Integration with Data Lakehouses 382 Relationship with MCP 386 Enterprise Adoption and Ecosystem 387 Declarative Self-Improving Python 388 Core Concepts 388 DSPy in the Data Lakehouse 390 DSPy Integration with MLOps and Model Management 393 Advanced DSPy Capabilities for Lakehouse Workloads 394 A2A and DSPy Together: Multiagent Intelligence on the Lakehouse 395 Example: Retail Analytics Assistant on a Databricks Lakehouse 396 Problem Overview 396 Why A2A + DSPy? 396 Architecture Overview 397 Example Interaction 398 Running the Example 401 Summary 402 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405 x | Table of Contents

📄 Page 13

Preface AI is no longer a research curiosity confined to academic papers and proof-of- concept demos. It is reshaping how organizations derive value from data, how soft‐ ware systems are built, and how decisions are made at scale. At the center of this transformation sits a powerful and rapidly maturing platform: the data lakehouse. Combining the openness and flexibility of a data lake with the governance and per‐ formance of a data warehouse, the lakehouse has become the natural home for machine learning (ML) and, more recently, for the generative AI (GenAI) and agentic systems that are defining the next era of intelligent applications. What makes this moment so exciting is the pace of convergence. Traditional machine learning, large language models, feature engineering, model serving, AI agents, and the protocols that connect them are all maturing simultaneously, and the Databricks Data Intelligence Platform sits at the intersection of every one of these disciplines. This book is about that intersection. It is about understanding not just what each piece does in isolation, but how they fit together to form a coherent, production- grade AI and ML practice built on the lakehouse. Whether you are training your first scikit-learn model on Databricks, building a Retrieval-Augmented Generation (RAG) pipeline, orchestrating multiagent work‐ flows, or designing the integration layer that connects your language models to enter‐ prise data systems through the Model Context Protocol (MCP), this book covers the full spectrum. The goal is to give you both the conceptual grounding and the practi‐ cal, hands-on knowledge to move confidently across all of it. Who Should Read This Book This book is written for data and AI practitioners who are working with, or planning to work with, the Databricks platform in a professional context. Specifically, it will be most valuable to the following readers: xi

📄 Page 14

• Data engineers and data architects who want to understand how ML and GenAI workloads fit into the lakehouse architecture they already manage and what new infrastructure and governance considerations those workloads introduce • Data scientists and ML engineers who want to move beyond notebook experi‐ mentation and build reproducible, governed, production-ready ML systems using MLflow, Unity Catalog Feature Engineering, and Databricks Model Serving • Software engineers and technical leads who are extending their organizations into GenAI and agentic architectures and want to understand how tools such as the MCP and Agent2Agent (A2A) protocol integrate with enterprise data systems • Technology architects and technical decision-makers who need a comprehensive mental model of the Databricks Data Intelligence Platform and how its capabili‐ ties across ML, GenAI, and agentic systems relate to one another Readers will get the most out of this book if they have a working familiarity with Python and some exposure to data concepts such as tables, schemas, and SQL. Prior experience with Databricks is helpful but not required. The early chapters provide the foundational context needed to engage productively with the more advanced material in later chapters. Why I Wrote This Book When I began working with the Databricks platform in an enterprise context, I found no shortage of documentation on individual features: how to configure a cluster, how to log an experiment with MLflow, how to register a model. What I could not find was a single, coherent resource that connected all of those pieces into a unified pic‐ ture of how to build and operate AI and ML systems at scale on the lakehouse. That gap is what motivated me to write this book. The landscape has become significantly more complex since that time. GenAI has moved from the margins to the center of enterprise technology strategy. Agentic sys‐ tems, where AI models reason, plan, and take actions across tools and data sources, are rapidly moving from prototype to production. The MCP has emerged as a key integration standard. The Databricks platform has evolved accordingly, absorbing and integrating capabilities across all of these areas with remarkable speed. This book is the resource I wish I had: a practical, architecture-aware guide that cov‐ ers the full span of AI and ML on the data lakehouse, from foundational concepts through the most current agentic and integration patterns. My hope is that it gives you not only the knowledge to use these tools, but also the understanding to make good decisions about how and when to apply them in your own environment. xii | Preface

📄 Page 15

Navigating This Book This book is organized to take you from foundational concepts through advanced, production-oriented architectures. The chapters build on one another, but each is written to be useful as a standalone reference as well. Chapter 1, “An Overview of Machine Learning, AI, and GenAI” Traces the evolution of AI and covers the four ML subdomains: supervised, unsupervised, semi-supervised, and reinforcement learning. The chapter then surveys GenAI architectures (GANs, VAEs, and transformer-based models) and establishes why the lakehouse architecture is a natural fit for AI and ML work‐ loads. Chapter 2, “The Databricks Data Intelligence Platform” Provides a detailed orientation to the platform, and walks through cluster types, access modes, workspaces, and notebooks, and serves as a practical guide to set‐ ting up an ML environment. Chapter 3, “An Introduction to ML on Databricks” Covers the end-to-end ML workflow on the platform. The chapter frames MLflow’s role clearly alongside other lifecycle tools and explains why its simplic‐ ity, framework agnosticism, and native Databricks integration make it the back‐ bone of ML operations throughout the book. Chapter 4, “End-to-End ML with MLflow” Puts the preceding concepts into practice with a complete hotel booking cancel‐ lation prediction pipeline. Starting from exploratory data analysis, it works through feature engineering stored as a managed Delta table in Unity Catalog, training four classification models with MLflow autologging, and selecting the best performer programmatically. Chapter 5, “Feature Engineering in the Unity Catalog” Explains how feature engineering works in the current Databricks platform. This chapter covers the FeatureEngineeringClient, FeatureLookups, training set con‐ struction with automatic lineage capture, and the distinction between offline Delta tables and online stores for real-time serving. Chapter 6, “ML at Scale” Addresses the fundamental scalability limitations of single-node frameworks and explains how Apache Spark and Delta Lake together form the foundation for enterprise ML. The chapter concludes with a hands-on end-to-end pipeline using the NYC Yellow Taxi dataset, building a distributed Random Forest regressor with Optuna-based hyperparameter tuning and Unity Catalog model registration using the “Champion” alias pattern. Preface | xiii

📄 Page 16

Chapter 7, “GenAI in the Lakehouse: Foundations and Architecture” Establishes the technical foundations that distinguish generative from predictive AI. It examines the four primary generative architectures: VAEs, GANs, diffusion models, and transformers, explaining why autoregressive transformers dominate language modeling. Chapter 8, “GenAI in a Databricks Lakehouse Environment” Grounds GenAI in practical Databricks workflows. This chapter begins with prompt engineering techniques and introduces the Databricks AI Playground for interactive experimentation. The chapter then builds a complete RAG pipeline. Chapter 9, “AI Agents in the Lakehouse” Defines what an agent is and, just as importantly, when not to use one. It introdu‐ ces a reference architecture covering the orchestrator, model, tool and data layer, and observability plane, then walks through the agent lifecycle and the memory shapes suited to different tasks. The chapter concludes with a hands-on ReAct diagnostic analytics agent built on a medallion architecture. Chapter 10, “The Model Context Protocol” Introduces MCP as the emerging open standard for connecting AI agents to external tools and data sources. It walks through a complete custom Databricks Unity Catalog MCP server. Chapter 11, “Agent-to-Agent Communication and the DSPy Framework” Closes the book with two advanced topics that define the frontier of enterprise agentic AI: the A2A protocol and DSPy (Declarative Self-Improving Python). It concludes with a Retail Analytics Assistant case study in which A2A, DSPy, and MCP work together on a Databricks lakehouse to investigate a business question across specialized agents. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program ele‐ ments such as variable or function names, databases, data types, environment variables, statements, and keywords. xiv | Preface

📄 Page 17

This element signifies a tip or suggestion. This element signifies a general note. This element indicates a warning or caution. Using Code Examples Supplemental material (code examples, exercises, etc.) is available for download at https://oreil.ly/supp-MLGenAI_DataLakehouse. The code for this book is organized across three GitHub repositories. Most chapters draw on the main supplemental repository. Two chapters use dedicated repositories that match the specific toolchains they cover: Chapter 10 uses the MCP repository at https://oreil.ly/mcpRepo, and Chapter 11 uses the A2A and DSPy repository at https://oreil.ly/a2a_dspyRepo. Each of these chapters tells you which repository its examples come from, so you can clone the one you need as you go. If you have a technical question or a problem using the code examples, please email support@oreilly.com. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “ML and Generative AI in the Data Lakehouse by Bennie Haelen (O’Reilly). Copyright 2026 Bennie Haelen, 978-1-098-17849-9.” Preface | xv

📄 Page 18

If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit https://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 141 Stony Circle, Suite 195 Santa Rosa, CA 95401 800-889-8969 (in the United States or Canada) 707-827-7019 (international or local) 707-829-0104 (fax) support@oreilly.com https://oreilly.com/about/contact.html We have a web page for this book, where we list errata and any additional informa‐ tion. You can access this page at https://oreil.ly/mlGenAI_DataLakehouse. For news and information about our books and courses, visit https://oreilly.com. Find us on LinkedIn: https://linkedin.com/company/oreilly. Watch us on YouTube: https://youtube.com/oreillymedia. xvi | Preface

📄 Page 19

Acknowledgments Writing a technical book of this scope is never a solitary effort, and I am grateful to everyone who contributed their time, knowledge, and patience to bringing it to life. I want to begin by thanking the team at O’Reilly who made this book possible. Andy Kwan, the acquisitions editor, believed in this project from the start and helped bring it into being. My editor, Jeff Bleiel, shaped this book in ways large and small through his guidance, patience, and editorial instincts, and working with Jeff has been a genu‐ ine privilege. Clare Laylock, the production editor, steered the manuscript through production with care and professionalism. I am deeply grateful to the technical reviewers who gave their time and expertise to improve this book: Cathy Snell, John Miner, Sathiesh Veera, and Seethamareddy Gowda. Their feedback was direct, thoughtful, and essential. Any errors that remain are entirely my own. Finally, and most importantly, I want to thank my wonderful wife, Jenny. Every book is a long project that asks more of the people closest to the author than it ever should. That she was willing to go through it all a second time, after living through the writ‐ ing of my first book, Delta Lake: Up and Running, says everything about her generos‐ ity and grace. Her support, encouragement, and patience made this possible. This book is as much hers as it is mine. Preface | xvii

📄 Page 20

(This page has no text content)