Kubeflow Operations Guide Managing Cloud and On-Premise Deployment (Josh Patterson, Michael Katzenellenbogen etc.) (z-library.sk, 1lib.sk, z-lib.sk)

Josh Patterson, Michael Katzenellenbogen & Austin Harris Kubeflow Operations Guide Managing Cloud and On-Premise Deployment

Praise for Kubeflow Operations Guide “This book is the go-to resource for enterprise deployment of Kubeflow from on-premise to the cloud. It will take you through how to think about Kubeflow on an operational level and then through the ways a team needs to think about integrating with their infrastructure for resources such as GPUs and Identity management.” —Jeremy Lewi, Cofounder of Kubeflow, Principal Software Engineer, Primer “Patterson, Katzenellenbogen, and Harris have pulled together a terrific book that describes not just the components of setting up a production-ready Kubeflow deployment, but the tactical steps necessary to do so on-premises or on any of the hyperscale clouds. This is an essential book for understanding how to bring Kubeflow from experimentation to enterprise-ready.” —David Aronchick, Cofounder of Kubeflow “A concise guide that covers planning, installing, and managing ML infrastructure across on-premises and cloud. This book provides a sorely needed step-by-step tutorial for using Kubeflow to support notebooks and autoscaled ML pipelines across hybrid cloud setups.” —Lak Lakshmanan, Director of Analytics and AI Solutions, Google Cloud “Kubeflow Operations is a great resource that dives deep into the operational aspects of running real-world Kubeflow and Kubernetes clusters. This book also includes best practices for managing Kubernetes security, multitenancy, traffic routing, service mesh, GPUs, autoscaling, and capacity planning.” —Chris Fregly, Developer Advocate, AI and Machine Learning at AWS

“The Josh Patterson, Michael Katzenellenbogen, and Austin Harris book on Kubeflow should be a valuable roadmap for any data engineer or data scientist who is trying to build a modern data driven system. TBs/sec data streams, and online complex DL/ML-based decision models are becoming mainstream. With the availability of 400 Gb/s NDR Infiniband networking and PFLOPS CPU/GPU processing power on a single chip the role of the data scientist is often reduced to assembling available tools and monitoring the whole process rather than actively analyzing data and/or developing models. Data is driving both the feature generation and the model building. This is what this book is about.” —Alex Kozlov, Ph.D., Senior Data Scientist, NVIDIA “Josh Patterson is a skilled practitioner who has helped many companies deploy and use Kubeflow successfully. He has also been deeply involved in the Kubeflow community for several years, giving him in-depth knowledge of the topic and a unique perspective not present in other Kubeflow guides. It is my pleasure to recommend this book.” —Hamel Husain, Staff Machine Learning Engineer, GitHub “Kubeflow is a great way to consistently manage MLOps workflows across many clouds (including on-premises). Setting up and managing a hybrid Kubeflow is nontrivial and the authors do a great job at demystifying the whole process of explaining practical issues faced by MLOps engineers, starting from the guts of Kubeflow to deployment and operating in different clouds. This book fills a gap in the MLOps space very nicely and is highly recommended for both MLOps as well as the data scientist persona.” —Debo Dutta, Distinguished Engineer, Cisco “Kubeflow is quickly emerging as the open-source MLOps platform of choice in enterprise IT, and this book masterfully covers the ins and outs of Kubeflow operations. It should be required reading for all MLOps engineers.” —Mike Oglesby, MLOps Engineer, NetApp

“Kubeflow is a favored development platform to simplify building and deploying AI capabilities into modern applications that utilize Kubernetes to scale and evolve efficiently. The Kubeflow Operations Guide provides valuable insights for planning, implementing, and operating Kubeflow.” —Zeki Yasar, Principal Solutions Architect, ePlus Technology, Inc. “This book provides an exceptional deep dive into the operation of Kubeflow on-premise or via cloud providers. Kubeflow is a vital project in the machine learning engineering ecosystem and this publication provides a missing puzzle piece in the ecosystem: an excellent guide on how to set up and operate your machine learning engineering stack with Kubeflow or how to deploy machine learning models with KFServing effectively. I see this book as the go-to reference for machine learning or DevOps engineers wanting to understand a production Kubeflow setup. I wish the book would have been around when I set up my first clusters running Kubeflow; it would have saved me hours.” —Hannes Hapke, Senior Machine Learning Engineer at SAP Concur “This book helped me to fully get my head around all the different parts of the Kubeflow system and understand what role Kubeflow plays in helping build a more reliable and reproducible data science deployment pipeline. From security to Jupyter implementation and on to deployment, this book was the guide that helped me see how the pieces fit together.” —JD Long, RenaissanceRe “This book is a must-read guide for any DevOps team considering standardizing model deployments. Learn from the best and understand how machine learning works.” —Axel Damian Sirota, Machine Learning Research Engineer

Josh Patterson, Michael Katzenellenbogen, and Austin Harris Kubeflow Operations Guide Managing Cloud and On-Premise Deployment Boston Farnham Sebastopol TokyoBeijing

978-1-492-05327-9 [LSI] Kubeflow Operations Guide by Josh Patterson, Michael Katzenellenbogen, and Austin Harris Copyright © 2021 Josh Patterson, Michael Katzenellenbogen, and Austin Harris. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Jonathan Hassell Development Editor: Michele Cronin Production Editor: Deborah Baker Copyeditor: Piper Editorial, LLC Proofreader: Sonia Saruba Indexer: Potomac Indexing, LLC Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea December 2020: First Edition Revision History for the First Edition 2020-12-04: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781492053279 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Kubeflow Operations Guide, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the authors, and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

For my sons Ethan, Griffin, and Dane: Go forth, be bold, be persistent. —J. Patterson

Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv 1. Introduction to Kubeflow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Machine Learning on Kubernetes 1 The Evolution of Machine Learning in Enterprise 2 It’s Harder Than Ever to Run Enterprise Infrastructure 4 Identifying Next-Generation Infrastructure (NGI) Core Principles 6 Kubernetes for Production Application Deployment 8 Enter: Kubeflow 12 What Problems Does Kubeflow Solve? 14 Origin of Kubeflow 16 Who Uses Kubeflow? 17 Common Kubeflow Use Cases 18 Running Notebooks on GPUs 18 Shared Multitenant Machine Learning Environment 21 Building a Transfer Learning Pipeline 21 Deploying Models to Production for Application Integration 23 Components of Kubeflow 24 Machine Learning Tools 26 Applications and Scaffolding 28 Machine Learning Model Inference Serving with KFServing 35 Platforms and Clouds 37 Summary 39 2. Kubeflow Architecture and Best Practices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Kubeflow Architecture Overview 41 Kubeflow and Kubernetes 43 Ways to Run a Job on Kubeflow 44 ix

Machine Learning Metadata Service 44 Artifact Storage 45 Istio Operations in Kubeflow 45 Kubeflow Multitenancy Architecture 48 Multitenancy and Isolation 48 Multiuser Architecture 49 Multiuser Authorization Flow 49 Kubeflow Profiles 50 Multiuser Isolation 52 Notebook Architecture 53 Notebook Server Launcher UI 53 Notebook Controller 55 Pipelines Architecture 56 Kubeflow Best Practices 57 Managing Job Dependencies 57 Using GPUs 60 Experiment Management 62 Summary 63 3. Planning a Kubeflow Installation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 Security Planning 65 Components That Extend the Kubernetes API 66 Components Running Atop Kubernetes 66 Background and Motivation 67 Kubeflow and Deployed Applications 68 Integration 69 Users 70 Profiling Users 70 Varying Skillsets 72 Workloads 73 Cluster Utilization 73 Data Patterns 75 GPU Planning 75 Planning for GPUs 76 Models that Benefit from GPUs 77 Infrastructure Planning 79 Kubernetes Considerations 79 On-Premise 80 Cloud 81 Placement 82 Container Management 83 Serverless Container Operations with Knative 83 x | Table of Contents

Sizing and Growing 84 Forecasting 84 Storage 85 Scaling 86 Summary 87 4. Installing Kubeflow On-Premise. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 Kubernetes Operations from the Command Line 89 Installing kubectl 89 Using kubectl 93 Using Docker 95 Basic Install Process 97 Installing On-Premise 97 Considerations for Building Kubernetes Clusters 97 Gateway Host Access to Kubernetes Cluster 99 Active Directory Integration and User Management 99 Kerberos Integration 100 Storage Integration 100 Container Management and Artifact Repositories 103 Accessing and Interacting with Kubeflow 104 Common Command-Line Operations 104 Accessible Web UIs 104 Installing Kubeflow 105 System Requirements 105 Set Up and Deploy 105 Summary 107 5. Running Kubeflow on Google Cloud. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 Overview of the Google Cloud Platform 110 Storage 111 Google Cloud Identity-Aware Proxy 112 Google Cloud Security and the Cloud Identity-Aware Proxy 114 GCP Projects for Application Deployments 118 GCP Service Accounts 119 Signing Up for Google Cloud Platform 120 Installing the Google Cloud SDK 120 Update Python 121 Download and Install Google Cloud SDK 121 Installing Kubeflow on Google Cloud Platform 121 Create a Project in the GCP Console 122 Enabling APIs for a Project 123 Set Up OAuth for GCP Cloud IAP 125 Table of Contents | xi

Deploy Kubeflow Using the Command-Line Interface 131 Accessing the Kubeflow UI Post-Installation 141 Summary 142 6. Running Kubeflow on Amazon Web Services. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 Overview of Amazon Web Services 143 Storage 144 Amazon Storage Pricing 145 Amazon Cloud Security 145 AWS Compute Services 145 Managed Kubernetes on EKS 146 Signing Up for Amazon Web Services 146 Installing the AWS CLI 147 Update Python 147 Install the AWS CLI 147 Kubeflow on Amazon Web Services 150 Installing kubectl 151 Install the eksctl CLI for Amazon EKS 151 Install AWS IAM Authenticator 151 Install jq 151 Using Managed Kubernetes on Amazon EKS 152 Create an EKS Service Role 152 Create an AWS VPC 154 Creating EKS Clusters 157 Deploying an EKS Cluster with eksctl 158 Understanding the Deployment Process 158 Kubeflow Configuration and Deployment 159 Customize the Kubeflow Deployment 161 Customize Authentication 161 Resizing EKS Clusters 161 Deleting EKS Clusters 161 Adding Logging 162 Troubleshooting Deployments 164 Summary 164 7. Running Kubeflow on Azure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 Overview of the Azure Cloud Platform 165 Key Azure Components 166 Storage on Azure 167 The Azure Security Model 170 Service Accounts 172 Resources and Resource Groups 172 xii | Table of Contents

Azure Virtual Machines 173 Containers and Managed Azure Kubernetes Services 174 The Azure CLI 175 Installing the Azure CLI 175 Installing Kubeflow on Azure Kubernetes 175 Azure Login and Configuration 176 Create an AKS Cluster for Kubeflow 177 Kubeflow Installation 180 Authorizing Network Access to Deployment 187 Summary 187 8. Model Serving and Integration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189 Basic Concepts of Model Management 189 Understanding Training Models Versus Model Inference 190 Building an Intuition for Model Integration 192 Scaling Model Inference Throughput 195 Model Management 198 Introduction to KFServing 199 Advantages of Using KFServing 201 Core Concepts in KFServing 202 Supported Pre-Built Model Servers 210 KFServing Security Model 214 Managing Models with KFServing 215 Installing KFServing on a Kubernetes Cluster 215 Deploying a Model on KFServing 218 Managing Model Traffic with Canarying 224 Deploying a Custom Transformer 226 Roll Back a Deployed Model 228 Removing a Deployed Model 229 Summary 229 A. Infrastructure Concepts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231 B. An Overview of Kubernetes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 C. Istio Operations and Kubeflow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267 Table of Contents | xiii

Preface What Is in This Book? This book focuses on the DevOps and MLOps sides of deploying and operating Kubeflow. The authors feel that this is compelling and relevant content for today’s practicing DevOps/MLOps teams as this sector is still changing. Many machine learning platforms today take different approaches to the architecture and solution space of managing machine learning workflows. The difficulty of considering all aspects of operating a machine learning platform is where this story kicks off in Chapter 1: “Where are we today and what do we need to be thinking about from ground zero for machine learning platforms?” This book starts by taking you through today’s machine learning infrastructure land‐ scape and explaining the challenges and trade-offs faced by many enterprise teams today. We then go on to outline the core principles needed to support the full life cycle of machine learning operations and explain how Kubernetes solves some of the issues that arise. We’ll further show the remaining functional gaps in how Kubernetes fits into the MLOps picture, and how Kubeflow functions to complete the picture. This book has three major parts. The first section (Chapters 1, 2, and 3) focuses on understanding the core concepts of Kubeflow and its architecture. Chapter 1 covers machine learning architecture concerns, such as “Why is Kuber‐ netes compelling here?” and “What does Kubeflow add beyond Kubernetes?” It includes a discussion of understanding today’s machine learning platforms. Chapter 2 moves oon to the architecture of Kubeflow. Chapter 3 walks you through ways to plan a Kubeflow deployment. The second part (Chapters 4, 5, 6, and 7) covers how to install Kubeflow 1.0.2 on- premise and on the three major cloud vendors: Google Cloud Platform, Amazon Web Services, and Microsoft Azure. These chapters are intended to walk engineers through the steps required to deploy Kubeflow both on-premise, and on each of the xv

major cloud platforms. Some readers may skip some of this material depending on how they are deploying Kubeflow. Chapter 8 closes the book by focusing on deploying models into production for infer‐ ence with KFServing. This chapter is compelling in that it starts by defining model inference and then outlining considerations you need to take into account when wir‐ ing the output of a saved model into a production application. Chapter 8 closes out with a deep dive into KFServing, the model deployment framework included with Kubeflow. Finally, the appendixes give background information on infrastructure core concepts, Istio and the control plane, and also core Kubernetes concepts. This book does not cover specific use case examples in machine learning, as there are many existing books that already cover that topic. Who Is This Book For? DevOps and MLOps teams will benefit from this book the most as the book focuses on both the architecture of Kubeflow and also its operational side. There are many infrastructure debates to be had in every organization and this book should help arm a DevOps team with at least a grounding in the trade-offs to look for in a machine learning platform architecture. Data scientists may find its material good background information from the user per‐ spective but they may get bored with so much discussion around topics such as Kubernetes and identity management. However, with some patience, a data scientist may find value in the book as they can better understand what is happening behind the scenes, allowing them to be better-informed users of Kubeflow. This book assumes you are familiar with basic Kubernetes concepts and can already build machine learning code. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program ele‐ ments such as variable or function names, databases, data types, environment variables, statements, and keywords. xvi | Preface

Constant width bold Shows commands or other text that should be typed literally by the user. Constant width italic Shows text that should be replaced with user-supplied values or by values deter‐ mined by context. This element signifies a tip or suggestion. This element signifies a general note. This element indicates a warning or caution. Using Code Examples Supplemental material (code examples, etc.) is available for download at https:// github.com/jpatanooga/kubeflow_ops_book_dev. If you have a technical question or a problem using the code examples, please email bookquestions@oreilly.com. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Kubeflow Operations Guide by Josh Patterson, Michael Katzenellenbogen, and Austin Harris (O’Reilly). Preface | xvii

Copyright 2021 Josh Patterson, Michael Katzenellenbogen, and Austin Harris, 978-1-492-05327-9.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit http://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/Kubeflow_Operations_Guide. Email bookquestions@oreilly.com to comment or ask technical questions about this book. For news and information about our books and courses, visit http://oreilly.com. Find us on Facebook: http://facebook.com/oreilly Follow us on Twitter: http://twitter.com/oreillymedia Watch us on YouTube: http://www.youtube.com/oreillymedia xviii | Preface