Kubernetes Best Practices Blueprints for Building Successful Applications on Kubernetes - Second Edition (Brendan Burns, Eddie Villalba, Dave Strebel etc.) (Z-Library)

Kub ernetes B est Pra ctices Kub ernetes B est Pra ctices Brendan Burns, Eddie Villalba, Dave Strebel & Lachlan Evenson Kubernetes Best Practices Blueprints for Building Successful Applications on Kubernetes Second Edition2E

“Written by luminaries in the cloud native space, Kubernetes Best Practices is a masterclass in managing container orchestration at scale.” —Joseph Sandoval Principal Product Manager, Adobe Inc. CLOUD COMPUTING Kubernetes Best Practices In this practical guide, four Kubernetes professionals with deep experience in distributed systems, open source, and enterprise application development will guide you through the process of building applications with this container orchestration system. Together, they distill decades of experience from companies that successfully run Kubernetes in production and provide concrete code examples to back the methods presented in this book. Revised to cover all the latest Kubernetes features, new tooling, and deprecations, this book is ideal for those who are familiar with basic Kubernetes concepts but want to get up to speed on the latest best practices. You’ll learn exactly what you need to know to build your best app with Kubernetes the first time. • Set up and develop applications in Kubernetes • Learn patterns for monitoring, securing your systems, and managing upgrades, rollouts, and rollbacks • Integrate services and legacy applications and develop higher-level platforms on top of Kubernetes • Run machine learning workloads in Kubernetes • Ensure pod and container security • Build higher-level application patterns on top of Kubernetes and implement an operator • Understand issues increasingly critical to the successful implementation of Kubernetes: chaos engineering/testing, GitOps, service mesh, observability, and managing multiple clusters Twitter: @oreillymedia linkedin.com/company/oreilly-media youtube.com/oreillymedia US $65.99 CAN $82.99 ISBN: 978-1-098-14216-2 Brendan Burns is a distinguished engineer at Microsoft Azure and cofounder of the Kubernetes open source project. Eddie Villalba is the engineering manager and application platform practice lead for North America at Google Cloud, focused on the cloud native ecosystem and Kubernetes. Dave Strebel is a principal cloud native architect focused on open source cloud and Kubernetes. Lachlan Evenson is a principal program manager on the Microsoft Azure cloud native ecosystem. 2E

Praise for Kubernetes Best Practices, Second Edition Written by luminaries in the cloud native space, Kubernetes Best Practices is a masterclass in managing container orchestration at scale. From setup to security, this book is a comprehensive resource that not only teaches but empowers. Cut your learning curve in half and build better applications faster with the proven strategies in this essential read. —Joseph Sandoval, Principal Product Manager, Adobe Inc. Just because we can do something, doesn’t mean we should. Cloud native is a large topic so there are ample opportunities to go awry. These expert authors focus their deep knowledge on the key recipes to help keep your Kubernetes deliveries on the rails. —Jonathan Johnson, Cloud Native Architect, Presenter, Trainer Your roadmap to building successful applications with Kubernetes; laden with expert insights and real-world best practices. —Dr. Roland Huß, Senior Principal Software Developer, Red Hat A trove of wisdom about container management from true mavens. Quote this book in meetings! It’s not stealing ideas—they want you to read this book. —Jess Males, Devops Engineer, TriumphPay The Kubernetes ecosystem has expanded significantly over time. The specific, actionable recommendations detailed in this excellent guide make the current complexity approachable for the growing community. —Bridget Kromhout, Principal Product Manager, Microsoft

Having written a book on Kubernetes and reviewed numerous others, I can attest to the uniqueness and depth of Kubernetes Best Practices. This book is a masterclass for those familiar with Kubernetes, designed specifically for teams operating and managing Kubernetes. It offers a systematic approach to understanding best practices, covering essential areas crucial for large-scale application deployment, from developer workflows to global application distribution, policy, governance, and the seamless integration of external services. Every page is infused with technical insights, presenting a comprehensive perspective that I haven’t encountered in other Kubernetes literature. It not only serves as a blueprint for designing clusters but also provides a flexible guide, pinpointing the what and why, while allowing readers to adapt the intricate how to their specific organizational contexts. —Bilgin Ibryam, Coauthor of Kubernetes Patterns, Principal Product Manager, Diagrid

Brendan Burns, Eddie Villalba, Dave Strebel, and Lachlan Evenson Kubernetes Best Practices Blueprints for Building Successful Applications on Kubernetes SECOND EDITION Boston Farnham Sebastopol TokyoBeijing

978-1-098-14216-2 [LSI] Kubernetes Best Practices by Brendan Burns, Eddie Villalba, Dave Strebel, and Lachlan Evenson Copyright © 2024 Brendan Burns, Eddie Villalba, Dave Strebel, and Lachlan Evenson. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: John Devins Development Editor: Jill Leonard Production Editor: Beth Kelly Copyeditor: Piper Editorial Consulting, LLC Proofreader: Piper Editorial Consulting, LLC Indexer: nSight, Inc. Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea November 2019: First Edition October 2023: Second Edition Revision History for the First Edition 2023-10-04: First Release See https://www.oreilly.com/catalog/errata.csp?isbn=0636920805021 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Kubernetes Best Practices, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the authors, and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. This work is part of a collaboration between O’Reilly and Microsoft. See our statement of editorial independence.

Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv 1. Setting Up a Basic Service. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Application Overview 1 Managing Configuration Files 2 Creating a Replicated Service Using Deployments 3 Best Practices for Image Management 4 Creating a Replicated Application 4 Setting Up an External Ingress for HTTP Traffic 6 Configuring an Application with ConfigMaps 8 Managing Authentication with Secrets 9 Deploying a Simple Stateful Database 12 Creating a TCP Load Balancer by Using Services 16 Using Ingress to Route Traffic to a Static File Server 17 Parameterizing Your Application by Using Helm 19 Deploying Services Best Practices 21 Summary 21 2. Developer Workflows. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Goals 23 Building a Development Cluster 24 Setting Up a Shared Cluster for Multiple Developers 25 Onboarding Users 26 Creating and Securing a Namespace 29 Managing Namespaces 30 Cluster-Level Services 31 Enabling Developer Workflows 31 Initial Setup 32 v

Enabling Active Development 33 Enabling Testing and Debugging 34 Setting Up a Development Environment Best Practices 34 Summary 35 3. Monitoring and Logging in Kubernetes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 Metrics Versus Logs 37 Monitoring Techniques 37 Monitoring Patterns 38 Kubernetes Metrics Overview 39 cAdvisor 39 Metrics Server 40 kube-state-metrics 40 What Metrics Do I Monitor? 41 Monitoring Tools 42 Monitoring Kubernetes Using Prometheus 44 Logging Overview 48 Tools for Logging 49 Logging by Using a Loki-Stack 50 Alerting 53 Best Practices for Monitoring, Logging, and Alerting 54 Monitoring 54 Logging 55 Alerting 55 Summary 55 4. Configuration, Secrets, and RBAC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 Configuration Through ConfigMaps and Secrets 57 ConfigMaps 58 Secrets 58 Common Best Practices for the ConfigMap and Secrets APIs 59 Best Practices Specific to Secrets 64 RBAC 65 RBAC Primer 66 RBAC Best Practices 68 Summary 69 5. Continuous Integration, Testing, and Deployment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 Version Control 72 Continuous Integration 72 Testing 73 Container Builds 73 vi | Table of Contents

Container Image Tagging 74 Continuous Deployment 75 Deployment Strategies 75 Testing in Production 80 Setting Up a Pipeline and Performing a Chaos Experiment 81 Setting Up CI 81 Setting Up CD 84 Performing a Rolling Upgrade 84 A Simple Chaos Experiment 85 Best Practices for CI/CD 85 Summary 86 6. Versioning, Releases, and Rollouts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 Versioning 88 Releases 88 Rollouts 89 Putting It All Together 90 Best Practices for Versioning, Releases, and Rollouts 93 Summary 94 7. Worldwide Application Distribution and Staging. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 Distributing Your Image 96 Parameterizing Your Deployment 97 Load-Balancing Traffic Around the World 98 Reliably Rolling Out Software Around the World 98 Pre-Rollout Validation 99 Canary Region 102 Identifying Region Types 103 Constructing a Global Rollout 103 When Something Goes Wrong 104 Worldwide Rollout Best Practices 105 Summary 106 8. Resource Management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 Kubernetes Scheduler 107 Predicates 107 Priorities 108 Advanced Scheduling Techniques 109 Pod Affinity and Anti-Affinity 109 nodeSelector 110 Taints and Tolerations 110 Pod Resource Management 112 Table of Contents | vii

Resource Request 112 Resource Limits and Pod Quality of Service 113 PodDisruptionBudgets 115 Managing Resources by Using Namespaces 116 ResourceQuota 117 LimitRange 119 Cluster Scaling 120 Application Scaling 121 Scaling with HPA 122 HPA with Custom Metrics 123 Vertical Pod Autoscaler 123 Resource Management Best Practices 124 Summary 124 9. Networking, Network Security, and Service Mesh. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 Kubernetes Network Principles 125 Network Plug-ins 128 Kubenet 129 Kubenet Best Practices 129 The CNI Plug-in 129 CNI Best Practices 130 Services in Kubernetes 130 Service Type ClusterIP 131 Service Type NodePort 132 Service Type ExternalName 134 Service Type LoadBalancer 134 Ingress and Ingress Controllers 136 Gateway API 137 Services and Ingress Controllers Best Practices 139 Network Security Policy 140 Network Policy Best Practices 142 Service Meshes 143 Service Mesh Best Practices 145 Summary 145 10. Pod and Container Security. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 Pod Security Admission Controller 147 Enabling Pod Security Admission 148 Pod Security levels 148 Activating Pod Security Using Namespace Labels 149 Workload Isolation and RuntimeClass 150 Using RuntimeClass 151 viii | Table of Contents

Runtime Implementations 151 Workload Isolation and RuntimeClass Best Practices 152 Other Pod and Container Security Considerations 153 Admission Controllers 153 Intrusion and Anomaly Detection Tooling 153 Summary 153 11. Policy and Governance for Your Cluster. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 Why Policy and Governance Are Important 155 How Is This Policy Different? 155 Cloud Native Policy Engine 156 Introducing Gatekeeper 156 Example Policies 157 Gatekeeper Terminology 157 Defining Constraint Templates 158 Defining Constraints 159 Data Replication 160 UX 161 Using Enforcement Action and Audit 161 Mutation 163 Testing Policies 163 Becoming Familiar with Gatekeeper 163 Policy and Governance Best Practices 164 Summary 165 12. Managing Multiple Clusters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 Why Multiple Clusters? 167 Multicluster Design Concerns 169 Managing Multiple Cluster Deployments 171 Deployment and Management Patterns 171 The GitOps Approach to Managing Clusters 173 Multicluster Management Tools 175 Kubernetes Federation 176 Managing Multiple Clusters Best Practices 176 Summary 177 13. Integrating External Services with Kubernetes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 Importing Services into Kubernetes 179 Selector-Less Services for Stable IP Addresses 180 CNAME-Based Services for Stable DNS Names 181 Active Controller-Based Approaches 182 Exporting Services from Kubernetes 183 Table of Contents | ix

Exporting Services by Using Internal Load Balancers 184 Exporting Services on NodePorts 184 Integrating External Machines and Kubernetes 185 Sharing Services Between Kubernetes 186 Third-Party Tools 187 Connecting Cluster and External Services Best Practices 188 Summary 188 14. Running Machine Learning in Kubernetes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189 Why Is Kubernetes Great for Machine Learning? 189 Machine Learning Workflow 190 Machine Learning for Kubernetes Cluster Admins 191 Model Training on Kubernetes 192 Distributed Training on Kubernetes 195 Resource Constraints 195 Specialized Hardware 196 Libraries, Drivers, and Kernel Modules 197 Storage 197 Networking 198 Specialized Protocols 198 Data Scientist Concerns 199 Machine Learning on Kubernetes Best Practices 199 Summary 201 15. Building Higher-Level Application Patterns on Top of Kubernetes. . . . . . . . . . . . . . . . 203 Approaches to Developing Higher-Level Abstractions 203 Extending Kubernetes 204 Extending Kubernetes Clusters 205 Extending the Kubernetes User Experience 206 Making Containerized Development Easier 207 Developing a “Push-to-Deploy” Experience 207 Design Considerations When Building Platforms 208 Support Exporting to a Container Image 208 Support Existing Mechanisms for Service and Service Discovery 209 Building Application Platforms Best Practices 209 Summary 210 16. Managing State and Stateful Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 Volumes and Volume Mounts 212 Volume Best Practices 213 Kubernetes Storage 213 PersistentVolume 213 x | Table of Contents

PersistentVolumeClaims 214 StorageClasses 215 Kubernetes Storage Best Practices 216 Stateful Applications 217 StatefulSets 218 Operators 220 StatefulSet and Operator Best Practices 221 Summary 222 17. Admission Control and Authorization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 Admission Control 224 What Are They? 224 Why Are They Important? 224 Admission Controller Types 225 Configuring Admission Webhooks 226 Admission Control Best Practices 228 Authorization 231 Authorization Modules 231 Authorization Best Practices 234 Summary 234 18. GitOps and Deployment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235 What Is GitOps? 236 Why GitOps? 237 GitOps Repo Structure 238 Managing Secrets 240 Setting Up Flux 241 GitOps Tooling 243 GitOps Best Practices 244 Summary 244 19. Security. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 Cluster Security 246 etcd Access 246 Authentication 246 Authorization 246 TLS 247 Kubelet and Cloud Metadata Access 247 Secrets 247 Logging and Auditing 247 Cluster Security Posture Tooling 248 Cluster Security Best Practices 248 Table of Contents | xi

Workload Container Security 248 Pod Security Admission 249 Seccomp, AppArmor, and SELinux 249 Admission Controllers 249 Operators 249 Network Policy 250 Runtime Security 250 Workload Container Security Best Practices 250 Code Security 251 Non-Root and Distroless Containers 251 Container Vulnerability Scanning 252 Code Repository Security 252 Code Security Best Practices 252 Summary 253 20. Chaos Testing, Load Testing, and Experiments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255 Chaos Testing 255 Goals for Chaos Testing 256 Prerequisites for Chaos Testing 256 Chaos Testing Your Application’s Communication 257 Chaos Testing Your Application’s Operation 258 Fuzz Testing Your Application for Security and Resiliency 259 Summary 259 Load Testing 259 Goals for Load Testing 259 Prerequisites for Load Testing 260 Generating Realistic Traffic 261 Load Testing Your Application 262 Tuning Your Application Using Load Tests 262 Summary 263 Experiments 263 Goals for Experiments 263 Prerequisites for an Experiment 264 Setting Up an Experiment 264 Summary 265 Chaos Testing, Load Testing, and Experiments Summary 266 21. Implementing an Operator. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267 Operator Key Components 268 Custom Resource Definitions 268 Creating Our API 270 Controller Reconciliation 277 xii | Table of Contents

Resource Validation 278 Controller Implementation 279 Operator Life Cycle 284 Version Upgrades 284 Operator Best Practices 285 Summary 286 22. Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291 Table of Contents | xiii

Preface Who Should Read This Book Kubernetes is the de facto standard for cloud native development. It is a powerful tool that can make your next application easier to develop, faster to deploy, and more reliable to operate. However, unlocking the power of Kubernetes requires using it correctly. This book is intended for anyone who is deploying real-world applications to Kubernetes and is interested in learning patterns and practices they can apply to the applications that they build on top of Kubernetes. Importantly, this book is not an introduction to Kubernetes. We assume that you have a basic familiarity with the Kubernetes API and tools, and that you know how to create and interact with a Kubernetes cluster. If you are looking to learn Kubernetes, there are numerous great resources out there, such as Kubernetes: Up and Running (O’Reilly), that can give you an introduction. Instead, this book is a resource for anyone who wants to dive deep on how to deploy specific applications and workloads on Kubernetes. It should be useful to you whether you are about to deploy your first application onto Kubernetes or you’ve been using Kubernetes for years. Why We Wrote This Book Between the four of us, we have significant experience helping a wide variety of users deploy their applications onto Kubernetes. Through this experience, we have seen where people struggle, and we have helped them find their way to success. When sitting down to write this book, we attempted to capture these experiences so that many more people could learn by reading the lessons that we learned from these real-world experiences. It’s our hope that by committing our experiences to writing, we can scale our knowledge and allow you to be successful deploying and managing your application on Kubernetes on your own. xv

Navigating This Book Although you might read this book from cover to cover in a single sitting, that is not really how we intended you to use it. Instead, we designed this book to be a collection of standalone chapters. Each chapter gives a complete overview of a particular task that you might need to accomplish with Kubernetes. We expect people to dive into the book to learn about a specific topic or interest, and then leave the book alone, only to return when a new topic comes up. Despite this standalone approach, some themes span the book. There are several chapters on developing applications on Kubernetes. Chapter 2 covers developer workflows. Chapter 5 discusses continuous integration and testing. Chapter 15 covers building higher-level platforms on top of Kubernetes, and Chapter 16 discusses managing state and stateful applications. In addition to developing applications, there are several chapters on operating services in Kubernetes. Chapter 1 covers the setup of a basic service, and Chapter 3 covers monitoring and metrics. Chapter 4 covers configuration management, while Chapter 6 covers versioning and releases. Chapter 7 covers deploying your application around the world. There are also several chapters on cluster management, including Chapter 8 on resource management, Chapter 9 on networking, Chapter 10 on pod security, Chap‐ ter 11 on policy and governance, Chapter 12 on managing multiple clusters, and Chapter 17 on admission control and authorization. Finally, some chapters are truly independent; these cover machine learning (Chapter 14) and integrating with exter‐ nal services (Chapter 13). Though it can be useful to read all the chapters before you actually attempt the topic in the real world, our primary hope is that you will treat this book as a reference. It is intended as a guide as you put these topics to practice in the real world. New to This Edition We wanted to complement the 1st edition with four new chapters that cover emerg‐ ing tools and patterns as Kubernetes continues to mature and provide best practices. These new chapters are Chapter 18 on GitOps, Chapter 19 on security, Chapter 20 on chaos testing, and Chapter 21 on implementing an operator. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. xvi | Preface

Constant width Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords. Constant width bold Shows commands or other text that should be typed literally by the user. Constant width italic Shows text that should be replaced with user-supplied values or by values deter‐ mined by context. This element signifies a tip or suggestion. This element signifies a general note. This element indicates a warning or caution. Using Code Examples Supplemental material (code examples, exercises, etc.) is available for download at https://oreil.ly/KBPsample. If you have a technical question or a problem using the code examples, please send email to bookquestions@oreilly.com. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant Preface | xvii

amount of example code from this book into your product’s documentation does require permission. We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Kubernetes Best Practi‐ ces by Brendan Burns, Eddie Villalba, Dave Strebel, and Lachlan Evenson (O’Reilly). Copyright 2024 Brendan Burns, Eddie Villalba, Dave Strebel, and Lachlan Evenson, 978-1-098-14216-2.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, conferences, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in- depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, please visit http://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-889-8969 (in the United States or Canada) 707-829-7019 (international or local) 707-829-0104 (fax) support@oreilly.com https://www.oreilly.com/about/contact.html We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/kubernetes-best-practices2. For news and information about our books and courses, visit https://oreilly.com. Find us on LinkedIn: https://linkedin.com/company/oreilly-media xviii | Preface