Kubernetes Operators (Jason Dobies, Joshua Wood)（Z-Library）

Jason Dobies & Joshua Wood Kubernetes Operators Automating the Container Orchestration Platform Sponsored by

Praise for Kubernetes Operators “Kubernetes has emerged as the world’s most powerful container orchestration platform, but its true power is hidden behind an extensible API and automation framework that will redefine how future platforms are built and operated; this book is the missing manual.” —Kelsey Hightower, Technologist, Google Cloud “The Kubernetes Operators book by Jason and Josh is something that should not be missing on your (digital) bookshelf, if you’re serious about Kubernetes. It’s hands-on, covers the why and the how, and enables you to successfully apply the operator pattern in your environment. I’d consider this book the perfect followup to Programming Kubernetes.” —Michael Hausenblas, Amazon Web Services “This book is essential for anyone looking to adopt the Operator Paradigm for their critical workloads. It provides a comprehensive overview of design principles, implementation paths and traps, and utilization of the existing ecosystem.” —Anish Asthana, Software Engineer, Red Hat “Working with Jason over the past several years, I have always wanted a dump of what’s in his head, and now I have it! Josh and Jason have created an essential guide for anyone creating operators, and it will be a significant advantage for us as we look to mature our operator into the Auto Pilot phase with the goal of becoming a true ‘Kubernetes Application Reliability Engineering’ function for our customers.” —Dave Meurer, Technical Global Alliances, Synopsys, Inc. “Another brilliant publication by Josh and Jason that provides market-leading data for Kubernetes Operators.” —Joe Gomes, Global Alliances, Synopsys, Inc.

Jason Dobies and Joshua Wood Kubernetes Operators Automating the Container Orchestration Platform Boston Farnham Sebastopol TokyoBeijing

978-1-492-04805-3 [LSI] Kubernetes Operators by Jason Dobies and Joshua Wood Copyright © 2020 Red Hat, Inc. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: John Devins Development Editor: Virginia Wilson Production Editor: Deborah Baker Copyeditor: Rachel Head Proofreader: Sonia Saruba Indexer: Ellen Troutman-Zaig Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Rebecca Demarest March 2020: First Edition Revision History for the First Edition 2020-01-13: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781492048046 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Kubernetes Operators, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the authors, and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. This work is part of a collaboration between O’Reilly and Red Hat, Inc. See our statement of editorial independence (https://oreil.ly/editorial-independence).

To my kids, Leanne and Austin, know that it is never easy to have to tell you “No, daddy has to work.” Realize that all of it—the meetings, the trips, the book—all of it is for you two. I have your backs in whatever the future holds for you, and I can’t wait to see the awesome things you two do. —Jason To Shayna. —Joshua

Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii 1. Operators Teach Kubernetes New Tricks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 How Kubernetes Works 1 Example: Stateless Web Server 3 Stateful Is Hard 4 Operators Are Software SREs 5 How Operators Work 5 Kubernetes CRs 6 How Operators Are Made 6 Example: The etcd Operator 7 The Case of the Missing Member 7 Who Are Operators For? 8 Operator Adoption 8 Let’s Get Going! 9 2. Running Operators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Setting Up an Operator Lab 11 Cluster Version Requirements 11 Authorization Requirements 12 Standard Tools and Techniques 13 Suggested Cluster Configurations 13 Checking Your Cluster Version 14 Running a Simple Operator 15 A Common Starting Point 15 Fetching the etcd Operator Manifests 16 CRs: Custom API Endpoints 16 Who Am I: Defining an Operator Service Account 17 vii

Deploying the etcd Operator 19 Declaring an etcd Cluster 20 Exercising etcd 21 Scaling the etcd Cluster 22 Failure and Automated Recovery 23 Upgrading etcd Clusters 24 Cleaning Up 26 Summary 27 3. Operators at the Kubernetes Interface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 Standard Scaling: The ReplicaSet Resource 29 Custom Resources 30 CR or ConfigMap? 30 Custom Controllers 31 Operator Scopes 31 Namespace Scope 31 Cluster-Scoped Operators 32 Authorization 32 Service Accounts 32 Roles 33 RoleBindings 33 ClusterRoles and ClusterRoleBindings 33 Summary 34 4. The Operator Framework. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Operator Framework Origins 35 Operator Maturity Model 36 Operator SDK 36 Installing the Operator SDK Tool 37 Operator Lifecycle Manager 37 Operator Metering 38 Summary 39 5. Sample Application: Visitors Site. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Application Overview 41 Installation with Manifests 43 Deploying MySQL 43 Backend 45 Frontend 47 Deploying the Manifests 49 Accessing the Visitors Site 49 Cleaning Up 49 viii | Table of Contents

Summary 50 6. Adapter Operators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 Helm Operator 53 Building the Operator 53 Fleshing Out the CRD 57 Reviewing Operator Permissions 57 Running the Helm Operator 57 Ansible Operator 58 Building the Operator 58 Fleshing Out the CRD 60 Reviewing Operator Permissions 60 Running the Ansible Operator 60 Testing an Operator 61 Summary 62 Resources 62 7. Operators in Go with the Operator SDK. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 Initializing the Operator 64 Operator Scope 64 Custom Resource Definitions 66 Defining the Go Types 67 The CRD Manifest 68 Operator Permissions 68 Controller 69 The Reconcile Function 71 Operator Writing Tips 72 Retrieving the Resource 72 Child Resource Creation 73 Child Resource Deletion 76 Child Resource Naming 77 Idempotency 77 Operator Impact 78 Running an Operator Locally 79 Visitors Site Example 80 Summary 81 Resources 81 8. Operator Lifecycle Manager. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 OLM Custom Resources 83 ClusterServiceVersion 84 CatalogSource 84 Table of Contents | ix

Subscription 85 InstallPlan 85 OperatorGroup 85 Installing OLM 86 Using OLM 88 Exploring the Operator 92 Deleting the Operator 93 OLM Bundle Metadata Files 94 Custom Resource Definitions 94 Cluster Service Version File 95 Package Manifest File 95 Writing a Cluster Service Version File 95 Generating a File Skeleton 95 Metadata 97 Owned CRDs 98 Required CRDs 101 Install Modes 102 Versioning and Updating 102 Writing a Package Manifest File 103 Running Locally 104 Prerequisites 104 Building the OLM Bundle 107 Installing the Operator Through OLM 109 Testing the Running Operator 111 Visitors Site Operator Example 111 Summary 111 Resources 111 9. Operator Philosophy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 SRE for Every Application 113 Toil Not, Neither Spin 114 Automatable: Work Your Computer Would Like 114 Running in Place: Work of No Enduring Value 114 Growing Pains: Work That Expands with the System 115 Operators: Kubernetes Application Reliability Engineering 115 Managing Application State 116 Golden Signals Sent to Software 116 Seven Habits of Highly Successful Operators 118 Summary 119 10. Getting Involved. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 Feature Requests and Reporting Bugs 121 x | Table of Contents

Contributing 122 Sharing Operators 123 Summary 123 A. Running an Operator as a Deployment Inside a Cluster. . . . . . . . . . . . . . . . . . . . . . . . . . 125 B. Custom Resource Validation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 C. Role-Based Access Control (RBAC). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 Table of Contents | xi

Preface Kubernetes is a popular container orchestrator. It harnesses many computers together into one large computing resource and establishes a means of addressing that resource through the Kubernetes application programming interface (API). Kuber‐ netes is open source software with origins at Google, developed over the last five years by a large group of collaborators under the auspices of the Cloud Native Com‐ puting Foundation (CNCF) (https://www.cncf.io/). An Operator extends Kubernetes to automate the management of the entire lifecycle of a particular application. Operators serve as a packaging mechanism for distribut‐ ing applications on Kubernetes, and they monitor, maintain, recover, and upgrade the software they deploy. Who This Book Is For If you’ve deployed applications on a Kubernetes cluster, you’ll be familiar with some of the challenges and aspirations that forged the Operator pattern. If you’ve main‐ tained foundation services like databases and filesystems in their own ghetto outside your orchestrated clusters, and you yearn to bring them into the neighborhood, this guide to Kubernetes Operators is for you. What You Will Learn This book explains what an Operator is and how Operators extend the Kubernetes API. It shows how to deploy and use existing Operators, and how to create and dis‐ tribute Operators for your applications using the Red Hat Operator Framework (https://github.com/operator-framework). We relate good practices for designing, building, and distributing Operators, and we explain the thinking that animates Operators with Site Reliability Engineering (SRE) principles. After describing Operators and their concepts in the first chapter, we’ll suggest ways to get access to a Kubernetes cluster where you can do the exercises in the rest of the xiii

book. With a cluster running, you’ll deploy an Operator and observe its behavior when its application fails, scales, or gets upgraded to a new version. Later, we will explore the Operator SDK and show you how to use it to build an Operator to naturalize an example application as a first-class Kubernetes citizen. With that practical foundation in place, we will discuss the SRE ideas from which Opera‐ tors derive and the goals they share: reducing operations effort and cost, increasing service reliability, and spurring innovation by freeing teams from repetitive mainte‐ nance work. Operator Framework and SDK The Operator pattern emerged at CoreOS (https://coreos.com) as a way to automate increasingly complex applications on Kubernetes clusters, including managing Kubernetes itself and the etcd (https://github.com/coreos/etcd) key-value store at its heart. Work on Operators continued through an acquisition by Red Hat, leading to the 2018 release of the open source Operator Framework and SDK. The examples in this book use the Red Hat Operator SDK and the distribution mechanisms that join it in the Operator Framework. Other Operator Tools A community has grown up around Operators, with more than a hundred Operators for an array of applications from many vendors and projects available in Red Hat’s distribution channels alone. Several other Operator construction tools exist. We won’t discuss them in detail, but after you read this book you’ll be able to compare any of them with the Operator SDK and Framework. Other open source tools available for building Operators include Kopf (https://oreil.ly/JCL-S) for Python, Kubebuilder (https://oreil.ly/8zdbj) from the Kubernetes project, and the Java Operator SDK (https://oreil.ly/yXhVg). Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program ele‐ ments such as variable or function names, databases, data types, environment variables, statements, and keywords. Constant width bold Shows commands or other text that should be typed literally by the user. xiv | Preface

Constant width italic Shows text that should be replaced with user-supplied values or by values deter‐ mined by context. This element signifies a tip or suggestion. This element signifies a general note. This element indicates a warning or caution. Using Code Examples Supplemental material (code examples, exercises, etc.) is available for download at https://github.com/kubernetes-operators-book/. If you have a technical question or a problem using the code examples, please send email to bookquestions@oreilly.com. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Kubernetes Operators by Jason Dobies and Joshua Wood (O’Reilly). Copyright 2020 Red Hat, Inc., 978-1-492-04805-3.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. Preface | xv

O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, conferences, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in- depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, please visit http://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/Kubernetes_Operators. Email bookquestions@oreilly.com to comment or ask technical questions. For more about our books, courses, and conferences, see http://www.oreilly.com. Find us on Facebook: http://facebook.com/oreilly Follow us on Twitter: http://twitter.com/oreillymedia Watch us on YouTube: http://www.youtube.com/oreillymedia Acknowledgments We’d like to thank Red Hat and the OpenShift Advocacy team there for their support, in particular the steadfast and all-trades assistance of Ryan Jarvinen. We also thank the many people who reviewed, checked, suggested, and otherwise gave their time to make this work more coherent and complete, among them Anish Asthana, Evan Cor‐ dell, Michael Gasch, Michael Hausenblas, Shawn Hurley, and Jess Males. xvi | Preface

CHAPTER 1 Operators Teach Kubernetes New Tricks An Operator is a way to package, run, and maintain a Kubernetes application. A Kubernetes application is not only deployed on Kubernetes, it is designed to use and to operate in concert with Kubernetes facilities and tools. An Operator builds on Kubernetes abstractions to automate the entire lifecycle of the software it manages. Because they extend Kubernetes, Operators provide application- specific automation in terms familiar to a large and growing community. For applica‐ tion programmers, Operators make it easier to deploy and run the foundation services on which their apps depend. For infrastructure engineers and vendors, Oper‐ ators provide a consistent way to distribute software on Kubernetes clusters and reduce support burdens by identifying and correcting application problems before the pager beeps. Before we begin to describe how Operators do these jobs, let’s define a few Kuber‐ netes terms to provide context and a shared language to describe Operator concepts and components. How Kubernetes Works Kubernetes automates the lifecycle of a stateless application, such as a static web server. Without state, any instances of an application are interchangeable. This simple web server retrieves files and sends them on to a visitor’s browser. Because the server is not tracking state or storing input or data of any kind, when one server instance fails, Kubernetes can replace it with another. Kubernetes refers to these instances, each a copy of an application running on the cluster, as replicas. A Kubernetes cluster is a collection of computers, called nodes. All cluster work runs on one, some, or all of a cluster’s nodes. The basic unit of work, and of replication, is 1

the pod. A pod is a group of one or more Linux containers with common resources like networking, storage, and access to shared memory. The Kubernetes pod documentation (https://oreil.ly/ziz5q) is a good starting point for more information about the pod abstraction. At a high level, a Kubernetes cluster can be divided into two planes. The control plane is, in simple terms, Kubernetes itself. A collection of pods comprises the control plane and implements the Kubernetes application programming interface (API) and cluster orchestration logic. The application plane, or data plane, is everything else. It is the group of nodes where application pods run. One or more nodes are usually dedicated to running applica‐ tions, while one or more nodes are often sequestered to run only control plane pods. As with application pods, multiple replicas of control plane components can run on multiple controller nodes to provide redundancy. The controllers of the control plane implement control loops that repeatedly compare the desired state of the cluster to its actual state. When the two diverge, a controller takes action to make them match. Operators extend this behavior. The schematic in Figure 1-1 shows the major control plane components, with worker nodes running application workloads. 2 | Chapter 1: Operators Teach Kubernetes New Tricks