Gitops Cookbook Kubernetes Automation in Practice (Natale Vinto, Alex Bueno) (Z-Library)

GitOps Cookbook Kubernetes Automation in Practice Natale Vinto and Alex Soto Bueno

GitOps Cookbook by Natale Vinto and Alex Soto Bueno Copyright © 2023 Natale Vinto and Alex Soto Bueno. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: John Devins Development Editor: Shira Evans Production Editor: Kate Galloway Copyeditor: Kim Cofer Proofreader: Liz Wheeler Indexer: nSight, Inc. Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea January 2023: First Edition Revision History for the First Edition 2023-01-03: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781492097471 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. GitOps Cookbook, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the authors, and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. This work is part of a collaboration between O’Reilly and Red Hat. See our statement of editorial independence. 978-1-492-09747-1 [LSI]

Dedication To Alessia and Sofia, the most beautiful chapters of my life. —Natale [Ada i Alexandra] Sabeu que sou flipants, encara que sortiu del fang. —Alex

Foreword A few years ago, during a trip to Milan for a Red Hat event, I ran into a passionate colleague at the Red Hat office. We spoke at length about how customers in Italy adopt containers to speed up application development on OpenShift. While his name slipped my mind at the time, his enthusiasm about the subject didn’t, especially since he was also hospitable enough to take me to an espresso bar near the office to show me what real coffee tastes like. A while later, I was introduced to a developer advocate in a meeting who would speak at a conference about CI/CD using products like OpenShift Pipelines and OpenShift GitOps that my teams delivered at the time. At that moment, I instantly recognized Natale. Many who attended that talk thought it was insightful, given his firsthand grasp of challenges that customers experience when delivering applications and his hands-on approach to technology. Application delivery is a complex process involving many systems and teams with numerous handoffs between these parties, often synonymous with delays and back-and-forth talks at each point. Automation has long been a key enabler for improving this process and has become particularly popular within the DevOps movement. Continuous integration, infrastructure as code, and numerous other practices became common in many organizations as they navigated their journey toward adopting DevOps. More recently, and coinciding with the increased adoption of Kubernetes, GitOps as a blueprint for implementing a subset of DevOps practices has become an area I frequently get asked about. While neither the term nor the practices GitOps advocates are new, it does combine. It presents the existing knowledge in a workflow that is simple, easy to understand, and can be implemented in a standard way across many teams. Although the path to adopting the GitOps workflow is simple and concrete, many technical choices need to be made to fit within each organization’s security, compliance, operational, and other requirements. Therefore, I am particularly thrilled about the existence of this book and the practical guides it provides to assist these teams in making choices that are right for their applications, teams, and organizations. Siamak Sadeghianfar Product Management, Red Hat

Preface We wrote this book for builders. Whether you are a developer, DevOps engineer, site reliability engineer (SRE), or platform engineer dealing with Kubernetes, you are building some good stuff. We would like to share our experience from what we have learned in the field and in the community about the latest Kubernetes automation insights for pipelines and CI/CD workloads. The book contains a comprehensive list of the most popular available software and tools in the Kubernetes and cloud native ecosystem for this purpose. We aim to provide a list of practical recipes that might help your daily job or are worth exploring further. We are not sticking to a particular technology or project for implementing Kubernetes automation. However, we are opinionated on some of our choices to deliver a concise GitOps pathway. The book is organized in sequential chapters, from the basics to advanced topics in the Kubernetes ecosystem, following the GitOps principles. We hope you’ll find these recipes valuable and inspiring for your projects! Chapter 1 is an introduction to GitOps principles and why they are continuously becoming more common and essential for any new IT project. Chapter 2 covers the installation requirements to run these recipes in a Kubernetes cluster. Concepts and tools like Git, Container Registry, Container Runtime, and Kubernetes are necessary for this journey. Chapter 3 walks you through a complete overview of containers and why they are essential for application development and deployment today. Kubernetes is a container-orchestration platform; however, it doesn’t build containers out of the box. Therefore, we’ll provide a list of practical recipes for making container apps with the most popular tools available in the cloud native community. Chapter 4 gives you an overview of Kustomize, a popular tool for managing Kubernetes resources. Kustomize is interoperable, and you find it often used within CI/CD pipelines. Chapter 5 explores Helm, a trendy tool to package applications in Kubernetes. Helm is also a templating system that you can use to deploy apps in CI/CD workloads. Chapter 6 walks you through cloud native CI/CD systems for Kubernetes. It gives a comprehensive list of recipes for the continuous integration part with Tekton, the Kubernetes- native CI/CD system. Additionally, it also covers other tools such as Drone and GitHub Actions. Chapter 7 kicks off the pure GitOps part of the book as it sticks to the Continuous Deployment phase with Argo CD, a popular GitOps tool for Kubernetes. Chapter 8 goes into the advanced topics for GitOps with Argo CD, such as secrets management, progressive application delivery, and multicluster deployments. This concludes the most common use cases and architectures you will likely work with today and tomorrow

following the GitOps approach. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords. Constant width bold Shows commands or other text that should be typed literally by the user. Constant width italic Shows text that should be replaced with user-supplied values or by values determined by context. TIP This element signifies a tip or suggestion. NOTE This element signifies a general note. WARNING This element indicates a warning or caution. Using Code Examples Supplemental material (code examples, exercises, etc.) is available for download at https://github.com/gitops-cookbook. If you have a technical question or a problem using the code examples, please send email to bookquestions@oreilly.com. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for

permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “GitOps Cookbook by Natale Vinto and Alex Soto Bueno (O’Reilly). Copyright 2023 Natale Vinto and Alex Soto Bueno, 978-1-492-09747-1.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. O’Reilly Online Learning NOTE For more than 40 years, O’Reilly Media has provided technology and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit http://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/gitops-cookbook. Email bookquestions@oreilly.com to comment or ask technical questions about this book. For news and information about our books and courses, visit https://oreilly.com.

Find us on LinkedIn: https://linkedin.com/company/oreilly-media. Follow us on Twitter: https://twitter.com/oreillymedia. Watch us on YouTube: https://youtube.com/oreillymedia. Acknowledgments We both want to thank our tech reviewers Peter Miron and Andy Block for their accurate review that helped us improve the reading experience with this book. Thanks also to the people at O’Reilly who helped us during the whole writing cycle. Many thanks to our colleagues Aubrey Muhlac and Colleen Lobner for the great support with publishing this book. Thanks to Kamesh Sampath and all the people who helped us during the early release phases with comments and suggestions that we added to the book—your input is much appreciated! Alex Soto During these challenging times, I’d like to acknowledge Santa (aquest any sí), Uri (don’t stop the music), Guiri (un ciclista), Gavina, Gabi (thanks for the support), and Edgar and Ester (life is good especially on Friday); my friends Edson, Sebi (the best fellow traveler), Burr (I learned a lot from you), Kamesh, and all the Red Hat developers team, we are the best. Jonathan Vila, Abel Salgado, and Jordi Sola for the fantastic conversations about Java and Kubernetes. Last but certainly not least, I’d like to acknowledge Anna for being here; my parents Mili and Ramon for buying my first computer; my daughters Ada and Alexandra, “sou les ninetes dels meus ulls.” Natale Vinto Special thanks to Alessia for the patience and motivation that helped me while writing this book. And to my parents for everything they made for me, grazie mamma e papà, you are the best!

Chapter 1. Introduction With the advent of practices such as infrastructure as code (IaC), software development has pushed the boundaries of platforms where you can run applications. This becomes more frequent with programmable, API-driven platforms such as public clouds and open source infrastructure solutions. While some years ago developers were only focusing on application source code, today they also have the opportunity to code the infrastructure where their application will run. This gives control and enables automation, which significantly reduces lead time. A good example is with Kubernetes, a popular open source container workload orchestration platform and the de facto standard for running production applications, either on public or private clouds. The openness and extensibility of the platform enables automation, which reduces risks of delivery and increases service quality. Furthermore, this powerful paradigm is extended by another increasingly popular approach called GitOps. 1.1 What Is GitOps? GitOps is a methodology and practice that uses Git repositories as a single source of truth to deliver infrastructure as code. It takes the pillars and approaches from DevOps culture and provides a framework to start realizing the results. The relationship between DevOps and GitOps is close, as GitOps has become the popular choice to implement and enhance DevOps, platform engineering, and SRE. GitOps is an agnostic approach, and a GitOps framework can be built with tools such as Git, Kubernetes, and CI/CD solutions. The three main pillars of GitOps are: Git is the single source of truth Treat everything as code Operations are performed through Git workflows There is an active community around GitOps, and the GitOps Working Group defines a set of GitOps Principles (currently in version 1.0.0) available at OpenGitOps: Declarative A system managed by GitOps must have its desired state expressed declaratively. Versioned and immutable The desired state is stored in a way that enforces immutability and versioning and retains a complete version history. Pulled automatically Software agents automatically pull the desired state declarations from the source.

Continuously reconciled Software agents continuously observe the actual system state and attempt to apply the desired state. 1.2 Why GitOps? Using the common Git-based workflows that developers are familiar with, GitOps expands upon existing processes from application development to deployment, app lifecycle management, and infrastructure configuration. Every change throughout the application lifecycle is traced in the Git repository and is auditable. This approach is beneficial for both developers and operations teams as it enhances the ability to trace and reproduce issues quickly, improving overall security. One key point is to reduce the risk of unwanted changes (drift) and correct them before they go into production. Here is a summary of the benefits of the GitOps adoption in four key aspects: Standard workflow Use familiar tools and Git workflows from application development teams Enhanced security Review changes beforehand, detect configuration drifts, and take action Visibility and audit Capture and trace any change to clusters through Git history Multicluster consistency Reliably and consistently configure multiple environments and multiple Kubernetes clusters and deployment 1.3 Kubernetes CI/CD Continuous integration (CI) and continuous delivery (CD) are methods used to frequently deliver apps by introducing automation into the stages of app development. CI/CD pipelines are one of the most common use cases for GitOps. In a typical CI/CD pipeline, submitted code checks the CI process while the CD process checks and applies requirements for things like security, infrastructure as code, or any other boundaries set for the application framework. All code changes are tracked, making updates easy while also providing version control should a rollback be needed. CD is the GitOps domain and it works together with the CI part to deploy apps in multiple environments, as you can see in Figure 1-1.

Figure 1-1. Continuous integration and continuous delivery With Kubernetes, it’s easy to implement an in-cluster CI/CD pipeline. You can have CI software create the container image representing your application and store it in a container image registry. Afterward, a Git workflow such as a pull request can change the Kubernetes manifests illustrating the deployment of your apps and start a CD sync loop, as shown in Figure 1-2. Figure 1-2. Application deployment model This cookbook will show practical recipes for implementing this model on Kubernetes acting as a CI/CD and GitOps platform. 1.4 App Deployment with GitOps on Kubernetes

As GitOps is an agnostic, platform-independent approach, the application deployment model on Kubernetes can be either in-cluster or multicluster. An external GitOps tool can use Kubernetes just as a target platform for deploying apps. At the same time, in-cluster approaches run a GitOps engine inside Kubernetes to deploy apps and sync manifests in one or more Kubernetes clusters. The GitOps engine takes care of the CD part of the CI/CD pipeline and accomplishes a GitOps loop, which is composed of four main actions as shown in Figure 1-3: Deploy Deploy the manifests from Git. Monitor Monitor either the Git repo or the cluster state. Detect drift Detect any change from what is described in Git and what is present in the cluster. Take action Perform an action that reflects what is on Git (rollback or three-way diff). Git is the source of truth, and any change is performed via a Git workflow. Figure 1-3. GitOps loop

In Kubernetes, application deployment using the GitOps approach makes use of at least two Git repositories: one for the app source code, and one for the Kubernetes manifests describing the app’s deployment (Deployment, Service, etc.). Figure 1-4 illustrates the structure of a GitOps project on Kubernetes. Figure 1-4. Kubernetes GitOps loop The following list outlines the items in the workflow: 1. App source code repository 2. CI pipeline creating a container image 3. Container image registry 4. Kubernetes manifests repository 5. GitOps engine syncing manifests to one or more clusters and detecting drifts 1.5 DevOps and Agility GitOps is a developer-centric approach to continuous delivery and infrastructure operations, and a developer workflow through Git for automating processes. As DevOps is complementary to Agile software development, GitOps is complementary to DevOps for infrastructure automation and application lifecycle management. As you can see in Figure 1-5, it’s a developer workflow for automating operations.

One of the most critical aspects of the Agile methodology is to reduce the lead time, which is described more abstractly as the time elapsed between identifying a requirement and its fulfillment. Figure 1-5. GitOps development cycle Reducing this time is fundamental and requires a cultural change in IT organizations. Seeing applications live provides developers with a feedback loop to redesign and improve their code and make their projects thrive. Similarly to DevOps, GitOps also requires a cultural adoption in business processes. Every operation, such as application deployment or infrastructure change, is only possible through Git workflows. And sometimes, this means a cultural shift. The “Teaching Elephants to Dance (and Fly!)” speech from Burr Sutter gives a clear idea of the context. The elephant is where your organization is today. There are phases of change between traditional and modern environments powered by GitOps tools. Some organizations have the

luxury of starting from scratch, but for many businesses, the challenge is teaching their lumbering elephant to dance like a graceful ballerina.

Chapter 2. Requirements This book is about GitOps and Kubernetes, and as such, you’ll need a container registry to publish the containers built throughout the book (see Recipe 2.1). Also, a Git service is required to implement GitOps methodologies; you’ll learn how to register to public Git services like GitHub or GitLab (see Recipe 2.2). Finally, it would be best to have a Kubernetes cluster to run the book examples. Although we’ll show you how to install Minikube as a Kubernetes cluster (see Recipe 2.3), and the book is tested with Minikube, any Kubernetes installation should work as well. Let’s prepare your laptop to execute the recipes provided in this book. 2.1 Registering for a Container Registry Problem You want to create an account for a container registry service so you can store generated containers. Solution You may need to publish some containers into a public container registry as you work through this book. Use Docker Hub (docker.io) to publish containers. If you already have an account with docker.io, you can skip the following steps. Otherwise, keep reading to learn how to sign up for an account. Discussion Visit DockerHub to sign up for an account. The page should be similar to Figure 2-1.

Figure 2-1. DockerHub registration page When the page is loaded, fill in the form by setting a Docker ID, Email, and Password, and click the Sign Up button. When you are registered and your account confirmed, you’ll be ready to publish containers under the previous step’s Docker ID. See Also Another popular container registry service is quay.io. It can be used on the cloud (like docker.io) or installed on-premises. Visit the website to get more information about Quay. The page should be similar to Figure 2-2.