GitOps and Kubernetes Continuous Deployment with Argo CD, Jenkins X, and Flux (Billy Yuen, Alexander Matyushentsev etc.) (Z-Library)

📄 File Format: PDF

💾 File Size: 12.6 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

M A N N I N G Billy Yuen Alexander Matyushentsev Todd Ekenstam Jesse Suen Continuous Deployment with Argo CD, Jenkins X, and Flux

📄 Page 2

302 Chapter 3 Chapter 4 Chapter 5Chapter 6 Chapter 7 Chapter 8 Chapter 10 Chapter 11 Chapter 9 Environment management Access control and security Secrets Observability Pipelines Deployment strategies Argo CD Jenkins X Flux Chapter 2 Kubernetes and GitOps Chapter 1 Why GitOps? Part 1: Background Part 2: Patterns and processes Part 3: Tools

📄 Page 3

GitOps and Kubernetes Continuous Deployment with Argo CD, Jenkins X, and Flux

📄 Page 4

ii

📄 Page 5

GitOps and Kubernetes CONTINUOUS DEPLOYMENT WITH ARGO CD, JENKINS X, AND FLUX BILLY YUEN ALEXANDER MATYUSHENTSEV TODD EKENSTAM AND JESSE SUEN M A N N I N G SHELTER ISLAND

📄 Page 6

For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact Special Sales Department Manning Publications Co. 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 Email: orders@manning.com ©2021 by Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps. Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine. Manning Publications Co. Development editor: Dustin Archibald 20 Baldwin Road Technical development editor: Al Krinker PO Box 761 Review editor: Aleks Dragosavljević Shelter Island, NY 11964 Production editor: Deirdre S. Hiam Proofreader: Katie Tennant Technical proofreader: Sam Brown Typesetter and cover designer: Marija Tudor ISBN 9781617297977 Printed in the United States of America

📄 Page 7

contents preface xi acknowledgments xii about this book xiii about the authors xviii about the cover illustration xx PART 1 BACKGROUND ................................................. 1 1 Why GitOps? 3 1.1 Evolution to GitOps 4 Traditional Ops 4 ■ DevOps 6 ■ GitOps 7 1.2 Developer benefits of GitOps 9 Infrastructure as code 9 ■ Self-service 10 ■ Code reviews 11 Git pull requests 12 1.3 Operational benefits of GitOps 13 Declarative 13 ■ Observability 15 ■ Auditability and compliance 16 ■ Disaster recovery 19 2 Kubernetes and GitOps 20 2.1 Kubernetes introduction 20 What is Kubernetes? 21 ■ Other container orchestrators 22 Kubernetes architecture 22 ■ Deploying to Kubernetes 25 2.2 Declarative vs. imperative object management 30 How declarative configuration works 34v

📄 Page 8

CONTENTSvi2.3 Controller architecture 37 Controller delegation 37 ■ Controller pattern 38 NGINX operator 40 2.4 Kubernetes + GitOps 44 2.5 Getting started with CI/CD 44 Basic GitOps operator 45 ■ Continuous integration pipeline 47 PART 2 PATTERNS AND PROCESSES ............................. 53 3 Environment management 55 3.1 Introduction to environment management 56 Components of an environment 57 ■ Namespace management 59 ■ Network isolation 64 ■ Preprod and prod clusters 67 3.2 Git strategies 68 Single branch (multiple directories) 69 ■ Multiple branches 70 Multirepo vs. monorepo 70 3.3 Configuration management 71 Helm 72 ■ Kustomize 76 ■ Jsonnet 79 ■ Configuration management summary 83 3.4 Durable vs. ephemeral environments 83 4 Pipelines 86 4.1 Stages in CI/CD pipelines 86 GitOps continuous integration 88 ■ GitOps continuous delivery 94 4.2 Driving promotions 98 Code vs. manifest vs. app config 98 ■ Code and image promotion 99 ■ Environment promotion 101 ■ Putting it all together 102 4.3 Other pipelines 102 Rollback 103 ■ Compliance pipeline 106 5 Deployment strategies 109 5.1 Deployment basics 110 Why ReplicaSet is not a good fit for GitOps 111 ■ How Deployment works with ReplicaSets 114 ■ Traffic routing 120 Configuring minikube for other strategies 122

📄 Page 9

CONTENTS vii5.2 Blue-green 123 Blue-green with Deployment 125 ■ Blue-green with Argo Rollouts 130 5.3 Canary 133 Canary with Deployment 134 ■ Canary with Argo Rollouts 138 5.4 Progressive delivery 140 Progressive delivery with Argo Rollouts 140 6 Access control and security 148 6.1 Introduction to access control 149 What is access control? 149 ■ What to secure 150 ■ Access control in GitOps 153 6.2 Access limitations 155 Git repository access 155 ■ Kubernetes RBAC 163 ■ Image registry access 168 6.3 Patterns 171 Full access 171 ■ Deployment repo access 172 ■ Code access only 172 6.4 Security concerns 173 Preventing image pull from untrusted registries 173 Cluster-level resources in a Git repository 174 7 Secrets 176 7.1 Kubernetes Secrets 177 Why use Secrets? 177 ■ How to use Secrets 178 7.2 GitOps and Secrets 181 No encryption 181 ■ Distributed Git repos 181 ■ No granular (file-level) access control 181 ■ Insecure storage 181 Full commit history 182 7.3 Secrets management strategies 182 Storing Secrets in Git 182 ■ Baking Secrets into the container image 182 ■ Out-of-band management 183 ■ External Secrets management systems 184 ■ Encrypting Secrets in Git 185 Comparison of strategies 186 7.4 Tooling 187 HashiCorp Vault 187 ■ Vault Agent Sidecar Injector 190 Sealed Secrets 194 ■ Kustomize Secret generator plugin 198

📄 Page 10

CONTENTSviii8 Observability 203 8.1 What is observability? 204 Event logging 205 ■ Metrics 209 ■ Tracing 212 Visualization 217 ■ Importance of observability in GitOps 219 8.2 Application health 219 Resource status 220 ■ Readiness and liveness 224 Application monitoring and alerting 225 8.3 GitOps observability 227 GitOps metrics 227 ■ Application sync status 228 Configuration drift 232 ■ GitOps change log 234 PART 3 TOOLS ........................................................ 239 9 Argo CD 241 9.1 What is Argo CD? 241 Main use cases 242 ■ Core concepts 243 ■ Sync and health statuses 244 ■ Architecture 246 9.2 Deploy your first application 249 Deploying the first application 249 ■ Inspect the application using the user interface 251 9.3 Deep dive into Argo CD features 253 GitOps-driven deployment 253 ■ Resource hooks 254 Postdeployment verification 257 9.4 Enterprise features 257 Single sign-on 258 ■ Access control 261 ■ Declarative management 263 10 Jenkins X 267 10.1 What is Jenkins X? 267 10.2 Exploring Prow, Jenkins X pipeline operator, and Tekton 269 10.3 Importing projects into Jenkins X 273 Importing a project 274 ■ Promoting a release to the production environment 281 11 Flux 284 11.1 What is Flux? 284 What Flux does 285 ■ Docker registry scanning 286 Architecture 288

📄 Page 11

CONTENTS ix11.2 Simple application deployment 289 Deploying the first application 289 ■ Observing application state 290 ■ Upgrading the deployment image 291 ■ Using Kustomize for manifest generation 292 ■ Securing deployment using GPG 294 11.3 Multitenancy with Flux 296 appendix A Setting up a test Kubernetes cluster 299 appendix B Setting up GitOps tools 302 appendix C Configuring GPG key 313 index 315

📄 Page 12

CONTENTSx

📄 Page 13

preface As Intuit embarked on the journey from on-premises to cloud-native, the journey itself presented an opportunity to reinvent our build and deployment process. Similar to many large enterprises, our old deployment process was data-center-centric with sepa- rate QA, Ops, and Infrastructure teams. Code could take weeks to get deployed, and developers had no access to infrastructure when there were production issues. Infra- structure issues could take a long time to resolve and required many groups’ collabo- ration. As Marianna Tessel (Intuit CTO) and Jeff Brewer (Intuit SBSEG chief architect) decided to bet big on Kubernetes and Docker, we were fortunate to be the first team to fully migrate one of our production applications with Kubernetes and Docker. Along the way, we got to reinvent our CI/CD pipeline and adopt the GitOps process. Jesse and Alex created Argo CD (CNCF incubator project) to address enterprise needs for GitOps. Todd and his team created world-class cluster management tools so we can scale out to hundreds of clusters with ease. Having a standard like Kubernetes and Docker enables all engineers to speak a common language in terms of infrastructure and deployment. Engineers can easily contribute to other projects and deploy as soon as the development process is com- plete. GitOps also allows us to know exactly who and what gets changed in our environ- ments, which is especially important if you are subject to compliance requirements. We cannot imagine going back to the old way we did deployment, and we hope that this book can help accelerate your journey to embrace GitOps! xi

📄 Page 14

acknowledgments This book turned out to be an 18-month journey that required a lot of work and addi- tional research to tell the complete story. We believe that we have delivered what we set out to do, and it is a great book for anyone who wants to adopt GitOps and Kuber- netes. There are quite a few people we’d like to thank for helping us along the way. At Man- ning, we would like to thank our development editor, Dustin Archibald, project editor, Deirdre Hiam, proofreader, Katie Tennant, and reviewing editor, Aleks Dragosavljevic. We want to thank Marianna Tessel and Jeff Brewer, who provided us the opportu- nity and freedom to transform and experiment with GitOps and Kubernetes. We would also like to thank Pratik Wadher, Saradhi Sreegiriaju, Mukulika Kupas, and Edward Lee for their guidance throughout the process. We want to call out Viktor Far- cic and Oscar Medina for their insightful contributions to the Jenkins X chapter. To all the reviewers: Andres Damian Sacco, Angelo Simone Scotto, Björn Neuhaus, Chris Viner, Clifford Thurber, Conor Redmond, Diego Casella, James Liu, Jaume López, Jeremy Bryan, Jerome Meyer, John Guthrie, Marco Massenzio, Matthieu Evrin, Mike Ensor, Mike Jensen, Roman Zhuzha, Samuel Brown, Satej Kumar Sahu, Sean T. Booker, Wendell Beckwith, and Zorodzayi Mukuya, we say thank you. Your suggestions helped make this a better book. For Jeff Brewer, who inspired us all for this awesome transformation journey!xii

📄 Page 15

about this book Who this book is for This book is intended for both Kubernetes infrastructure and operation engineers and software developers who want to deploy applications to Kubernetes through a declarative model using the GitOps process. It will benefit anyone looking to improve the stability, reliability, security, and auditability of their Kubernetes clusters while at the same time reducing operational costs through automated continuous software deployments. Readers are expected to have a working knowledge of Kubernetes (Deployment, Pod, Service, and Ingress resources, for example) as well as an understanding of mod- ern software development practices including continuous integration/continuous deployment (CI/CD), revision control systems (such as Git), and deployment/infra- structure automation. Who this book is not for Advanced users who have successfully implemented a mature GitOps system may be better off reading a more advanced book on their chosen tool. This book is not intended to cover all aspects of Kubernetes in depth. While we cover many Kubernetes concepts that are relevant to GitOps, readers looking for a comprehensive guide to Kubernetes should look at the other great books and online resources available on the topic. How this book is organized: A roadmap This book describes the benefits of GitOps on Kubernetes, including flexible configu- ration management, monitoring, robustness, multienvironment support, and security.xiii

📄 Page 16

ABOUT THIS BOOKxivYou will learn the best practices, techniques, and tools to achieve these benefits, which enable enterprises to use Kubernetes to accelerate application development without compromising on stability, reliability, or security. You will also gain in-depth understanding of the following topics:  Multiple-environment management with branching, namespace, and configu- ration  Access control with Git, Kubernetes, and pipelines  Pipeline considerations with CI/CD, promotion, push/pull, and release/roll- back  Observability and drift detection  Managing Secrets  Deployment strategy selection among rolling update, blue/green, canary, and progressive delivery This book takes a hands-on approach with tutorials and exercises to develop the skills you need to embrace GitOps using Kubernetes. After reading this book, you will know how to implement a declarative continuous delivery system for your applications run- ning on Kubernetes. This book contains hands-on tutorials on  Getting started with managing Kubernetes application deployments  Configuration and environment management using Kustomize  Writing your own basic Kubernetes continuous delivery (CD) operator  Implementing CI/CD using Argo CD,1 Jenkins X,2 and Flux3 IMPERATIVE VS. DECLARATIVE There are two basic ways to deploy Kubernetes: imperatively using many kubectl commands or declaratively by writing mani- fests and using kubectl apply. The former is useful for learning and interac- tive experimentation. The latter is best for reproducible deployments and tracking changes. This book is intended for you to follow along, running the hands-on portion of the tutorials, using your own test Kubernetes cluster. Appendix A describes several options for creating a test cluster. There are many code listings contained in the book. All code listings and addi- tional supporting material can be found in the publicly accessible GitHub repository for this book: https://github.com/gitopsbook/resources We encourage you to clone or fork this repository and use it as you work through the tutorials and exercises in the book. 1 https://argoproj.github.io/argo-cd. 2 https://jenkins-x.io. 3 https://github.com/fluxcd/flux.

📄 Page 17

ABOUT THIS BOOK xv The following tools and utilities should be installed on your workstation:  Kubectl (v1.16 or later)  Minikube (v1.4 or later)  Bash or the Windows Subsystem for Linux (WSL) Most tutorials and exercises can be completed using a minikube running on your workstation. If not, we will mention if the cluster running on a cloud provider is needed, and you can refer to appendix A for details on creating the cluster. NOTE You may incur additional costs for running a test Kubernetes cluster on a cloud provider. While we have attempted to reduce the cost of the rec- ommended test configuration as much as possible, remember you are respon- sible for these costs. We recommend you delete your test cluster after completing each tutorial or exercise. This book has 3 parts that cover 11 chapters. Part 1 covers the background and intro- duces GitOps and Kubernetes:  Chapter 1 walks you through the journey of software deployment evolution and how GitOps became the latest practice. It also covers the many key concepts and benefits of GitOps.  Chapter 2 provides key concepts on Kubernetes and why its declarative nature is perfect for GitOps. It also covers the core operator concept and how to imple- ment a simple GitOps operator. Part 2 goes over the patterns and processes to adopt the GitOps process:  Chapter 3 discusses the definition of an environment and how Kubernetes Namespaces nicely map as environments. It also covers branching strategy and config management to your environment implementation.  Chapter 4 goes deep into the GitOps CI/CD pipeline with comprehensive descriptions of all stages necessary for a complete pipeline. It also covers code, image, and environment promotion as well as the rollback mechanism.  Chapter 5 describes various deployment strategies, including rolling update, blue/green, canary, and progressive delivery. It also covers how to implement each strategy by using native Kubernetes resources and other open source tools.  Chapter 6 discusses GitOps-driven deployment’s attack surfaces and how to mit- igate each area. It also reviews Jsonnet, Kustomize, and Helm and how to choose the right configuration management pattern for your use cases.  Chapter 7 discusses various strategies for managing Secrets for GitOps. It also covers several Secret management tools as well as native Kubernetes Secrets.  Chapter 8 explains the core concepts of observability and why it is important to GitOps. It also describes various methods to implement observability with GitOps and Kubernetes.

📄 Page 18

ABOUT THIS BOOKxviPart 3 goes over several enterprise-grade GitOps tools:  Chapter 9 discusses the intent and architecture for Argo CD. It also covers con- figuring application deployment using Argo CD and how to secure Argo CD in production.  Chapter 10 discusses the intent and architecture for Jenkins X. It also covers configuring application deployment and promotion to various environments.  Chapter 11 discusses the intent and motivation for Flux. It also covers configur- ing application deployment using Flux and multitenancy. The book was organized to read all the chapters in sequential order. However, if there is a particular area of interest you’d like to jump into, we recommend you read the prerequisite chapters. For example, if you would like to jump right into learning to use Argo CD, we recommend you read chapters 1, 2, 3, and 5 before reading chapter 9. About the code This book contains many examples of source code both in numbered listings and inline with normal text. In both cases, source code is formatted in a fixed-width font to separate it from ordinary text. Sometimes, code is also in bold to highlight Chapter 3 Chapter 4 Chapter 5Chapter 6 Chapter 7 Chapter 8 Chapter 10 Chapter 11 Chapter 9 Environment management Access control and security Secrets Observability Pipelines Deployment strategies Argo CD Jenkins X Flux Chapter 2 Kubernetes and GitOps Chapter 1 Why GitOps? Part 1: Background Part 2: Patterns and processes Part 3: Tools Appendix A Set up a test Kubernetes cluster Appendix B Set up GitOps tools Appendix C Configure GPG key Appendixes: optional information

📄 Page 19

ABOUT THIS BOOK xviicode that has changed from previous steps in the chapter, such as when a new feature adds to an existing line of code. In many cases, the original source code has been reformatted; we’ve added line breaks and reworked indentation to accommodate the available page space in the book. Additionally, comments in the source code have often been removed from the listings when the code is described in the text. Code annotations accompany many of the listings, highlighting important concepts. Source code for the examples in this book is available for download from https://github.com/gitopsbook /resources. liveBook discussion forum Purchase of GitOps and Kubernetes includes free access to a private web forum run by Manning Publications where you can make comments about the book, ask technical questions, and receive help from the authors and from other users. To access the forum, go to https://livebook.manning.com/book/GitOps-and-Kubernetes/discussion. You can also learn more about Manning’s forums and the rules of conduct at https:// livebook.manning.com/#!/discussion. Manning’s commitment to our readers is to provide a venue where a meaningful dialogue between individual readers and between readers and authors can take place. It is not a commitment to any specific amount of participation on the part of the authors, whose contribution to the forum remains voluntary (and unpaid). We sug- gest you try asking them some challenging questions lest their interest stray! The forum and the archives of previous discussions will be accessible from the publisher’s website as long as the book is in print.

📄 Page 20

about the authors BILLY YUEN is a principal engineer with Intuit’s Platform team, focusing on AWS and Kubernetes adoption, system resiliency, and monitoring. Previously, Billy worked on Netflix’s Edge Services team to build the next generation of edge-service infrastruc- ture to support millions of customers (more than 3 billion requests per day) with high scalability, resilience to failure, and rapid innovation. Billy was a speaker at Java One 2016 and Velocity NY 2016 on “Operational Excellence with Netflix Hystrix,” “CI/CD at Lightspeed” at KubeCon 2018, and “Automated Canary Release” at Container World 2019. ALEXANDER MATYUSHENTSEV is a principal engineer on the Intuit Platform team, focusing on building tools that make it easier to use Kubernetes. Alexander is passion- ate about open source, cloud-native infrastructure, and tools that increase developers’ productivity. He is one of the core contributors to the Argo Workflows and Argo CD projects. Alexander was a speaker at KubeCon 2019 on “How Intuit Does Canary and Blue-Green Deployments with a K8s Controller.” TODD EKENSTAM is a principal engineer at Intuit, building a platform for secure, multitenant Kubernetes infrastructure supporting applications serving Intuit’s approximately 50 million customers. Todd has worked on a variety of large-scale dis- tributed systems projects during his career of more than 25 years, including hierarchi- cal storage management, peer-to-peer database replication, enterprise storage virtualization, and two-factor authentication SaaS. Todd has presented at academic, government, and industry conferences, most recently as a guest speaker on “Introduc- tion to Open Policy Agent” at KubeCon 2018. JESSE SUEN is a principal engineer on the Intuit Platform team, developing microservices-based, distributed applications for Kubernetes. He was an earlyxviii