Consul Up Running Service Mesh for Any Runtime or Cloud (Luke Kysow) (Z-Library)

Kysow C onsul: U p & R unning C onsul: U p & R unning Luke Kysow Foreword by Mitchell Hashimoto & Armon Dadgar Consul Up & Running Service Mesh for Any Runtime or Cloud

SERVICE MESH From the Foreword “Luke Kysow has been part of the Consul engineering team for many years, personally implementing many of its incredible features. He has a particular talent in making complex topics approachable by anyone, and he does so beautifully in this book.” —Mitchell Hashimoto & Armon Dadgar cofounders of HashiCorp Consul: Up and Running US $59.99 CAN $74.99 ISBN: 978-1-098-10614-0 Twitter: @oreillymedia linkedin.com/company/oreilly-media youtube.com/oreillymedia With the advent of microservices, Kubernetes, public cloud, and hybrid computing, site reliability and DevOps engineers are facing more complexity than ever before. Service mesh is an exciting new technology that promises to help tackle this complexity. A service mesh provides you with a unified control plane to manage application networking across these distinct platforms. With this definitive guide, you’ll learn how to automate networking for simple and secure application delivery with Consul. Author Luke Kysow, Consul engineer at HashiCorp, demonstrates how this service mesh solution provides a software-driven approach to security, observability, reliability, and traffic management. Once you learn how to deploy Consul on multiple platforms, you’ll be able to take control of application traffic, prevent outages, view metrics, integrate with legacy systems, and more. • Dive into the characteristics of service meshes, zero trust networking, and traffic-shaping patterns • Deploy Consul on Kubernetes and virtual machines • Learn how to secure, monitor, and manage your application traffic with Consul • Use this guide to deploy and operate applications as a platform operator, DevOps engineer, or developer Luke Kysow is a principal engineer at HashiCorp, where he works on Consul. He has extensive experience developing and operating applications in cloud and hybrid environments and has helped many companies, large and small, adopt Consul. He’s also the cocreator of Atlantis, a popular open source Terraform CI/CD tool. Kysow

Luke Kysow Foreword by Mitchell Hashimoto and Armon Dadgar Consul: Up and Running Service Mesh for Any Runtime or Cloud Boston Farnham Sebastopol TokyoBeijing

978-1-098-10614-0 [LSI] Consul: Up and Running by Luke Kysow Copyright © 2022 Luke Kysow. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: John Devins Development Editor: Corbin Collins Production Editor: Gregory Hyman Copyeditor: Liz Wheeler Proofreader: nSight, Inc. Indexer: nSight, Inc. Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea June 2022: First Edition Revision History for the First Edition 2022-06-01: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781098106140 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Consul: Up and Running, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the author, and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

To Isha, Kate, Mom, and Dad For your love and support

Table of Contents Foreword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi 1. Service Mesh 101. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 How a Service Mesh Works 2 Sidecar Proxies 2 Control Plane 3 Concrete Example 4 Why Use a Service Mesh 6 Security 6 Observability 7 Reliability 8 Traffic Control 9 Features in Combination 10 When to Use a Service Mesh 11 Summary 12 2. Introduction to Consul. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Architecture 14 Consul Servers 15 Consul Clients 17 Sidecar Proxies 19 Example Use Case 19 Consul Versus Other Meshes 22 Consul’s Other Features 23 Summary 23 v

3. Deploying Consul. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Deploying Consul on Kubernetes 25 Provisioning a Kubernetes Cluster 25 Installing Consul with the consul-k8s CLI 29 Deploying Consul on VMs 34 Provisioning a Local VM 34 Installing and Configuring Consul 36 systemd 38 Interacting with Consul 40 Consul’s UI 40 Consul’s CLI 44 Consul’s API 45 Summary 46 4. Adding Services to the Mesh. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 Birdwatcher Example Service 47 Deploying Services on Kubernetes 49 Adding Kubernetes Services to the Mesh 53 Deploying Services on VMs 61 Registering VM Services with Consul 64 Deploying Sidecar Proxies on VMs 67 Configuring Routing on VMs 70 Summary 72 5. Ingress Gateways. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 Why You Need an Ingress Gateway 73 Deploying an Ingress Gateway on Kubernetes 76 Deploying an Ingress Gateway on VMs 78 Config Entries 82 Config Entries on Kubernetes 83 Config Entries on VMs 84 Configuring Ingress Gateways 86 Configuring Ingress Gateways on Kubernetes 87 Configuring Ingress Gateways on VMs 91 Testing Out Your Ingress Gateway 93 Ingress Gateways in Production 95 Summary 95 6. Security. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 Zero Trust Networking 98 Encryption 101 TLS Encryption 102 vi | Table of Contents

Consul Encryption 104 Authentication 105 Authorization and Intentions 109 Configuring Intentions with Consul’s UI 111 Configuring Intentions with Config Entries 114 Application Aware Intentions 122 Summary 128 7. Observability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 Metrics 130 Deploying and Configuring Prometheus 133 Emitting Metrics 135 Viewing Consul UI Metrics 138 Grafana 142 Distributed Tracing 156 How Tracing Works 156 Instrumenting Your Services 157 Tracing Collectors 158 Viewing Service Traces 164 Enabling Tracing for the Service Mesh 166 Analyzing Service Mesh Traces 171 Summary 172 8. Reliability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 Health Checking 176 Active Versus Passive Health Checking 176 Configuring Active Health Checks 178 Passive Health Checks 188 Retries 191 Timeouts 194 Summary 198 9. Traffic Control. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199 Deployment Strategies 199 Rolling Deployments 200 Blue/Green Deployments 201 Canary Deployments 202 Load Balancers Versus the Service Mesh 203 Traffic Control Config Entries 204 Service Resolvers 204 Service Splitters 206 Service Routers 207 Table of Contents | vii

Canary Deployments with Consul 209 Deploying backend v2 on Kubernetes 213 Deploying backend v2 on VMs 214 Canary Deployment Continued 217 Other Traffic Control Use Cases 223 Summary 225 10. Advanced Use Cases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227 Multi-cluster Federation 227 Consul API Gateway 228 Terminating Gateways 229 HashiCorp Vault Integration 229 Connect Native 230 Network Infrastructure Automation 230 Securing Consul 230 ACLs 230 Gossip Encryption 231 Control Plane TLS 231 Consul Enterprise 232 HashiCorp Cloud Platform 233 Amazon Elastic Container Service (ECS) 233 Nomad 233 Conclusion 234 Appendix. Common Errors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 viii | Table of Contents

Foreword We started HashiCorp in 2012 to solve the challenges introduced by the rise of public cloud. The products we introduced were built in roughly the same order in which they’d be experienced by a new team building in the cloud. Vagrant was first, since creating a development environment was the first challenge we faced. Packer was second, to translate those development environments into cloud images. With the applications successfully deployed, the next challenge was networking between the multiple deployed images, and thus Consul was born. The word microservice wasn’t used then, and the problem space was admittedly much smaller: we needed a way to find the address of a healthy instance of another application or service. The reality of cloud introduced several new technical chal‐ lenges: global availability, automation friendliness, and the expectation that applica‐ tion instances came up and went down constantly. The initial release of Consul in 2014 solved all these challenges. The beauty of new paradigms is that first-order challenges—once solved—give rise to second-order capabilities. The first-order challenge was service discovery in the world of public cloud. The second-order capability was then microservices, improved monitoring, more dynamic routing, and enhanced security by leveraging this new software-driven networking layer. Consul was the natural place to enable these new capabilities, and over the years Consul has evolved to solve these difficult, modern networking challenges. From latency-aware routing at the DNS layer to automatic TLS between services to HTTP- aware load balancing and more, Consul has grown into a fully featured service mesh. These capabilities enable teams to take full advantage of what public cloud has to offer while simultaneously getting more out of on-premises environments. Teams can deliver more applications across more regions safely, and teams that use multiple cloud platforms or on-premises datacenters can communicate across those environ‐ ments in a consistent manner. And this is all possible without any modifications to the deployed applications. ix

Today, Consul is downloaded millions of times per year and is deployed into every‐ thing from small hobbyist home labs to the infrastructure of the world’s largest companies. It has been proven in challenging production environments time and time again. Luke Kysow has been part of the Consul engineering team for many years, personally implementing many of its incredible features. He has a particular talent in making complex topics approachable by anyone, and he does so beautifully in this book. — Mitchell Hashimoto and Armon Dadgar Cofounders of HashiCorp and creators of Consul x | Foreword

Preface The sheer volume of software required by today’s world has necessitated an evolu‐ tion in how we structure our engineering organizations. We’ve learned that smaller, independent teams work better than larger, highly coupled ones. Since Conway’s law—that companies will produce systems to match their organizational structure— is inevitable, this evolution has precipitated the rise of microservices: smaller, inde‐ pendent services owned by smaller, independent teams. As a result of these forces, companies are now running hundreds and even thousands of services in production. The rise of microservices has enabled development teams to scale up and ship code faster, but it has also caused an exponential increase in complexity for operations teams. What was once an in-memory function call is now a cross-continent API request that can fail in unexpected and spectacular ways. What was once a single monitoring dashboard is now a byzantine maze of metrics, logs, and traces. A security model that was once a simple firewall now must protect against a myriad of ever-evolving attack vectors and threats. Finally, what was once a single monolithic service is now hundreds of services built using different technologies and deployed on multiple runtimes: virtual machines (VMs), Kubernetes, serverless platforms, and more. Operations teams, also known as DevOps and site reliability engineering (SRE), thus face a monumental challenge. In the midst of this complexity, they must harden secu‐ rity, increase reliability, simplify observability, and speed application delivery—and they must do so in a way that works across multiple runtimes and languages. Service mesh is an exciting new technology that promises a solution to these problems. Consul is a fully featured service mesh from HashiCorp, the company that also created Terraform, Vault, Nomad, Packer, and Vagrant. A small operations team can leverage Consul to impact security, reliability, observability, and application delivery across their entire stack—all without requiring developers to modify their underlying microservices. xi

In this book, you’ll learn to install, configure, and operate Consul in order to tame complexity and take back control of your infrastructure. I’m excited for you to start on your service mesh journey with Consul—let’s dig in and get up and running! Who Should Read This Book If you’re a platform or operations engineer tasked with maintaining a growing micro‐ services environment on Kubernetes or VMs, then this book is for you. If you’re a microservices developer interested in increasing reliability or experiment‐ ing with advanced deployment strategies such as blue/green and canarying, this book is also for you. Or perhaps your organization is already using Consul and you’re looking to learn how it works at a deeper level and how to utilize it better. This book will also be helpful for security engineers and higher-level decision-makers (managers, directors, VPs of engineering, and CTOs) to provide an overview of the concepts behind a service mesh and the value it provides. This book assumes general knowledge of microservices development and networking concepts such as load balancers. It contains instructions for installing Consul on Kubernetes or Linux VMs and assumes that you will be familiar with one of those platforms. It contains exercises that you can complete on Windows, macOS, or Linux machines. Navigating This Book The book starts with service mesh fundamentals: what a service mesh is and how it works. Next, you’ll learn what makes Consul unique, its architecture, and the specific protocols it uses. With that groundwork in place, you’ll be ready to deploy Consul onto Kubernetes or VMs and add your services into the service mesh. You’ll then learn to use Consul to secure your systems with zero trust networking, add observability, increase reliability, and control traffic. In the final chapter, I cover advanced topics such as multi-cluster deployment. Throughout the book, I include exercises for both Kubernetes and VMs, so you can utilize these concepts with an actual microservices application. If you wish to follow along with the exercises, I recommend you complete the chapters in order since they often rely on one another. xii | Preface

Join the official Consul: Up and Running Discord server to chat with other readers and the author. What Is Not in This Book This book does not cover Consul features unrelated to its service mesh functionality. For example, Consul’s key/value store and Domain Name System (DNS) service dis‐ covery are not covered. Also, this book is not a detailed production-ready operations guide to Consul. The aim is to familiarize readers with Consul’s concepts and get them “up and running” with its functionality. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords. Constant width bold Shows commands or other text that should be typed literally by the user. Also used occasionally in program listings to highlight text of interest. This element signifies a tip or suggestion. This element signifies a general note. This element indicates a warning or caution. Preface | xiii

Using Code Examples Supplemental material (code examples, exercises, etc.) is available for download at https://oreil.ly/consul-examples. If you have a technical question or a problem using the code examples, please send email to bookquestions@oreilly.com. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Consul: Up and Running by Luke Kysow (O’Reilly). Copyright 2022 Luke Kysow, 978-1-098-10614-0.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit https://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 xiv | Preface

800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/consul-up-and-running. Email bookquestions@oreilly.com to comment or ask technical questions about this book. For news and information about our books and courses, visit https://oreilly.com. Find us on LinkedIn: https://linkedin.com/company/oreilly-media Follow us on Twitter: https://twitter.com/oreillymedia Watch us on YouTube: https://youtube.com/oreillymedia Acknowledgments First and foremost, I would like to thank all of the contributors to Consul over the years. Consul’s creators Armon Dadgar and Mitchell Hashimoto, current and former HashiCorp employees, and hundreds of community contributors have made Consul the unique software it is today. It is a privilege to work with you all and write about Consul. Thanks to Paul Banks and Matthew Keeler for their Consul knowledge, Sabeen Syed for her support, and Hannah Hearth for designing the UI for the sample application. Finally, I owe a debt of gratitude to my reviewers, Nitya Dhanushkodi, Brandon McRae, Guy Barros, and Isha, along with my O’Reilly editors, Corbin Collins, Liz Wheeler, and Gregory Hyman. Thank you so much for your insights, suggestions, and encouragement. This book wouldn’t be what it is without you. Preface | xv

CHAPTER 1 Service Mesh 101 To get started on your service mesh journey, you need to know three things: what a service mesh is, how it works, and why you should use it (and when you should not). There is no universally accepted definition for a service mesh, but I define it as follows: A service mesh is an infrastructure layer that enables you to control the network communication of your workloads from a single control plane. We can break that definition down into parts to better understand it: By infrastructure layer, I mean that a service mesh is not part of your services; it is deployed and operated independently. Since it is not aware of service-specific busi‐ ness logic, but it affects every service, it is considered infrastructure or middleware. Figure 1-1 shows a typical software stack. Services and applications run on top of infrastructure. Service mesh is at the first infrastructure layer with storage, metrics, and other higher-level infrastructure requirements. Under that is VMs, Kubernetes, or any compute provider or orchestrator where everything runs. At the bottom is actual hardware (bare metal). By control the network communication of your workloads, I mean that a service mesh controls the traffic entering and leaving a microservice, database, or anything else that does network communication. For example, a service mesh might disallow incoming traffic based on a rule (such as it’s missing a required header), or it might encrypt outgoing traffic. A service mesh has complete control over all traffic entering and leaving the services. 1

1 Some meshes use other technology such as iptables or eBPF to control traffic rather than a separate proxy process. Figure 1-1. A typical software stack Finally, by from a single control plane, I mean a single location from which service mesh operators can interact with the service mesh. Suppose operators want to change the configuration for multiple services. In that case, they don’t need to reconfigure a dozen subsystems or modify the services themselves; instead, they configure the service mesh once, and it handles propagating out all changes. Hopefully, this definition gives you some idea of what a service mesh is, but I often find that I need to understand how something actually works before I fully grasp what it is. How a Service Mesh Works A service mesh is made up of sidecar proxies and the control plane. Sidecar Proxies A proxy is an application that traffic is routed through on the way to its destination. Popular proxies you may have heard of are NGINX, HAProxy, and Envoy. In most service meshes, all service traffic (inbound and outbound) is routed through a local proxy dedicated to each service instance.1 Figure 1-2 shows what a service mesh looks like with two service instances: frontend and backend. When frontend calls backend, frontend’s local proxy captures the outgoing request. frontend’s proxy then forwards the request to the backend service. When the request reaches the backend service, again, it is captured by backend’s local proxy and inspected. If the request is allowed, backend’s proxy forwards it to the actual backend service. 2 | Chapter 1: Service Mesh 101