Cloud Native Go Building Reliable Services in Unreliable Environments (Matthew A. Titmus) (z-library.sk, 1lib.sk, z-lib.sk)

Matthew A. Titmus Cloud Native Go Building Reliable Services in Unreliable Environments

Praise for Cloud Native Go This book does a great job of bringing the high level concept of “Cloud Native” down to earth and implemented using the modern computing language Go. The marriage of the two concepts works well, and the result is great inspiration. —Lee Atchison Owner, Atchison Technology LLC This is the first book I’ve come across that covers such a breadth and depth of modern cloud native practices in such a practical way. The patterns presented here have clear examples to solve real problems that are faced by engineers on a daily basis. —Alvaro Atienza Site Reliability Engineer, Flatiron Health Matt’s expertise in the art and science of building reliable systems in a fundamentally unreliable world are clearly (and humorously) captured in the pages within. Join him as he introduces you to the fundamental building blocks and system designs that enable large scale, reliable systems to be constructed from the ephemeral and unreliable components that comprise the underlying cloud infrastructure of today’s modern computing environment. —David Nicponski Principal Engineer, Robinhood

Over the past few years, two infrastructure trends have been happening: Go has been increasingly used for infrastructure, in addition to backend; and the infrastructure is moving to the cloud. This book summarizes the state of the art of the combination of the two. —Natalie Pistunovich Lead Developer Advocate, Aerospike I came in knowing next to nothing about Go, and left feeling like an expert. I would go so far as to say that simply reading this book made me a better engineer. —James Quigley Systems Reliability Engineer, Bloomberg

Matthew A. Titmus Cloud Native Go Building Reliable Services in Unreliable Environments Boston Farnham Sebastopol TokyoBeijing

978-1-492-07633-9 [LSI] Cloud Native Go by Matthew A. Titmus Copyright © 2021 Matthew A. Titmus. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institu‐ tional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Suzanne McQuade Development Editor: Amelia Blevins Production Editor: Daniel Elfanbaum Copyeditor: Piper Editorial Consulting, LLC Proofreader: nSight, Inc. Indexer: nSight, Inc. Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea April 2021: First Edition Revision History for the First Edition 2020-04-20: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781492076339 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Cloud Native Go, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the author, and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

For you, Dad. Your gentleness, wisdom, and humility are dearly missed. Also, you taught me to code, so any mistakes in this book are technically your fault.

Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv Part I. Going Cloud Native 1. What Is a “Cloud Native” Application?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 The Story So Far 4 What Is Cloud Native? 6 Scalability 6 Loose Coupling 7 Resilience 8 Manageability 10 Observability 11 Why Is Cloud Native a Thing? 12 Summary 13 2. Why Go Rules the Cloud Native World. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 The Motivation Behind Go 15 Features for a Cloud Native World 16 Composition and Structural Typing 16 Comprehensibility 18 CSP-Style Concurrency 19 Fast Builds 20 Linguistic Stability 21 Memory Safety 22 Performance 22 vii

Static Linking 23 Static Typing 24 Summary 25 Part II. Cloud Native Go Constructs 3. Go Language Foundations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 Basic Data Types 30 Booleans 30 Simple Numbers 31 Complex Numbers 31 Strings 32 Variables 32 Short Variable Declarations 33 Zero Values 33 The Blank Identifier 35 Constants 35 Container Types: Arrays, Slices, and Maps 36 Arrays 36 Slices 37 Maps 41 Pointers 42 Control Structures 44 Fun with for 44 The if Statement 47 The switch Statement 48 Error Handling 49 Creating an Error 50 Putting the Fun in Functions: Variadics and Closures 50 Functions 50 Variadic Functions 54 Anonymous Functions and Closures 55 Structs, Methods, and Interfaces 57 Structs 57 Methods 58 Interfaces 59 Composition with Type Embedding 61 The Good Stuff: Concurrency 64 Goroutines 64 viii | Table of Contents

Channels 64 Select 67 Summary 69 4. Cloud Native Patterns. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 The Context Package 72 What Context Can Do for You 73 Creating Context 74 Defining Context Deadlines and Timeouts 74 Defining Request-Scoped Values 75 Using a Context 75 Layout of this Chapter 76 Stability Patterns 77 Circuit Breaker 77 Debounce 80 Retry 84 Throttle 86 Timeout 90 Concurrency Patterns 92 Fan-In 93 Fan-Out 95 Future 97 Sharding 101 Summary 106 5. Building a Cloud Native Service. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 Let’s Build a Service! 107 What’s a Key-Value Store? 108 Requirements 108 What Is Idempotence and Why Does It Matter? 108 The Eventual Goal 110 Generation 0: The Core Functionality 110 Your Super Simple API 111 Generation 1: The Monolith 112 Building an HTTP Server with net/http 112 Building an HTTP Server with gorilla/mux 114 Building a RESTful Service 117 Making Your Data Structure Concurrency-Safe 121 Generation 2: Persisting Resource State 123 What’s a Transaction Log? 125 Storing State in a Transaction Log File 126 Table of Contents | ix

Storing State in an External Database 137 Generation 3: Implementing Transport Layer Security 145 Transport Layer Security 146 Private Key and Certificate Files 147 Securing Your Web Service with HTTPS 148 Transport Layer Summary 149 Containerizing Your Key-Value Store 150 Docker (Absolute) Basics 151 Building Your Key-Value Store Container 158 Externalizing Container Data 162 Summary 163 Part III. The Cloud Native Attributes 6. It’s All About Dependability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 What’s the Point of Cloud Native? 168 It’s All About Dependability 168 What Is Dependability and Why Is It So Important? 169 Dependability: It’s Not Just for Ops Anymore 171 Achieving Dependability 172 Fault Prevention 174 Fault Tolerance 176 Fault Removal 176 Fault Forecasting 178 The Continuing Relevance of the Twelve-Factor App 178 I. Codebase 179 II. Dependencies 179 III. Configuration 180 IV. Backing Services 182 V. Build, Release, Run 183 VI. Processes 184 VII. Data Isolation 184 VIII. Scalability 185 IX. Disposability 186 X. Development/Production Parity 186 XI. Logs 187 XII. Administrative Processes 188 Summary 189 x | Table of Contents

7. Scalability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 What Is Scalability? 192 Different Forms of Scaling 193 The Four Common Bottlenecks 194 State and Statelessness 195 Application State Versus Resource State 196 Advantages of Statelessness 196 Scaling Postponed: Efficiency 197 Efficient Caching Using an LRU Cache 198 Efficient Synchronization 201 Memory Leaks Can…fatal error: runtime: out of memory 206 On Efficiency 209 Service Architectures 209 The Monolith System Architecture 210 The Microservices System Architecture 211 Serverless Architectures 213 Summary 217 8. Loose Coupling. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 Tight Coupling 220 Tight Coupling Takes Many Forms 221 Communications Between Services 224 Request-Response Messaging 224 Common Request-Response Implementations 225 Issuing HTTP Requests with net/http 226 Remote Procedure Calls with gRPC 230 Loose Coupling Local Resources with Plug-ins 241 In-Process Plug-ins with the plugin Package 241 HashiCorp’s Go Plug-in System over RPC 247 Hexagonal Architecture 255 The Architecture 255 Implementing a Hexagonal Service 256 Summary 264 9. Resilience. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265 Keeping on Ticking: Why Resilience Matters 266 What Does It Mean for a System to Fail? 267 Building for Resilience 268 Cascading Failures 269 Preventing Overload 270 Table of Contents | xi

Play It Again: Retrying Requests 275 Backoff Algorithms 276 Circuit Breaking 280 Timeouts 281 Idempotence 286 Service Redundancy 290 Designing for Redundancy 291 Autoscaling 293 Healthy Health Checks 294 What Does It Mean for an Instance to Be “Healthy”? 295 The Three Types of Health Checks 295 Failing Open 300 Summary 300 10. Manageability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303 What Is Manageability and Why Should I Care? 304 Configuring Your Application 306 Configuration Good Practice 307 Configuring with Environment Variables 307 Configuring with Command-Line Arguments 308 Configuring with Files 314 Viper: The Swiss Army Knife of Configuration Packages 329 Feature Management with Feature Flags 334 The Evolution of a Feature Flag 334 Generation 0: The Initial Implementation 335 Generation 1: The Hard-Coded Feature Flag 335 Generation 2: The Configurable Flag 336 Generation 3: Dynamic Feature Flags 337 Summary 341 11. Observability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343 What Is Observability? 344 Why Do We Need Observability? 345 How Is Observability Different from “Traditional” Monitoring? 345 The “Three Pillars of Observability” 346 OpenTelemetry 347 The OpenTelemetry Components 348 Tracing 350 Tracing Concepts 350 Tracing with OpenTelemetry 352 Putting It All Together: Tracing 363 xii | Table of Contents

Metrics 369 Push Versus Pull Metric Collection 371 Metrics with OpenTelemetry 374 Putting It All Together: Metrics 384 Logging 387 Better Logging Practices 388 Logging with Go’s Standard log Package 391 The Zap Logging Package 394 Summary 400 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403 Table of Contents | xiii

1 Including CNCF Sandbox, Incubating, and Graduated code-based (non-specification) projects, as of February 2021. Preface It’s a magical time to be a technologist. We have Docker to build containers, and Kubernetes to orchestrate them. Prome‐ theus lets us monitor them. Consul lets us discover them. Jaeger lets us trace the rela‐ tionships between them. These are just a few examples, but there are many, many more, all representative of a new generation of technologies: all of them are “cloud native,” and all of them are written in Go. The term “cloud native” feels ambiguous and buzzwordy, but it actually has a pretty specific definition. According to the Cloud Native Computing Foundation, a sub- foundation of the renowned Linux Foundation, a cloud native application is one that’s designed to be scalable in the face of a wildly changing load, resilient in the face of environmental uncertainty, and manageable in the face of ever-changing require‐ ments. In other words, a cloud native application is built for life in a cruel, uncertain universe. Incorporating lessons learned from years of building cloud-based software, Go was created about a decade ago as the first major language designed specifically for the development of cloud native software. This was largely because the common server languages in use at the time simply weren’t a great fit for writing the kinds of dis‐ tributed, process-intensive applications that Google produces a lot of. Since that time, Go has emerged as the lingua franca of cloud native development, being used in everything from Docker to Harbor, Kubernetes to Consul, InfluxDB to CockroachDB. Ten out of fifteen of the Cloud Native Computing Foundation’s grad‐ uated projects, and forty-two of sixty-two1 of its projects overall, are written mostly or entirely in Go. And more arrive every day. Preface | xv

Who Should Read This Book This book is directed at intermediate-to-advanced developers, particularly web appli‐ cation engineers and DevOps specialists/site reliability engineers. Many will have been using Go to build web services, but may be unfamiliar with the subtleties of cloud native development—or even have a clear idea of what “cloud native” is—and have subsequently found their services to be difficult to manage, deploy, or observe. For these readers, this work will provide a solid foundation in not just how to build a cloud native service, but it will show why these techniques matter at all, as well as offer concrete examples to understand this sometimes abstract topic. It’s expected that many readers may be more familiar with other languages, but lured by Go’s reputation as the language of cloud native development. For these readers, this book will present best practices for adopting Go as their cloud native develop‐ ment language, and help them solve their own cloud native management and deploy‐ ment issues. Why I Wrote This Book The way that applications are designed, built, and deployed is changing. Demands of scale are forcing developers to spread their services’ efforts across legions of servers: the industry is going “cloud native.” But this introduces a host of new problems: how do you develop or deploy or manage a service running on ten servers? A hundred? A thousand? Unfortunately, the existing books in the “cloud native” space focus on abstract design principles, and contain only rudimentary examples of how to do any of this, or none at all. This book seeks to fill a need in the marketplace for a practical demonstration of complex cloud native design principles. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program ele‐ ments such as variable or function names, databases, data types, environment variables, statements, and keywords. xvi | Preface

Constant width bold Shows commands or other text that should be typed literally by the user. Constant width italic Shows text that should be replaced with user-supplied values or by values deter‐ mined by context. This element signifies a tip or suggestion. This element signifies a general note. This element indicates a warning or caution. Using Code Examples Supplemental material (code examples, exercises, etc.) is available for download at https://github.com/cloud-native-go/examples. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require per‐ mission. We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: "Cloud Native Go by Matthew A. Tit‐ mus (O’Reilly). Copyright 2021 Matthew A. Titmus, 978-1-492-07633-9.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. Preface | xvii

O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit http://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/cloud-native-go. Email bookquestions@oreilly.com to comment or ask technical questions about this book. For more information about our books, courses, conferences, and news, see our web‐ site at http://www.oreilly.com. Find us on Facebook: http://facebook.com/oreilly Follow us on Twitter: http://twitter.com/oreillymedia Watch us on YouTube: http://www.youtube.com/oreillymedia Acknowledgments First and foremost, I’d like to thank my wife and son. You’re the motivation for every good thing I’ve done since you’ve entered my life, and the guiding stars that let me keep my direction true and my eyes on the sky. xviii | Preface