Learning OpenTelemetry Setting Up and Operating a Modern Observability System (Austin Parker, Ted Young)（Z-Library）

OBSERVABILIT Y “Ted and Austin live and breathe OpenTelemetry. If you’re looking for a book to understand not only the how, but also the why of this emerging standard for cloud native observability, look no further.” —Daniel Gomez Blanco Principal Software Engineer, Skyscanner Learning OpenTelemetry linkedin.com/company/oreilly-media youtube.com/oreillymedia OpenTelemetry is a revolution in observability data. Instead of multiple disconnected systems, OpenTelemetry combines tracing, metrics, and logs into a single integrated stream of data, providing valuable and accurate correlations that were previously unavailable. This practical guide shows you how to set up, operate, and troubleshoot the OpenTelemetry observability system. Authors Ted Young and Austin Parker, leaders and founding contributors of the OpenTelemetry project, cover every OpenTelemetry component, as well as observability best practices for many popular cloud, platform, and data services such as Kubernetes and AWS Lambda. You’ll learn how OpenTelemetry enables OSS libraries and services to provide their own native instrumentation—a first in the industry. Ideal for application developers, OSS maintainers, operators and infrastructure teams, and managers and team leaders, this book guides you through: • The principles of modern observability • All OpenTelemetry components and how they fit together • A practical approach to instrumenting platforms and applications • Methods for installing, operating, and troubleshooting an OpenTelemetry-based observability solution • Ways to roll out and maintain end-to-end observability across a large organization • How to write and maintain consistent, high-quality instrumentation without a lot of work Ted Young is one of the cofounders of the OpenTelemetry project. Over the past 20 years, he has designed and built a variety of large-scale distributed systems, including visual FX pipelines and container scheduling systems. Austin Parker, director of open source at honeycomb.io, is a cofounder of the OpenTelemetry project and a member of the OpenTelemetry Governance Committee. In addition, Austin is an author of Distributed Tracing in Practice (O’Reilly). 9 7 8 1 0 9 8 1 4 7 1 8 1 5 4 5 9 9 US $45.99 CAN $57.99 ISBN: 978-1-098-14718-1 Young & Pa rker

Ted Young and Austin Parker Learning OpenTelemetry Setting Up and Operating a Modern Observability System Boston Farnham Sebastopol TokyoBeijing

978-1-098-14718-1 [LSI] Learning OpenTelemetry by Ted Young and Austin Parker Copyright © 2024 Ted Young and Austin Parker. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: John Devins Development Editor: Sarah Grey Production Editor: Aleeya Rahman Copyeditor: Arthur Johnson Proofreader: Sharon Wilkey Indexer: nSight, Inc. Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea March 2024: First Edition Revision History for the First Edition 2024-03-05: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781098147181 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Learning OpenTelemetry, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the authors and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

For Dylan Mae —Austin For the OpenTelemetry cofounders —Ted

Table of Contents Foreword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi 1. The State of Modern Observability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 The Times They Are A-Changin’ 2 Observability: Key Terms to Know 3 A Brief History of Telemetry 5 The Three Browser Tabs of Observability 6 Emerging Complications 7 The Three Pillars Were an Accident 8 A Single Braid of Data 8 Conclusion 9 2. Why Use OpenTelemetry?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Production Monitoring: The Status Quo 12 The Challenges of Production Debugging 13 The Importance of Telemetry 14 Hard and Soft Context 14 Telemetry Layering 17 Semantic Telemetry 19 What Do People Need? 20 Developers and Operators 20 Teams and Organizations 21 Why Use OpenTelemetry? 22 Universal Standards 22 Correlated Data 23 Conclusion 23 v

3. OpenTelemetry Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Primary Observability Signals 26 Traces 27 Metrics 29 Logs 30 Observability Context 31 The Context Layer 32 Attributes and Resources 34 Semantic Conventions 37 OpenTelemetry Protocol 38 Compatibility and Future-Proofing 38 Conclusion 40 4. The OpenTelemetry Architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Application Telemetry 41 Library Instrumentation 42 The OpenTelemetry API 43 The OpenTelemetry SDK 43 Infrastructure Telemetry 44 Telemetry Pipelines 44 What’s Not Included in OpenTelemetry 44 Hands-On with the OpenTelemetry Demo 45 Running the Demo 46 Architecture and Design 47 Managing Application Performance with OpenTelemetry 49 Finding Needles in Haystacks 52 Observability Pipelines in the Demo 56 The New Observability Model 57 Conclusion 59 5. Instrumenting Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 Agents and Automated Setup 62 Installing the SDK 63 Registering Providers 63 Providers 64 TracerProvider 64 MeterProvider 68 LoggerProvider 69 Shutting Down Providers 70 Custom Providers 70 Configuration Best Practices 71 Remote Configuration 72 vi | Table of Contents

Attaching Resources 72 Resource Detectors 72 Service Resources 73 Installing Instrumentation 74 Instrumenting Application Code 74 How Much Is Too Much? 75 Layering Spans and Metrics 75 Browser and Mobile Clients 76 The Complete Setup Checklist 77 Packaging It All Up 78 Conclusion 79 6. Instrumenting Libraries. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 The Importance of Libraries 82 Why Provide Native Instrumentation? 83 Observability Works by Default in Native Instrumentation 84 Native Instrumentation Lets You Communicate with Your Users 84 Native Instrumentation Shows That You Care About Performance 85 Why Aren’t Libraries Already Instrumented? 86 How OpenTelemetry Is Designed to Support Libraries 87 OpenTelemetry Separates the Instrumentation API and the Implementation 88 OTel Maintains Backward Compatibility 88 OTel Keeps Instrumentation Off by Default 89 Shared Libraries Checklist 90 Shared Services Checklist 92 Conclusion 92 7. Observing Infrastructure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 What Is Infrastructure Observability? 94 Observing Cloud Providers 95 Collecting Cloud Metrics and Logs 96 Metamonitoring 100 Collectors in Containers 101 Observing Platforms 101 Kubernetes Platforms 101 Serverless Platforms 103 Queues, Service Buses, and Other Async Workflows 104 Conclusion 106 8. Designing Telemetry Pipelines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 Common Topologies 108 No Collector 108 Table of Contents | vii

Local Collector 109 Collector Pools 111 Pipeline Operations 116 Filtering and Sampling 116 Transforming, Scrubbing, and Versioning 120 Transforming Telemetry with OTTL 121 Privacy and Regional Regulations 122 Buffering and Backpressure 122 Changing Protocols 123 Collector Security 123 Kubernetes 124 Managing Telemetry Costs 125 Conclusion 126 9. Rolling Out Observability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 The Three Axes of Observability 128 Deep Versus Wide 128 Code Versus Collection 131 Centralized Versus Decentralized 132 Moving from Innovation to Differentiation 135 Observability as Testing 135 Green Observability 135 AI Observability 136 OpenTelemetry Rollout Checklist 137 Conclusion 138 A. The OpenTelemetry Project. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 B. Further Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 viii | Table of Contents

Foreword In the ever-evolving landscape of cloud-native technologies, observing an applica‐ tion’s performance and health is no longer a luxury, but a critical imperative. As microservice architectures become the norm, distributed systems sprawl, and data volumes explode, traditional monitoring tools struggle to keep pace. This is where OpenTelemetry emerges as a game changer, offering a standardized and vendor- neutral approach to observability. OpenTelemetry is not just a technology; it also represents a paradigm shift, “crossing the chasm” from just monitoring to complete observability. OpenTelemetry is changing the industry from silos to unified telemetry. As the authors Ted Young and Austin Parker explain, OpenTelemetry is about embracing a unified telemetry data-driven approach to observability, leveraging open standards like the OpenTelemetry Protocol (OTLP), and being empowered to build and operate fully observable, resilient, high-performing cloud-native applications. Learning OpenTelemetry serves as your comprehensive guide to unlocking the power of OpenTelemetry. Whether you’re a seasoned engineer grappling with the complexi‐ ties of distributed tracing, a newcomer seeking to understand the fundamentals, or an organization embarking on its observability journey, this book equips you with the knowledge and practical insights to navigate this transformative technology. The authors emphasize that observability requires understanding the cloud-native paradigm’s broader context and inherent challenges. For example, microservice archi‐ tectures, while offering agility and scalability, introduce new complexities. Traditional monitoring tools, designed for monolithic applications, often struggle to capture the intricate interactions and dependencies between services. This lack of coherent visi‐ bility leads to visibility gaps, making it difficult to pinpoint performance bottlenecks, diagnose issues, and ensure application health. Learning OpenTelemetry highlights how OpenTelemetry addresses these challenges head-on by providing a unified and vendor-agnostic approach to collecting and exporting telemetry data. This unified approach uses metrics, traces, logs, and profil‐ ing to offer a correlated view of your application’s health and performance. ix

The authors delve into the intricacies of OpenTelemetry, guiding us through core concepts of OpenTelemetry and pursuing instrumentation strategies for different programming languages and frameworks such as shared libraries and shared services. They illuminate best practices for collecting and processing telemetry data using the OpenTelemetry Collector; they survey deployment patterns for scaling telemetry collection for platforms such as Kubernetes, serverless and data streaming. They’ll show you how to build scalable telemetry pipelines by balancing wide approaches with deep ones, centralized architectures with decentralized ones, and more. The final chapter explores advanced topics, such as generative AI, FinOps, and cloud sustainability. We live in exciting times. As the worlds of cloud-native services and AI applications converge, it’s critical to use telemetry data to understand large-scale model behavior. That’s why the next giant leap in OpenTelemetry’s journey will be to provide an open framework to fully support observability for smart, distributed GenAI applications. Observability, as a practice, must incorporate viable AI models to collect and analyze telemetry at massive scale. So, open this book, dive into the world of OpenTelemetry, and unlock the power of observability for your cloud-native journey. Remember, the path to mastery starts with a single step, and this book is a guide to your first and following steps in that journey. Enjoy! — Alolita Sharma Palo Alto, California February 2024 Alolita Sharma is an OpenTelemetry Governance Committee member and has been contributing to the OpenTelemetry project for over five years. She is co-chair for the CNCF Observability Technical Advisory Group (TAG) and leads Apple’s AIML observa‐ bility practice. She contributes to open source and open standards in OpenTelemetry, Observability TAG, Unicode and W3C. Alolita has also provided strong leadership for observability, infrastructure, and search engineering at AWS and has managed engineering teams at IBM, PayPal, Twitter, and Wikipedia. x | Foreword

Preface Over the past decade, observability has gone from a niche discipline talked about at events like Monitorama or Velocity (RIP) to a multibillion-dollar industry that touches every part of the cloud native world. The key to effective observability, though, is high-quality telemetry data. OpenTelemetry is a project that aims to pro‐ vide this data and, in doing so, kick off the next generation of observability tools and practices. If you’re reading this book, it’s highly likely that you’re an observability practitioner— perhaps a developer or an SRE—who is interested in how to profile and understand complex systems in production. You may have picked it up because you’re interested in what OpenTelemetry is, how it fits together, and what makes it different from historical monitoring frameworks. Or maybe you’re just trying to understand what all the hype is about. After all, in just five years, OpenTelemetry has gone from an idea to one of the most popular open source projects in the world. Regardless of why you’re here, we’re glad you came. Our goal in writing this book was not to create a “missing manual” for OpenTeleme‐ try—you can find lots of documentation and tutorials and several other fantastic books that dive deep into implementing OpenTelemetry in specific languages (see Appendix B for details on those). Our goal was to present a comprehensive guide to learning OpenTelemetry itself. We want you to understand not just what the different parts are but how they fit together and why. This book should equip you with the foundational knowledge you’ll need not only to implement OpenTelemetry in a production system but also to extend OpenTelemetry itself—either as a contributor to the project or by making it part of an organizational observability strategy. In general, this book has two main parts. In Chapters 1 through 4, we discuss the current state of monitoring and observability and show you the motivation behind OpenTelemetry. These chapters help you understand the foundational concepts that underpin the entire project. They’re invaluable not just for first-time readers but for anyone who’s been practicing observability for a while. Chapters 5 through 9 xi

move into specific use cases and implementation strategies. We discuss the “how” behind the concepts introduced in earlier chapters and give you pointers on actually implementing OpenTelemetry in a variety of applications and scenarios. If you’re already well versed in observability topics, you might be considering skip‐ ping ahead to the latter part of the book. While we can’t stop you, you’ll probably get something out of reviewing the initial chapters. Regardless, as long as you go into this book with an open mind, you should get something out of it—and keep coming back, time after time. We hope this book becomes the foundation for the next chapter of your observability journey. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords. Constant width bold Shows commands or other text that should be typed literally by the user. Constant width italic Shows text that should be replaced with user-supplied values or by values deter‐ mined by context. This element signifies a tip or suggestion. This element signifies a general note. This element indicates a warning or caution. xii | Preface

Using Code Examples Supplemental material (code examples, exercises, etc.) is available for download at https://github.com/orgs/learning-opentelemetry-oreilly/. If you have a technical question or a problem using the code examples, please send email to support@oreilly.com. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Learning OpenTelemetry by Ted Young and Austin Parker (O’Reilly). Copyright 2024 Austin Parker and Ted Young, 9781098147181.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit https://oreilly.com. Preface | xiii

1 “Nothing without effort.” How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-889-8969 (in the United States or Canada) 707-827-7019 (international or local) 707-829-0104 (fax) support@oreilly.com https://www.oreilly.com/about/contact.html We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/learning-opentelemetry. For news and information about our books and courses, visit https://oreilly.com. Find us on LinkedIn: https://linkedin.com/company/oreilly-media Watch us on YouTube: https://youtube.com/oreillymedia Acknowledgments The authors would like to thank their entire team at O’Reilly for constant support, encouragement, and grace. Special thanks go out to our acquisitions editor, John Devins, and our developmental editor, Sarah Grey. We also thank our tech reviewers for their invaluable feedback, as well as Alolita Sharma for her contribution. This book would not be possible without the work of every OpenTelemetry contribu‐ tor across the years. Austin I’d like to thank my coauthor for convincing me that writing another book would be a good idea, actually. To my partner, Mandy: thank you for putting up with the long hours and unpredicta‐ ble nature of writing. Tada gan iarracht.1 I would also like to thank the many people whom I used as sounding boards over the past year or so, and whose friendship and ideas have made it into these words; they include (but are not limited to) Phillip Carter, Alex Hidalgo, Jessica Kerr, Reese Lee, xiv | Preface

Rynn Mancuso, Ana Margarita Medina, Ben Sigelman, Pierre Tessier, Amy Tobey, Adriana Villela, Hazel Weakly, and Christine Yen. All y’all great humans. Ted I’d like to thank my coauthor for being convinced that writing another book would be a good idea, actually. I would like to thank all the maintainers of the OpenTracing and OpenCensus projects. Both projects have the same goal: to create a universal standard for describ‐ ing the computer operations of distributed systems. Choosing to put egos aside, merge the projects, and accept a years-long setback as we started over with OpenTele‐ metry was a difficult decision. I appreciate the bravery and trust that it took to do this. I would also like to thank the maintainers of the Elastic Common Schema project. This was another case in which having two standards meant that we had no stand‐ ards. Their willingness to merge ECS into the OpenTelemetry Semantic Conventions was another important step toward our shared goal of a universally accepted teleme‐ try system. It’s a common (and funny) joke to point at OpenTelemetry and bring up the clas‐ sic XKCD comic #927, “How Standards Proliferate”. But I must say, au contraire, Monsieur chuckles! OpenTelemetry did create a new standard, but in the process it deprecated three other standards. So we are now at minus two standards. I believe this may be a record in the history of standardization. I’m hoping for at least minus four before we’re done. Preface | xv

1 Henry Glassie, Passing the Time in Ballymenone: Culture and History of an Ulster Community (Philadelphia: University of Pennsylvania Press, 1982). CHAPTER 1 The State of Modern Observability History is not the past but a map of the past, drawn from a particular point of view, to be useful to the modern traveler. —Henry Glassie, US historian1 This is a book about the difficult problems inherent to large-scale distributed com‐ puter systems, and about how to apply OpenTelemetry to help solve those problems. Modern software engineering is obsessed with end-user experience, and end users demand blazing-fast performance. Surveys show that users will abandon ecommerce sites that take more than two seconds to load. You’ve probably spent a fair amount of time trying to optimize and debug application performance issues, and if you’re like us, you’ve been frustrated by how inelegant and inefficient this process can be. There’s either not enough data or too much of it, and what data there is can be riddled with inconsistencies or unclear measurements. Engineers are also faced with stringent uptime requirements. That means identifying and mitigating any issues before they cause a meltdown, not just waiting for the system to fail. And it means moving quickly from triage to mitigation. To do that, you need data. But you don’t need just any data; you need correlated data—data that is already organized, ready to be analyzed by a computer system. As you will see, data with that level of organization has not been readily available. In fact, as systems have scaled and become more heterogeneous, finding the data you need to analyze an issue has become even harder. If it was once like looking for a needle in a haystack, it’s now more like looking for a needle in a stack of needles. 1

2 Carl E. Sagan (author and presenter), in Cosmos: A Personal Voyage, season 1, episode 2, “One Voice in the Cosmic Fugue,” produced by Adrian Malone (Arlington, VA: Public Broadcasting Service, 1980). OpenTelemetry solves this problem. By turning individual logs, metrics, and traces into a coherent, unified graph of information, OpenTelemetry sets the stage for the next generation of observability tools. And since the software industry is broadly adopting OpenTelemetry already, that next generation of tools is being built as we write this. The Times They Are A-Changin’ Technology comes in waves. As we write this in 2024, the field of observability is riding its first real tsunami in at least 30 years. You’ve chosen a good time to pick up this book and gain a new perspective! The advent of cloud computing and cloud native application systems has led to seismic shifts in the practice of building and operating complex software systems. What hasn’t changed, though, is that software runs on computers, and you need to understand what those computers are doing in order to understand your software. As much as the cloud has sought to abstract away fundamental units of computing, our ones and zeros are still using bits and bytes. Whether you are running a program on a multiregion Kubernetes cluster or a laptop, you will find yourself asking the same questions: “Why is it slow?” “What is using so much RAM?” “When did this problem start?” “Where is the root cause?” “How do I fix this?” The astronomer and science communicator Carl Sagan said, “You have to know the past to understand the present.”2 That certainly applies here: to see why a new approach to observability is so important, you first need to be familiar with tradi‐ tional observability architecture and its limitations. This may look like a recap of rudimentary information! But the observability mess has been around for so long that most of us have developed quite the pile of precon‐ ceptions. So even if you’re an expert—especially if you’re an expert—it is important to have a fresh perspective. Let’s start this journey by defining several key terms we will use throughout this book. 2 | Chapter 1: The State of Modern Observability