Mastering Distributed Observability in Rust - Implement OpenTelemetry in a real-world, multi-container e-commerce architecture (Manjunath Gangappa, Rajkumar Rangaraj)（Z-Library）

Mastering Distributed Observability in Rust Implement OpenTelemetry in a real-world, multi-container e-commerce architecture Manjunath Gangappa Rajkumar Rangaraj

Mastering Distributed Observability in Rust Copyright © 2026 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. Portfolio Director: Kunal Chaudhary Relationship Lead: Srishti Seth Program Manager: K. Loganathan Project Manager: Ankit Maroli Content Engineer: Sushma Reddy Technical Editor: Irfa Ansari Indexer: Rekha Nair Production Designer: Prashant Ghare Growth Lead: Vinishka Kalra First published: June 2026 Production reference: 1230626 Published by Packt Publishing Ltd. Grosvenor House 11 St Paul's Square Birmingham B3 1RB, UK. ISBN 978-1-80667-179-3 www.packtpub.com

To my wife Shreedevi, for her unwavering support; Kushan and Gagan, for their constant inspiration; and my parents, my foundation. — Manjunath Gangappa To my wife, Amsaveni, for her patience, love, and encouragement through this journey. To my sons, Nikil Pranav and Reyan, whose curiosity, energy, and smiles make every day brighter. To my parents, Rangaraj and Chandravadivu, whose sacrifices and values shaped the person I am today. And to my sister, Latha, for her constant love, encouragement, and support. — Rajkumar Rangaraj

Foreword I serve on the OpenTelemetry Governance Committee and maintain the OpenTelemetry semantic conventions, including HTTP, database, and Generative AI (GenAI) conventions, the parts of the standard used throughout this book. Which is why I was eager to read this book. OpenTelemetry only delivers on its promise when the conventions work cleanly in real services, written by engineers who treat the standard as a core design concern. The authors do exactly that: low-cardinality span names, database telemetry that builds on the stable conventions instead of reinventing them, GenAI conventions applied to a real service that calls an LLM, and the reasoning behind each choice explained clearly. As a semantic conventions maintainer, that is the most useful kind of validation a specification can get. What struck me first about the book itself is the teaching method. The authors build one running system, OpenTel E-Commerce, and every chapter extends it. The system is deliberately engineered to fail the way production systems fail: inventory that leaks across a distributed transaction, query plans that regress under load, connection pools that saturate, Tokio executors that starve, promotional traffic that exposes hidden bottlenecks. A single mystery runs through the whole book, "The Mystery of the Slow Checkout," introduced in Chapter 1, and the reader returns to it with flamegraphs, async profiling, database failure-mode triage, and finally a full incident postmortem. It reads less like a reference manual and more like a detective novel with telemetry as the evidence. The second thing that struck me is the scope. The book starts with the three pillars and a vendor-neutral pipeline (Jaeger, Prometheus, Grafana, Loki, all fed from a single instrumented codebase) and then keeps going. SLOs and error budgets translate telemetry into business impact. Flamegraphs and async profiling close the loop between where time disappears and why the code is slow. A later chapter reuses the same trace infrastructure as a security sensor, with Collector tail sampling to make retention affordable. The final chapter extends the model to GenAI-augmented services, where tokens become a new cost dimension and a new failure mode. This is what OpenTelemetry has always promised: once you have a real telemetry foundation built to the standard, you can answer most of the hard questions your system raises.

The Rust focus is what makes this book stand out. Rust has become a serious choice for backend infrastructure, and the community has invested heavily in observability tools and libraries that conform to the OpenTelemetry specification. The authors take full advantage. Chapter 2, Reading Rust Memory from Traces, is the kind of chapter I wish existed for every language: stack versus heap, Arc and Mutex contention, Send and Sync, async context propagation, each one connected back to what you will actually see in a span. If you write Rust for a living, this is a chapter you will reread. If you do not, it is still the clearest demonstration of how language details show up in telemetry that any OpenTelemetry user would recognize. One chapter in particular stayed with me. Chapter 11 opens with a premise that is grounded in real observability work: three completely different database failures (a slow query, a saturated connection pool, and a contended lock) produce the same span in Jaeger. The chapter is about extending the instrumentation just enough to tell them apart. That kind of teaching only comes from operating systems that have actually broken in production, and the book is full of moments like it. The authors do not teach observability as a catalog of features. They teach it the way it is actually learned: by walking from alert to root cause, again and again, until the signals start to mean something. That is the kind of judgment a specification cannot give you on its own. Read this before your next incident, while there is still time to build the signals you will wish you had. You will see OpenTelemetry working the way it was designed to work, in the hands of authors who clearly operated systems that needed it. — Trask Stalnaker OpenTelemetry Governance Committee Semantic Conventions Maintainer

Contributors About the authors With 19 years of experience in the software industry, Manjunath Gangappa has witnessed the evolution of enterprise technology firsthand. His career, which spans from hands-on Senior Software Engineering to his current role as a Director of Software Engineering, has been driven by a singular passion: architecting high-performance systems that scale. Manjunath possesses a deep technical versatility, with extensive expertise in Java, Python, and Rust across diverse domains including SaaS, FinTech, and Blockchain. Currently, he leads an R&D team at Mastercard, where he tackles the extreme complexities of global payment transactions. His work focuses on the cutting edge of observability and large-scale metrics, engineering the systems that ensure reliability in mission-critical environments. Rajkumar Rangaraj is a Principal Software Engineer at Microsoft, based in the Seattle area. He believes that observability is a first-class engineering discipline, not an afterthought added when production systems break. With more than two decades in software engineering, he has built diagnostic tools, telemetry SDKs, and AI-assisted onboarding agents used by Microsoft and its customers to operate distributed systems at scale. Raj leads technical strategy for Azure Monitor's .NET and Java observability ecosystems and is a maintainer of the OpenTelemetry .NET project across the SDK, Contrib, and Auto-Instrumentation repositories. He invented an out-of-process auto-instrumentation model for .NET that addressed a long- standing observability gap the industry had carried for nearly a decade. His current work extends these patterns into Rust-based pipelines for edge and cloud environments, the direct bridge between his engineering practice and this book.

About the reviewers Lalit Kumar Bhasin is a software engineer with extensive experience in observability, distributed systems, cloud-native telemetry, and systems programming. He has worked deeply with OpenTelemetry, Rust, C++, eBPF, and high-performance telemetry pipelines, with a focus on building reliable and efficient observability infrastructure for modern distributed applications. Lalit is actively involved in the OpenTelemetry ecosystem and has contributed to work around telemetry collection, exporters, SDKs, and performance-focused observability components. His technical interests include distributed tracing, metrics, logs, Rust-based telemetry systems, eBPF-based instrumentation, and scalable data pipelines for cloud-native environments. I would like to thank my family, especially my wife Bhawna, my daughter Vaanya, and my parents, Indra Lal and Saroj, for their constant support, patience, and encouragement during the review of this book. Antonio Souza is a senior software engineer with nearly 20 years of industry experience. His work focuses on building highly available, scalable web applications, with a strong commitment to contributing to the open source ecosystem. He has contributed to the Rust language and other Rust ecosystem projects, including Tokio. He is also the creator and maintainer of Rapina, an open source web framework written in Rust and designed to improve the developer experience. In less than four months, Rapina has grown to over 140 GitHub stars and an active community of more than 20 contributors. Beyond Rapina, Antonio's areas of focus include distributed systems, infrastructure, backend engineering, and applied AI/ML. He is the founder of Zap Tech, a company dedicated to developing open source solutions and delivering tailored software development services. I would like to thank, first and foremost, God, for granting me the wisdom and resilience to remain in such a competitive industry for so many years. I am also deeply grateful to my family — and in particular to my late father, an engineer, who sparked and nurtured my passion for technology when I was just a boy. A special thank you goes to my wife, Mariana, for her patience and support through the countless weekends I spent buried in code and study.

Table of Contents Preface xxix Free benefits with your book .......................................................................... xxxvii Part 1: Foundation 1 Chapter 1: Introduction to Observability 3 Introduction ........................................................................................................ 3 How This Book Works • 3 Why Observability Matters in Distributed Systems ................................................ 4 What Each Tool Reveals (and Misses) • 4 From Guesswork to Understanding • 5 The Three Pillars of Observability in Action ........................................................... 6 Traces: Following the Request's Journey • 7 Logs: The Narrative of What Happened • 8 Metrics: The Pulse of the System • 9 Bringing It Together • 10 What Is OpenTelemetry? .................................................................................... 10 Building an Observability Mindset ....................................................................... 12 Chapter Summary ............................................................................................... 12 Subscribe to Deep Engineering ............................................................................ 13 Chapter 2: Reading Rust Memory From Traces 15 Introduction ....................................................................................................... 15 What You'll See in Your Traces ............................................................................. 17 Allocation Patterns: Stack vs. Heap ...................................................................... 17 Why Stack Allocations Don't Appear in Memory Profiles • 18 Heap Allocations: The Observability Hotspots • 18 Understanding Rust's Memory Model • 19

Ownership • 19 Borrowing - The Golden Rule of Observability • 22 Clone vs. Move: Performance Impact in Traces • 26 Lock Contention: Mutex and Arc in Distributed Traces ........................................ 29 How Lock Waits Appear as Span Duration • 30 Async Runtime Observability ............................................................................... 31 Task Scheduling: What Tokio Does Behind the Scenes • 32 Blocking Operations in Async Code (The Silent Killer) • 32 Task Stalls and Backpressure in Traces • 33 Send and Sync: Why They Matter for Tracing ...................................................... 34 Thread-Safe Span Propagation • 35 Context Propagation Across Async Boundaries • 35 Memory Leaks and Reference Cycles ................................................................... 36 How Arc Cycles Appear in Memory Profiles • 36 Detecting Leaks Through Metrics • 37 Summary ........................................................................................................... 38 Chapter 3: The OpenTel E-Commerce System 39 Introduction ...................................................................................................... 39 OpenTel E-Commerce: System Architecture ........................................................ 40 Domain Modeling: From Database to Rust ........................................................... 41 Identities and Types • 42 Service Boundaries and Responsibilities .............................................................. 43 OtelMart Gateway (BFF) • 43 Products Service • 43 Inventory Service • 43 Orders Service • 44 Navigating the Codebase .................................................................................... 45 Inter-Service Communication: The Distributed Transaction ............................... 46 The "Happy Path" • 46 The "Unhappy Path" • 46 The Backend Request Journey ............................................................................. 48 Table of Contents x

The Canonical Checkout Request (HTTP Entry to Database) • 48 Mapping Backend Service Hops • 49 The Current State of Logging .............................................................................. 50 The "Black Box" Problem • 50 Structured Logging (A Half Step) • 51 Why Observability is Non-Negotiable ................................................................. 52 Summary ........................................................................................................... 53 Subscribe to Deep Engineering ........................................................................... 53 Part 2: Instrumentation 55 Chapter 4: Instrumenting the Request Journey 57 Introduction ...................................................................................................... 57 The Transformation: Before and After OpenTelemetry ........................................ 58 Where We Left Off • 58 The Transformation • 59 From Concepts to Implementation ....................................................................... 61 Traces and Spans: The Hierarchical View • 62 SpanContext: The Propagation Mechanism • 62 Span Events: Marking Moments in Time • 64 Seeing Timing and Events in Action • 65 Understanding Span Lifecycle • 65 How OpenTelemetry Fits into the Rust Ecosystem • 68 Adding the tracing and OpenTelemetry Crates .................................................... 69 Required Dependencies • 70 The Bridge Layer (Invisible Plumbing) • 71 Configuration & Export (One-Time Setup) • 72 Feature Flags • 73 Version Compatibility • 74 Initializing the Tracer Provider ........................................................................... 74 Step 1: Resource - Service Identity • 75 Step 2: OTLP Exporter - Data Egress • 77 xi Table of Contents

Step 3: TracerProvider - The Span Factory • 78 Step 4: Connecting to the tracing Ecosystem • 79 Graceful Shutdown • 80 The Complete telemetry.rs Module • 80 Integrating into main.rs • 82 Trace Taxonomy: Naming Conventions & Semantic Attributes ............................ 83 The Five Fundamental Span Kinds • 83 Span Hierarchy Rules • 86 Span Naming Best Practices • 87 OpenTelemetry Semantic Conventions • 87 Designing a Taxonomy for OpenTel E-Commerce • 91 Applying the Taxonomy in Code • 93 Instrumenting HTTP Boundaries ........................................................................ 94 Server-Side: Automatic HTTP Instrumentation • 95 Manual Instrumentation for Business Logic • 96 When to Use Each Approach • 98 Combining Automatic and Manual Instrumentation • 99 Async Functions and Span Context • 99 Recording Errors in Spans • 101 Service to Service Communication ..................................................................... 103 The Propagation Problem • 103 Injection: Adding Context to Outgoing Requests • 104 Extraction: Reading Context from Incoming Requests • 105 Replacing Manual Correlation IDs • 106 Verifying Context Propagation • 107 Viewing Distributed Traces in Jaeger ................................................................ 108 Setting Up the Telemetry Backend • 108 Adding Jaeger to docker-compose.yml • 108 Starting the Stack • 110 Alternative: OpenTelemetry Collector • 110 Adding the Collector • 110 Collector Configuration • 111 Table of Contents xii

Switching Services to Use the Collector • 112 Triggering a Trace • 112 Viewing in Jaeger • 113 What We Can Now See • 114 Identifying the Bottleneck • 115 Evaluating Trace Quality • 115 Summary .......................................................................................................... 116 Chapter 5: Data Layer Instrumentation 119 Introduction: Following the Request into SQL (Hop 3) ........................................ 119 OpenTelemetry Semantic Conventions for Databases ........................................ 120 Why Conventions Matter • 120 Core Database Attributes • 120 Attribute Requirements • 121 The Cardinality Rule • 122 Query Text Sanitization • 122 Designing a Stable Database Span Schema ......................................................... 123 Span Naming Strategy • 123 Required Attributes for All Services • 124 Service-Specific Attributes • 125 Business Context Attributes • 125 What We Explicitly Exclude • 126 SQLx Instrumentation Strategy ......................................................................... 126 SQLx Configuration in OpenTel E-Commerce • 126 Why We Instrument Manually • 127 Compile-Time Verification • 127 The Repository Layer Pattern • 128 The Database Wrapper • 128 Connecting Database Spans to HTTP Spans ........................................................ 130 Automatic Context Propagation Within a Service • 130 Cross Service Propagation • 130 Instrumenting Read Queries .............................................................................. 131 xiii Table of Contents

Single-Row Fetch • 131 List of Queries • 132 Handling "Not Found" • 133 Instrumenting Write Queries ............................................................................. 134 Inventory Reservation • 134 Recording Values After Execution with tracing::field::Empty • 135 Handling Database Errors • 136 Instrumenting Transactions .............................................................................. 137 Why Transactions Deserve Their Own Spans • 137 The Transaction Wrapper • 139 Using the Transaction Wrapper • 140 Transaction Span Hierarchy • 142 Connection Pool Visibility .................................................................................. 142 What SQLx Handles Automatically • 143 When Pool Time Becomes Visible • 143 The Diagnostic Pattern • 143 Viewing Database Spans in Jaeger ...................................................................... 144 Triggering Trace Generation • 144 Before Database Instrumentation (Chapter 4) • 144 After Database Instrumentation (Chapter 5) • 144 Reading the Trace: Healthy Request • 145 Reading the Trace: Checkout Flow • 146 Reading the Trace: Failed Checkout • 147 Reading the Trace: Leaked Reservation • 148 Summary .......................................................................................................... 149 Subscribe to Deep Engineering .......................................................................... 151 Chapter 6: Metrics That Matter (RED + USE + KPIs) 153 Introduction ..................................................................................................... 153 From Traces to Metrics ...................................................................................... 154 Every metric has three parts: • 154 Understanding Time Series • 155 Table of Contents xiv

The Cardinality Problem • 156 Metrics as System Pulse • 158 The Three Metric Types • 159 Counters: Things That Only Go Up • 159 Gauges: Snapshots in Time • 160 Histograms: Understanding Distributions • 161 What You See in Prometheus • 162 Aggregation Temporality: Cumulative vs Delta • 163 Synchronous vs. Asynchronous: Push vs. Pull • 164 The Instrument Cheat Sheet • 165 RED Metrics: Request-Oriented Observability .................................................... 167 Rate: The Traffic Volume • 167 Errors: The Warning Sign • 168 Duration: User Experience • 169 RED vs Business Metrics • 169 USE Metrics: Resource-Oriented Observability ................................................... 170 Utilization: Percentage in Use • 171 Saturation and Errors: The Breaking Points • 171 Saturation: The Waiting Line • 171 Errors: The Hard Limit • 172 Business KPIs with Custom Metrics ................................................................... 172 Automatic HTTP Metrics vs Business Metrics • 173 Conversion Rate: Measuring the Checkout Funnel • 174 Instrumenting the Conversion Funnel • 175 Tracking Funnel Stages in the Handler • 176 Querying Conversion Rate • 177 Cart Abandonment: Understanding Time-to-Checkout • 178 Tracking Time-to-Checkout • 178 Analyzing Checkout Duration • 179 Revenue per Request: Business Impact • 180 Average Order Value (AOV) • 180 Revenue Distribution (Percentiles) • 180 xv Table of Contents

Total Revenue Rate • 181 Connecting Business KPIs to Technical Metrics • 181 Why Business KPIs Matter • 182 Implementing Metrics in Rust ........................................................................... 182 Dependencies • 182 Extending the Telemetry Module • 183 Creating Service Meters • 186 Registering Database Pool Metrics • 187 Instrumenting the Orders Service • 187 Automatic HTTP Metrics with axum-otel-metrics • 187 Instrumenting the Checkout Handler • 189 Instrumenting the Inventory Service • 190 Inventory Metrics Definition • 190 Instrumenting the OtelMart Gateway • 192 Gateway Metrics Definition • 192 Upstream Service Client with Metrics • 194 Why These Metrics Matter for a Gateway • 195 Instrumenting the Products Service • 196 Products Metrics Setup • 196 Why Products Need Minimal Custom Metrics • 196 The Metrics Pipeline: Direct OTLP to Prometheus • 197 Why Direct OTLP to Prometheus • 197 How Metrics Flow Through the Pipeline • 198 Connecting the Infrastructure • 198 Configuring Services to Export • 200 Verifying the Pipeline • 200 Dashboards: Visualizing the Request Journey .................................................... 201 Understanding Metric Name Transformations • 201 Connecting Grafana to Prometheus • 202 Building the RED Dashboard • 202 Request Rate: The Store Traffic • 202 Error Rate: Broken Registers • 203 Table of Contents xvi

Latency: Checkout Speed (Wait Times) • 203 Business Metrics: Checkout Success Rate • 204 Building the USE Dashboard • 204 Pool Utilization • 204 Idle Connections (Pool Headroom) • 205 Inventory Reservation Rate • 205 Dashboard Design Principles • 205 PromQL Patterns for Exploration • 205 Generating Realistic Traffic • 206 Observing Metrics in Grafana • 207 Correlating Metrics with Traces • 208 What Patterns to Look For • 208 The Journey from Instrumentation to Insight • 209 Lessons Learned: How Metrics Change Developer Thinking .............................. 209 Lesson 1: Averages Lie • 209 Lesson 2: Business Metrics Matter More Than Technical Ones • 209 Lesson 3: Cardinality Discipline Saves Infrastructure • 209 Lesson 4: The RED/USE Combination Covers Most Needs • 210 Lesson 5: Metrics and Traces Complement Each Other • 210 Summary ......................................................................................................... 210 Chapter 7: Log Strategy Without Log Hell 213 Introduction: Logs as Trace Annotations ............................................................ 213 The Problem with Traditional Logging ............................................................... 214 What Log Hell Looks Like • 214 The Five Paths to Log Hell • 215 Structured Logging with Trace Context .............................................................. 216 Structured Records as the Universal Format • 216 The Anatomy of a Good Log Entry • 217 From eprintln! to Structured tracing • 217 The tracing Crate's Structured Approach • 218 Service-Level Context via Resource • 219 xvii Table of Contents

Correlation IDs: Linking Logs to Traces • 219 Automatic Context Propagation • 220 The Click-Through Workflow • 220 Log Levels and Filtering .................................................................................... 220 The Five Levels • 220 When to Use Each Level • 220 When to Log What • 222 Always Log • 222 The Decision Framework • 222 Avoiding Log Spam • 222 The Production Default Rule • 222 Dynamic Log Levels Without Redeployment • 223 Sampling for High-Volume Paths • 224 Integrating Logs with OpenTelemetry .............................................................. 225 Log Exporters • 225 Architecture Overview • 225 Adding the OpenTelemetry Logs SDK • 225 Updating telemetry.rs • 227 Adding Loki to Docker Compose • 230 Configuring Loki for OTLP Ingestion • 231 Configuring Grafana Data Sources • 231 Verifying the Pipeline • 232 Unified Observability Pipeline • 233 Logging Standards • 233 Migration Checklist: Replacing eprintln! • 233 Orders Service • 234 Compensating Transaction Logging • 235 Inventory Service • 236 Products Service • 238 Gateway Service (OtelMart) • 239 The Three Pillars Unified • 239 Searching Logs by Trace ID ................................................................................ 241 Table of Contents xviii

Best Practices for Production Logging ............................................................... 243 Never Log • 243 PII Protection • 243 Lazy Formatting • 244 Sensitive Data Redaction • 244 Common Anti-Patterns and How to Fix Them • 245 Summary ......................................................................................................... 246 Subscribe to Deep Engineering ......................................................................... 248 Part 3: Advanced Observability 249 Chapter 8: Business Intelligence Views From Telemetry 251 Introduction ..................................................................................................... 251 The Language of Reliability: SLIs, SLOs, and SLAs .............................................. 252 Service Level Indicators: What You Measure • 252 What Makes a Good SLI • 253 Service Level Objectives: What You Promise (Internally) • 254 The Nines Table: What SLOs Actually Mean • 254 Service Level Agreements: What You Promise (Externally, with Consequences) • 255 How Others Define Their SLIs: Real-World Examples • 256 Business Case: Why OpenTel E-Commerce Needs SLOs ...................................... 256 Latency and Revenue: The Connection That Matters • 256 Revenue Per Minute: The Dollar Baseline • 257 Defining SLIs for OpenTel E-Commerce ............................................................ 258 SLI 1: Checkout Availability • 258 SLI 2: Checkout Latency • 258 SLI 3: Checkout Correctness • 259 New Telemetry for This Chapter (Minimal) • 260 SLOs and Error Budgets ..................................................................................... 261 Setting the Checkout SLO • 261 The Error Budget: Doing the Math • 262 Error Budgets Change Culture • 263 xix Table of Contents