Enabling Microservice Success Managing Technical, Organizational, and Cultural Challenges (Sarah Wells) (Z-Library)

Enabling Microservice Success Managing Technical, Organizational, and Cultural Challenges Sarah Wells Foreword by Sam Newman

SOF T WARE ARCHITEC TURE Enabling Microservice Success linkedin.com/company/oreilly-media youtube.com/oreillymedia Microservices can be a very effective approach for delivering value to your organization and to your customers. If you get them right, microservices help you to move fast by making changes to small parts of your system hundreds of times a day. But if you get them wrong, microservices will just make everything more complicated. In this book, technical engineering leader Sarah Wells provides practical, in-depth advice for moving to microservices. Having built her first microservice architecture in 2013 for the Financial Times, Sarah discusses the approaches you need to take from the start and explains the potential problems most likely to trip you up. You’ll also learn how to maintain the architecture as your systems mature while minimizing the time you spend on support and maintenance. With this book, you will: • Learn the impact of microservices on software development patterns and practices • Identify the organizational changes you need to make to successfully build and operate this architecture • Determine the steps you must take before you move to microservices • Understand the traps to avoid when you create a microservice architecture—and learn how to recover if you fall into one Sarah Wells is a technology leader, consultant, and conference speaker with a focus on microservices, engineering enablement, observability, and DevOps. She has over 20 years of experience as a developer, principal engineer, and tech director across product, platform, SRE, and DevOps teams. She spent over a decade working at the Financial Times as it transitioned from 12 releases a year to more than 20,000 and adopted the cloud, microservices, and DevOps. 9 7 8 1 0 9 8 1 3 0 7 9 4 5 6 5 9 9 US $65.99 CAN $82.99 ISBN: 978-1-098-13079-4 “This superb book acts as a reference for what to expect when moving to a ‘fast f low’ way of working using small, empowered teams. Essential reading for any forward-looking organization.” —Matthew Skelton Coauthor of Team Topologies and CEO at Conflux

Praise for Enabling Microservice Success It may be cliché to say this is Sarah’s magnum opus, but you won’t find a more condensed and inspiring guide to designing, building, and running microservices at scale than this book. The case studies and hard-won experiences jump off the page and allow the reader to avoid many operational traps that are easy to fall into. —Daniel Bryant, architect, technical consultant, and coauthor of Mastering API Architecture Small, empowered teams with aligned autonomy are a feature of high-performing organizations because the rapid feedback and agility that this setup provides helps organizations to be nimble and responsive. This superb book acts as a reference for what to expect when moving to a “fast flow” way of working using small, empowered teams. I love the emphasis on healthy organization dynamics as a key factor for success with small decoupled services, because ultimately success with any modern technology initiative rests on a humane approach with people, not just technology. Essential reading for any forward-looking organization. —Matthew Skelton, coauthor of Team Topologies and CEO at Conflux This book is packed full of great advice and practical tips to give you the best chance of succeeding with microservices. —Sam Newman, independent consultant and author Sarah Wells has been building and running microservices since before they were cool, and her experience shines through in every chapter. This in-depth book has everything you’ll need to go beyond the theory and make microservices work in the messy real world. —Tanya Reilly, senior principal engineer and author of The Staff Engineer’s Path

Sarah distills the core technical and organizational foundations so that the development and management of microservices can be a huge success. When I get asked what a good microservices design and architecture looks like, I can now say: look no further than this book, which is filled to the brim with practitioner guidance. —Suhail Patel, staff software engineer at Monzo Sarah’s hands-on experience permeates this book, providing readers with invaluable insights beyond the fundamentals of building and deploying microservices. Sarah delves into the complexities of evolving and scaling architectures from both technical and organizational perspectives to enable continued delivery and growth of business value. This makes the book essential for engineers and leaders alike embarking on this journey. — Nicky Wrightson, head of engineering at topi Whether you are thinking about moving to a microservices architecture or have one already, this book is essential reading. Sarah distills many years of running microservices architectures and seeing what works—and what doesn’t!—into actionable steps you can take right now, whatever stage you are at. —Anna Shipman, CTO at Kooth Digital Health

Sarah Wells Foreword by Sam Newman Enabling Microservice Success Managing Technical, Organizational, and Cultural Challenges Boston Farnham Sebastopol TokyoBeijing

978-1-098-13079-4 [LSI] Enabling Microservice Success by Sarah Wells Copyright © 2024 Sarah Wells. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editors: Melissa Duffield & Louise Corrigan Development Editor: Jill Leonard Production Editor: Clare Laylock Copyeditor: Kim Cofer Proofreader: Piper Editorial Consulting, LLC Indexer: nSight, Inc. Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea April 2024: First Edition Revision History for the First Edition 2024-03-26: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781098130794 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Enabling Microservice Success, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the author and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

Table of Contents Foreword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix Part I. Context 1. Understanding Microservices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Defining the Microservices Architectural Style 4 A Suite of Services 4 Each Running in Its Own Process 5 Communicating with Lightweight Mechanisms 5 Built Around Business Capabilities 5 Independently Deployable 6 “Small” 6 With a Bare Minimum of Centralized Management 7 Heterogeneous 7 Forerunners and Alternatives 8 The Monolith 8 Modular Monoliths 10 Service-Oriented Architecture 11 The Microservices Ecosystem 12 Infrastructure as Code 12 Continuous Delivery 13 The Public Cloud 15 New Deployment Options 16 v

DevOps 20 Observability 21 Advantages of Microservices 22 Independently Scalable 22 Robust 23 Easy to Release Small Changes Frequently 23 Support Flexible Technology Choices 24 Challenges of Microservices 24 Latency 25 Estate Complexity 25 Operational Complexity 25 Data Consistency 27 Security 28 Finding the Right Level of Granularity 28 Handling Change 30 Require Organizational Change 32 Change the Developer Experience 32 In Summary 33 2. Effective Software Delivery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Regularly Delivering Business Value 36 High Deployment Frequency 38 Short Lead Time for Changes 39 Running Experiments 40 Separating Deploying Code from Releasing Functionality 41 Handling Work That Goes Across Team Boundaries 42 Adapting to Changing Priorities 44 Maintaining Appropriate Service Levels 47 When a Release Goes Wrong 47 Knowing When Something Important Is Broken 48 Restore Some Level of Service Quickly 51 Avoid Failure Cascades 52 Spending Most of Your Time on Meaningful Work 53 Not Having to Start Again 56 Keeping Risk at an Acceptable Level 58 How Microservices Measure Up 59 In Summary 60 3. Are Microservices Right for You?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 Reasons to Choose Microservices 62 Scaling the Organization 62 Developer Experience 65 vi | Table of Contents

Separating Out Areas with Compliance and Security Requirements 66 Scaling for Load 66 Increasing Robustness 67 Increasing Flexibility 68 Conditions for Success 68 Domain Understanding 69 Products Not Projects 69 Leadership Support 70 Teams That Want Autonomy 70 Processes That Enable Autonomy 71 Technical Maturity 71 Managing Change 72 Sticking with a Monolithic Architecture 73 Enable Zero-Downtime Deployments 73 Build a Modular Monolith 74 Everything Is Distributed Now 78 The Rise of Cloud Native 79 SaaS Makes Sense 79 Recommendations 80 Starting from Scratch 81 Replacing an Existing Monolith 81 Measuring Success 84 In Summary 84 Part II. Organizational Structure and Culture 4. Conway’s Law and Finding the Right Boundaries. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 Conway’s Law 90 The Inverse Conway Maneuver 93 Possible Boundaries 94 Business Domains 95 Locations 96 Technologies 97 Compliance 97 Tolerance for Failure 98 Frequency of Changes 98 Recommendations 99 Identifying When Boundaries Are Wrong 100 In Summary 103 Table of Contents | vii

5. Building Effective Teams. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 Organizational Culture 105 Open 106 Learning 107 Empowering 108 Optimized for Change 108 The Westrum Model 109 Effective Teams 111 Motivated through Autonomy, Mastery, and Purpose 111 Aligned to Business Domain 112 Appropriately Sized 113 Cross-Functional and T-shaped 114 Strong Ownership 116 Long Lived 116 Sustainable Cognitive Load 117 High Trust and High Psychological Safety 118 Part of a Group 119 Optimizing for Flow 119 Stream-Aligned 121 Enabling 122 Complicated Subsystem 122 Platform 122 In Summary 125 6. Enabling Autonomy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 What Is Autonomy? 128 Why Does Autonomy Matter? 128 Limits to Autonomy 129 The Right Amount of Communication 129 Interaction Styles 130 Collaboration 131 X-as-a-Service 133 Facilitating 134 Ways of Working That Support Autonomy 134 Aligning on Outcomes 135 Light Touch Governance 136 Trust but Verify 137 Agreeing and Aligning on Technology 137 The Role of the Individual Contributor 139 Minimum Viable Competencies 140 Making Space for Learning 141 Responsibilities of Autonomous Teams 142 viii | Table of Contents

Active Ownership 142 Communication and Cooperation 143 Compliance with Standards 144 Maintaining a Team Page 144 In Summary 146 7. Engineering Enablement and Paving the Road. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 What’s in a Name? 148 Building a Platform 150 Platform Services 151 Organization-Level Concerns 152 Building the Thinnest Viable Platform 154 Build for the Needs of the Majority 156 Platform as a Product 157 Beyond the Platform 159 Vendor Engineering 159 APIs, Templates, Libraries, and Examples 160 A Service Catalog 161 Insights 161 Paving the Road 163 What Capabilities to Include 165 Make It Optional 166 Keep It Small 168 How to Go Off Road 168 Bringing the Treasure Back 170 Internal Developer Portals 170 Building a Platform People Actually Use 171 Making Sure What You Build Meets a Need 172 Market It 173 Look for Signs You Are Getting It Wrong 174 Principles for Building a Paved Road 175 Optional 176 Provides Value 176 Self-Service 177 Owned and Supported 178 Easy to Use 179 Guides People to Do the Right Thing 181 Composable and Extendable 181 Measuring Impact 182 When to Invest in Engineering Enablement 184 In Summary 187 Table of Contents | ix

8. Ensuring “You Build It, You Run It”. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189 Why Microservices Implies DevOps 190 Release on Demand 191 Work on Operational Features 192 Building Things Differently 192 Good Runbooks 193 Running on Someone Else’s Servers 195 Getting Comfortable in Production 195 Supporting Your System in Production 196 Assign Dedicated In-Hours Ops Support 196 Improve Alerts and Documentation 197 Identify the Haunted Forests 198 Practice 199 Out-of-Hours Support 201 Allow People to Opt Out 201 Formal Rotas Versus Best Endeavors 202 Make Sure Calls Are Rare 203 Only for Critical Systems 204 Provide Support and Guidance 205 Incident Management 205 Blameless Culture 206 Raising an Incident 208 Roles to Assign 209 During the Incident 209 After the Incident 210 Learning from Incidents 211 In Summary 213 Part III. Building and Operating 9. Active Service Ownership. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217 Responding to the Log4Shell Vulnerability 218 A Counter Example: Equifax and a Struts Vulnerability 220 Ownership During Active Development 220 Strong Ownership 221 Weak Ownership 221 Collective Ownership 222 Once a Service Is Feature Complete 223 No Ownership 224 Nominal Ownership 225 x | Table of Contents

Active Ownership 225 What Active Ownership Means 225 Code Stewardship 226 Upgrades and Patching 226 Migrations 227 Production Support 228 Documentation 228 Knowing Your Estate 229 Your Own Software 229 Dependencies 230 Third-Party Software 230 What You Need from a Service Catalog 232 Graph-Based Model 233 API-Driven 234 Extensible 235 Flexible Schema 235 Provides Different Views Across the Estate 236 Transferring Ownership 237 What Does a Good Transfer Look Like? 238 Meeting Quality Expectations 238 Operational Handover 239 Replacing 239 What to Do If You’re Struggling 240 Make the Business Case 240 Start with Critical Systems 241 Make Your Best Guess at Owners 242 Deliver Value from the Data 242 Aim for Continuous Improvement 242 Look for Teams That Are Overwhelmed 243 Services Shouldn’t Live Forever 244 In Summary 244 10. Getting Value from Testing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247 Why Do We Test? 248 Building the Thing Right 251 Building the Right Thing 252 Picking Up Regressions 253 Meeting Quality-of-Service Requirements 254 Shifting Testing Left 254 What Makes a Good Test? 255 Fast and Early Feedback 256 Easy to Change 256 Table of Contents | xi

Finds Real Problems 257 Types of Testing 257 The Testing Pyramid 258 Unit Tests 259 Service Tests 259 End-to-End Tests 261 Contract Tests 262 Consistency Tests 263 Exploratory Tests 264 Cross-Functional Testing 264 Testing in Production 265 Is It Safe? 266 Staging Is Not Production-Like 267 Your Customers Can Surprise You 267 You Can’t Test for Every Variation 268 You Don’t Have to Roll a Change Out to Everyone 268 Monitoring as Testing 271 Testing Your Infrastructure 273 Chaos Engineering 274 Testing Failovers and Restores 275 Quality Is About More Than Testing 275 What to Do If You’re Struggling 276 Not Enough Automated Testing 276 Tests That Aren’t Providing Value 276 In Summary 277 11. Governance and Standardization: Finding the Balance. . . . . . . . . . . . . . . . . . . . . . . . . 279 Why Governance Matters 280 Know Your Estate 280 What Sort of Information Is Relevant? 281 Guardrails and Policies 282 Automating Guardrails 283 What to Include 284 The FT’s Guardrails 284 Aligning on Guardrails 293 Tech Governance Group 293 Benefits of the TGG 297 Choosing Technologies 298 The Technology Lifecycle 298 Save Innovation for Key Business Outcomes 302 Use Boring Technology 305 Limit the Alternatives 306 xii | Table of Contents

Be Clear on Where Duplication Is Acceptable 307 Expect Things to Change 308 Insight Leads to Action 308 Governance in Other Organizations 309 Governance at Monzo 309 Governance at Skyscanner 311 What to Do If You’re Struggling 312 In Summary 313 12. Building Resilience In. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315 What Is Resilience? 315 Resilience for Distributed Systems 317 Resilience for Microservices 321 Understanding Your Service Level Requirements 322 Service Level Objectives 322 Error Budgets 324 Building Resilient Services 324 Redundancy 325 Fast Startup and Graceful Shutdown 325 Set Appropriate Timeouts 326 Back Off and Retry 326 Make Your Requests Idempotent 327 Protect Yourself 328 Testing Service Resilience 328 Make Building Resilient Services Easy 328 Building Resilient Systems 329 Caching 329 Handling Cascading Failures 330 Fallback Behavior 330 Avoiding Unnecessary Work 331 Go Asynchronous 332 Failover 332 Backup and Restore 333 Disaster Recovery 334 Building a Resilient Platform 334 Resilience to External Issues 334 Internal Tooling 335 Validating Your Resilience Choices 337 Chaos Engineering 337 Testing Backup and Restore 339 Practice Makes Perfect 339 Load Testing 339 Table of Contents | xiii

Learn from Incidents 340 One Thing at a Time 340 What to Do If You’re Struggling 340 In Summary 341 13. Running Your System in Production. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343 Operational Challenges of Microservices 344 Different Technologies Mean Different Support Knowledge Is Needed 345 Ephemeral Infrastructure 346 Rapid Change 347 Alert Overload 347 Complex Systems Run in Degraded Mode 348 Building Observability In 348 Logging 349 Monitoring and Metrics 351 Log Aggregation 352 OpenTelemetry 354 Focus on Events 355 Distributed Tracing 355 Archiving Observability Data 356 Building Your Own Tools 356 Spotting Issues 358 Getting Alerting Right 359 Healthchecks 359 Monitoring Business Outcomes 364 Understanding What Normal Looks Like 365 Mitigation 365 Troubleshooting 366 Maintaining Useful Documentation 366 Knowing What’s Changed 368 Problems with External Systems 369 Tooling Characteristics 369 Learning from Incidents 370 What to Do If You’re Struggling 370 In Summary 371 14. Keeping Things Up-to-Date. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373 Why Is This a Challenge? 374 Minimizing the Impact of Change 375 Think About the Long Term 375 A Reason to Be on the Paved Road 375 Choose Managed Services and SaaS Options 376 xiv | Table of Contents

Provide APIs 377 Immutable and Ephemeral Infrastructure 378 Decommission and Deprecate 378 Types of Change 378 Emergency Changes 379 Minor Planned Changes 379 Major Planned Changes 379 Responding to Change 380 Understand the Landscape 380 Define Guiding Policies 382 Making a Decision 382 Who Gets to Decide? 383 Scheduling Work 383 Managing Change 384 Clarity 385 Communication 386 Empathy 389 Execution 393 What to Do If You’re Struggling 394 In Summary 394 Afterword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397 A. Microservices Assessment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 401 B. Recommended Reading. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 407 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409 Table of Contents | xv

Foreword People who build software have always been busy, but we seem to be more time poor every year. The expectations of our users and the organizations we work for become greater as software plays a more important role in our day-to-day lives. Against this backdrop, the mantra of “shift left” has pushed more responsibility around things like usability, testing, security, and operations into teams that in the past would be much more limited in their scope. I don’t mean to suggest that the dismantling of silos in software delivery is a bad thing. The movement toward teams with more end-to-end responsibility that came to the fore with the Agile Manifesto and was turbocharged through DevOps is definitely moving us to a more effective way of building software. But at the same time, it means we’re asking ever more of the people in those teams who are absorbing all these new responsibilities. With all this going on, it is no wonder that people reach for easy answers. When a bandwagon comes rolling along with promises of a brighter future, better hair, and a more magnetic personality, how can we resist? The issue of course is that so many attractive industry trends don’t deliver on that initial hype, and once the bandwagon has disappeared over the horizon, we’re left with the legacy of decisions made in haste. Microservice architectures are no different—while they’ve been great for some people, for others, they’ve left chaos in their wake. Microservices are simple in concept, but the devil is in the details. There are so many aspects to getting the most out of this style of architecture, while also dealing with the complexity it brings. The problem is, getting the most out of microservices often is about nuance. The development of any software-based system requires people and technology coming together, and with microservices the people and organizational aspects are often greatly overlooked. People assume the answer must be a new pro‐ gramming language, more Kubernetes clusters, or perhaps a new vendor. But this approach results in an orgy of technological overload that ends up looking at less than half of the story. xvii

Most of my work revolves around helping teams navigate this world, and I know firsthand how maddening it can be for people when I’m asked questions and respond with the dreaded “it depends.” In most cases, it’s about finding time to stop and think, to break the problem down, to not follow some dogmatic approach but to consider your own context. That’s all well and good, but fundamentally people are still time poor. So having someone who has seen firsthand how to get the most out of micro‐ services lay out some home truths for you, in the way Sarah has in this book, can really help speed things up. This book is all about the nuance, about the multiple different choices that you’ll face in getting from early adoption to some degree of maturity with microservices. It’s rare we’re offered a binary choice in this space, and rarer still that there is one right answer. But what Sarah does so well in this book is lay out the options and explain the context, giving you information to make the right choice for your situation. That doesn’t mean that this is a book without opinions—rather, it is one where Sarah shares her own view, but still gives you the space to think, reason, and pick your own path. In this increasingly busy world, you might find it difficult to justify spending the time to sit down and read a book. But trust me, if you’re struggling with the reality of microservices, the time spent learning the lessons this book shares will be paid back several times over. — Sam Newman Independent consultant and author East Kent, February 2024 xviii | Foreword