Refactoring at Scale Regaining Control of Your Codebase (Maude Lemaire) (Z-Library)

📄 File Format: PDF

💾 File Size: 11.6 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

Refactoring at Scale Regaining Control of Your Codebase Maude Lemaire

📄 Page 2

(This page has no text content)

📄 Page 3

Maude Lemaire Refactoring at Scale Regaining Control of Your Codebase Boston Farnham Sebastopol TokyoBeijing

📄 Page 4

978-1-492-07553-0 [LSI] Refactoring at Scale by Maude Lemaire Copyright © 2021 Maude Lemaire. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Melissa Duffield Development Editor: Jeff Bleiel Production Editor: Christopher Faucher Copyeditor: nSight, Inc. Proofreader: Christina Edwards Indexer: Potomac Indexing, LLC Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea October 2020: First Edition Revision History for the First Edition 2020-10-13: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781492075530 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Refactoring at Scale, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the author, and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

📄 Page 5

Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Part I. Introduction 1. Refactoring. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 What Is Refactoring? 4 What Is Refactoring at Scale? 6 Why Should You Care About Refactoring? 8 Benefits of Refactoring 9 Developer Productivity 9 Identifying Bugs 10 Risks of Refactoring 11 Serious Regressions 11 Unearthing Dormant Bugs 12 Scope Creep 12 Unnecessary Complexity 12 When to Refactor 13 Small Scope 13 Code Complexity Actively Hinders Development 13 Shift in Product Requirements 14 Performance 14 Using a New Technology 15 When Not to Refactor 16 For Fun or Out of Boredom 16 Because You Happened to Be Passing By 16 To Making Code More Extendable 18 iii

📄 Page 6

When You Don’t Have Time 18 Our First Refactoring Example 18 Simplifying Conditionals 21 Extracting Magic Numbers 22 Extracting Self-Contained Logic 23 2. How Code Degrades. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 Why Understanding Code Degradation Matters 28 Requirement Shifts 29 Scalability 29 Accessibility 30 Device Compatibility 30 Environmental Changes 31 External Dependencies 32 Unused Code 32 Changes in Product Requirements 33 Tech Debt 36 Working Around Technology Choices 36 Persistent Lack of Organization 39 Moving Too Quickly 40 Applying Our Knowledge 41 Part II. Planning 3. Measuring Our Starting State. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Why Is Measuring the Impact of a Refactor Difficult? 46 Measuring Code Complexity 47 Halstead Metrics 47 Cyclomatic Complexity 50 NPath Complexity 53 Lines of Code 55 Test Coverage Metrics 57 Documentation 60 Formal Documentation 61 Informal Documentation 62 Version Control 64 Commit Messages 64 Commits in Aggregate 65 Reputation 66 Building a Complete Picture 69 iv | Table of Contents

📄 Page 7

4. Drafting a Plan. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 Defining Your End State 72 On the Road 72 At Work 72 Mapping the Shortest Distance 74 On the Road 74 At Work 75 Identifying Strategic Intermediate Milestones 76 On the Road 77 At Work 77 Choosing a Rollout Strategy 80 Dark Mode/Light Mode 81 Smart DNA’s Rollout 86 Cleaning Up Artifacts 87 Referencing Metrics in Your Plan 88 Interpolating Goal Metrics to Intermediate Milestones 89 Distinct Milestone Metrics 89 Estimating 90 Sharing Your Plan with Other Teams 91 Transparency 92 Perspective 92 Refined Plan 94 5. Getting Buy-In. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 Why Your Manager Is Not Onboard 98 Managers Aren’t Coding 98 Managers Are Evaluated Differently 99 Managers See the Risk 99 Managers Need to Coordinate 100 Strategies for Making a Compelling Argument 101 Using Conversational Devices 102 Building an Alignment Sandwich 104 Relying on Evidence 107 Playing Hardball 108 Buy-In Shapes the Refactor 109 6. Building the Right Team. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 Identifying Different Kinds of Experts 112 Matchmaking 113 Experts of Many Trades 115 Revisiting Active Contributors 115 Table of Contents | v

📄 Page 8

Biases in Our Expert List 116 Types of Refactoring Teams 117 Owners 117 Proposed Approach 119 Cleanup Crews 120 The Pitch 122 Metrics 123 Generosity 124 Opportunity 124 Bartering 125 Repeat 126 A Few Outcomes 126 Realistic Scenario 127 Worst-Case Scenario 127 Fostering Strong Teams 128 Part III. Execution 7. Communication. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 Within Your Team 132 Stand-Ups 133 Weekly Syncs 135 Retrospectives 137 Outside Your Team 138 When Kicking Off the Project 138 During Project Execution 141 Always Iterate 146 8. Strategies for Execution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 Team Building 147 Pair Programming 148 Keeping Everyone Motivated 150 Keeping a Tally 152 Intermediate Metric Measurements 152 Unearthed Bugs 153 Clean-Up Items 154 Out-of-Scope Items 154 Programming Productively 155 Prototyping 155 Keep Things Small 156 vi | Table of Contents

📄 Page 9

Test, Test, Test 157 Asking the “Stupid” Question 157 Conclusion 158 9. Making the Refactor Stick. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 Fostering Adoption 160 Education 161 Active Education 161 Passive Education 164 Reinforcement 165 Progressive Linting 165 Code Analysis Tools 166 Gates Versus Guardrails 166 Integrating Improvement into the Culture 167 Part IV. Case Studies 10. Case Study: Redundant Database Schemas. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 Slack 101 172 Slack Architecture 101 174 Scalability Problems 177 Booting Up the Slack Client 177 File Visibility 178 Mentions 178 Consolidating the Tables 180 Gathering the Scattered Queries 181 Developing a Migration Strategy 183 Quantifying Our Progress 186 Attempting to Keep the Team Motivated 187 Communicating Our Progress 188 Tidying Up 190 Lessons Learned 192 Develop a Well-Defined, Well-Communicated Plan 192 Understand the Code’s History 193 Ensure Adequate Test Coverage 194 Keep Your Team Motivated 194 Focus on Strategic Milestones 195 Identify and Rely on Meaningful Metrics 195 Takeaways 196 Table of Contents | vii

📄 Page 10

11. Case Study: Migrating to a New Database. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197 Workspace-Sharded Data 198 Migrating channels_members to Vitess 199 Sharding Scheme 200 Developing a New Schema 201 Detangling JOINs 203 A Difficult Rollout 208 Backfill Mode 209 Dark Mode 210 Light Mode 215 Sunset Mode 216 Tidying Up 217 Lessons Learned 219 Set Realistic Estimates 219 Source the Teammates You Need 220 Plan Scope Carefully 220 Choose a Single Place for Project Communication 221 Design a Thoughtful Rollout Plan 221 Takeaways 222 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 viii | Table of Contents

📄 Page 11

Preface While there are a number of books about refactoring, most of them deal with the nitty-gritty of improving small bits of code one line at a time. I believe that the most difficult part of a refactor is usually not finding the precise way to improve the code at hand, but rather everything else that needs to happen around it. In fact, I might also go so far as to say that for any large software project, the little things rarely matter; coordinating complex changes is the biggest challenge of all. Refactoring at Scale is my attempt at helping you figure out those difficult pieces. It’s the culmination of many years of experience carrying out all sorts of refactoring projects of various scales. During my time at Slack, many of the projects I’ve led have allowed the company to scale dramatically; our product has gone from being able to support customers with 25,000 employees to those with a whopping 500,000. The strategies we developed to refactor effectively needed to tolerate explosive organiza‐ tional growth, with our engineering team growing nearly sixfold during the same period. Successfully planning and executing on a project that affects both a significant portion of your codebase and an increasing number of engineers is no small feat. I hope this book gives you the tools and resources you need to do just that. Who Should Read This Book If you work in a large, complex codebase alongside dozens (or more) of other engi‐ neers, this book is for you! If you’re a junior engineer seeking ways to start building more senior skills by making a difference at your company, a large refactoring effort can be a great way to achieve that. These kinds of projects have broad, meaningful impact extending well beyond your individual team. (They’re also not so glamorous that a senior engineer might snap it up right away.) They’re a great opportunity for you to acquire new professio‐ nal skills (and strengthen the ones you already have). This book will teach you how to navigate this kind of project smoothly from start to finish. ix

📄 Page 12

This book is also a valuable resource for highly technical senior engineers who can code themselves out of any problem, but are feeling frustrated that others aren’t understanding the value of their work. If you’re feeling isolated and are looking for ways to level-up others around you, this book can teach you the strategies you need to help others see important technical problems through your eyes. For the technical managers seeking to help guide their team through a large-scale refactor, this book can help you understand how to better support your team every step of the way. There isn’t a substantial amount of technical content contained within these pages, so if you are involved with a large-scale refactor in just about any capacity (engineering manager, product manager, project manager), you can benefit from the ideas herein. Why I Wrote This Book When I set out on my first large-scale refactor, I understood why the code needed to change and how it needed to change, but what puzzled me most was how to introduce those changes safely, gradually, and without stepping on everyone else’s toes. I was eager to have cross-functional impact and didn’t pause to acknowledge the ramifica‐ tions the refactor might have on others’ work, nor how I might motivate them to help me complete it. I simply plowed through. (You can read about this refactor in Chapter 10!) In the years that followed, I refactored many, many more lines of code and ended up on the receiving end of a few ill-executed refactors. The lessons I’d learned from these experiences felt important, so I began speaking about them at a number of conferen‐ ces. My talks resonated with hundreds of engineers, all of whom, like me, had experi‐ enced problems effectively refactoring large surface areas of code within their own companies. It seemed clear that there was some sort of gap in our software education, specifically around this core aspect of what it means to write software professionally. In many ways, this book attempts to teach the important things that aren’t covered in a typical computer science curriculum, simply because they are too difficult to teach in a classroom. Perhaps they cannot be taught in a book either, but why not give it a try? x | Preface

📄 Page 13

Navigating This Book This book is split into four parts and organized in rough chronological order of the work required to plan and execute a large-scale refactor, outlined as follows. • Part I introduces important concepts behind refactoring. — Chapter 1 discusses the basics of refactoring and how refactoring at scale dif‐ fers from smaller refactors. — Chapter 2 describes the many ways code can degrade and how this plays into effective refactoring. • Part II covers everything you need to know about planning a successful refactor. — Chapter 3 provides an overview of the many metrics you can use to measure the problems your refactor seeks to solve before any improvements are made. — Chapter 4 explains the important components of a comprehensive execution plan and how to go about drafting one. — Chapter 5 discusses different approaches to get engineering leadership to sup‐ port your refactor. — Chapter 6 describes how to identify which engineers are best suited to work on the refactor and tips for recruiting them. • Part III focuses on what you can do to make sure that your refactor goes well while it is underway. — Chapter 7 explores how best to promote good communication within your team and with any external stakeholders. — Chapter 8 looks at a number of ways to maintain momentum throughout the refactor. — Chapter 9 provides a few suggestions for how to ensure that the changes introduced by your refactor stick around. • Part IV contains two case studies, both pulled from projects I was involved with while working at Slack. These refactors affected a significant portion of our core application, truly at scale. I hope these will help illustrate the concepts discussed in Parts I–III of the book. This ordering is not prescriptive; just because we’ve reached a new phase doesn’t mean we shouldn’t revisit our previous assumptions if necessary. For example, you might be kicking off your refactor with a strong sense of the team you’ll be working with, only to discover halfway through drafting your execution plan that you’ll need to bring in more engineers than you had initially anticipated. That’s ok; it happens all the time! Preface | xi

📄 Page 14

Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program ele‐ ments such as variable or function names, databases, data types, environment variables, statements, and keywords. This element signifies a tip or suggestion. This element signifies a general note. Using Code Examples Supplemental material (code examples, exercises, etc.) is available for download at https://github.com/qcmaude/refactoring-at-scale. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Refactoring at Scale by Maude Lem‐ aire (O’Reilly). Copyright 2021 Maude Lemaire, 978-1-492-07553-0.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. xii | Preface

📄 Page 15

O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit http://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/refactoring-at-scale. Email bookquestions@oreilly.com to comment or ask technical questions about this book. For news and information about our books and courses, visit http://oreilly.com. Find us on Facebook: http://facebook.com/oreilly Follow us on Twitter: http://twitter.com/oreillymedia Watch us on YouTube: http://youtube.com/oreillymedia Preface | xiii

📄 Page 16

Acknowledgments Writing a book is not an easy task, and this one was no exception. Refactoring at Scale would not have been possible without the contributions of many people. First, I’d like to thank my editor at O’Reilly, Jeff Bleiel. Jeff turned an inexperienced writer (me) into a published author. His feedback was always spot-on, helping me organize my thoughts more cohesively, and encouraging me to cut whenever I was being too wordy (which is something that happened quite frequently). I simply can’t imagine working with a better editor. Second, I want to thank the handful of friends and colleagues who read early versions of a few chapters: Morgan Jones, Ryan Greenberg, and Jason Liszka. Their feedback assured me that my ideas were sound and would be valuable to a wide range of read‐ ers. For the words of encouragement and thought-provoking conversations, thanks go to Joann, Kevin, Chase, and Ben. I’d like to thank Maggie Zhou for all her help cowriting the second case study chapter (Chapter 11). She is one of the most thoughtful, intelligent, energetic coworkers I’ve ever had the pleasure to work with and I’m thrilled for the world to read about our adventures together! A huge thank you to my technical reviewers, David Cottrell and Henry Robinson. David has been a close friend since university and has led a number of large-scale refactors in his many years at Google. He’s since founded his own company. Henry is a colleague at Slack who’s made countless open-source contributions and seen explo‐ sive growth at Silicon Valley companies firsthand. They are both incredibly conscien‐ tious engineers, and the book greatly benefited from their guidance and wisdom. I am endlessly grateful for the many hours they spent verifying its contents. Any inaccura‐ cies in the final manuscript are mistakes of my own. Thank you to everyone who’s ever refactored something with me. There are too many of you to name, but you know who you are. You all have had a hand in shaping the ideas in this book. Thank you to my family (Simon, Marie-Josée, François-Rémi, Sophie, Sylvia, Gerry, Stephanie, and Celia) for cheering me on from the sidelines. Finally, thank you to my husband, Avery. Thank you for your patience, for giving me the time, space, and encouragement to write. Thank you for letting me hijack count‐ less afternoons to talk through an idea or two (or three or four). Thank you for believing in me. This book is just as much yours as it is mine. I love you. xiv | Preface

📄 Page 17

PART I Introduction

📄 Page 18

(This page has no text content)

📄 Page 19

CHAPTER 1 Refactoring Someone once asked me what it was that I liked so much about refactoring. What kept me coming back to these types of projects at work so often? I told her that there was something addicting about it. Maybe it’s the simple act of tidying, like neatly cata‐ loging and ordering your spices; or maybe it’s the joy of decluttering and finally dep‐ recating something, like bringing a bag of forgotten clothes to Goodwill; or maybe yet it’s the little voice in my head reminding me that these tiny, incremental changes will amount to a significant improvement in my colleagues’ daily lives. I think it’s the combination of it all. There’s something in the act of refactoring that can appeal to us all, whether we’re building new product features or working on scaling an infrastructure. We all must strike a balance in our work between writing more or writing less code. We must strive to understand the downstream effects of our changes, whether intentional or not. Code is a living, breathing thing. When I think about the code that I’ve written living on for another five, ten years, I can’t help but wince a little bit. I certainly hope that by that time, someone will have come along and either removed it entirely or replaced it with something cleaner and, most importantly, more suited to the needs of the application at that time. This is what refactoring is all about. In this chapter, we’ll start by defining a few concepts. We’ll propose a basic definition for refactoring in the general case and build on top of it to develop a separate defini‐ tion for refactoring at scale. To frame some of the motivations of this book, we’ll dis‐ cuss why we should care about refactoring and what advantages we can bring to our teams if we’ve honed this skill. Next, we’ll dive into some of the benefits to expect from refactoring and some of the risks we should keep in mind when considering whether to do it. With our knowledge of the trade-offs, we’ll consider some scenarios when the time is right and when the time is wrong. Finally, we’ll walk through a short example to bring these concepts to life. 3

📄 Page 20

What Is Refactoring? Very simply put, refactoring is the process by which we restructure existing code (the factoring) without changing its external behavior. Now if you think that this defini‐ tion is incredibly generic, don’t worry; it purposefully is! Refactoring can take many equally effective forms, depending on the code it’s applied to. To illustrate this, we’ll define a “system” as any defined set of code that produces a set of outputs from a set of inputs. Say we have a concrete implementation of such a system called S, pictured in Figure 1-1. The system was built under a tight deadline, encouraging the authors to cut some corners. Over time, it’s become a large pile of tangled code. Thankfully, con‐ sumers of the system aren’t exposed to the internal mess of the system directly; they interact with S, using a defined interface and rely on it to provide consistent results. Figure 1-1. A simple system with inputs and outputs A few brave developers cleaned up the internals of the system, which we’ll now call S’, picture in Figure 1-2. While it might be a tidier system, to the consumers of S’, abso‐ lutely nothing has changed. Figure 1-2. A simple refactored system with inputs and outputs System S could be anything; it could be a single if statement, a ten-line function, a popular open source library, a multimillion-line application, or anything in between. (Inputs and outputs could be equally diverse.) The system could operate on database entries, collections of files, or data streams. Outputs aren’t limited to returned values, but could also include a number of side effects such as printing to the console or issu‐ 4 | Chapter 1: Refactoring