Learning DevSecOps A Practical Guide to Processes and Tools (Steve Suehring) (Z-Library)

Steve Suehring Learning DevSecOps A Practical Guide to Processes and Tools Suehring

SECURIT Y Learning DevSecOps linkedin.com/company/oreilly-media youtube.com/oreillymedia How do some organizations maintain 24-7 internet-scale operations? How can organizations integrate security while continuously deploying new features? How do organizations increase security within their DevOps processes? This practical guide helps you answer those questions and more. Author Steve Suehring provides unique content to help practitioners and leadership successfully implement DevOps and DevSecOps. Learning DevSecOps emphasizes prerequisites that lead to success through best practices and then takes you through some of the tools and software used by successful DevSecOps-enabled organizations. You’ll learn how DevOps and DevSecOps can eliminate the walls that stand between development, operations, and security so that you can tackle the needs of other teams early in the development lifecycle. This book helps you: • Learn why DevSecOps is about culture and processes, with tools to support the processes • Understand why DevSecOps practices are key elements to deploying software in a 24-7 environment • Deploy software using a DevSecOps toolchain and create scripts to assist • Integrate processes from other teams earlier in the software development lifecycle • Help team members learn the processes important for successful software development Steve Suehring has worked in academia and the tech industry in a variety of roles related to DevSecOps. He has also written several technology books and has served as an editor for LinuxWorld Magazine. 9 7 8 1 0 9 8 1 4 4 8 6 9 5 5 5 9 9 US $55.99 CAN $69.99 ISBN: 978-1-098-14486-9 “Learning DevSecOps expertly weaves theory with real-life applications to provide a comprehensive yet accessible roadmap to this critical f ield in today’s software development arena. With emphasis on cultural shift and strategic planning, this book is indispensable in grasping the pivotal principles of successfully implementing DevSecOps in any organization.” —David Volm Application security engineer, FanDuel Group Suehring

978-1-098-14486-9 [LSI] Learning DevSecOps by Steve Suehring Copyright © 2024 Steve Suehring. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Simina Calin Development Editor: Melissa Potter Production Editors: Jonathon Owen and Clare Laylock Copyeditor: nSight, Inc. Proofreader: Piper Editorial Consulting, LLC Indexer: nSight, Inc. Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea May 2024: First Edition Revision History for the First Edition 2024-05-15: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781098144869 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Learning DevSecOps, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the author, and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii 1. The Need for DevSecOps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Developing Software 2 Developing Agility 4 Developing Broken Software 6 Operating in a Darkroom 7 Security as an Afterthought 8 Culture First 9 Processes over Tools 10 Promoting the Right Skills 10 DevSecOps as Process 11 The DevSecOps SDLC 13 Summary 15 2. Foundational Knowledge in 25 Pages or Less. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 The Command-Line Interface 18 Command Line Versus Terminal Versus Shell 18 Why Do I Need the Command Line? 19 Getting Started with the Command Line 20 Protocols: A High-Level Overview 20 Protocol Layers 21 Two Protocols Plus Another 22 Basic Internet Protocols 23 Data Security: Confidentiality, Integrity, and Availability 31 Development Overview for Scripting 33 Commands and Built-ins 34 Basic Programmatic Constructs: Variables, Data, and Data Types 34 iii

Making Decisions with Conditionals 35 Looping 38 Lists and Arrays 39 Summary 39 3. Integrating Security. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Integrating Security Practices 41 Implementing Least Privilege 42 Maintaining Confidentiality 44 Data in Flight 45 Data at Rest 48 Verifying Integrity 50 Checksums 50 Verifying Email 52 Providing Availability 53 Service-Level Agreements and Service-Level Objectives 54 Identifying Stakeholders 54 Identifying Availability Needs 54 Defining Availability and Estimating Costs 55 What About Accountability? 57 Site Reliability Engineering 57 Code Traceability and Static Analysis 59 Becoming Security Aware 61 Finding Formal Training 61 Obtaining Free Knowledge 62 Enlightenment Through Log Analysis 63 Practical Implementation: OWASP ZAP 63 Creating a Target 64 Installing ZAP 65 Getting Started with ZAP: Manual Scan 66 Summary 74 4. Managing Code and Testing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 Examining Development 77 Be Intentional and Deliberate 78 Don’t Repeat Yourself 78 Managing Source Code with Git 79 A Simple Setup for Git 79 Using Git (Briefly) 82 Branching and Merging 86 Examining the Gitflow Pattern 87 Examining the Trunk-Based Pattern 89 iv | Table of Contents

Testing Code 90 Unit Testing 90 Integration Testing 91 System Testing 91 Automating Tests 91 Summary 94 5. Moving Toward Deployment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 Managing Configuration as Code and Software Bill of Materials (SBOM) 97 Using Docker 101 Container and Image Concepts 102 Obtaining Images 103 Deploying Safely with Blue-Green Deployment 112 Summary 113 6. Deploy, Operate, and Monitor. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 Continuous Integration and Continuous Deployment 115 Building and Maintaining Environments with Ansible 116 Using Jenkins for Deployment 117 Creating a Pipeline 126 Monitoring 131 Summary 134 7. Plan and Expand. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 Scaling Up with Kubernetes 137 Understanding Basic Kubernetes Terms 138 Installing Kubernetes 138 Deploying with Kubernetes 144 Defining a Deployment 144 Defining a Service 147 Moving Toward Microservices 149 Connecting the Resources 150 Integrating Helm 153 Summary 154 8. Beyond DevSecOps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 DevSecOps Patterns 155 Shifting Left and Adding CI/CD 156 Multicloud Integration 156 Integrated and Automatic Security 156 Linux Everywhere 157 Refactor and Redeploy 157 Table of Contents | v

Summary 157 A. Ports and Protocols. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 B. Command Reference. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 vi | Table of Contents

Preface DevSecOps jobs are abundant, but looking at the requirements for those jobs, it’s quickly evident that there is no agreement on what DevSecOps actually entails. That’s what made this book quite difficult to write. I’ve written books on everything from MySQL to JavaScript to Windows Server to Linux Firewalls. Each of those technolo‐ gies has a well-defined scope. Writing on Linux firewalls does not require covering several different technologies and skills in the same book. But DevSecOps is not as well-defined. Writing on DevSecOps exposes the fissures in how we define technolo‐ gies, between the actual hands-on work and the hype. Even the term “DevSecOps” is not as widely used as the term “DevOps.” Granted, “DevSecOps” does not roll off the tongue as easily as “DevOps,” but it’s more than that. Simply lending a voice to the definition of DevSecOps is one of the reasons that I wrote this book. The goal of this book is not to be a comprehensive step-by-step guide to implement‐ ing DevSecOps, whatever the term means. That book is impossible to write because of the rapid changes in tools and the highly customized needs of each organization moving toward DevSecOps. Rather, the goal of this book is to provide patterns of success while also exposing some of the technologies and practices involved in large DevSecOps deployments. The book does not cover every software tool that an organization might use in DevSecOps. This is not an omission, or if it is, the omission is intentional so that the focus can remain on processes and people rather than technology and tools. Tech and tooling will change, but having the best people implementing the best processes will always work. What Is DevSecOps? What is DevSecOps? It depends on who you ask. As defined in this book, DevSecOps is a set of agile and iterative practices that help to deliver software and technology systems rapidly, accurately, and repeatedly, emphasizing processes and people above tools. vii

DevSecOps is about culture first. I’ve worked for organizations that were so far away from agile and iterative as to be spinning backward in the software development life‐ cycle. In such organizations, technologies are chosen by unqualified people without any consideration for workflow, productivity, or best practices, much less the end user. Deadlines are chosen before we even know what we’re building. Contrast, then, DevSecOps cultures, where testing and security are natural extensions within the development process rather than additions later on. Automation and scripting are heavily emphasized in DevOps and DevSecOps. Computing has a great way of reinventing itself over and over again. Many of the practices shown in this book have been around since the early days of computing. Mainframes allowed for slices of computing time and resources, and that’s what we do today with cloud provisioning, just on a grander scale. Many of the things we do today as modern DevSecOps practices have been around for decades on Linux. Scripting and automation is not new, but formalizing it and getting buy-in from everyone involved in an organization is the value brought by DevSecOps. That’s the essence of DevSecOps: enabling people to use processes and tools to rapidly and repeatedly improve the quality of software. Who Is This Book For? This book is for anyone interested in learning about DevSecOps and its predecessor, DevOps. You might be involved in development, operations, or security and want to learn about the melding of all three into a set of tools and processes for making production-level deployments easier. To get maximum value from the entire book, you should have a computing background, but everyone interested in DevSecOps will benefit from Chapter 1, even those without a computing background. Being able to write code, commit and push the code, and have tests automatically executed on that code is one such practice in DevSecOps. Scaling across multiple cloud providers is common as well. All of this is done seamlessly. Of course, all of that automation needs people who understand not only the goals of the automation but how to configure it. With that in mind, if you’re interested in learning about the pro‐ cesses involved in DevSecOps while also being exposed to some of the technologies involved, then this book should be helpful. How This Book Is Organized This book is organized into eight chapters. With a few exceptions, the chapters are largely standalone, meaning that you can read only the chapters or sections of chapters that you find valuable. Chapter 1, “The Need for DevSecOps”, helps to frame the story for the rest of the book. The chapter demonstrates how software was developed with methodologies viii | Preface

like Waterfall and Agile and how software is developed with DevSecOps. Chapter 1 also discusses the need to tear down departmental silos and places an emphasis on the importance of culture in DevSecOps. Chapter 2, “Foundational Knowledge in 25 Pages or Less”, condenses some of the most basic knowledge needed to be successful at DevSecOps—or at least lays the foundation for gaining such knowledge. If there was a way to condense weeks’ worth of computing course material into one place, this is hopefully that place. Chapter 3, “Integrating Security”, continues with some of the foundational material from Chapter 2 but with a focus on security. Building on that foundation, we’ll discuss the OWASP ZAP tool in Chapter 3. Chapter 4, “Managing Code and Testing”, looks at git and the Gitflow pattern in DevSecOps. The chapter also covers the various levels of testing. Chapter 5, “Moving Toward Deployment”, introduces management of configuration as code along with Docker. We’ll also build a local registry for Docker. Chapter 6, “Deploy, Operate, and Monitor”, examines Ansible and Jenkins for deployment and code building. Both of these technologies are widely used, though they’re certainly not the only technologies that perform deployment and build tasks. Chapter 6 also discusses monitoring, with an emphasis on best practices for monitoring. Chapter 7, “Plan and Expand”, integrates Kubernetes into the DevSecOps coverage, clustering and expanding the deployment of software in an organization. Chapter 8, “Beyond DevSecOps”, wraps up the book with coverage of five patterns and takeaways from successful DevSecOps organizations. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords. Constant width italic Shows text that should be replaced with user-supplied values or by values deter‐ mined by context. Preface | ix

This element signifies a tip or suggestion. This element signifies a general note. This element indicates a warning or caution. O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit https://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-889-8969 (in the United States or Canada) 707-827-7019 (international or local) 707-829-0104 (fax) support@oreilly.com https://www.oreilly.com/about/contact.html x | Preface

We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/LearningDevSecOps. For news and information about our books and courses, visit https://oreilly.com. Find us on LinkedIn: https://linkedin.com/company/oreilly-media. Watch us on YouTube: https://youtube.com/oreillymedia. Acknowledgments Thank you to the technical reviewers, Patrick Dubois, David Volm, Swapnil Shevate, and Vladislav Bilay, for their time, effort, and feedback. They helped identify areas that needed additional coverage and gave their expertise in other areas of the book. Thanks to Rob, Jim, and Jaclyn at Partners and to Tim for the assistance with review questions. Preface | xi

CHAPTER 1 The Need for DevSecOps Software is created to solve problems. However, too often, creating software comes with its own set of problems, sometimes even creating new problems along the way. An organization makes a decision whether to develop customized software or to purchase prebuilt software. The prebuilt option is most economical for commodity software like an office productivity suite. But custom development is often needed for development of advanced solutions in business functional areas. Custom solutions are created in pursuit of the ultimate goals of gaining competitive advantage or increasing efficiency. The process of developing software changed significantly in the late 1990s and into the early 2000s. That major shift went from an intense focus on gathering require‐ ments to a focus on iteration and speed. The iterative manner in which software is developed features repeatable processes and automation that enable rapid delivery of new features, incorporating feedback loops throughout the development lifecycle. Together with organizational cultural changes that promote an open source, transpar‐ ent mentality, the result is cross-functional teams concerned more with quality than territory and merging of multiple teams: Development, Operations, and Security— DevSecOps. This chapter looks at the drivers behind the DevSecOps movement. The process of software development is the initial focus. The evolution of software development methodologies provides the background needed to fully understand, and thus be successful at, DevSecOps. The chapter continues with an emphasis on the importance of cultural changes for organizations moving toward DevSecOps. 1

Developing Software To achieve their goals, organizations allocate some of their resources to create soft‐ ware. It’s important to consider that these resources could be invested elsewhere where the resources might gain a higher return. For example, investing $100,000 into marketing might result in more customers than investing those funds into streamlining the customer sign-up process on the website. Even if money is not a concern, speed is. The ability to create and then deploy software quickly is a limiting factor on any effort to gain competitive advantage or increase efficiency. After a certain point, adding more developers to a project does not get that project done any faster. Just the opposite. As more developers are added, coherent communication becomes impossible. Software starts as an idea. Taking that idea and turning it into working software requires forethought and planning. A software development project can be managed using several processes, depending in part on the type of software being developed. Software development involves defining the requirements, designing the solution, developing and coding, and finally testing the software just prior to release. This process is illustrated in Figure 1-1. Figure 1-1. A process for software development The four stages, sometimes called a software development lifecycle (SDLC), can be conceptualized as a waterfall, with each stage producing one or more artifacts, which are then passed or fall to the next stage, more like Figure 1-2. Figure 1-2. Completing each phase of a project in waterfall development 2 | Chapter 1: The Need for DevSecOps

When using a methodology like waterfall to create software, each stage is completed prior to moving on to the next stage. This is illustrated within Figure 1-1 where requirements are gathered and documented before moving on to the design phase, labeled “Design solution” in Figure 1-1. If a new requirement is discovered during the design phase or additional questions lead to new requirements, those elements are frequently added into a follow-on project. At the end of the requirements-gathering phase, the project formally has a scope defined, which includes all of the features of the software. These features incorporate the primary functions of the software along with additional features that aren’t tech‐ nically required for the software to function but are expected. These nonfunctional requirements are items like responsiveness or speed, security, and other behaviors of the application. Without capturing and adding the nonfunctional requirements, the resulting software product will leave users frustrated and underwhelmed. Consider a business requirement: enabling a customer to find a product and place an order. Prior to computers, this business requirement was fulfilled in any number of ways, including the customer walking into a store, finding, and then purchasing the product or using a catalog to find the product, calling the company, and placing the order via telephone. With computers and the internet, this business requirement is now frequently accomplished through the web. Fulfilling the business requirement of enabling a customer to find a product and place an order using a website leaves significant space to find a solution. Uploading a PDF of the catalog to the website and providing a form that enables the customer to email their order fulfills the minimal functional requirements for the site. However, even though the requirement is fulfilled, most users would expect something different and probably wouldn’t order with such a clumsy process that lacks many of the fea‐ tures that customers take for granted within the user experience of ordering products online. Instead, nonfunctional requirements also need to be captured. A few exploratory questions to the stakeholder or project sponsor would reveal rich detail about the intent for the solution. For example: • How will products be represented on the site (photos, narrative, technical specifi‐ cations, and so on)? • Who will take product photos and produce them for the web, and who will write the narrative production description? • How will inventory be updated so that customers can’t order products that are out of stock? • How will orders be placed? • How will employees be alerted when a new order is placed? Developing Software | 3

• Who will maintain the online catalog with new products? • What forms of payment are accepted? • Do customers need to create accounts, track order history, track shipping? These questions represent just a small fraction of the questions that would need to be answered during an initial exploratory or feasibility meeting. Some of these questions are already or will quickly become functional requirements during the feasibility phase or during the requirements-gathering phase. However, absent someone in the meeting who has deployed a project such as this, there would surely be missed requirements. The scope of the project, then, defines those elements that are included and delineates other elements that are not meant to be included within a project. Anything not specifically included is assumed to be excluded and thus out of scope for the project. If a fundamental requirement was missed, the project sponsor will face the unhappy choice of redefining scope or moving forward without that requirement and then adding the missed feature in a later follow-up project. Months or even years of calendar time can elapse between the idea and the imple‐ mentation. The delay between idea and released software product makes waiting for a missed feature even more painful for the project sponsor. Within that delay, any competitive advantage that might have been realized can quickly evaporate when a competitor who didn’t miss the requirement releases their own version. The following sections examine some of the problems and associated solutions sur‐ rounding modern software development. Developing Agility In response to the lag between project definition and completion, organizations have turned toward iterative processes like Agile and Scrum as a means to rapidly deliver value to the stakeholder. With an iterative software development process, all four stages described earlier (requirements, design, development, and testing) are performed. Rather than attempting to capture all requirements for all possible aspects of the project, iterative development focuses on the features that are of the highest value to the stakeholder. The highest-value features are then expanded through a round of requirements gathering before being designed, developed, tested, and released, with a short cycle of two to four weeks. Figure 1-3 shows how each phase of the SDLC is handled with an iterative process like Agile. 4 | Chapter 1: The Need for DevSecOps

Figure 1-3. Iterating through each phase and then starting over with an Agile-like process As illustrated in Figure 1-3, each phase is completed, but there is no attempt to gather full requirements because of the learning process associated with iterative development. If a requirement is missed, the stakeholder can choose to not release the feature or add the missed requirement in the next iteration. With an iterative development process, the next release is only weeks away rather than months or years away. Contrast that to a missed requirement in a waterfall process, where the next release may be months or years away, and you can see the clear benefit of this process. Iterative development also enables rapid response to changing market conditions. For example, you might have the best idea for the next killer app, start development on that app, but then have your competitor release essentially the same app. In a waterfall model, you would need to scrap the project entirely. With an iterative process, focus can be shifted toward features that might be missing from the competitor’s app. Agile software development features several ceremonies such as sprint planning, daily stand-up, sprint review, sprint retrospective, and backlog grooming. An overall backlog or list of all of the possible features known at a given moment is created and prioritized. From that prioritized list of features, a sprint backlog is created. The sprint backlog is a commitment from the development team of which features will be implemented during the current iteration. The sprint backlog is created based on availability of team members and their estimation of effort, also called level of effort (LOE), for each individual item on the backlog. At the end of the sprint, a sprint review is conducted where the team shows off what it has accomplished during that iteration. After the sprint review has been completed, the team examines what might have been done differently during the sprint within the retrospective. A team might answer three questions during the retrospective: • What should we start doing? • What should we stop doing? • What should we continue doing? Developing Software | 5

These three questions enable the team to reflect on what worked, what didn’t work, and what they might change moving into the next iteration. With the retrospective complete, the team can move toward backlog grooming, where the product backlog is refined and reprioritized. The stakeholder or product owner is usually involved in the backlog refinement process to set priority for the team. Developing Broken Software Flawed requirements lead to flawed software, or software that doesn’t meet the orig‐ inal requirement. Flawed software can happen regardless of whether that original requirement was successfully elicited from the project sponsor. The end result is dissatisfaction, broken functionality, and security problems. When examining the requirements, developers are often left with questions. These questions range from the mundane, such as where to place the curly braces for a conditional in some languages, to the critical, such as obtaining credentials for a database connection. In the latter case, development may need to stop while those credentials are obtained. In other cases, developers simply answer the question to the best of their ability and keep moving forward. Developing software in a silo, devoid of interaction with anyone other than develop‐ ers, leads to broken software. In the siloed development style, using a waterfall or similar methodology, the developers examine and interpret requirements to the best of their ability. Consider the following question: “In which web browsers should the site work?” along with a common answer: “Browsers? I’ve been developing using Chrome; I didn’t think about the site working in other browsers.” Figure 1-4 illustrates development in a silo, where developers, operations staff, and security engineers don’t communicate well. Figure 1-4. Siloed development in an organization leads to lack of visibility 6 | Chapter 1: The Need for DevSecOps

Deadlines dictate the number of features and the quality of those features. The deadline for delivery may be such that there is no time to even identify the issues that might occur when testing using a different browser or different viewport such as a phone, much less fix those issues. If cross-browser testing was not included as a step in the project and the browsers in which the site must work were not specified in the requirements, then it’s anyone’s guess as to which browsers the site will work in. Deadlines, or the timeline of the project, is one of the three levers that can be controlled within a software development project. The other two levers are cost and features. The adage is that a given project can choose two of the three, meaning that if the project needs to be done quickly and with many features, then costs will increase. Likewise, if a project needs many features but low costs, then completing the project will take longer. Finally, if costs must be kept as low as possible while still meeting the deadline, then features are the first thing to be sacrificed. Figure 1-5 illustrates the concept of the software development triangle. Figure 1-5. Choose two of the three elements at any one time The next problem I’ll address is the handoff between development and QA. Operating in a Darkroom Somewhere between development and testing lies an all-too-often awkward handoff between those who developed the software and those who are now charged with deploying, operating, and supporting the software in its production environment: the operations team. The operations team may be known by many names, including net‐ work administrators, system administrators, or engineering (site reliability engineer [SRE], production engineers, and the like), among others. The operations team needs to take software that may never have been tested on a computing environment like the one in production and run it according to the service-level agreement (SLA) needed by the organization. That software may have only been tested on developer workstations and then a small quality assurance (QA) environment. The QA environment may have an entirely different configuration—for example, it may be lacking a load balancer, may be deployed in a different region, Developing Software | 7