Container Security Fundamental Technology Concepts that Protect Containerized Applications (Liz Rice) (Z-Library)

Liz Rice Container Security Fundamental Technology Concepts that Protect Containerized Applications Boston Farnham Sebastopol TokyoBeijing

978-1-492-05670-6 [LSI] Container Security by Liz Rice Copyright © 2020 Vertical Shift Ltd. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: John Devins Developmental Editor: Virginia Wilson Production Editor: Nan Barber Copyeditor: Arthur Johnson Proofreader: Kim Wimpsett Indexer: Devon Thomas Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Rebecca Demarest April 2020: First Edition Revision History for the First Edition 2020-04-03: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781492056706 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Container Security, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the authors and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. This work is part of a collaboration between O’Reilly and Aqua Security. See our statement of editorial independence.

Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix 1. Container Security Threats. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Risks, Threats, and Mitigations 2 Container Threat Model 3 Security Boundaries 6 Multitenancy 7 Shared Machines 8 Virtualization 8 Container Multitenancy 9 Container Instances 10 Security Principles 10 Least Privilege 10 Defense in Depth 11 Reducing the Attack Surface 11 Limiting the Blast Radius 11 Segregation of Duties 11 Applying Security Principles with Containers 11 Summary 12 2. Linux System Calls, Permissions, and Capabilities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 System Calls 13 File Permissions 14 setuid and setgid 16 Linux Capabilities 19 Privilege Escalation 21 Summary 22 iii

3. Control Groups. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Cgroup Hierarchies 23 Creating Cgroups 24 Setting Resource Limits 26 Assigning a Process to a Cgroup 27 Docker Using Cgroups 28 Cgroups V2 29 Summary 30 4. Container Isolation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Linux Namespaces 32 Isolating the Hostname 33 Isolating Process IDs 35 Changing the Root Directory 38 Combine Namespacing and Changing the Root 41 Mount Namespace 42 Network Namespace 43 User Namespace 45 User Namespace Restrictions in Docker 48 Inter-process Communications Namespace 48 Cgroup Namespace 49 Container Processes from the Host Perspective 50 Container Host Machines 52 Summary 53 5. Virtual Machines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 Booting Up a Machine 55 Enter the VMM 57 Type 1 VMMs, or Hypervisors 57 Type 2 VMM 58 Kernel-Based Virtual Machines 59 Trap-and-Emulate 59 Handling Non-Virtualizable Instructions 60 Process Isolation and Security 61 Disadvantages of Virtual Machines 62 Container Isolation Compared to VM Isolation 63 Summary 63 6. Container Images. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 Root Filesystem and Image Configuration 65 Overriding Config at Runtime 66 iv | Table of Contents

OCI Standards 66 Image Configuration 67 Building Images 68 The Dangers of docker build 68 Daemonless Builds 69 Image Layers 69 Storing Images 71 Identifying Images 72 Image Security 73 Build-Time Security 74 Provenance of the Dockerfile 74 Dockerfile Best Practices for Security 75 Attacks on the Build Machine 77 Image Storage Security 77 Running Your Own Registry 77 Signing Images 78 Image Deployment Security 78 Deploying the Right Image 78 Malicious Deployment Definition 79 Admission Control 79 GitOps and Deployment Security 80 Summary 80 7. Software Vulnerabilities in Images. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 Vulnerability Research 83 Vulnerabilities, Patches, and Distributions 84 Application-Level Vulnerabilities 85 Vulnerability Risk Management 85 Vulnerability Scanning 85 Installed Packages 86 Container Image Scanning 87 Immutable Containers 87 Regular Scanning 88 Scanning Tools 89 Sources of Information 89 Out-of-Date Sources 89 Won’t Fix Vulnerabilities 89 Subpackage Vulnerabilities 90 Package Name Differences 90 Additional Scanning Features 90 Scanner Errors 90 Table of Contents | v

Scanning in the CI/CD Pipeline 91 Prevent Vulnerable Images from Running 93 Zero-Day Vulnerabilities 93 Summary 94 8. Strengthening Container Isolation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 Seccomp 95 AppArmor 97 SELinux 98 gVisor 100 Kata Containers 102 Firecracker 103 Unikernels 103 Summary 104 9. Breaking Container Isolation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 Containers Run as Root by Default 105 Override the User ID 106 Root Requirement Inside Containers 107 Rootless Containers 109 The --privileged Flag and Capabilities 111 Mounting Sensitive Directories 113 Mounting the Docker Socket 114 Sharing Namespaces Between a Container and Its Host 115 Sidecar Containers 115 Summary 116 10. Container Network Security. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117 Container Firewalls 117 OSI Networking Model 119 Sending an IP Packet 120 IP Addresses for Containers 121 Network Isolation 122 Layer 3/4 Routing and Rules 123 iptables 123 IPVS 125 Network Policies 125 Network Policy Solutions 127 Network Policy Best Practices 128 Service Mesh 129 Summary 130 vi | Table of Contents

11. Securely Connecting Components with TLS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 Secure Connections 131 X.509 Certificates 132 Public/Private Key Pairs 133 Certificate Authorities 134 Certificate Signing Requests 136 TLS Connections 136 Secure Connections Between Containers 138 Certificate Revocation 138 Summary 139 12. Passing Secrets to Containers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 Secret Properties 141 Getting Information into a Container 142 Storing the Secret in the Container Image 143 Passing the Secret Over the Network 144 Passing Secrets in Environment Variables 144 Passing Secrets Through Files 145 Kubernetes Secrets 145 Secrets Are Accessible by Root 146 Summary 148 13. Container Runtime Protection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 Container Image Profiles 149 Network Traffic Profiles 150 Executable Profiles 150 File Access Profiles 152 User ID Profiles 152 Other Runtime Profiles 153 Container Security Tools 153 Drift Prevention 155 Summary 156 14. Containers and the OWASP Top 10. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 Injection 157 Broken Authentication 157 Sensitive Data Exposure 158 XML External Entities 158 Broken Access Control 158 Security Misconfiguration 159 Cross-Site Scripting XSS 159 Table of Contents | vii

Insecure Deserialization 159 Using Components with Known Vulnerabilities 160 Insufficient Logging and Monitoring 160 Summary 161 Conclusions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 Appendix. Security Checklist. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 viii | Table of Contents

Preface Many organizations are running applications in cloud native environments, using containers and orchestration to facilitate scalability and resilience. If you’re a member of the Operations, the DevOps, or even the DevSecOps team setting up these envi‐ ronments for your company, how do you know whether your deployments are secure? If you’re a security professional with experience in traditional server-based or virtual machine–based systems, how can you adapt your existing knowledge for container-based deployments? And as a developer in the cloud native world, what do you need to think about to improve the security of your containerized applications? This book delves into some of the key underlying technologies that containers and cloud native rely on, to leave you better equipped to assess the security risks and potential solutions applicable to your environment and to help you avoid falling into bad practices that will leave your technology deployments exposed. In this book you will learn about many of the building block technologies and mecha‐ nisms that are commonly used in container-based systems, and how they are con‐ structed in the Linux operating system. Together we will dive deep into the underpinnings of how containers work and how they communicate so that you are well versed not just in the “what” of container security but also, and more impor‐ tantly, in the “why.” My goal in writing this book is to help you better understand what’s happening when you deploy containers. I want to encourage you to build men‐ tal models that allow you to make your own assessment of potential security risks that could affect your deployments. This book primarily considers the kind of “application containers” that many busi‐ nesses are using these days to run their business applications in systems such as Kubernetes and Docker. This is in contrast to “system containers” such as LXC and LXD from the Linux Containers Project. In an application container, you are encour‐ aged to run immutable containers with as little code as is necessary to run the appli‐ cation, whereas in a system container environment the idea is to run an entire Linux distribution and treat it more like a virtual machine. It’s considered perfectly normal to SSH into a system container, but application container security experts will look at ix

you askance if you want to SSH into an application container (for reasons covered later in this book). However, the basic mechanisms used to create application and sys‐ tem containers alike are control groups, namespaces, and changing the root directory, so this book will give you a solid foundation from which you may wish to explore the differences in approach taken by the different container projects. Who This Book Is For Whether you consider yourself a developer, a security professional, an operator, or a manager, this book will suit you best if you like to get into the nitty-gritty of how things work, and if you enjoy time spent in a Linux terminal. If you are looking for an instruction manual that gives a step-by-step guide to secur‐ ing containers, this may not be the book for you. I don’t believe there is a one-size- fits-all approach that would work for every application in every environment and every organization. Instead, I want to help you understand what is happening when you run applications in containers, and how different security mechanisms work, so that you can judge the risks for yourself. As you’ll find out later in this book, containers are made with a combination of fea‐ tures from the Linux kernel. Securing containers involves using a lot of the same mechanisms as you would use on a Linux host. (I use the term “host” to cover both virtual machines and bare-metal servers.) I lay out how these mechanisms work and then show how they apply in containers. If you are an experienced system adminis‐ trator, you’ll be able to skip over some sections to get to the container-specific information. I assume that you have some basic familiarity with containers, and you have probably at least toyed with Docker or Kubernetes. You will understand terms like “pulling a container image from a registry” or “running a container” even if you don’t know exactly what is happening under the covers when you take these actions. I don’t expect you to know the details of how containers work—at least, not until you have read the book. What This Book Covers We’ll start in Chapter 1 by considering threat models and attack vectors that affect container deployments, and the aspects that differentiate container security from tra‐ ditional deployment security. The remainder of the book is concerned with helping you build a thorough understanding of containers and these container-specific threats, and with how you can defend against them. Before you can really think about how to secure containers, you’ll need to know how they work. Chapter 2 sets the scene by describing some core Linux mechanisms such x | Preface

as system calls and capabilities that will come into play when we use containers. Then in Chapters 3 and 4, we’ll delve into the Linux constructs that containers are made from. This will give you an understanding of what containers really are and of the extent to which they are isolated from each other. We’ll compare this with virtual machine isolation in Chapter 5. In Chapter 6 you’ll learn about the contents of container images and consider some best practices for building them securely. Chapter 7 addresses the need to identify container images with known software vulnerabilities. In Chapter 8 we will look at some optional Linux security measures that can be applied to harden containers beyond the basic implementation we saw in Chapter 4. We will look into ways that container isolation can be compromised through danger‐ ous but commonplace misconfigurations in Chapter 9. Then we will turn to the communications between containers. Chapter 10 looks at how containers communicate and explores ways to leverage the connections between them to improve security. Chapter 11 explains the basics of keys and certificates, which containerized components can use to identify each other and set up secure net‐ work connections between themselves. This is no different for containers than it is for any other component, but this topic is included since keys and certificates are often a source of confusion in distributed systems. In Chapter 12 we will see how cer‐ tificates and other credentials can be safely (or not so safely) passed to containers at runtime. In Chapter 13 we will consider ways in which security tooling can prevent attacks at runtime, taking advantage of the features of containers. Finally, Chapter 14 reviews the top 10 security risks published by the Open Web Application Security Project and considers container-specific approaches for address‐ ing them. Spoiler alert: some of the top security risks are addressed in exactly the same way whether your application is containerized or not. A Note about Kubernetes These days the majority of folks using containers are doing so under the Kubernetes orchestrator. An orchestrator automates the process of running different workloads in a cluster of machines, and there are places in this book where I will assume that you have a basic grasp of this concept. In general, I have tried to stay focused on con‐ cepts that act at the level of the underlying containers—the “data plane” in a Kuber‐ netes deployment. Because Kubernetes workloads run in containers, this book is relevant to Kubernetes security, but it is not a comprehensive treatment of everything related to securing Kubernetes or cloud native deployments. There are many other concerns around the Preface | xi

configuration and use of the control plane components that are outside the scope of this book. If you would like more on this topic, you might be interested in the O’Reilly Kubernetes Security report (which I coauthored with Michael Hausenblas). Examples There are lots of examples in this book, and I encourage you to try them out for yourself. In the examples I assume that you are comfortable with basic Linux command-line tools like ps and grep, and with the basic day-to-day activities of running container applications through the use of tools like kubectl or docker. This book will use the former set of tools to explain a lot more about what’s happening when you use the latter! To follow along with the examples in this book, you will need access to a Linux machine or virtual machine. I created the examples using an Ubuntu 19.04 virtual machine running under VirtualBox on my Mac; I also use Vagrant to create, start, and stop my virtual machines. You should be able to achieve similar results on differ‐ ent Linux distributions and using virtual machines from your favorite cloud provider. How to Run Containers For many people, their main (perhaps only) experience of running containers directly is with Docker. Docker democratized the use of containers by providing a set of tools that developers generally found easy to use. From a terminal, you manipulate con‐ tainers and container images using the docker command. This docker tool is really a thin layer making API calls to Docker’s main component: a daemon that does all the hard work. Within the daemon is a component called containerd that is invoked whenever you want to run a container. The containerd component makes sure that the container image you want to run is in place, and it then calls a runc component to do the business of actually instantiating a container. If you want to, you can run a container yourself by invoking containerd or even runc directly. The containerd project was donated by Docker to the Cloud Native Com‐ puting Foundation (CNCF) back in 2017. Kubernetes uses an interface called the Container Runtime Interface (CRI) beneath which users can opt for a container runtime of their choice. The most commonly used options today are the aforementioned containerd and CRI-O (which originated from Red Hat before being donated to the CNCF). The docker CLI is just one option for managing containers and images. There are several others you can use to run the kind of application containers covered in this xii | Preface

book. Red Hat’s podman tool, originally conceived to remove reliance on a daemon component, is one such option. The examples in this book use a variety of different container tools to illustrate that there are multiple container implementations that share many common features. Feedback There is a website at containersecurity.tech to accompany this book. You are invited to raise issues there with feedback and any corrections that you’d like to see in future editions. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program ele‐ ments such as variable or function names, databases, data types, environment variables, statements, and keywords. Constant width bold Shows commands or other text that should be typed literally by the user. Constant width italic Shows text that should be replaced with user-supplied values or by values deter‐ mined by context. This element signifies a general note. Using Code Examples Supplemental material (code examples, exercises, etc.) is available for download at https://containersecurity.tech. If you have a technical question or a problem using the code examples, please send email to bookquestions@oreilly.com. Preface | xiii

This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require per‐ mission. We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Container Security by Liz Rice (O’Reilly). Copyright 2020 Vertical Shift Ltd., 978-1-492-05670-6.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, please visit http:// oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/container-security. xiv | Preface

Email bookquestions@oreilly.com to comment or ask technical questions about this book. For more information about our books, courses, and news, see our website at http:// www.oreilly.com. Find us on Facebook: http://facebook.com/oreilly Follow us on Twitter: http://twitter.com/oreillymedia Watch us on YouTube: http://www.youtube.com/oreillymedia Acknowledgments I’m grateful to many people who have helped and supported me through the process of writing this book. • My editor at O’Reilly, Virginia Wilson, for keeping everything on track and mak‐ ing sure the book is up to scratch. • The technical reviewers who provided thoughtful comments and actionable feed‐ back: Akhil Behl, Alex Pollitt, Andrew Martin, Erik St. Martin, Phil Estes, Rani Osnat, and Robert P. J. Day. • My colleagues at Aqua Security who taught me so much about container security over the years. • Phil Pearl—my husband, my best critic and coach, and my best friend. Preface | xv

CHAPTER 1 Container Security Threats In the last few years, the use of containers has exploded. The concepts around con‐ tainers existed for several years before Docker, but most observers agree that it was Docker’s easy-to-use command-line tools that started to popularize containers among the developer community from its launch in 2013. Containers bring many advantages: as described in Docker’s original tagline, they allow you to “build once, run anywhere.” They do this by bundling together an appli‐ cation and all its dependencies and isolating the application from the rest of the machine it’s running on. The containerized application has everything it needs, and it is easy to package up as a container image that will run the same on my laptop and yours, or in a server in a data center. A knock-on effect of this isolation is that you can run multiple different containers side by side without them interfering with each other. Before containers, you could easily end up with a dependency nightmare where two applications required different versions of the same packages. The easiest solution to this problem was simply to run the applications on separate machines. With containers, the dependencies are isolated from each other so it becomes straightforward to run multiple apps on the same server. People quickly realized that they could take advantage of containerization to run multiple applications on the same host (whether it’s a virtual machine or a bare- metal server) without having to worry about dependencies. The next logical step was to spread containerized applications across a cluster of servers. Orchestrators such as Kubernetes automate this process so that you no longer have to manually install an app on a particular machine; you tell the orchestrator what containers you want to run, and it finds a suitable location for each one. 1

From a security perspective, many things are the same in a containerized environ‐ ment as they are in a traditional deployment. There are attackers out in the world who want to steal data, or modify the way a system behaves, or use other people’s compute resources to mine their own cryptocurrencies. This doesn’t change when you move to containers. However, containers do change a lot about the way that applications run, and there are a different set of risks as a result. Risks, Threats, and Mitigations A risk is a potential problem, and the effects of that problem if it were to occur. A threat is a path to that risk occurring. A mitigation is a countermeasure against a threat—something you can do to prevent the threat or at least reduce the likelihood of its success. For example, there is a risk that someone could steal your car keys from your house and thus drive off in your car. The threats would be the different ways they might steal the keys: breaking a window to reach in and pick them up; putting a fishing rod through your letter box; knocking on your door and distracting you while an accom‐ plice slips in quickly to grab the keys. A mitigation for all these threats might be to keep your car keys out of sight. Risks vary greatly from one organization to another. For a bank holding money on behalf of customers, the biggest risk is almost certainly that money being stolen. An ecommerce organization will worry about the risks of fraudulent transactions. An individual running a personal blog site might fear someone breaking in to imperso‐ nate them and post inappropriate comments. Because privacy regulations differ between nations, the risk of leaking customers’ personal data varies with geography— in many countries the risk is “only” reputational, while in Europe the General Data Protection Regulation (GDPR) allows for fines of up to 4% of a company’s total revenue. Because the risks vary greatly, the relative importance of different threats will also vary, as will the appropriate set of mitigations. A risk management framework is a process for thinking about risks in a systematic way, enumerating the possible threats, prioritizing their importance, and defining an approach to mitigation. Threat modeling is a process of identifying and enumerating the potential threats to a system. By systematically looking at the system’s components and the possible modes of attack, a threat model can help you identify where your system is most vulnerable to attack. 2 | Chapter 1: Container Security Threats