Podman in Action Secure, rootless containers for Kubernetes, microservices, and more (Daniel Walsh)（Z-Library）

M A N N I N G Daniel Walsh Secure, rootless containers for Kubernetes, microservices, and more

Podman and Docker feature comparison Feature Podman Docker Description Supports all OCI and Docker images ✔ ✔ Both pull and run container images from container regis- tries (i.e., quay.io and docker.io) Launches OCI container engines ✔ ✔ Launch containers using runc, crun, Kata, gVisor, and OCI container engines Simple command-line interface ✔ ✔ Podman and Docker share the same CLI. Integration with systemd ✔ ✘ Podman supports running systemd inside of the container as well as many systemd features. Fork/exec model ✔ ✘ The container is a direct descendant of the podman command. Fully support user name- space ✔ ✘ Only Podman supports running containers in separate user namespaces. Client–server model ✔ ✔ Docker is a RESTful API daemon. Podman supports REST- ful API via a systemd socket=activated service. Supports docker- compose ✔ ✔ compose scripts work against both restful APIs. Podman’s works in rootless mode. Supports docker-py ✔ ✔ docker-py python bindings work against both restful APIs. Podman’s works in rootless mode. Podman also supports podman-py for running advanced features. Daemonless ✔ ✘ The podman command runs like a traditional command-line tool, while Docker requires multiple root-running daemons. Supports Kubernetes-like pods ✔ ✘ Podman supports running multiple containers within the same pod. Supports Kubernetes yaml ✔ ✘ Podman can launch containers and pods based on Kuber- netes yaml. It can also generate Kuberenetes.yaml from running containers. Supports Docker swarm ✘ ✔ Podman believes the future for orchestrated multi-node containers is Kubernetes and does not plan on implement- ing Swarm. Customizable registries ✔ ✘ Podman allows you to configure registries for short name expansion. Docker is hard coded to docker.io when you specify a short name. Customizable defaults ✔ ✘ Podman supports fully customizing all of its defaults includ- ing security, namespaces, volumes, and more. Mac OS support ✔ ✔ Podman and Docker support running containers on a Mac via a VM running Linux. Windows support ✔ ✔ Podman and Docker support running containers on a Win- dows WSL2 or a VM running Linux. Linux support ✔ ✔ Podman and Docker are supported on all major Linux distri- butions.

Podman in Action SECURE, ROOTLESS CONTAINERS FOR KUBERNETES, MICROSERVICES, AND MORE DANIEL WALSH MANN I NG SHELTER ISLAND

For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact Special Sales Department Manning Publications Co. 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 Email: orders@manning.com ©2023 by Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps. Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine. The author and publisher have made every effort to ensure that the information in this book was correct at press time. The author and publisher do not assume and hereby disclaim any liability to any party for any loss, damage, or disruption caused by errors or omissions, whether such errors or omissions result from negligence, accident, or any other cause, or from any usage of the information herein. Manning Publications Co. Development editor: Toni Arritola 20 Baldwin Road Technical development editor: Joshua White PO Box 761 Technical editor: Roman Zhuzha Shelter Island, NY 11964 Review editor: Aleksandar Dragosavljević Production editor: Andy Marinkovich Copy editor: Christian Berk Proofreader: Katie Tennant Technical proofreader: Alain Lompo Typesetter: Dennis Dalinnik Cover designer: Marija Tudor ISBN: 9781633439689 Printed in the United States of America

In memory of my mother, Joan P. Walsh

v brief contents PART 1 FOUNDATIONS ..............................................................1 1 ■ Podman: A next-generation container engine 3 2 ■ Command line 27 3 ■ Volumes 67 4 ■ Pods 76 PART 2 DESIGN.......................................................................87 5 ■ Customization and configuration files 89 6 ■ Rootless containers 106 PART 3 ADVANCED TOPICS . ...................................................125 7 ■ Integration with systemd 127 8 ■ Working with Kubernetes 151 9 ■ Podman as a service 166 PART 4 CONTAINER SECURITY . ..............................................187 10 ■ Security container isolation 189 11 ■ Additional security considerations 216

contents preface xi acknowledgments xii about this book xiii about the author xvi about the cover illustration xvii PART 1 FOUNDATIONS ....................................................1 1 Podman: A next-generation container engine 3 1.1 About all these terms 4 1.2 A brief overview of containers 7 Container images: A new way to ship software 9 ■ Container images lead to microservices 11 ■ Container image format 13 Container standards 14 1.3 Why use Podman when you have Docker? 15 Why have only one way to run containers? 15 ■ Rootless containers 16 ■ Fork/exec model 17 ■ Podman is daemonless 19 ■ User-friendly command line 19 ■ Support for REST API 21 ■ Integration with systemd 21 ■ Pods 22 Customizable registries 23 ■ Multiple transports 25 ■ Complete customizability 25 ■ User-namespace support 26 1.4 When not to use Podman 26vi

CONTENTS vii2 Command line 27 2.1 Working with containers 28 Exploring containers 28 ■ Running the containerized application 30 ■ Stopping containers 34 ■ Starting containers 35 ■ Listing containers 36 ■ Inspecting containers 36 ■ Removing containers 37 ■ exec-ing into a container 38 ■ Creating an image from a container 39 2.2 Working with container images 41 Differences between a container and an image 42 ■ Listing images 44 ■ Inspecting images 45 ■ Pushing images 46 podman login: Logging into a container registry 48 ■ Tagging images 50 ■ Removing images 53 ■ Pulling images 55 Searching for images 58 ■ Mounting images 59 2.3 Building images 60 Format of a Containerfile or Dockerfile 61 ■ Automating the building of our application 64 3 Volumes 67 3.1 Using volumes with containers 68 Named volumes 70 ■ Volume mount options 72 ■ podman run - -mount command option 75 4 Pods 76 4.1 Running pods 76 4.2 Creating a pod 79 4.3 Adding a container to a pod 80 4.4 Starting a pod 82 4.5 Stopping a pod 83 4.6 Listing pods 84 4.7 Removing pods 84 PART 2 DESIGN .............................................................87 5 Customization and configuration files 89 5.1 Configuration files for storage 91 Storage location 91 ■ Storage drivers 94 5.2 Configuration files for registries 96 registries.conf 96

CONTENTSviii5.3 Configuration files for engines 100 5.4 System configuration files 104 6 Rootless containers 106 6.1 How does rootless Podman work? 109 Images contain content owned by multiple user identifiers (UIDs) 110 6.2 Rootless Podman under the covers 118 Pulling the image 119 ■ Creating a container 120 Setting up the network 120 ■ Starting the container monitor: conmon 121 ■ Launching the OCI runtime 121 ■ The containerized application runs until completion 124 PART 3 ADVANCED TOPICS...........................................125 7 Integration with systemd 127 7.1 Running systemd within a container 128 Containerized systemd requirements 131 ■ Podman container in systemd mode 131 ■ Running an Apache service within a systemd container 132 7.2 Journald for logging and events 134 Log driver 135 ■ Events 136 7.3 Starting containers at boot 137 Restarting containers 137 ■ Podman containers as systemd services 138 ■ Distributing systemd unit files to manage Podman containers 141 ■ Automatically updating Podman containers 142 7.4 Running containers in notify unit files 145 7.5 Rolling back failed containers after update 147 7.6 Socket-activated Podman containers 147 8 Working with Kubernetes 151 8.1 Kubernetes YAML files 153 8.2 Generating Kubernetes YAML files with Podman 153 8.3 Generating Podman pods and containers from Kubernetes YAML 157 Shutting down pods and containers based on a Kubernetes YAML file 158 ■ Building images using Podman and Kubernetes YAML files 159

CONTENTS ix8.4 Running Podman within a container 162 Running Podman within a Podman container 163 ■ Running Podman within a Kubernetes pod 164 9 Podman as a service 166 9.1 Introducing the Podman service 167 Systemd services 168 9.2 Podman-supported APIs 171 9.3 Python libraries for interacting with Podman 173 Using docker-py with the Podman API 174 ■ Using podman-py with the Podman API 175 ■ Which Python library should you use? 176 9.4 Using docker-compose with the Podman service 177 9.5 podman --remote 180 Local connections 180 ■ Remote connections 182 Setting up SSH on the client machine 184 ■ Configuring a connection 185 PART 4 CONTAINER SECURITY .....................................187 10 Security container isolation 189 10.1 Read-only Linux kernel pseudo filesystems 191 Unmasking the masked paths 192 ■ Masking additional paths 193 10.2 Linux capabilities 194 Dropped Linux capabilities 195 ■ Dropped CAP_SYS_ADMIN 196 Dropping capabilities 197 ■ Adding capabilities 197 ■ No new privileges 198 ■ Root with no capabilities is still dangerous 198 10.3 UID isolation: User namespace 198 Isolating containers using the - -userns=auto flag 199 ■ User- namespaced Linux capabilities 201 ■ Rootless Podman with the - -userns=auto flag 202 ■ User volumes with the - -userns=auto flag 202 10.4 Process isolation: PID namespace 204 10.5 Network isolation: Network namespace 205 10.6 IPC isolation: IPC namespace 206 10.7 Filesystem isolation: Mount namespace 206

CONTENTSx10.8 Filesystem isolation: SELinux 207 SELinux type enforcement 207 ■ SELinux Multi-Category Security separation 211 10.9 System call isolation seccomp 213 10.10 Virtual machine isolation 214 11 Additional security considerations 216 11.1 Daemon versus the fork/exec model 217 Access to the docker.sock 217 ■ Auditing and logging 218 11.2 Podman secret handling 220 11.3 Podman image trust 221 Podman image signing 224 11.4 Podman image scanning 228 Read-only containers 229 11.5 Security in depth 229 Podman uses all security mechanisms simultaneously 230 Where should you run your containers? 230 appendix A Podman-related container tools 232 appendix B OCI runtimes 246 appendix C Getting Podman 254 appendix D Contributing to Podman 259 appendix E Podman on macOS 262 appendix F Podman on Windows 269 index 281

preface I have been working on computer security for close to 40 years, and for the past 20 years, I’ve focused on container technologies. When Docker showed up about 10 years ago, it triggered a revolution in the way the people distributed and ran applications on the internet. As I worked on Docker, I felt it could have been designed better. Working with a root-running daemon and then adding more and more daemons felt like the wrong approach. Instead, I felt we could use low-level operating systems con- cepts to create a tool that ran the same containerized applications in the same man- ner but with more security and requiring fewer privileges. With this in mind, my team at Red Hat set out to build a series of tools to help developers and administrators run containers in the most secure way possible. Out of this effort came Podman. I started blogging on subjects like SELinux in the early 2000s and have been writ- ing articles ever since. I have written hundreds of articles on containers and security over the years, but I wanted to consolidate the ideas and describe the technology of Podman in a single book I could point users and customers to. This book introduces Podman and how to use it. It also dives deep into the tech- nology and the different parts of the Linux operating system that we take advantage of. Since I am a security engineer, I also spend a couple of chapters describing how the security of containers works. Reading this book should give you a better under- standing of what containers are, how they work, and how to work with different fea- tures of Podman. You will even learn a lot more about Docker. As Podman grows in popularity and infiltrates your infrastructure, this book will be a handy reference to guide your way.xi

acknowledgments I extend thanks to all the people who helped me write this book. This includes mem- bers of the Podman team, who have written articles that helped me understand some of the technology I did not fully comprehend and have helped build a great product. Thank you, Brent Baude, Matt Heon, Valentin Rothberg, Giuseppe Scrivano, Urvashi Mohnani, Nalin Dahyabhai, Lokesh Mandvekar, Miloslav Trmac, Jason Greene, Jhon Honce, Scott McCarty, Tom Sweeney, Ashley Cui, Ed Santiago, Chris Evich, Aditya Rajan, Paul Holzinger, Preethi Thomas, and Charlie Doern. I also want to thank the countless open source contributors who have made Linux containers and Podman possible. I thank the entire team at Manning, but especially Toni Arritola. Toni taught me how to better focus my ideas and has been a great partner on this journey. She’s had to deal with me, an old mathematics major who was never great at writing, and she helped make this book possible. To all the reviewers—Alain Lompo, Alessandro Campeis, Allan Makura, Amanda Debler, Anders Björklund, Andrea Monacchi, Camal Cakar, Clifford Thurber, Conor Redmond, David Paccoud, Deepak Sharma, Federico Kircheis, Frans Oilinki, Gowtham Sadasivam, Ibrahim Akkulak, James Liu, James Nyika, Jeremy Chen, Kent Spillner, Kevin Etienne, Kirill Shirinkin, Kosmas Chatzimichalis, Krzysztof Kamyczek, Larry Cai, Michael Bright, Mladen Knežić, Oliver Korten, Richard Meinsen, Roman Zhuzha, Rui Liu, Satadru Roy, Seung-jin Kim, Simeon Leyzerzon, Simone Sguazza, Syed Ahmed, Thomas Peklak, and Vivek Veerappan—thank you, your suggestions helped make this a better book.xii

about this book Podman in Action describes how users can build, manage, and run containers. My goal in writing it was to explain how easy it is to transfer skills you might have learned in Docker to Podman as well as how easy it is to use Podman if you have never used a container engine before. Podman in Action also teaches you how to use advanced fea- tures like pods and guides you on your journey toward building applications ready to run on the edge of or inside Kubernetes. Finally, Podman in Action explains all of the security features of the Linux kernel used to isolate containers from the system as well as from other containers. Who should read this book? Podman in Action is written for software developers who are looking to understand, develop, and work with containers, as well as system administrators who need to run containers in production. Reading this book will give you a deeper understanding of what containers are. Having knowledge of Linux processes and familiarity working with the Linux shells is necessary to get the full benefit of the book. The book should have something for everyone on their quest to use containers. Users with a deep understanding of Docker will learn about advanced features of Podman not available from Docker and will get an even deeper understanding of how Docker works. Novice users will learn the basics of containers and pods. xiii

ABOUT THIS BOOKxivHow this book is organized: A roadmap Podman in Action is split into four parts and six appendixes: Part 1, “Foundations,” comprises four chapters and provides readers an introduc- tion to Podman. Chapter 1 explains what Podman does, why it was created, and why it is important. The next two chapters introduce the command-line interface and how to use volumes within containers. Finally, chapter 4 introduces the con- cept of pods and how Podman works with them. There should be something for everyone in these chapters, but if you have great experience with Docker, you should be able to skim over much of the content in chapter 2. Part 2, “Design,” comprises two chapters in which I dig deep into Podman’s design. You will learn about rootless containers and how they work and will come out of these chapters with a better understanding of user namespaces and the security of rootless containers. You will also learn how to customize the con- figuration of your Podman environment. Part 3, “Advanced topics,” comprises three chapters and moves beyond the basics of Podman. In chapter 7 you will see how Podman can work in produc- tion through its integration with systemd. It covers running systemd inside a container and how you can use it as a container manager. You will learn how to set up edge servers with Podman containers, where systemd manages the life cycle of the container. Podman makes it easy to generate systemd unit files to help you put your containerized applications into production. In chapter 8 you will learn how Podman can be used to help you move contain- ers into Kubernetes. Podman supports launching containers with the same YAML files that Kubernetes uses as well as the ability to generate Kubernetes YAML from your current containers. In chapter 9 you will see Podman run- ning as a service, allowing remote access to Podman containers. Using Pod- man as a service allows you to use other programming languages and tools to manage Podman containers. You will see how docker-compose can work with Podman containers. You will also learn how to use the Python libraries like podman-py and docker-py to communicate with the Podman service for man- aging containers. Part 4, “Container security,” comprises two chapters, in which I discuss import- ant security considerations. Chapter 10 covers features used to ensure con- tainer isolation. This chapter covers security subsystems of Linux, like SELinux, seccomp, Linux capabilities, kernel file systems, and namespaces. Chapter 11 then examines the security considerations I consider best practices for running your containers in as secure a manner as possible. Additionally, there are six appendixes covering Podman-related subjects: Appendix A covers all of the Podman-related tools, including Buildah, Skopeo, and CRI-O.

ABOUT THIS BOOK xv Appendix B dives into the different OCI runtimes available to Podman as well as Docker. It covers runc, crun, Kata, and gVisor. Appendix C describes how you can get Podman onto your local system, whether that system is a Linux, Mac, or Windows box. Appendix D describes the Podman open source community and how you can join. Appendixes E and F dive into running Podman on Mac and Windows boxes. liveBook discussion forum Each purchase of Podman in Action includes free access to liveBook, Manning’s online reading platform. Using liveBook’s exclusive discussion features, you can attach com- ments to the book globally or to specific sections or paragraphs. It’s a snap to make notes for yourself, ask and answer technical questions, and receive help from the author and other users. To access the forum, go to https://livebook.manning.com/ book/podman-in-action/discussion. You can also learn more about Manning's forums and the rules of conduct at https://livebook.manning.com/discussion. Manning’s commitment to our readers is to provide a venue where a meaningful dialogue between individual readers and between readers and the author can take place. It is not a commitment to any specific amount of participation on the part of the author, whose contribution to the forum remains voluntary (and unpaid). We sug- gest you try asking him some challenging questions lest his interest stray! The forum and the archives of previous discussions will be accessible from the publisher’s website as long as the book is in print. Author online You can follow Dan Walsh on Twitter and GitHub @rhatdan. He regularly blogs at https://www.redhat.com/sysadmin/users/dwalsh as well as on several other sites. There are many videos of talks Dan has presented available on YouTube as well.

about the author DANIEL WALSH leads the team that created Podman, Buildah, Skopeo, CRI-O, and their related tools. Dan is a senior distin- guished engineer at Red Hat, having joined in August 2001. He has worked in the computer security field for over 40 years. Dan is sometimes referred to as Mr. SELinux after leading the devel- opment of SELinux at Red Hat prior to leading the container team. Dan has a BA in mathematics from the College of the Holy Cross and an MS in computer science from Worcester Polytechnic Institute. On Twitter and GitHub you can find him @rhatdan. You can email him at dwalsh@redhat.com.xvi

about the cover illustration The figure on the cover of Podman in Action is captioned “La vandale,” or “The van- dal,” and is taken from a collection by Jacques Grasset de Saint-Sauveur, published in 1797. Each illustration is finely drawn and colored by hand. In those days, it was easy to identify where people lived and what their trade or sta- tion in life was just by their dress. Manning celebrates the inventiveness and initiative of the computer business with book covers based on the rich diversity of regional cul- ture centuries ago, brought back to life by pictures from collections such as this one.xvii