Podman for DevOps Build secure, rootless containers, and integrate them into real DevOps and AI workflows (Alessandro Arrichiello, Gianni Salinetti) (z-library.sk, 1lib.sk, z-lib.sk)

Podman for DevOps Build secure, rootless containers, and integrate them into real DevOps and AI workfl ows P od m an for D evO ps SECOND EDITION Alessandro Arrichiello Gianni Salinetti A lessan d ro A rrich iello G ian n i Salin etti • Understand the fundamentals and history of container technology • Compare Podman and Docker to choose the right container engine • Run and manage containers on various Linux environments using Podman • Build containers from scratch using Buildah and manage them with Skopeo • Secure containers using rootless execution, SELinux, and image signing • Troubleshoot and monitor containers using system tools and the Podman CLI • Connect containers with advanced networking and integrate them with systemd • Manage containers and explore AI/ML use cases with Podman Desktop WHAT YOU WILL LEARN Containers are transforming how modern applications are built, deployed, and scaled. Podman offers a powerful, secure alternative to Docker by eliminating the daemon and embracing rootless container execution. If you’re ready to move beyond legacy workfl ows and gain full control over container management, this practical guide is for you. You’ll begin with container fundamentals and a side-by-side comparison of Docker and Podman to ease the transition. Then, you’ll run your fi rst container, manage its lifecycle, and use Buildah to build images from scratch. You will use Skopeo to transfer and inspect images across registries. As you progress, you’ll secure your environment with rootless containers, signed images, and SELinux policies. You’ll also confi gure container networking, integrate workloads with systemd services, and troubleshoot issues using native Linux tools. The fi nal chapters focus on modern developer workfl ows, showing how to migrate existing Docker workloads, manage containers and Kubernetes resources visually using Podman Desktop, and leverage Podman AI Lab to experiment with, run, and manage AI/ML models locally in a containerized environment. By the end of this book, you’ll be able to build, run, and secure containers, automate workfl ows, and confi dently manage deployments across DevOps and AI-powered environments. Podman for DevOps www.packtpub.com Get a free PDF of this book packtpub.com/unlock/9781835886625 SECOND EDITION

Podman for DevOps Second Edition Build secure, rootless containers, and integrate them into real DevOps and AI workflows Alessandro Arrichiello Gianni Salinetti

Podman for DevOps Second Edition Copyright © 2025 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. Portfolio Director: Kartikey Pandey Relationship Lead: Aaron Tanna Project Manager: Sonam Pandey Content Engineer: Arun Nadar Technical Editor: Simran Ali Copy Editor: Safis Editing Indexer: Tejal Soni Proofreader: Arun Nadar Production Designer: Jyoti Kadam Growth Lead: Shreyans Singh First published: April 2022 Second edition: February 2025 Production reference: 1250226 Published by Packt Publishing Ltd. Grosvenor House 11 St Paul's Square Birmingham B3 1RB, UK. ISBN 978-1-83588-662-5 www.packtpub.com

Contributors About the authors Alessandro Arrichiello is a senior specialist solution architect at Red Hat, within the EMEA Telco Center of Excellence. A dedicated free and open source software (FOSS) enthusiast, his journey with GNU/Linux systems began at the age of 14, evolving into a career focused on designing and architecting complex enterprise systems. With over a decade of experience at Red Hat, Alessandro has held various roles, from platform consultant to solution architect, specializing in OpenShift, Kubernetes, and AppDev for major telecommunications accounts. Alessandro is a prolific contributor to the open source community, regularly authoring articles on container architecture and technology for the Red Hat Developer blog. He is a seasoned public speaker who has presented at major industry events, including Devconf.cz, on topics such as OpenShift security and hybrid cloud management. Gianni Salinetti is an associate principal account solution architect at Red Hat, specializing in cloud-native computing and hybrid cloud strategies. Since beginning his journey with GNU/ Linux in 2001, Gianni has built a career rooted in a deep passion for open source software. His current expertise spans cloud-native applications, automation, platform engineering, and systems performance tuning, with a dedicated focus on the evolving field of AI infrastructure design. A former award-winning Red Hat instructor, Gianni was recognized as the best EMEA instructor for DevOps, cloud, and middleware. He has trained hundreds of professionals on Ansible, Kubernetes, and Red Hat OpenShift. Beyond architecture, Gianni has contributed to the Red Hat Developer blog and is a regular speaker at webinars and industry events, where he shares insights on the future of platform engineering.

About the reviewer Matthew Heon is a principal software engineer for Red Hat with a focus on containerization technology, with a particular focus on container networking and security. He began working on Linux containers in 2014 as a Docker contributor before becoming one of the original authors of Podman in 2017. After almost 9 years of work on Podman, he now leads Red Hat's Container Tools team and is a core maintainer of the Podman, Buildah, and Skopeo projects. He currently focuses on increasing Podman's adoption in environments such as the financial sector, automotive usage, and disconnected edge deployments.

Table of Contents Preface xvii Free Benefits with Your Book ............................................................................ xxii Part 1: From Theory to Practice: Let's Run Our First Container with Podman 1 Chapter 1: Introduction to Container Technology 3 Technical requirements ........................................................................................ 3 Book conventions ................................................................................................. 4 What are containers? ............................................................................................ 4 Resource usage with cgroups • 7 Running isolated processes • 8 Isolating mounts • 9 Container images to the rescue • 9 Security considerations • 12 Container engines and runtimes • 13 Containers versus virtual machines • 15 Virtual machines in containers • 17 Bootable containers • 18 Why do I need a container? .................................................................................. 19 Open source • 19 Portability • 19 DevOps facilitators • 20 Cloud readiness • 20 Infrastructure optimization • 21 Microservices • 21 AI/ML • 22 Where do containers come from? ........................................................................ 22

Chroot and Unix v7 • 22 FreeBSD jails • 23 Solaris Containers (also known as Solaris Zones) • 24 Linux Containers (LXC) • 25 Linux namespaces • 25 Mandatory access control • 25 Control groups • 26 Docker • 26 rkt • 27 OCI and CRI-O • 28 Podman • 29 Where are containers used today? ....................................................................... 30 Summary ........................................................................................................... 34 Further reading .................................................................................................. 34 Chapter 2: Comparing Podman and Docker 37 Technical requirements ...................................................................................... 37 Docker container daemon architecture ............................................................... 38 The Docker daemon • 39 Interacting with the Docker daemon • 40 The Docker REST API • 40 Docker client commands • 42 Docker images • 43 Docker registries • 44 What does a running Docker architecture look like? • 45 containerd architecture • 47 Podman daemonless architecture ....................................................................... 49 Podman commands and REST API • 49 Podman building blocks • 51 The libpod library • 52 The runc and crun OCI container runtimes • 53 Conmon • 56 Table of Contents vi

Rootless containers • 57 OCI images • 58 Manifest • 58 Image index • 59 Image layout • 59 Filesystem layers • 59 Image configuration • 59 The main differences between Docker and Podman ............................................. 63 Command-line interface comparison • 64 Running a container • 66 Summary ........................................................................................................... 68 Further reading .................................................................................................. 69 Chapter 3: Running the First Container 71 Technical requirements ....................................................................................... 71 Choosing an operating system and installation method ....................................... 72 Choosing between Linux distributions and another OS • 72 Installing Podman on Fedora • 74 Installing Podman on CentOS Stream • 74 Installing Podman on RHEL • 74 Installing Podman on Rocky Linux • 75 (Not) Installing Podman on Fedora CoreOS and Fedora Silverblue • 76 Installing Podman on Debian • 76 Installing Podman on Ubuntu • 76 Installing Podman on openSUSE • 77 Installing Podman on Gentoo • 77 Installing Podman on Arch Linux • 77 Installing Podman on Raspberry Pi OS • 77 Installing Podman on macOS • 78 Installing Podman on Windows • 79 Building Podman from source • 80 Preparing your environment ............................................................................... 81 vii Table of Contents

Customizing the container registries search list • 82 Optional – enable socket-based services • 82 Optional – customize Podman's behavior • 85 Running your first container ............................................................................... 87 Interactive and pseudo-tty • 87 Detaching from a running container • 89 Network port publishing • 90 Configuration and environment variables • 91 Summary ........................................................................................................... 93 Further reading .................................................................................................. 93 Chapter 4: Managing Running Containers 95 Technical requirements ...................................................................................... 95 Managing container images ................................................................................ 96 Searching for images • 97 Pulling and viewing images • 99 Inspecting images' configurations and contents • 101 Deleting images • 104 Operations with running containers .................................................................. 105 Viewing and handling container status • 105 Pausing and unpausing containers • 112 Inspecting processes inside containers • 114 Monitoring container stats • 117 Inspecting container information ...................................................................... 119 Capturing logs from containers .......................................................................... 122 Executing processes into a running container ..................................................... 125 Running containers in pods ............................................................................... 127 Summary .......................................................................................................... 130 Chapter 5: Implementing Storage for the Container's Data 131 Technical requirements ..................................................................................... 131 Why does storage matter for containers? ............................................................ 132 Table of Contents viii

Containers' storage features .............................................................................. 133 Storage driver • 134 The storage.conf configuration file • 145 Copying files in and out of a container ................................................................ 146 Using podman cp • 146 Interacting with overlayfs • 147 Persisting changes with podman commit • 147 Attaching host storage to a container ................................................................. 149 Managing and attaching bind mounts to a container • 149 The -v|--volume option • 150 The --mount option • 150 Managing and attaching volumes to a container • 151 Mounting volumes with the --mount option • 155 Volume drivers • 155 Volumes in builds • 156 Mounting volumes across containers • 157 SELinux considerations for mounts • 158 Attaching other types of storage to a container ................................................... 163 Attaching tmpfs storage • 163 Attaching images • 164 Attaching devpts • 165 Summary .......................................................................................................... 166 Further reading ................................................................................................. 166 Part 2: Building Your Container from Scratch with Buildah 169 Chapter 6: Meet Buildah – Building Containers from Scratch 171 Technical requirements ..................................................................................... 171 Building basic images with Podman ................................................................... 172 Understanding builds under the hood • 172 Exploring Dockerfile and Containerfile instructions • 176 Running builds with Podman • 179 ix Table of Contents

Introducing Buildah, Podman's companion ...................................................... 184 Preparing our environment .............................................................................. 189 Verifying the installation • 191 Buildah configuration files • 192 Choosing our build strategy ............................................................................... 193 Building a container image starting from an existing base image • 194 Building a container image starting from scratch • 194 Building a container image starting from a Dockerfile • 195 Building images from scratch ............................................................................. 195 Building images from a Dockerfile .................................................................... 200 Summary ......................................................................................................... 205 Further reading ................................................................................................ 205 Chapter 7: Integrating with Existing Application Build Processes 207 Technical requirements .................................................................................... 207 Multistage container builds .............................................................................. 208 Multistage builds with Dockerfiles • 209 Multistage builds with Buildah native commands • 212 Running Buildah inside a container ................................................................... 216 Running rootless Buildah containers with volume stores • 217 Running Buildah containers with bind-mounted stores • 221 Running native Buildah commands inside containers • 224 Integrating Buildah with custom builders ......................................................... 225 Including Buildah in our Go build tool • 225 Running Quarkus-native executables in containers • 233 Summary ......................................................................................................... 234 Further reading ................................................................................................ 234 Chapter 8: Choosing the Container Base Image 237 Technical requirements .................................................................................... 238 The Open Container Initiative image format ..................................................... 238 OCI Image Manifest • 239 Table of Contents x

Where do container images come from? ............................................................ 240 Docker Hub container registry service • 241 Quay.io container registry service • 243 Red Hat Container Registry • 244 Trusted container image sources ...................................................................... 247 Managing trusted registries • 248 Introducing the Universal Base Image ............................................................... 254 The UBI Standard image • 255 The UBI Minimal image • 256 The UBI Micro image • 258 The UBI Init image • 260 Other UBI-based images • 263 Summary ......................................................................................................... 263 Further reading ................................................................................................ 264 Chapter 9: Pushing Images to a Container Registry 265 Technical requirements .................................................................................... 265 What is a container registry? ............................................................................. 266 Repository management • 266 Pushing container images • 267 Tag management • 267 Pulling container images • 267 Authentication management • 267 Cloud-based and on-premises container registries ............................................ 268 On-premises container registries • 268 Cloud-based container registries • 269 Docker Hub cloud registry • 270 Red Hat Quay.io cloud registry • 270 Managing container images with Skopeo .......................................................... 276 Installing Skopeo • 277 Verifying the installation • 278 Copying images across locations • 279 xi Table of Contents

Inspecting remote images • 283 Synchronizing registries and local directories • 285 Deleting images • 289 Running a local container registry ..................................................................... 291 Running a containerized registry • 291 Customizing the registry configuration • 293 Using a local registry to sync repositories • 298 Managing registry garbage collection • 298 Summary ......................................................................................................... 299 Further reading ................................................................................................ 300 Chapter 10: Securing Containers 301 Technical requirements ..................................................................................... 301 Running rootless containers with Podman ........................................................ 302 The Podman Swiss Army knife – subuid and subgid • 303 Avoiding running containers with UID 0 ........................................................... 304 Signing our container images ........................................................................... 308 Signing images with Sigstore and Podman • 309 Configuring Podman to pull signed images • 313 Testing signature verification failures • 315 Managing keys with Podman image trust commands • 316 Managing signatures with Skopeo • 319 Generating a Sigstore key with Skopeo • 320 Using Rekor and Cosign to manage image signatures • 320 Customizing Linux kernel capabilities • 323 Understanding Linux Kernel capabilities • 323 Using capabilities in containers • 326 Customizing a container's capabilities • 328 Understanding SELinux interaction with containers .......................................... 331 Introducing Udica • 336 Summary .......................................................................................................... 341 Further reading ................................................................................................. 341 Table of Contents xii

Part 3: Managing and Integrating Containers Securely 343 Chapter 11: Troubleshooting and Monitoring Containers 345 Technical requirements .................................................................................... 345 Troubleshooting running containers ................................................................ 346 Permission denied while using storage volumes • 346 The SELinux layer: security contexts • 346 The rootless layer: user namespaces (UID/GID) • 348 Issues with the ping command in rootless containers • 349 Monitoring containers with health checks ......................................................... 352 Inspecting your container build results ............................................................. 358 Troubleshooting builds from Dockerfiles • 359 Troubleshooting builds with Buildah-native commands • 363 Advanced troubleshooting with nsenter ............................................................ 365 Troubleshooting a database client with nsenter • 367 Summary .......................................................................................................... 373 Further reading ................................................................................................. 373 Chapter 12: Implementing Container Networking Concepts 375 Technical requirements ..................................................................................... 375 Configuring container networking and setting up Podman ................................ 376 Configuring Netavark • 377 Creating a container with Podman and Netavark • 379 Managing networks with Podman • 382 Interconnecting two or more containers ........................................................... 388 Containers DNS resolution • 390 DNS resolution on a Netavark network backend • 392 Running containers inside a Pod • 394 Exposing containers outside our underlying host .............................................. 397 Port publishing • 397 Attaching the host network • 399 xiii Table of Contents

Host firewall configuration • 401 Rootless container network behavior ................................................................ 402 Summary ......................................................................................................... 407 Further reading ................................................................................................ 408 Chapter 13: Docker Migration Tips and Tricks 409 Technical requirements .................................................................................... 409 Migrating existing images and playing with a command's alias ......................... 410 Podman commands versus Docker commands ................................................... 413 Behavioral differences between Podman and Docker • 414 Missing commands in Podman • 415 Missing commands in Docker • 415 Using Docker Compose with Podman ................................................................. 416 Docker Compose quick start • 417 Configuring Podman to interact with docker-compose • 421 Running Compose workloads with Podman and docker-compose • 423 Using podman-compose • 428 Summary .......................................................................................................... 431 Further reading ................................................................................................. 431 Chapter 14: Interacting with systemd and Kubernetes 433 Technical requirements .................................................................................... 433 Setting up the prerequisites for the host OS ....................................................... 434 Integrating containers into systemd via Quadlets .............................................. 435 Generating Kubernetes YAML resources ............................................................ 444 Generating basic Pod resources from running containers • 445 Generating Pods and Services from running containers • 447 Generating a composite application in a single Pod • 450 Generating composite applications with multiple Pods • 457 Running Kubernetes resource files in Podman ................................................... 462 Testing the results in Kubernetes ...................................................................... 465 Setting up minikube • 466 Table of Contents xiv

Starting minikube • 468 Running generated resource files in Kubernetes • 469 Summary .......................................................................................................... 471 Further reading ................................................................................................ 472 Chapter 15: Managing Your Container, Kubernetes, and AI Workloads from a Graphical Interface 473 Technical requirements .................................................................................... 474 Setting up the prerequisites for the host operating system ................................. 474 Installing on Linux • 474 Installing with Flatpak • 475 Installing on macOS • 476 Installing on Windows • 477 Easily building, running, stopping, and managing containers with a user-friendly interface ............................................................................................................ 478 Running a container from an image • 478 Managing the container lifecycle • 480 Attaching persistent storage using volumes • 481 Working with container networking • 482 Connecting to and interacting directly with Kubernetes clusters ....................... 483 Connecting to a local cluster with minikube • 484 Connecting Podman Desktop to a Kubernetes cluster • 484 Creating a local kind cluster • 485 Exploring and managing Kubernetes resources • 488 Managing Pods • 488 Managing deployments and services • 489 Deploying local AI inference for app integration ................................................. 491 Introducing Podman AI Lab • 491 Enabling Podman AI Lab • 491 Running your first AI model • 492 Verifying the running model server • 494 Obtaining the API endpoint • 495 xv Table of Contents

Integrating an AI model running on Podman for building your own chatbot use case .......................................................................................................................... 496 Step 1: Starting an OpenAI-compatible inference server • 497 Step 2: Running the AnythingLLM application container • 497 Step 3: Connecting the model to the application • 499 Step 4: Testing your personal assistant • 501 Summary ......................................................................................................... 502 Further reading ................................................................................................ 503 Chapter 16: Unlock Your Exclusive Benefits 505 Unlock this Book’s Free Benefits in three Easy Steps .......................................... 506 Step 1 • 506 Step 2 • 507 Step 3 • 508 Need Help • 508 Other Books You May Enjoy 510 Index 513 Table of Contents xvi

Preface Containers have revolutionized application development on Linux, evolving from a simple packaging mechanism into the definitive foundation for DevOps and cloud-native platforms. While Docker popularized the concept, the industry has matured toward a specialized toolkit that aligns more closely with native Linux design and security standards. This book adopts a foundations-first philosophy; regardless of the engine you eventually use, mastering the underlying Linux primitives is essential for navigating the modern IT landscape. Podman represents this evolution by eliminating the central daemon and supporting rootless execution, treating containers as standard Linux processes rather than isolated artifacts. When combined with Buildah and Skopeo, it provides a modular toolkit that integrates seamlessly with system services, security policies, and orchestration layers. Today, container technology serves as the heartbeat for more than just application deployment. It is the fundamental unit for Kubernetes orchestration, the engine behind modern virtualization for running VMs alongside containers, and the backbone of AI and machine learning environments where isolated, GPU-accelerated pipelines are critical. The focus remains on architectural clarity and practical workflows grounded in real-world usage. This book explains not just how to run a command, but why the technology functions the way it does, empowering you to operate containers confidently across development, production, and the emerging wave of AI-driven infrastructure. Who this book is for This book is designed for system administrators, DevOps engineers, cloud practitioners, and developers seeking a practical, production-oriented approach to Linux containers. It is particularly tailored for those looking to transition beyond Docker-centric workflows to embrace the daemonless, rootless architecture provided by Podman and its ecosystem. Beyond standalone management, this guide serves as a critical foundation for those aspiring to master Kubernetes. A core premise of the text is that one cannot reliably operate Kubernetes workloads, whether standard applications, virtual machines via KubeVirt, or complex AI models, without first internalizing the underlying mechanics of container technology. Intermediate Linux knowledge is assumed, though prior experience with containers or Kubernetes is not required, as concepts are introduced progressively to ensure a smooth learning curve. Hands-on guidance bridges the gap between managing containers as native

system services and packaging applications for consistent, secure deployment. By providing a deep dive into the Linux fundamentals that power modern orchestration, this book ensures that you are prepared to manage the diverse workloads required in contemporary enterprise environments. What this book covers Chapter 1, Introduction to Container Technology, introduces the fundamental concepts behind containerization, including a brief history, the problems containers solve, and how they fit into modern software delivery and system management. Chapter 2, Comparing Podman and Docker, examines the architectural differences between Docker and Podman, focusing on daemon-based versus daemonless designs, security implications, and operational trade-offs. Chapter 3, Running the First Container, walks you through preparing a Linux environment, installing Podman, and running your first containers while introducing essential command- line concepts. Chapter 4, Managing Running Containers, covers container lifecycle management, including inspecting containers, viewing logs, executing commands, and managing containers individually or in pods. Chapter 5, Implementing Storage for the Container’s Data, explains how containers handle data, covering volumes, bind mounts, file copying, and persistent storage strategies. Chapter 6, Meet Buildah – Building Containers from Scratch, introduces Buildah and demonstrates how to create container images using both Dockerfiles and scratch-based workflows. Chapter 7, Integrating with Existing Application Build Processes, explores advanced build techniques, including multistage builds, running Buildah in containers, and integrating image builds into existing CI pipelines. Chapter 8, Choosing the Container Base Image, focuses on image formats, trusted sources, registries, and best practices for selecting secure and maintainable base images. Chapter 9, Pushing Images to a Container Registry, explains how container registries work and shows how to authenticate, push, pull, inspect, and manage images using Podman, Buildah, and Skopeo. Chapter 10, Securing Containers, covers container security fundamentals, including rootless containers, user namespaces, image signing, kernel capabilities, and SELinux integration. Preface xviii

Chapter 11, Troubleshooting and Monitoring Containers, teaches practical techniques for diagnosing container failures, monitoring health, inspecting builds, and using system tools to debug issues. Chapter 12, Implementing Container Networking Concepts, introduces container networking fundamentals, including project Netavark, container interconnection, port exposure, and rootless networking limitations. Chapter 13, Docker Migration Tips and Tricks, provides practical guidance for migrating existing Docker workflows to Podman, including command compatibility, image reuse, and Docker Compose integration. Chapter 14, Interacting with systemd and Kubernetes, shows how to integrate containers with systemd services and how to generate and run Kubernetes resources using Podman. Chapter 15, Your Container, introduces Podman Desktop and Podman AI Lab, demonstrating how to manage containers, interact with Kubernetes clusters, and run AI and machine learning workloads locally through a graphical interface. To get the most out of this book You should be comfortable working on the Linux command line and familiar with basic operating system concepts such as processes, filesystems, and networking. All examples are designed to run locally on Linux systems. The chapters build progressively, and readers new to containers are encouraged to work through the book in order. More experienced readers may choose to focus on advanced topics such as security, system integration, Kubernetes, and AI workflows. Rather than presenting isolated commands, the book emphasizes understanding system behavior, trade-offs, and real-world constraints. You are encouraged to experiment, inspect system state, and explore failure scenarios to build confidence operating containers in production. The tools and practices shown reflect common enterprise and open source workflows. Use them as a foundation and adapt them to your own infrastructure, security requirements, and operational goals. Download the example code files The code bundle for the book is hosted on GitHub at https://github.com/PacktPublishing/ Podman-for-DevOps-Second-Edition. We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing. Check them out! xix Preface