Certified Kubernetes Security Specialist (CKS) Study Guide In-Depth Guidance and Practice (Benjamin Muschko) (Z-Library)

Benjamin Muschko Certified Kubernetes Security Specialist (CKS) Study Guide In-Depth Guidance and Practice

KUBERNETES SECURIT Y “Benjamin did a stellar job! This is a perfect CKS study guide— it’s full of scenarios, examples, and exercises. I strongly recommend the study guide when preparing for the CKS certification exam.” —Robin Smorenburg Tech lead, architect, and engineer “This is a great guide containing clear explanations and examples that will prepare you well for the CKS exam.” —Michael Kehoe Sr Staff Engineer, Confluent, Coauthor of Cloud Native Infrastructure with Azure Certified Kubernetes Security Specialist (CKS) Study Guide US $55.99 CAN $69.99 ISBN: 978-1-098-13297-2 Twitter: @oreillymedia linkedin.com/company/oreilly-media youtube.com/oreillymedia Vulnerabilities in software and IT infrastructure pose a major threat to organizations. In response, the Cloud Native Computing Foundation (CNCF) developed the Certified Kubernetes Security Specialist (CKS) certification to verify an administrator’s proficiency to protect Kubernetes clusters and the cloud native software they contain. This practical book helps you fully prepare for the certification exam by walking you through all of the topics covered. Different from typical multiple-choice formats used by other certifications, this performance-based exam requires deep knowledge of the tasks it covers under intense time pressure. Author Benjamin Muschko shares his personal experience to help you learn the objectives, abilities, and tips and tricks you need to pass the CKS exam on the first attempt. • Identify, mitigate, and/or minimize threats to cloud native applications and Kubernetes clusters • Learn the ins and outs of Kubernetes’s security features, and external tools for security detection and mitigation purposes • Solve real-world Kubernetes problems in a hands-on, command-line environment • Effectively navigate and solve questions during the CKS exam Benjamin Muschko is a software engineer, consultant, and trainer with more than 20 years of experience in the industry. Ben is the author of study guides for the CKAD and CKA exams (O’Reilly) and Gradle in Action (Manning), and he holds the CKAD, CKA, and CKS certifications and is a CNCF Ambassador Spring 2023.

Benjamin Muschko Certified Kubernetes Security Specialist (CKS) Study Guide In-Depth Guidance and Practice Boston Farnham Sebastopol TokyoBeijing

978-1-098-13297-2 [LSI] Certified Kubernetes Security Specialist (CKS) Study Guide by Benjamin Muschko Copyright © 2023 Automated Ascent, LLC. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (https://oreilly.com). For more information, contact our corporate/institu‐ tional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: John Devins Development Editor: Michele Cronin Production Editor: Beth Kelly Copyeditor: Liz Wheeler Proofreader: Amnet Systems, LLC Indexer: Potomac Indexing, LLC Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea June 2023: First Edition Revision History for the First Edition 2023-06-08: First Release See https://oreilly.com/catalog/errata.csp?isbn=9781098132972 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Certified Kubernetes Security Specialist (CKS) Study Guide, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the author, and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix 1. Exam Details and Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Kubernetes Certification Learning Path 1 Kubernetes and Cloud Native Associate (KCNA) 2 Kubernetes and Cloud Native Security Associate (KCSA) 2 Certified Kubernetes Application Developer (CKAD) 2 Certified Kubernetes Administrator (CKA) 2 Certified Kubernetes Security Specialist (CKS) 2 Exam Objectives 3 Curriculum 3 Cluster Setup 4 Cluster Hardening 4 System Hardening 5 Minimize Microservice Vulnerabilities 5 Supply Chain Security 5 Monitoring, Logging, and Runtime Security 6 Involved Kubernetes Primitives 6 Involved External Tools 7 Documentation 7 Candidate Skills 8 Practicing and Practice Exams 8 Summary 9 2. Cluster Setup. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Using Network Policies to Restrict Pod-to-Pod Communication 11 Scenario: Attacker Gains Access to a Pod 12 Observing the Default Behavior 13 iii

Denying Directional Network Traffic 15 Allowing Fine-Grained Incoming Traffic 16 Applying Kubernetes Component Security Best Practices 18 Using kube-bench 18 The kube-bench Verification Result 19 Fixing Detected Security Issues 20 Creating an Ingress with TLS Termination 22 Setting Up the Ingress Backend 23 Creating the TLS Certificate and Key 25 Creating the TLS-Typed Secret 25 Creating the Ingress 26 Calling the Ingress 28 Protecting Node Metadata and Endpoints 28 Scenario: A Compromised Pod Can Access the Metadata Server 29 Protecting Metadata Server Access with Network Policies 30 Protecting GUI Elements 31 Scenario: An Attacker Gains Access to the Dashboard Functionality 31 Installing the Kubernetes Dashboard 32 Accessing the Kubernetes Dashboard 32 Creating a User with Administration Privileges 33 Creating a User with Restricted Privileges 35 Avoiding Insecure Configuration Arguments 37 Verifying Kubernetes Platform Binaries 37 Scenario: An Attacker Injected Malicious Code into Binary 37 Verifying a Binary Against Hash 38 Summary 39 Exam Essentials 40 Sample Exercises 41 3. Cluster Hardening. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Interacting with the Kubernetes API 43 Processing a Request 44 Connecting to the API Server 44 Restricting Access to the API Server 48 Scenario: An Attacker Can Call the API Server from the Internet 48 Restricting User Permissions 48 Scenario: An Attacker Can Call the API Server from a Service Account 52 Minimizing Permissions for a Service Account 53 Updating Kubernetes Frequently 59 Versioning Scheme 59 Release Cadence 60 Performing the Upgrade Process 60 iv | Table of Contents

Summary 62 Exam Essentials 62 Sample Exercises 63 4. System Hardening. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 Minimizing the Host OS Footprint 65 Scenario: An Attacker Exploits a Package Vulnerability 66 Disabling Services 66 Removing Unwanted Packages 67 Minimizing IAM Roles 68 Scenario: An Attacker Uses Credentials to Gain File Access 68 Understanding User Management 69 Understanding Group Management 70 Understanding File Permissions and Ownership 72 Minimizing External Access to the Network 73 Identifying and Disabling Open Ports 73 Setting Up Firewall Rules 74 Using Kernel Hardening Tools 75 Using AppArmor 75 Using seccomp 79 Summary 82 Exam Essentials 82 Sample Exercises 83 5. Minimizing Microservice Vulnerabilities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 Setting Appropriate OS-Level Security Domains 85 Scenario: An Attacker Misuses root User Container Access 86 Understanding Security Contexts 86 Enforcing the Usage of a Non-Root User 87 Setting a Specific User and Group ID 88 Avoiding Privileged Containers 89 Scenario: A Developer Doesn’t Follow Pod Security Best Practices 91 Understanding Pod Security Admission (PSA) 92 Enforcing Pod Security Standards for a Namespace 93 Understanding Open Policy Agent (OPA) and Gatekeeper 95 Installing Gatekeeper 96 Implementing an OPA Policy 96 Managing Secrets 99 Scenario: An Attacker Gains Access to the Node Running etcd 99 Accessing etcd Data 100 Encrypting etcd Data 101 Understanding Container Runtime Sandboxes 103 Table of Contents | v

Scenario: An Attacker Gains Access to Another Container 104 Available Container Runtime Sandbox Implementations 105 Installing and Configuring gVisor 105 Creating and Using a Runtime Class 106 Understanding Pod-to-Pod Encryption with mTLS 107 Scenario: An Attacker Listens to the Communication Between Two Pods 108 Adopting mTLS in Kubernetes 108 Summary 109 Exam Essentials 110 Sample Exercises 110 6. Supply Chain Security. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 Minimizing the Base Image Footprint 113 Scenario: An Attacker Exploits Container Vulnerabilities 114 Picking a Base Image Small in Size 114 Using a Multi-Stage Approach for Building Container Images 116 Reducing the Number of Layers 118 Using Container Image Optimization Tools 119 Securing the Supply Chain 119 Signing Container Images 119 Scenario: An Attacker Injects Malicious Code into a Container Image 119 Validating Container Images 120 Using Public Image Registries 122 Scenario: An Attacker Uploads a Malicious Container Image 122 Whitelisting Allowed Image Registries with OPA GateKeeper 123 Whitelisting Allowed Image Registries with the ImagePolicyWebhook Admission Controller Plugin 126 Implementing the Backend Application 127 Configuring the ImagePolicyWebhook Admission Controller Plugin 127 Static Analysis of Workload 130 Using Hadolint for Analyzing Dockerfiles 130 Using Kubesec for Analyzing Kubernetes Manifests 131 Scanning Images for Known Vulnerabilities 134 Summary 136 Exam Essentials 136 Sample Exercises 138 7. Monitoring, Logging, and Runtime Security. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 Performing Behavior Analytics 142 Scenario: A Kubernetes Administrator Can Observe Actions Taken by an Attacker 142 Understanding Falco 142 vi | Table of Contents

Installing Falco 143 Configuring Falco 144 Generating Events and Inspecting Falco Logs 146 Understanding Falco Rule File Basics 147 Overriding Existing Rules 150 Ensuring Container Immutability 151 Scenario: An Attacker Installs Malicious Software 151 Using a Distroless Container Image 152 Configuring a Container with a ConfigMap or Secret 152 Configuring a Read-Only Container Root Filesystem 153 Using Audit Logs to Monitor Access 154 Scenario: An Administrator Can Monitor Malicious Events in Real Time 154 Understanding Audit Logs 154 Creating the Audit Policy File 155 Configuring a Log Backend 156 Configuring a Webhook Backend 158 Summary 159 Exam Essentials 159 Sample Exercises 160 Appendix: Answers to Review Questions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189 Table of Contents | vii

Preface The Kubernetes certification program has been around since 2018, or five years as of this writing. During this time, security has become more and more important every‐ where, including the Kubernetes world. Recently, the role of Certified Kubernetes Security Specialist (CKS) has been added to the certification track to address the need. Security can have different facets, and the way you address those concerns can be very diverse. That’s where the Kubernetes ecosystem comes into play. Apart from Kubernetes built-in security features, many tools have evolved that help with identifying and fixing security risks. As a Kubernetes administrator, you need to be familiar with the wide range of concepts and tools to harden your clusters and applications. The CKS certification program was created to verify competence on security-based topics, and it requires a successful pass of the Certified Kubernetes Administrator (CKA) exam before you can register. If you are completely new to the Kubernetes certification program, then I would recommend exploring the CKA or Certified Kubernetes Application Developer (CKAD) program first. In this study guide, I will explore the topics covered in the CKS exam to fully prepare you to pass the certification exam. We’ll look at determining when and how you should apply the core concepts of Kubernetes and external tooling to secure cluster components, cluster configuration, and applications running in a Pod. I will also offer tips to help you better prepare for the exam and share my personal experience with getting ready for all aspects of it. The CKS is different from the typical multiple-choice format of other certifications. It’s completely performance based and requires you to demonstrate deep knowledge of the tasks at hand under immense time pressure. Are you ready to pass the test on the first go? ix

Who This Book Is For This book is for anyone who already passed the CKA exam and wants to broaden their knowledge in the realm of security. Given that you need to pass the CKA exam before signing up for the CKS, you should already be familiar with the format of the exam questions and environment. Chapter 1 only briefly recaps the general aspects of the exam curriculum, but it highlights the information specific to the CKS exam. If you have not taken the CKA exam yet, I recommend taking a step by reading the Certified Kubernetes Administrator (CKA) Study Guide (O’Reilly). The book will provide you with the foundation you need to get started with the CKS. What You Will Learn The content of the book condenses the most important aspects relevant to the CKS exam. Cloud-provider-specific Kubernetes implementations like AKS or GKE do not need to be considered. Given the plethora of configuration options available in Kubernetes, it’s almost impossible to cover all use cases and scenarios without duplicating the official documentation. Test takers are encouraged to reference the Kubernetes documentation as the go-to compendium for broader exposure. External tools relevant to the CKS exam, such as Trivy or Falco, are only covered on a high level. Refer to their documentation to explore more features, functionality, and configuration options. Structure of This Book The outline of the book follows the CKS curriculum to a T. While there might be a more natural, didactical structure for learning Kubernetes in general, the curriculum outline will help test takers prepare for the exam by focusing on specific topics. As a result, you will find yourself cross-referencing other chapters of the book depending on your existing knowledge level. Be aware that this book covers only the concepts relevant to the CKS exam. Founda‐ tional Kubernetes concepts and primitives are not discussed. Refer to the Kubernetes documentation or other books if you want to dive deeper. Practical experience with Kubernetes is key to passing the exam. Each chapter con‐ tains a section named “Sample Exercises” with practice questions. Solutions to those questions can be found in the Appendix. Conventions Used in This Book The following typographical conventions are used in this book: x | Preface

Italic Indicates new terms, URLs, and email addresses. Constant width Used for filenames, file extensions, and program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords. Constant width bold Shows commands or other text that should be typed literally by the user. This element signifies a tip or suggestion. This element signifies a general note. This element indicates a warning or caution. Using Code Examples Some code snippets in the book use the backslash character (\) to break up a single line into multiple lines to make it fit the page. You will need to rectify the code manually if you are copy-pasting it directly from the book content to a terminal or editor. The better choice is to refer to the code book’s GitHub repository, which already has the proper formatting. The GitHub repository is distributed under the Apache License 2.0. The code is free to use in commercial and open source projects. If you encounter an issue in the source code or if you have a question, open an issue in the GitHub issue tracker. I’ll be happy to have a conversation and fix any issues that might arise. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code Preface | xi

from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. We appreciate, but generally do not require, attribution. An attri‐ bution usually includes the title, author, publisher, and ISBN. For example: “Certified Kubernetes Security Specialist (CKS) Study Guide by Benjamin Muschko (O’Reilly). Copyright 2023 Automated Ascent, LLC, 978-1-098-13297-2.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit http://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-889-8969 (in the United States or Canada) 707-829-7019 (international or local) 707-829-0104 (fax) support@oreilly.com https://www.oreilly.com/about/contact.html We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/cks-study-guide. For news and information about our books and courses, visit http://oreilly.com. xii | Preface

Find us on LinkedIn: https://linkedin.com/company/oreilly-media Follow us on Twitter: http://twitter.com/oreillymedia Watch us on YouTube: http://youtube.com/oreillymedia Follow the author on Twitter: https://twitter.com/bmuschko Follow the author on GitHub: https://github.com/bmuschko Follow the author’s blog: https://bmuschko.com Acknowledgments Every book project is a long journey and would not be possible without the help of the editorial staff and technical reviewers. Special thanks go to Robin Smorenburg, Werner Dijkerman, Michael Kehoe, and Liz Rice for their detailed technical guidance and feedback. I would also like to thank the editors at O’Reilly Media, John Devins and Michele Cronin, for their continued support and encouragement. Preface | xiii

CHAPTER 1 Exam Details and Resources This introductory chapter addresses the most pressing questions candidates ask when preparing for the Certified Kubernetes Security Specialist (CKS) exam. We will discuss the target audience for the certification, the curriculum, and the exam environment, as well as tips and tricks and additional learning resources. If you’re already familiar with the certification program, you can directly jump to any of the chapters covering the technical concepts. Kubernetes Certification Learning Path The CNCF offers four different Kubernetes certifications. Figure 1-1 categorizes each of them by target audience. You will find that the CKS is the most advanced certification you can acquire. It is the only one with a prerequisite of passing another certification first; all others are standalone programs. Figure 1-1. Kubernetes certifications learning path 1

Let’s have a very brief look at the details for each certification to see if the CKS is the right fit for you. Kubernetes and Cloud Native Associate (KCNA) KCNA is an entry-level certification program for anyone interested in cloud-native application development, runtime environments, and tooling. While the exam does cover Kubernetes, it does not expect you to actually solve problems in a practical manner. This exam is suitable for candidates interested in the topic with a broad exposure to the ecosystem. Kubernetes and Cloud Native Security Associate (KCSA) The certification focuses on basic knowledge of security concepts and their applica‐ tion in a Kubernetes cluster. The breadth and depth of the program is comparable to the KCNA, as it does not require solving problems hands-on. Certified Kubernetes Application Developer (CKAD) The CKAD exam focuses on verifying your ability to build, configure, and deploy a microservices-based application to Kubernetes. You are not expected to actually implement an application; however, the exam is suitable for developers familiar with topics like application architecture, runtimes, and programming languages. Certified Kubernetes Administrator (CKA) The target audience for the CKA exam are DevOps practitioners, system administra‐ tors, and site reliability engineers. This exam tests your ability to perform in the role of a Kubernetes administrator, which includes tasks like cluster, network, storage, and beginner-level security management, with a big emphasis on troubleshooting scenarios. Certified Kubernetes Security Specialist (CKS) The CKS exam expands on the topics verified by the CKA exam. Passing the CKA is a prerequisite before you can even sign up for the CKS exam. For this certification, you are expected to have a deeper knowledge of Kubernetes security aspects. The curricu‐ lum covers topics like applying best practices for building containerized applications and ensuring a secure Kubernetes runtime environment. 2 | Chapter 1: Exam Details and Resources

Exam Objectives Vulnerabilities in software and IT infrastructure, if exploited, can pose a major threat to organizations. The Cloud Native Computing Foundation (CNCF) developed the Certified Kubernetes Security Specialist (CKS) certification to verify a Kubernetes administrator’s proficiency to protect a Kubernetes cluster and the cloud native software operated in it. As part of the CKS exam, you are expected to understand Kubernetes core security features, as well as third-party tools and established practi‐ ces for securing applications and infrastructure. Kubernetes version used during the exam At the time of writing, the exam is based on Kubernetes 1.26. All content in this book will follow the features, APIs, and command- line support for that specific version. It’s certainly possible that future versions will break backward compatibility. While preparing for the certification, review the Kubernetes release notes and prac‐ tice with the Kubernetes version used during the exam to avoid unpleasant surprises. In this book, I am going to explain each of the security threats by providing a specific use case. We’ll start by talking about a scenario that allows an attacker to gain access to a cluster, inject malicious code, or use a vulnerability to hack into the system. Then, we’ll touch on the concepts, practices, and/or tools that will prevent that situation. With this approach, you’ll be able to evaluate the severity of a security risk and the need for implementing security measures. Curriculum The following overview lists the high-level sections, also called domains, of the CKS exam and their scoring weights: • 10%: Cluster Setup • 15%: Cluster Hardening • 15%: System Hardening • 20%: Minimize Microservice Vulnerabilities • 20%: Supply Chain Security • 20%: Monitoring, Logging, and Runtime Security Curriculum | 3

How the book works The outline of the book follows the CKS curriculum to a T. While there might be a more natural, didactical organization structure to learn Kubernetes in general, the curriculum outline will help test takers prepare for the exam by focusing on specific topics. As a result, you will find yourself cross-referencing other chapters of the book depending on your existing knowledge level. Let’s break down each domain in detail in the next sections. Cluster Setup This section covers Kubernetes concepts that have already been covered by the CKA exam; however, they assume that you already understand the basics and expect you to be able to go deeper. Here, you will be tested on network policies and their effects on disallowing and granting network communication between Pods within the same namespace and across multiple namespaces. The main focus will be on restricting communication to minimize the attack surface. Furthermore, the domain “cluster setup” will verify your knowledge of setting up an Ingress object with Transport Layer Security (TLS) termination. A big emphasis lies on identifying and fixing security vulnerabilities by inspecting the cluster setup. External tools like kube-bench can help with automating the process. As a result of executing the tool against your cluster, you will receive an actionable list of vulnerabilities. Changing the configuration settings of your cluster according to the recommendations can help with significantly reducing the security risk. Last, locking down cluster node endpoints, ports, and graphical user interfaces (GUIs) can help with making it harder for attackers to gain control of the cluster. You need to be aware of the default cluster settings so that you can limit access to them as much as possible. Kubernetes binaries and executables like kubectl, kubeadm, and the kubelet need to be checked against their checksum to ensure they haven’t been tampered with by a third party. You need to understand how to retrieve the checksum file for binaries and how to use it verify the validity of the executable. Cluster Hardening Most organizations start out with a cluster that allows developers and administrators alike to manage the Kubernetes installation, configuration, and management of any objects. While this is a convenient approach for teams getting comfortable with Kubernetes, it is not a safe and sound situation, as it poses the potential of opening the floodgates for attackers. Once access has been gained to the cluster, any malicious operation can be performed. 4 | Chapter 1: Exam Details and Resources