Linux System Administration for the 2020s The Modern Sysadmin Leaving Behind the Culture of Build and Maintain (Kenneth Hitchcock) (Z-Library)

Linux System Administration for the 2020s The Modern Sysadmin Leaving Behind the Culture of Build and Maintain — Kenneth Hitchcock

Linux System Administration for the 2020s The Modern Sysadmin Leaving Behind the Culture of Build and Maintain Kenneth Hitchcock

Linux System Administration for the 2020s ISBN-13 (pbk): 978-1-4842-7983-0 ISBN-13 (electronic): 978-1-4842-7984-7 https://doi.org/10.1007/978-1-4842-7984-7 Copyright © 2022 by Kenneth Hitchcock This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein. Managing Director, Apress Media LLC: Welmoed Spahr Acquisitions Editor: Divya Modi Development Editor: James Markham Coordinating Editor: Divya Modi Cover designed by eStudioCalamar Cover image designed by Pixabay Distributed to the book trade worldwide by Springer Science+Business Media New York, 1 New York Plaza, Suite 4600, New York, NY 10004-1562, USA. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail orders-ny@springer-sbm.com, or visit www.springeronline.com. Apress Media, LLC is a California LLC and the sole member (owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation. For information on translations, please e-mail booktranslations@springernature.com; for reprint, paperback, or audio rights, please e-mail bookpermissions@springernature.com. Apress titles may be purchased in bulk for academic, corporate, or promotional use. eBook versions and licenses are also available for most titles. For more information, reference our Print and eBook Bulk Sales web page at http://www.apress.com/bulk-sales. Any source code or other supplementary material referenced by the author in this book is available to readers on GitHub via the book's product page, located at https://github.com/ Apress/Linux- System- Administration- for- the- 2020s. For more detailed information, please visit http://www.apress.com/source- code. Printed on acid-free paper Kenneth Hitchcock Hampshire, UK

To Arlene, Dad, and Mom, without your support and sacrifices, I would not have had the successes I had.

v Table of Contents Part I: Laying the Foundation �������������������������������������������������������1 Chapter 1: Linux at a Glance �����������������������������������������������������������������3 Brief Unix to Linux History4 Open Source5 Linux Is Everywhere 6 Community Linux Distributions 8 Community 8 Upstream 8 Community Contributors 9 Common Distributions 10 Which Distribution Is Best for You 11 Before Committing 12 The Three Linux Distro Categories 13 Enterprise Linux Distributions 17 Red Hat 18 Canonical 20 SUSE 21 About the Author ������������������������������������������������������������������������������xvii About the Technical Reviewer �����������������������������������������������������������xix Acknowledgments �����������������������������������������������������������������������������xxi Introduction �������������������������������������������������������������������������������������xxiii

vi Community vs Enterprise 22 Knowledge Check 24 Summary24 Part II: Strengthening Core Skills ����������������������������������������������27 Chapter 2: New Tools to Improve the Administrative Experience �������29 Task Management 30 Starting a Process 30 Task Visualization Tooling 31 Killing Processes 34 Zombie Processes 35 Background Tasks 36 Running Time-Consuming Tasks 37 Ansible Introduction 39 Installing Ansible 40 Configuring Ansible 41 Ansible Inventory 42 Running Ansible 43 Playbooks 44 Roles 44 Modules 46 Sharing Your Ansible 46 Web Consoles 47 Cockpit47 Alternatives to Cockpit 49 Text Consoles 50 Installing 51 Using51 Summary52 Table of ConTenTs

vii Chapter 3: Estate Management ����������������������������������������������������������53 Outdated Ways of Working 54 Outdated Skills 54 Keeping Knowledge to Themselves 55 Over Engineering 55 Shell Scripting 56 Snowflakes 56 Reinventing the Wheel 57 Build Process 57 Manual Installation Methods 58 Automated Linux Installations 60 Using Images 64 Build Process Flow 66 System Patching 70 Update Types 70 Staging 71 Patch Management Systems 72 Planning 73 Rollback 74 Backup and Recovery 75 Important Directories and Files 76 Virtual Machine Backups 76 Disaster Recovery77 Common Bad Practices 79 Virtual Machine Templates 79 Patching or Lack Thereof 80 Firewall Disabled 80 SELinux Disabled or Permissive 80 Table of ConTenTs

viii Using Community Repositories 81 Scripts, Scripts, and More Scripts 81 Running As Root 82 Good Practices 82 Building Throwaway Systems82 Automate As Much As Possible 83 Search Before Creating 83 Sharing Knowledge and Collaborating 83 Source Control 84 Reassessing System Requirements 84 Summary84 Chapter 4: Estate Management Tools �������������������������������������������������87 Management Systems 88 Linux Platform Tools 88 Linux Platform Tools Available89 Selecting Your Linux Platform Tool 90 The Decision 91 Satellite Server 93 Satellite 6 94 SUSE Manager 99 Foreman 101 Spacewalk 102 Provisioning Tools 103 Cloudforms 104 Terraform 106 API and Extracting Useful Information 107 Don’t Reinvent the Wheel 107 Why to Not Write Your Own Tool 108 Table of ConTenTs

ix Best Tools to Use 108 Pipeline Tooling108 Automation Platforms 109 Shell Scripts 109 Summary109 Chapter 5: Automation ����������������������������������������������������������������������111 Automation in Theory 111 Idempotent Code 112 Knowing When and When Not to Automate 113 State Management 114 Automation Tooling 115 Automation Scripting Languages 115 Automation Platforms 119 Automation in Estate Management Tools 119 Ansible Automation Platform 120 Making the Decision 131 Automation with Management Tools 133 State Management 133 Enterprise Products 134 Use Case Example 134 Setting Up a SOE 137 Automate the Automation 139 Self-Healing 139 When to Self-Heal 145 How to Implement Self-Healing 145 Automation Best Practices 149 Do Not Reinvent the Wheel, Again … 149 Table of ConTenTs

x Things to Avoid 151 Shell Scripts 151 Restarting Services When Not Required 151 Using Old Versions 152 Correct Version Documentation 152 Good Practices 152 Summary153 Chapter 6: Containers �����������������������������������������������������������������������155 Getting Started 155 Virtual Machine vs Container 156 Container History 156 Container Runtimes 157 Container Images 163 Containers in Practice 166 Prerequisites 166 Creating Containers 167 Custom Images and Containers 171 Container Practices 177 Cloud Native 177 Good Practices 177 Bad Practices 180 Container Development 182 Development Considerations 182 Container Tooling 185 DevSecOps 189 DevSecOps Tooling 189 GitOps190 Table of ConTenTs

xi GitOps Toolbox 191 Container Orchestration 192 What Does It Do? 193 Why Not Use Podman? 193 Orchestration Options 194 Summary200 Part III: Day Two Practices and Keeping the Lights On ������������201 Chapter 7: Monitoring �����������������������������������������������������������������������203 Linux Monitoring Tools 204 Process Monitoring 204 Disk and IO 206 CPU 208 Memory 210 Virtual Memory 211 Network 212 Graphical Tools 217 Historical Monitoring Data 219 Central Monitoring 221 Nagios 221 Prometheus 224 Thanos 227 Enterprise Monitoring 229 Dashboards 232 Dashboarding Tools 233 Grafana 233 Table of ConTenTs

xii Application Monitoring 237 Tracing Tools 237 Exposing Metrics 239 Summary239 Chapter 8: Logging ���������������������������������������������������������������������������241 Linux Logging Systems 241 Rsyslog 241 Fluentd 244 Plugin Based 244 Used at Scale 245 Installation 245 Configuration 246 Understanding Logs 247 Where Are the Log Files 247 How to Read Log Files 247 Infrastructure Logs 248 Application Logs 249 Increasing Verbosity 250 Log Maintenance 251 Log Management Tools 252 Log Forwarding 253 Central Logging Systems 253 Summary257 Table of ConTenTs

xiii Chapter 9: Security ���������������������������������������������������������������������������259 Linux Security 259 Standard Linux Security Tools 260 Recommended Linux Security Configurations 264 DevSecOps 268 What Is It? 269 Everyone Is Responsible for Security 269 Tools 271 System Compliance 272 System Hardening 272 Vulnerability Scanning 276 Linux Scanning Tools 276 Container Image Scanning Tools 277 Container Platform Scanning Tools 280 Summary283 Chapter 10: Maintenance Tasks and Planning ���������������������������������285 What Maintenance Should Be Done 285 Patching286 Filesystem 288 Firewall 290 Backups 290 As Often As Possible 291 How Should Maintenance Be Done 292 Automation 292 Zero Downtime Environments 293 Maintenance Planning 294 Table of ConTenTs

xiv Agree Maintenance Window 294 Bite-Size Chunks 295 Automating Process and Task Together 295 Process Automation 296 Summary 297 Part IV: See, Analyze, Then Act ������������������������������������������������299 Chapter 11: Troubleshooting �������������������������������������������������������������301 See, Analyze, Then Act 301 Understand the Problem 302 Theorize Based on Evidence 306 Ask for Help 309 What to Do Before Asking for Help 309 How to Ask for Help 310 Things to Avoid When Troubleshooting 313 Live Debugging 313 Correlation vs Causation 314 Being a Lone Wolf 314 Guessing and Lying 314 Ghosts 315 All the Small Things 315 Keep Track of What You Have Tried 315 Measure Twice, Cut Once 315 Do Not Forget Your Retrospective 316 Summary 316 Table of ConTenTs

xv Chapter 12: Advanced Administration ����������������������������������������������319 System Analysis 319 Tools for the Sysadmin 320 System Tracing 323 Strace 323 Systemtap 325 System Tuning 328 Tuned 328 Summary 329 Index �������������������������������������������������������������������������������������������������331 Table of ConTenTs

xvii About the Author Kenneth Hitchcock is a principal consultant working for Red Hat, with over 20 years of experience in IT. Ken has spent the last 11 years predominantly focused on Red Hat products, certificating himself as a Red Hat Architect along the way. The last decade has been paramount in Ken’s understanding of how large Linux estates should be managed, and in the spirit of openness, he was inspired to share his knowledge and experiences in this book. Originally from Durban, South Africa, Ken now lives in the south of England, where he hopes to not only continue to inspire all he meets but also to continue improving himself and the industry he works in.

xix About the Technical Reviewer Zeeshan Shamim has been an IT professional in various capacities from management to DevOps for the past 15 odd years. He has worked in roles ranging from support to DevOps/sysadmin in various organizations ranging from big telecom firms to financial banks and is a proponent of open source technologies.

xxi Acknowledgments This book is based on all the experience and training I received over the years, all of which started while working for Justin Garlick and Alasdair Mackenzie. Thank you for all the opportunities to learn and for giving me the foundation to get started in the open source world. My eventual move to Red Hat opened opportunities to work with larger teams and allowed me to learn so much from so many great influential people in the various Red Hat teams. I am grateful for the guidance and friendship from Dan Hawker, Will McDonald, Vic Gabrie, Martin Sumner, Chris Brown, Paulo Menon, Zeeshan Shamim, and so many others I have not named who are from a truly special group of people that are always willing to help and make working at Red Hat so special. Thank you all for showing what it means to be open.

xxiii Introduction This book is divided into four main parts with each designed to expand from the previous. More subjects are introduced as you go along; some may require further reading, and others are explained. At the end of the book, you will be left either feeling happy that you are doing things the right way or have a thousand ideas on how to improve. Part 1 If you are reading this book with existing Linux knowledge, use Part 1 as a refresher or an opportunity to see things from a different perspective. It is entirely possible there is something you may not know or have possibly forgotten. For the reader new to Linux, this is not a book to teach you all the foundational skills either or bring you to the same level as readers with years of experience; that will require more effort on your part. It will, however, give you the keywords and subjects you will need to explore further on your own. Anyone who has ever been exposed to something new will understand the statement “you don’t know what you don’t know.” Part 1 is there to give you the breadcrumbs to these unknowns; further chapters will give you a bit more. The value in Part 1 will come from the structure it gives; it will show you what to learn and where to focus to build a solid foundation. The advanced users with many years of experience will most likely breeze through the first chapters and not gain anything new. All I can offer you is a different perspective on how I believe a solid Linux knowledge foundation can be laid.

xxiv Each of us comes from different backgrounds; we have different views on how things should be done and how we have always done them. Take Part 1 as either a refresher or a stepping stone to a greater understanding of the Linux and open source world. Part 2 Part 2 will explore how to improve ways of working with Linux systems and hopefully give you a few shortcuts along the way. It contains much of my experience as a consultant from the last ten years and will contain some interesting views on what I have experienced a little more recently. The ultimate goal of this part is to bring you up to speed with the latest estate management trends and tools. Everyone who is reading this book should gain some benefit from what I am sharing with you. It will start with new tooling and the new ways of working most organizations have started to adopt. Using these new tools, we will delve into estate management and how Linux systems have and should be provisioned. We will look into backing up and restoring platforms with a good understanding of the disaster recovery options available today. We will visit good and bad practices people commonly do and how to avoid them. We will then discuss best practices for running an efficient environment. Like we discussed community and enterprise Linux distros in Chapter 1, we will discuss community and enterprise estate management tools. We will look at how these tools can be leveraged to build a solution that can be truly inspirational. With good statement management, there needs to be a high degree of automation; in Chapter 5, we will explore in-depth automation concepts and practices to achieve higher productivity than what it was like to build systems ten years ago. Finally, we will discuss different aspects of containerization, when is the right time and what should and should not be containerized. InTroduCTIon