Efficient Linux at the Command Line Boost Your Command-Line Skills (Daniel J. Barrett) (Z-Library)

Daniel J. Barrett Author of Linux Pocket Guide Efficient Linux at the Command Line Boost Your Command-Line Skills

Daniel J. Barrett Efficient Linux at the Command Line Boost Your Command-Line Skills Boston Farnham Sebastopol TokyoBeijing

978-1-098-11340-7 [LSI] Efficient Linux at the Command Line by Daniel J. Barrett Copyright © 2022 Daniel Barrett. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (https://oreilly.com). For more information, contact our corporate/institu‐ tional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: John Devins Development Editor: Virginia Wilson Production Editor: Gregory Hyman Copyeditor: Kim Wimpsett Proofreader: Piper Editorial Consulting, LLC Indexer: Daniel J. Barrett Index Editor: Sue Klefstad Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea February 2022: First Edition Revision History for the First Edition 2022-02-16: First Release See https://oreilly.com/catalog/errata.csp?isbn=9781098113407 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Efficient Linux at the Command Line, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the author and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Part I. Core Concepts 1. Combining Commands. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Input, Output, and Pipes 4 Six Commands to Get You Started 6 Command #1: wc 6 Command #2: head 9 Command #3: cut 9 Command #4: grep 11 Command #5: sort 12 Command #6: uniq 14 Detecting Duplicate Files 16 Summary 18 2. Introducing the Shell. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Shell Vocabulary 20 Pattern Matching for Filenames 20 Evaluating Variables 23 Where Variables Come From 23 Variables and Superstition 24 Patterns Versus Variables 25 Shortening Commands with Aliases 26 Redirecting Input and Output 27 Disabling Evaluation with Quotes and Escapes 29 Locating Programs to Be Run 31 iii

Environments and Initialization Files, the Short Version 33 Summary 34 3. Rerunning Commands. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Viewing the Command History 36 Recalling Commands from the History 37 Cursoring Through History 37 History Expansion 39 Never Delete the Wrong File Again (Thanks to History Expansion) 41 Incremental Search of Command History 43 Command-Line Editing 44 Cursoring Within a Command 45 History Expansion with Carets 46 Emacs- or Vim-Style Command-Line Editing 47 Summary 48 4. Cruising the Filesystem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 Visiting Specific Directories Efficiently 50 Jump to Your Home Directory 50 Move Faster with Tab Completion 51 Hop to Frequently Visited Directories Using Aliases or Variables 52 Make a Big Filesystem Feel Smaller with CDPATH 54 Organize Your Home Directory for Fast Navigation 55 Returning to Directories Efficiently 58 Toggle Between Two Directories with “cd -” 58 Toggle Among Many Directories with pushd and popd 59 Summary 64 Part II. Next-Level Skills 5. Expanding Your Toolbox. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 Producing Text 68 The date Command 68 The seq Command 69 Brace Expansion (A Shell Feature) 70 The find Command 71 The yes Command 73 Isolating Text 73 grep: A Deeper Look 74 The tail Command 77 The awk {print} Command 78 iv | Table of Contents

Combining Text 80 The tac Command 81 The paste Command 81 The diff Command 82 Transforming Text 83 The tr Command 83 The rev Command 84 The awk and sed Commands 85 Toward an Even Larger Toolbox 93 Summary 94 6. Parents, Children, and Environments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 Shells Are Executable Files 96 Parent and Child Processes 97 Environment Variables 99 Creating Environment Variables 100 Superstition Alert: “Global” Variables 101 Child Shells Versus Subshells 102 Configuring Your Environment 103 Rereading a Configuration File 105 Traveling with Your Environment 106 Summary 106 7. 11 More Ways to Run a Command. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 List Techniques 107 Technique #1: Conditional Lists 108 Technique #2: Unconditional Lists 109 Substitution Techniques 110 Technique #3: Command Substitution 110 Technique #4: Process Substitution 112 Command-as-String Techniques 115 Technique #5: Passing a Command as an Argument to bash 116 Technique #6: Piping a Command to bash 117 Technique #7: Executing a String Remotely with ssh 119 Technique #8: Running a List of Commands with xargs 120 Process-Control Techniques 124 Technique #9: Backgrounding a Command 124 Technique #10: Explicit Subshells 129 Technique #11: Process Replacement 131 Summary 132 Table of Contents | v

8. Building a Brash One-Liner. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 Get Ready to Be Brash 137 Be Flexible 137 Think About Where to Start 138 Know Your Testing Tools 139 Inserting a Filename into a Sequence 140 Checking Matched Pairs of Files 143 Generating a CDPATH from Your Home Directory 145 Generating Test Files 147 Generating Empty Files 150 Summary 151 9. Leveraging Text Files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 A First Example: Finding Files 155 Checking Domain Expiration 156 Building an Area Code Database 158 Building a Password Manager 160 Summary 167 Part III. Extra Goodies 10. Efficiency at the Keyboard. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 Working with Windows 171 Instant Shells and Browsers 171 One-Shot Windows 172 Browser Keyboard Shortcuts 173 Switching Windows and Desktops 174 Web Access from the Command Line 174 Launching Browser Windows from the Command Line 175 Retrieving HTML with curl and wget 177 Processing HTML with HTML-XML-utils 178 Retrieving Rendered Web Content with a Text-Based Browser 182 Clipboard Control from the Command Line 183 Connecting Selections to stdin and stdout 184 Improving the Password Manager 186 Summary 188 11. Final Time-Savers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189 Quick Wins 189 Jumping Into Your Editor from less 189 Editing Files That Contain a Given String 190 vi | Table of Contents

Embracing Typos 190 Creating Empty Files Quickly 190 Processing a File One Line at a Time 191 Identifying Commands That Support Recursion 191 Read a Manpage 192 Longer Learning 192 Read the bash Manpage 192 Learn cron, crontab, and at 193 Learn rsync 194 Learn Another Scripting Language 195 Use make for Nonprogramming Tasks 195 Apply Version Control to Day-to-Day Files 197 Farewell 198 A. Linux Refresher. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199 B. If You Use a Different Shell. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213 Table of Contents | vii

1 This book displays the Linux prompt as a dollar sign. Your prompt may be different. 2 You’ll learn this mystery command’s purpose in Chapter 8. Preface This book will take your Linux command-line skills to the next level so you can work faster, smarter, and more efficiently. If you’re like most Linux users, you learned your early command-line skills on the job, or by reading an intro book, or by installing Linux at home and just trying things out. I’ve written this book to help you take the next step—to build intermediate to advanced skills at the Linux command line. It’s filled with techniques and concepts that I hope will transform how you interact with Linux and boost your productivity. Think of it as a second book on Linux use that takes you beyond the basics. A command line is the simplest of interfaces, yet also the most challenging. It’s simple because you’re presented with nothing but a prompt, which waits for you to run any command you may know:1 $ It’s challenging because everything beyond the prompt is your responsibility. There are no friendly icons, buttons, or menus to guide you. Instead, every command you type is a creative act. This is true for basic commands, like listing your files: $ ls and more elaborate commands like this one: $ paste <(echo {1..10}.jpg | sed 's/ /\n/g') \ <(echo {0..9}.jpg | sed 's/ /\n/g') \ | sed 's/^/mv /' \ | bash If you’re staring at the preceding command and thinking, “What the heck is that?” or “I would never need such a complex command,” then this book is for you.2 ix

What You’ll Learn This book will make you faster and more effective at three essential skills: • Choosing or constructing commands to solve a business problem at hand • Running those commands efficiently • Navigating the Linux filesystem with ease By the end, you’ll understand what happens behind the scenes when you run a com‐ mand, so you can better predict the results (and not develop superstitions). You’ll see a dozen different methods for launching commands and learn when to use each one for best advantage. You’ll also learn practical tips and tricks to make you more pro‐ ductive, such as: • Building complex commands out of simpler ones, step-by-step, to solve real- world problems, such as managing passwords or generating ten thousand test files • Saving time by organizing your home directory intelligently so you don’t have to hunt for files • Transforming text files and querying them like databases to achieve business goals • Controlling point-and-click features of Linux from the command line, such as copying and pasting with the clipboard, and retrieving and processing web data, without lifting your hands from the keyboard Most of all, you’ll learn general best practices so no matter which commands you run, you can become more successful in everyday Linux use and more competitive on the job market. This is the book I wish I had when I learned Linux. What This Book Is Not This book won’t optimize your Linux computer to make it run more efficiently. It makes you more efficient at interacting with Linux. This book is also not a comprehensive reference for the command line—there are hundreds of commands and features that I don’t mention. This book is about exper‐ tise. It teaches a carefully selected set of command-line knowledge in a practical order to build your skills. For a reference-style guide, try my previous book, Linux Pocket Guide (O’Reilly). x | Preface

Audience and Prerequisites This book assumes you have Linux experience; it’s not an introduction. It’s for users who want to improve their command-line skills, such as students, system administra‐ tors, software developers, site reliability engineers, test engineers, and general Linux enthusiasts. Advanced Linux users may find some useful material as well, especially if they learned to run commands by trial and error and want to strengthen their con‐ ceptual understanding. To benefit most from this book, you should already be comfortable with the following topics (if you’re not, see Appendix A for a quick refresher): • Creating and editing text files with a text editor such as vim (vi), emacs, nano, or pico • Basic file-handling commands such as cp (copy), mv (move or rename), rm (remove or delete), and chmod (change file permissions) • Basic file-viewing commands such as cat (view an entire file) and less (view one page at a time) • Basic directory commands such as cd (change directory), ls (list files in a direc‐ tory), mkdir (create directory), rmdir (remove directory), and pwd (display your current directory name) • The basics of shell scripts: storing Linux commands in a file, making the file exe‐ cutable (with chmod 755 or chmod +x), and running the file • Viewing Linux’s built-in documentation, known as manpages, with the man com‐ mand (example: man cat displays documentation on the cat command) • Becoming the superuser with the sudo command for full access to your Linux system (example: sudo nano /etc/hosts edits the system file /etc/hosts, which is protected from ordinary users) If you also know common command-line features such as pattern matching for file‐ names (with the * and ? symbols), input/output redirection (< and >), and pipes (|), you are off to a good start. Your Shell I assume your Linux shell is bash, which is the default shell for most Linux distribu‐ tions. Whenever I write “the shell,” I mean bash. Most of the ideas I present apply to other shells too, such as zsh or dash; see Appendix B to help translate this book’s Preface | xi

3 The version of bash on macOS is ancient and missing important features. To upgrade bash, see the article “Upgrading Bash on macOS” by Daniel Weibel. examples for other shells. Much of the material will work unchanged on the Apple Mac Terminal as well, which runs zsh by default but can also run bash.3 Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program ele‐ ments such as variable or function names, databases, data types, environment variables, statements, and keywords. Constant width bold Shows commands or other text that should be typed literally by the user. Also used occasionally in command output to highlight text of interest. Constant width italic Shows text that should be replaced with user-supplied values or by values deter‐ mined by context. Also used for brief notes on the right side of code listings. Constant width highlighted Used in complex program listings to call attention to specific text. This element signifies a tip or suggestion. This element signifies a general note. This element indicates a warning or caution. xii | Preface

Using Code Examples Supplemental material (code examples, exercises, etc.) is available for download at https://efficientlinux.com/examples. If you have a technical question or a problem using the code examples, please send email to bookquestions@oreilly.com. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Efficient Linux at the Command Line by Daniel J. Barrett (O’Reilly). Copyright 2022 Daniel Barrett, 978-1-098-11340-7.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit https://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Preface | xiii

Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/efficient-linux. Email bookquestions@oreilly.com to comment or ask technical questions about this book. For news and information about our books and courses, visit https://oreilly.com. Find us on Facebook: https://facebook.com/oreilly Follow us on Twitter: https://twitter.com/oreillymedia Watch us on YouTube: https://www.youtube.com/oreillymedia Acknowledgments This book was a joy to write. Thanks to the amazing folks at O’Reilly, especially edi‐ tors Virginia Wilson and John Devins, production editors Caitlin Ghegan and Greg‐ ory Hyman, content manager Kristen Brown, copyeditor Kim Wimpsett, index editor Sue Klefstad, and the ever-helpful tools team. I’m also very grateful to the book’s tech‐ nical reviewers, Paul Bayer, John Bonesio, Dan Ritter, and Carla Schroder, for many insightful comments and criticisms. Thanks also to the Boston Linux Users Group for title suggestions. Special thanks to Maggie Johnson at Google for her kind permission to write the book. I’d like to offer my deepest thanks to Chip Andrews, Matthew Diaz, and Robert Strandh, who were fellow students at The Johns Hopkins University 35 years ago. They noticed my new and growing interest in Unix and, to my utter surprise, recom‐ mended that the Computer Science Department hire me as their next system admin‐ istrator. Their small act of faith changed the trajectory of my life. (Robert also gets credit for the tip on touch-typing in Chapter 3.) Thanks also to the creators and maintainers of Linux, GNU Emacs, Git, AsciiDoc, and many other open source tools—without these smart and generous people, my career would have been very different indeed. As always, thank you to my wonderful family, Lisa and Sophia, for their love and patience. xiv | Preface

PART I Core Concepts The first four chapters aim to increase your efficiency quickly, covering concepts and techniques that should be immediately useful. You’ll learn to combine commands with pipes, understand the responsibilities of the Linux shell, rapidly recall and edit commands from the past, and navigate the Linux filesystem with great speed.

CHAPTER 1 Combining Commands When you work in Windows, macOS, and most other operating systems, you proba‐ bly spend your time running applications like web browsers, word processors, spreadsheets, and games. A typical application is packed with features: everything that the designers thought their users would need. So, most applications are self- sufficient. They don’t rely on other apps. You might copy and paste between applica‐ tions from time to time, but for the most part, they’re separate. The Linux command line is different. Instead of big applications with tons of features, Linux supplies thousands of small commands with very few features. The command cat, for example, prints files on the screen and that’s about it. ls lists the files in a directory, mv renames files, and so on. Each command has a simple, fairly well- defined purpose. What if you need to do something more complicated? Don’t worry. Linux makes it easy to combine commands so their individual features work together to accomplish your goal. This way of working yields a very different mindset about computing. Instead of asking “Which app should I launch?” to achieve some result, the question becomes “Which commands should I combine?” In this chapter, you’ll learn how to arrange and run commands in different combina‐ tions to do what you need. To keep things simple, I’ll introduce just six Linux com‐ mands and their most basic uses so you can focus on the more complex and interesting part—combining them—without a huge learning curve. It’s a bit like learning to cook with six ingredients, or learning carpentry with just a hammer and a saw. (I’ll add more commands to your Linux toolbox in Chapter 5.) You’ll combine commands using pipes, a Linux feature that connects the output of one command to the input of another. As I introduce each command (wc, head, cut, grep, sort, and uniq), I’ll immediately demonstrate its use with pipes. Some 3

1 On US keyboards, the pipe symbol is on the same key as the backslash (\), usually located between the Enter and Backspace keys or between the left Shift key and Z. examples will be practical for daily Linux use, while others are just toy examples to demonstrate an important feature. Input, Output, and Pipes Most Linux commands read input from the keyboard, write output to the screen, or both. Linux has fancy names for this reading and writing: stdin (pronounced “standard input” or “standard in”) The stream of input that Linux reads from your keyboard. When you type any command at a prompt, you’re supplying data on stdin. stdout (pronounced “standard output” or “standard out”) The stream of output that Linux writes to your display. When you run the ls command to print filenames, the results appear on stdout. Now comes the cool part. You can connect the stdout of one command to the stdin of another, so the first command feeds the second. Let’s begin with the familiar ls -l command to list a large directory, such as /bin, in long format: $ ls -l /bin total 12104 -rwxr-xr-x 1 root root 1113504 Jun 6 2019 bash -rwxr-xr-x 1 root root 170456 Sep 21 2019 bsd-csh -rwxr-xr-x 1 root root 34888 Jul 4 2019 bunzip2 -rwxr-xr-x 1 root root 2062296 Sep 18 2020 busybox -rwxr-xr-x 1 root root 34888 Jul 4 2019 bzcat ⋮ -rwxr-xr-x 1 root root 5047 Apr 27 2017 znew This directory contains far more files than your display has lines, so the output quickly scrolls off-screen. It’s a shame that ls can’t print the information one screen‐ ful at a time, pausing until you press a key to continue. But wait: another Linux com‐ mand has that feature. The less command displays a file one screenful at a time: $ less myfile You can connect these two commands because ls writes to stdout and less can read from stdin. Use a pipe to send the output of ls to the input of less: $ ls -l /bin | less This combined command displays the directory’s contents one screenful at a time. The vertical bar (|) between the commands is the Linux pipe symbol.1 It connects the 4 | Chapter 1: Combining Commands