Rust in Action Systems programming concepts and techniques (Timothy Samuel McNamara) (Z-Library)

📄 File Format: PDF

💾 File Size: 18.3 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

M A N N I N G Timothy Samuel McNamara Systems programming concepts and techniques

📄 Page 2

Raw Pointer The cousins mut T and* *const T are the free radicals of the pointer world. Lightning fast, but wildly unsafe. Powers • Speed • Can interact with the outside world Weaknesses • Unsafe Box<T> Store anything in a box. Accepts almost any type for long-term storage. The workhorse of a new, safe programming era. Powers • Store a value in central storage in a location called “the heap” Weaknesses • Size increase Rc<T> The reference counted pointer, Rc<T> is Rust's competent, yet miserly bookkeeper. It knows who has borrowed what and when. Powers • Shared access to values Weaknesses • Size increase • Runtime cost • Not threadsafe Cell<T> An expert in metamorphosis, Cell<T> confers the ability to mutate immutable values. Powers • Interior mutability Weaknesses • Size increase • Performance RefCell<T> Performs mutation on immutable references with RefCel<T>. Its mind-bending powers come with some costs. Weaknesses • Size increase • Runtime cost • Lack of compile- time guarantees Cow<T> Why write something down when you only need to read it? Perhaps you only want to make modifications. This is the role of Cow (copy on write). Powers • Avoids writes when only read access is used Weaknesses • Possible size increase Arc<T> Your program’s main storage system. Vec<T> keeps your data orderly as values are created and destroyed. Powers • Grows dynamically as required Weaknesses • Can over allocate size RawVec<T> The bedrock of Vec<T> and other dynamically sized types. Understands how to provide a home for your data as needed. Powers • Grows dynamically as required • Works with the memory allocator to find space Weaknesses • Not directly applicable from your code Unique<T> Sole owner of a value, a unique pointer is guaranteed to possess full control. Powers • Base for types such as Strings, requiring exclusive possession of values. Weaknesses • Not appropriate for application code directly Arc<T> Arc<T> is Rust’s ambassador. It can share values across threads, guaranteeing that these will not interfere with each other. Powers • Shared access to values • Threadsafe Weaknesses • Size increase • Runtime cost String Acting as a guide on how to deal with the uncertainties of user input, String shows us how to build safe abstractions. Powers • Grows dynamically as required • Guarantees correct encoding at runtime Weaknesses • Can over allocate size Shared<T> Sharing ownership is hard. Shared<T> makes life a little bit easier. Powers • Shared ownership • Can align memory to T’s width, even when empty Weaknesses • Not appropriate for application code directly Powers • Interior mutability • Can be nested within Rc and Arc, which only accept immutable refs

📄 Page 3

Rust in Action

📄 Page 4

(This page has no text content)

📄 Page 5

Rust in Action SYSTEMS PROGRAMMING CONCEPTS AND TECHNIQUES TIM MCNAMARA MANN I NG SHELTER ISLAND

📄 Page 6

For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact Special Sales Department Manning Publications Co. 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 Email: orders@manning.com ©2021 by Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps. Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine. Development editor: Elesha Hyde Technical development editor: René van den Berg Manning Publications Co. Review editor: Mihaela Batinic 20 Baldwin Road Production editor: Deirdre S. Hiam PO Box 761 Copy editor: Frances Buran Shelter Island, NY 11964 Proofreader: Melody Dolab Technical proofreader: Jerry Kuch Typesetter: Dennis Dalinnik Cover designer: Marija Tudor ISBN: 9781617294556 Printed in the United States of America

📄 Page 7

To everyone aspiring to write safer software.

📄 Page 8

(This page has no text content)

📄 Page 9

contents preface xv acknowledgments xvii about this book xix about the author xxii about the cover illustration xxiii 1 Introducing Rust 1 1.1 Where is Rust used? 2 1.2 Advocating for Rust at work 3 1.3 A taste of the language 4 Cheating your way to “Hello, world!” 5 ■ Your first Rust program 7 1.4 Downloading the book’s source code 8 1.5 What does Rust look and feel like? 8 1.6 What is Rust? 11 Goal of Rust: Safety 12 ■ Goal of Rust: Productivity 16 Goal of Rust: Control 18 1.7 Rust’s big features 19 Performance 19 ■ Concurrency 20 ■ Memory efficiency 20vii

📄 Page 10

CONTENTSviii1.8 Downsides of Rust 20 Cyclic data structures 20 ■ Compile times 20 ■ Strictness 21 Size of the language 21 ■ Hype 21 1.9 TLS security case studies 21 Heartbleed 21 ■ Goto fail; 22 1.10 Where does Rust fit best? 23 Command-line utilities 23 ■ Data processing 24 ■ Extending applications 24 ■ Resource-constrained environments 24 Server-side applications 25 ■ Desktop applications 25 Desktop 25 ■ Mobile 25 ■ Web 26 ■ Systems programming 26 1.11 Rust’s hidden feature: Its community 26 1.12 Rust phrase book 26 PART 1 RUST LANGUAGE DISTINCTIVES ..........................29 2 Language foundations 31 2.1 Creating a running program 33 Compiling single files with rustc 33 ■ Compiling Rust projects with cargo 33 2.2 A glance at Rust’s syntax 34 Defining variables and calling functions 35 2.3 Numbers 36 Integers and decimal (floating-point) numbers 36 ■ Integers with base 2, base 8, and base 16 notation 37 ■ Comparing numbers 38 ■ Rational, complex numbers, and other numeric types 43 2.4 Flow control 45 For: The central pillar of iteration 45 ■ Continue: Skipping the rest of the current iteration 47 ■ While: Looping until a condition changes its state 47 ■ Loop: The basis for Rust’s looping constructs 48 ■ Break: Aborting a loop 48 ■ If, if else, and else: Conditional branching 49 ■ Match: Type-aware pattern matching 51 2.5 Defining functions 52 2.6 Using references 53 2.7 Project: Rendering the Mandelbrot set 54

📄 Page 11

CONTENTS ix2.8 Advanced function definitions 56 Explicit lifetime annotations 56 ■ Generic functions 58 2.9 Creating grep-lite 60 2.10 Making lists of things with arrays, slices, and vectors 63 Arrays 64 ■ Slices 65 ■ Vectors 66 2.11 Including third-party code 67 Adding support for regular expressions 68 ■ Generating the third- party crate documentation locally 69 ■ Managing Rust toolchains with rustup 70 2.12 Supporting command-line arguments 70 2.13 Reading from files 72 2.14 Reading from stdin 74 3 Compound data types 77 3.1 Using plain functions to experiment with an API 78 3.2 Modeling files with struct 80 3.3 Adding methods to a struct with impl 84 Simplifying object creation by implementing new() 84 3.4 Returning errors 87 Modifying a known global variable 87 ■ Making use of the Result return type 92 3.5 Defining and making use of an enum 95 Using an enum to manage internal state 96 3.6 Defining common behavior with traits 98 Creating a Read trait 98 ■ Implementing std::fmt::Display for your own types 99 3.7 Exposing your types to the world 102 Protecting private data 102 3.8 Creating inline documentation for your projects 103 Using rustdoc to render docs for a single source file 104 Using cargo to render docs for a crate and its dependencies 104 4 Lifetimes, ownership, and borrowing 107 4.1 Implementing a mock CubeSat ground station 108 Encountering our first lifetime issue 110 ■ Special behavior of primitive types 112

📄 Page 12

CONTENTSx4.2 Guide to the figures in this chapter 114 4.3 What is an owner? Does it have any responsibilities? 115 4.4 How ownership moves 115 4.5 Resolving ownership issues 118 Use references where full ownership is not required 119 Use fewer long-lived values 123 ■ Duplicate the value 128 Wrap data within specialty types 131 PART 2 DEMYSTIFYING SYSTEMS PROGRAMMING ...........135 5 Data in depth 137 5.1 Bit patterns and types 137 5.2 Life of an integer 139 Understanding endianness 142 5.3 Representing decimal numbers 143 5.4 Floating-point numbers 144 Looking inside an f32 144 ■ Isolating the sign bit 146 Isolating the exponent 146 ■ Isolate the mantissa 148 Dissecting a floating-point number 150 5.5 Fixed-point number formats 152 5.6 Generating random probabilities from random bytes 157 5.7 Implementing a CPU to establish that functions are also data 158 CPU RIA/1: The Adder 159 ■ Full code listing for CPU RIA/1: The Adder 163 ■ CPU RIA/2: The Multiplier 164 ■ CPU RIA/3: The Caller 167 ■ CPU 4: Adding the rest 173 6 Memory 175 6.1 Pointers 176 6.2 Exploring Rust’s reference and pointer types 178 Raw pointers in Rust 183 ■ Rust’s pointer ecosystem 185 Smart pointer building blocks 186 6.3 Providing programs with memory for their data 187 The stack 188 ■ The heap 190 ■ What is dynamic memory allocation? 194 ■ Analyzing the impact of dynamic memory allocation 199

📄 Page 13

CONTENTS xi6.4 Virtual memory 202 Background 202 ■ Step 1: Having a process scan its own memory 203 ■ Translating virtual addresses to physical addresses 205 ■ Step 2: Working with the OS to scan an address space 208 ■ Step 3: Reading from and writing to process memory 211 7 Files and storage 212 7.1 What is a file format? 213 7.2 Creating your own file formats for data storage 214 Writing data to disk with serde and the bincode format 214 7.3 Implementing a hexdump clone 217 7.4 File operations in Rust 219 Opening a file in Rust and controlling its file mode 219 Interacting with the filesystem in a type-safe manner with std::fs::Path 220 7.5 Implementing a key-value store with a log-structured, append-only storage architecture 222 The key-value model 222 ■ Introducing actionkv v1: An in-memory key-value store with a command-line interface 222 7.6 Actionkv v1: The front-end code 224 Tailoring what is compiled with conditional compilation 226 7.7 Understanding the core of actionkv: The libactionkv crate 228 Initializing the ActionKV struct 228 ■ Processing an individual record 230 ■ Writing multi-byte binary data to disk in a guaranteed byte order 232 ■ Validating I/O errors with checksums 234 ■ Inserting a new key-value pair into an existing database 236 ■ The full code listing for actionkv 237 Working with keys and values with HashMap and BTreeMap 241 Creating a HashMap and populating it with values 243 Retrieving values from HashMap and BTreeMap 244 ■ How to decide between HashMap and BTreeMap 245 ■ Adding a database index to actionkv v2.0 246 8 Networking 251 8.1 All of networking in seven paragraphs 252 8.2 Generating an HTTP GET request with reqwest 254

📄 Page 14

CONTENTSxii8.3 Trait objects 256 What do trait objects enable? 256 ■ What is a trait object? 256 Creating a tiny role-playing game: The rpg project 257 8.4 TCP 260 What is a port number? 261 ■ Converting a hostname to an IP address 261 8.5 Ergonomic error handling for libraries 268 Issue: Unable to return multiple error types 269 ■ Wrapping downstream errors by defining our own error type 272 Cheating with unwrap() and expect() 277 8.6 MAC addresses 277 Generating MAC addresses 279 8.7 Implementing state machines with Rust’s enums 281 8.8 Raw TCP 282 8.9 Creating a virtual networking device 282 8.10 “Raw” HTTP 283 9 Time and timekeeping 293 9.1 Background 294 9.2 Sources of time 296 9.3 Definitions 296 9.4 Encoding time 297 Representing time zones 298 9.5 clock v0.1.0: Teaching an application how to tell the time 298 9.6 clock v0.1.1: Formatting timestamps to comply with ISO 8601 and email standards 299 Refactoring the clock v0.1.0 code to support a wider architecture 300 Formatting the time 301 ■ Providing a full command-line interface 301 ■ clock v0.1.1: Full project 303 9.7 clock v0.1.2: Setting the time 305 Common behavior 306 ■ Setting the time for operating systems that use libc 306 ■ Setting the time on MS Windows 308 ■ clock v0.1.2: The full code listing 310 9.8 Improving error handling 313

📄 Page 15

CONTENTS xiii9.9 clock v0.1.3: Resolving differences between clocks with the Network Time Protocol (NTP) 314 Sending NTP requests and interpreting responses 314 ■ Adjusting the local time as a result of the server’s response 316 ■ Converting between time representations that use different precisions and epochs 318 ■ clock v0.1.3: The full code listing 319 10 Processes, threads, and containers 328 10.1 Anonymous functions 329 10.2 Spawning threads 330 Introduction to closures 330 ■ Spawning a thread 331 Effect of spawning a few threads 331 ■ Effect of spawning many threads 333 ■ Reproducing the results 335 ■ Shared variables 338 10.3 Differences between closures and functions 340 10.4 Procedurally generated avatars from a multithreaded parser and code generator 341 How to run render-hex and its intended output 342 ■ Single- threaded render-hex overview 342 ■ Spawning a thread per logical task 351 ■ Using a thread pool and task queue 353 10.5 Concurrency and task virtualization 360 Threads 362 ■ What is a context switch? 362 ■ Processes 363 WebAssembly 363 ■ Containers 363 ■ Why use an operating system (OS) at all? 363 11 Kernel 365 11.1 A fledgling operating system (FledgeOS) 365 Setting up a development environment for developing an OS kernel 366 ■ Verifying the development environment 367 11.2 Fledgeos-0: Getting something working 368 First boot 368 ■ Compilation instructions 370 ■ Source code listings 370 ■ Panic handling 374 ■ Writing to the screen with VGA-compatible text mode 375 ■ _start(): The main() function for FledgeOS 377 11.3 fledgeos-1: Avoiding a busy loop 377 Being power conscious by interacting with the CPU directly 377 fledgeos-1 source code 378 11.4 fledgeos-2: Custom exception handling 379 Handling exceptions properly, almost 379 ■ fledgeos-2 source code 380

📄 Page 16

CONTENTSxiv11.5 fledgeos-3: Text output 381 Writing colored text to the screen 381 ■ Controlling the in-memory representation of enums 382 ■ Why use enums? 382 ■ Creating a type that can print to the VGA frame buffer 382 ■ Printing to the screen 383 ■ fledgeos-3 source code 383 11.6 fledgeos-4: Custom panic handling 385 Implementing a panic handler that reports the error to the user 385 Reimplementing panic() by making use of core::fmt::Write 385 Implementing core::fmt::Write 386 ■ fledge-4 source code 387 12 Signals, interrupts, and exceptions 390 12.1 Glossary 391 Signals vs. interrupts 391 12.2 How interrupts affect applications 393 12.3 Software interrupts 395 12.4 Hardware interrupts 395 12.5 Signal handling 395 Default behavior 395 ■ Suspend and resume a program’s operation 397 ■ Listing all signals supported by the OS 399 12.6 Handling signals with custom actions 400 Global variables in Rust 401 ■ Using a global variable to indicate that shutdown has been initiated 402 12.7 Sending application-defined signals 405 Understanding function pointers and their syntax 405 12.8 Ignoring signals 407 12.9 Shutting down from deeply nested call stacks 408 Introducing the sjlj project 409 ■ Setting up intrinsics in a program 409 ■ Casting a pointer to another type 412 Compiling the sjlj project 413 ■ sjlj project source code 414 12.10 A note on applying these techniques to platforms without signals 417 12.11 Revising exceptions 417 index 419

📄 Page 17

preface No one knows whether reading a technical book is going to be worth the effort. These books can be expensive, dull, and poorly written. Even worse, there’s a good chance that you won’t learn anything. Luckily, this book is written by someone who under- stands that. This book’s first aim is to teach you Rust. Rust in Action presents large, working projects to promote your learning. Over the course of the book, you’ll write a data- base, a CPU emulator, an operating system kernel, and several other interesting proj- ects. You’ll even dabble with generative art. Each project is designed to enable you to explore the Rust programming language at your own pace. For those readers who know little Rust, there are many opportunities to expand the projects in whatever direction you choose. There is more to learning a programming language than studying its syntax and semantics, however. You are also joining a community. Unfortunately, established communities can create invisible barriers for new entrants because of their shared knowledge, jargon, and practices. One such barrier for many new Rust programmers is the concept of systems pro- gramming. Lots of programmers come to Rust without a background in that area. To compensate for this, Rust in Action has a second aim—to teach you systems pro- gramming. And, among other topics, you’ll learn about how memory, digital time- keeping, and device drivers work in the book’s 12 chapters. I hope this enables you to feel more comfortable when becoming a member of the Rust community. And we need you!xv

📄 Page 18

PREFACExvi Our societies depend on software, yet critical security holes are accepted as normal and, perhaps, inevitable. Rust demonstrates that these are neither. Moreover, our computers are filled with bloated, energy-intensive applications. Rust provides a viable alternative for developing software that is less demanding on these finite resources. Rust in Action is about empowerment. This book’s ultimate objective is to convince you of that. Rust is not reserved for a select group of experts. It is a tool that’s available for everyone. Well done for making it this far through your learning journey; it’s my pleasure to take you a few more steps.

📄 Page 19

acknowledgments Thank you to Katie for preventing me from collapsing and for picking me up when I fell down anyway. Thanks also to Florence and Octavia for your hugs and smiles, even when Dad was unable to play because he was writing. I’m indebted to so many that it feels unfair to list only a select few. There are many members of the Rust community who have supported the book’s development. Thou- sands of readers submitted corrections, questions, and suggestions via the liveBook during the book’s development. Every contribution has helped me refine the text. Thank you. I’m especially grateful to a small number of readers, many of whom have become friends. To Aï Maiga, Ana Hobden, Andrew Meredith, Andréy Lesnikóv, Andy Grove, Arturo J. Pérez, Bruce Mitchener, Cecile Tonglet, Daniel Carosone, Eric Ridge, Esteban Kuber, Florian Gilcher, Ian Battersby, Jane Lusby, Javier Viola, Jonathan Turner, Lachezar Lechev, Luciano Mammino, Luke Jones, Natalie Bloomfield, Oleksandr Kaleniuk, Olivia Ifrim, Paul Faria, Paul J. Symonds, Philipp Gniewosz, Rod Elias, Stephen Oates, Steve Klabnik, Tannr Allard, Thomas Lockney, and William Brown; interacting with you over the last four years has been a special privilege. To the book’s reviewers, I extend my warm thanks to Afshin Mehrabani, Alastair Smith, Bryce Darling, Christoffer Fink, Christopher Haupt, Damian Esteban, Federico Hernandez, Geert Van Laethem, Jeff Lim, Johan Liseborn, Josh Cohen, Konark Modi, Marc Cooper, Morgan Nelson, Ramnivas Laddad, Riccardo Moschetti, Sanket Naik, Sumant Tambe, Tim van Deurzen, Tom Barber, Wade Johnson, William Brown, Wil- liam Wheeler, and Yves Dorfsman. All of your comments were read. Many of thexvii

📄 Page 20

ACKNOWLEDGMENTSxviiiimprovements in the latter stages of the book’s development are owed to your thoughtful feedback. Two team members at Manning deserve special credit for their patience, profes- sionalism, and positivity: Elesha Hyde and Frances Buran have skillfully guided the book through many, many drafts. Thank you also to the rest of the development editors, including Bert Bates, Jerry Kuch, Mihaela Batinić, Rebecca Rinehart, René van den Berg, and Tim van Deurzen. My thanks also extends to the production editors, including Benjamin Berg, Deirdre Hiam, Jennifer Houle, and Paul Wells. Rust in Action had 16 releases during its MEAP process, and these would have been impossible without the support of many. Thank you to Aleksandar Dragosavljević, Ana Romac, Eleonor Gardner, Ivan Martinović, Lori Weidert, Marko Rajkovic, Matko Hrvatin, Mehmed Pasic, Melissa Ice, Mihaela Batinic, Owen Roberts, Radmila Ercegovac, and Rejhana Markanovic. Thanks also to the members of the marketing team, including Branko Latincic, Candace Gillhoolley, Cody Tankersley, Lucas Weber, and Stjepan Jureković. You’ve been a tremendous source of encouragement for me. The wider Manning team has also been very responsive and helpful. To Aira Dučić, Andrew Waldron, Barbara Mirecki, Branko Latincic, Breckyn Ely, Christopher Kaufmann, Dennis Dalinnik, Erin Twohey, Ian Hough, Josip Maras, Julia Quinn, Lana Klasic, Linda Kotlyarsky, Lori Kehrwald, and Melody Dolab, thank you for your assis- tance during the book’s development. And to Mike Stephens, thanks for kicking this whole life-changing process off. You warned me that it would be hard. You were right.