Zero to Production in Rust An Opinionated Introduction to Backend Development (Luca Palmieri) (z-library.sk, 1lib.sk, z-lib.sk)

📄 File Format: PDF

💾 File Size: 2.3 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

PRODUCTION AN OPINIONATED INTRODUCTION TO BACKEND DEVELOPMENT Luca Palmieri ZERO TO IN RUST Sold to kartik.ynwa@gmail.com

📄 Page 2

Contents Foreword 9 Preface 10 What Is This Book About . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Cloud-native applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Working in a team . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Who Is This Book For . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 1 Getting Started 13 1.1 Installing The Rust Toolchain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 1.1.1 Compilation Targets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 1.1.2 Release Channels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 1.1.3 What Toolchains Do We Need? . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 1.2 Project Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 1.3 IDEs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 1.3.1 Rust-analyzer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 1.3.2 IntelliJ Rust . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 1.3.3 What Should I Use? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 1.4 Inner Development Loop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 1.4.1 Faster Linking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.4.2 cargo-watch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 1.5 Continuous Integration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 1.5.1 CI Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 1.5.1.1 Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 1.5.1.2 Code Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 1.5.1.3 Linting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 1.5.1.4 Formatting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 1.5.1.5 Security Vulnerabilities . . . . . . . . . . . . . . . . . . . . . . . . . . 19 1.5.2 Ready-to-go CI Pipelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 2 Building An Email Newsletter 21 2.1 Our Driving Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.1.1 Problem-based Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.1.2 Course-correcting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.2 What Should Our Newsletter Do? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 2.2.1 Capturing Requirements: User Stories . . . . . . . . . . . . . . . . . . . . . . . 22 2.3 Working In Iterations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 2.3.1 Coming Up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 3 Sign Up A New Subscriber 24 3.1 Our Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 3.2 Choosing A Web Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 3.3 Our First Endpoint: A Basic Health Check . . . . . . . . . . . . . . . . . . . . . . . . 25 3.3.1 Wiring Up actix-web . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 3.3.2 Anatomy Of An actix-web Application . . . . . . . . . . . . . . . . . . . . . . 26 3.3.2.1 Server - HttpServer . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 3.3.2.2 Application - App . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 3.3.2.3 Endpoint - Route . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.3.2.4 Runtime - tokio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 3.3.3 Implementing The Health Check Handler . . . . . . . . . . . . . . . . . . . . . 30 3.4 Our First Integration Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 3.4.1 How Do You Test An Endpoint? . . . . . . . . . . . . . . . . . . . . . . . . . . 32 3.4.2 Where Should I Put My Tests? . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 3.4.3 Changing Our Project Structure For Easier Testing . . . . . . . . . . . . . . . . 34 3.5 Implementing Our First Integration Test . . . . . . . . . . . . . . . . . . . . . . . . . . 36 3.5.1 Polishing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 1

📄 Page 3

3.5.1.1 Clean Up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 3.5.1.2 Choosing A Random Port . . . . . . . . . . . . . . . . . . . . . . . . . 39 3.6 Refocus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 3.7 Working With HTML Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 3.7.1 Refining Our Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 3.7.2 Capturing Our Requirements As Tests . . . . . . . . . . . . . . . . . . . . . . . 43 3.7.3 Parsing Form Data From A POST Request . . . . . . . . . . . . . . . . . . . . . 45 3.7.3.1 Extractors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3.7.3.2 Form And FromRequest . . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.7.3.3 Serialisation In Rust: serde . . . . . . . . . . . . . . . . . . . . . . . . 49 3.7.3.4 Putting Everything Together . . . . . . . . . . . . . . . . . . . . . . . 51 3.8 Storing Data: Databases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 3.8.1 Choosing A Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 3.8.2 Choosing A Database Crate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 3.8.2.1 Compile-time Safety . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 3.8.2.2 Query Interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 3.8.2.3 Async Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 3.8.2.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 3.8.2.5 Our Pick: sqlx . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 3.8.3 Integration Testing With Side-effects . . . . . . . . . . . . . . . . . . . . . . . . 54 3.8.4 Database Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 3.8.4.1 Docker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 3.8.4.2 Database Migrations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 3.8.5 Writing Our First Query . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 3.8.5.1 Sqlx Feature Flags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 3.8.5.2 Configuration Management . . . . . . . . . . . . . . . . . . . . . . . . 60 3.8.5.3 Connecting To Postgres . . . . . . . . . . . . . . . . . . . . . . . . . . 63 3.8.5.4 Our Test Assertion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 3.8.5.5 Updating Our CI Pipeline . . . . . . . . . . . . . . . . . . . . . . . . 65 3.9 Persisting A New Subscriber . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 3.9.1 Application State In actix-web . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 3.9.2 actix-web Workers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 3.9.3 The Data Extractor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 3.9.4 The INSERT Query . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 3.10 Updating Our Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 3.10.1 Test Isolation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 3.11 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 4 Telemetry 78 4.1 Unknown Unknowns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 4.2 Observability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 4.3 Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 4.3.1 The log Crate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 4.3.2 actix-web’s Logger Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . 80 4.3.3 The Facade Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 4.4 Instrumenting POST /subscriptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 4.4.1 Interactions With External Systems . . . . . . . . . . . . . . . . . . . . . . . . 83 4.4.2 Think Like A User . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 4.4.3 Logs Must Be Easy To Correlate . . . . . . . . . . . . . . . . . . . . . . . . . . 85 4.5 Structured Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 4.5.1 The tracing Crate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 4.5.2 Migrating From log To tracing . . . . . . . . . . . . . . . . . . . . . . . . . . 88 4.5.3 tracing’s Span . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 4.5.4 Instrumenting Futures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 4.5.5 tracing’s Subscriber . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 4.5.6 tracing-subscriber . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 4.5.7 tracing-bunyan-formatter . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 2

📄 Page 4

4.5.8 tracing-log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 4.5.9 Removing Unused Dependencies . . . . . . . . . . . . . . . . . . . . . . . . . . 95 4.5.10 Cleaning Up Initialisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 4.5.11 Logs For Integration Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 4.5.12 Cleaning Up Instrumentation Code - tracing::instrument . . . . . . . . . . . 101 4.5.13 Protect Your Secrets - secrecy . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 4.5.14 Request Id . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 4.5.15 Leveraging The tracing Ecosystem . . . . . . . . . . . . . . . . . . . . . . . . 108 4.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 5 Going Live 109 5.1 We Must Talk About Deployments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 5.2 Choosing Our Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 5.2.1 Virtualisation: Docker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 5.2.2 Hosting: DigitalOcean . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 5.3 A Dockerfile For Our Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 5.3.1 Dockerfiles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 5.3.2 Build Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 5.3.3 Sqlx Offline Mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 5.3.4 Running An Image . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 5.3.5 Networking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 5.3.6 Hierarchical Configuration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 5.3.7 Database Connectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 5.3.8 Optimising Our Docker Image . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 5.3.8.1 Docker Image Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 5.3.8.2 Caching For Rust Docker Builds . . . . . . . . . . . . . . . . . . . . . 122 5.4 Deploy To DigitalOcean Apps Platform . . . . . . . . . . . . . . . . . . . . . . . . . . 123 5.4.1 Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 5.4.2 App Specification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 5.4.3 How To Inject Secrets Using Environment Variables . . . . . . . . . . . . . . . 126 5.4.4 Connecting To Digital Ocean’s Postgres Instance . . . . . . . . . . . . . . . . . 127 5.4.5 Environment Variables In The App Spec . . . . . . . . . . . . . . . . . . . . . . 130 5.4.6 One Last Push . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 6 Reject Invalid Subscribers #1 132 6.1 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 6.1.1 Domain Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 6.1.2 Security Constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 6.2 First Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 6.3 Validation Is A Leaky Cauldron . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 6.4 Type-Driven Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136 6.5 Ownership Meets Invariants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 6.5.1 AsRef . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 6.6 Panics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142 6.7 Error As Values - Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144 6.7.1 Converting parse To Return Result . . . . . . . . . . . . . . . . . . . . . . . . 145 6.8 Insightful Assertion Errors: claim . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 6.9 Unit Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 6.10 Handling A Result . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 6.10.1 match . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 6.10.2 The ? Operator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 6.10.3 400 Bad Request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150 6.11 The Email Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 6.12 The SubscriberEmail Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 6.12.1 Breaking The Domain Sub-Module . . . . . . . . . . . . . . . . . . . . . . . . . 151 6.12.2 Skeleton Of A New Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 6.13 Property-based Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 3

📄 Page 5

6.13.1 How To Generate Random Test Data With fake . . . . . . . . . . . . . . . . . 154 6.13.2 quickcheck Vs proptest . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 6.13.3 Getting Started With quickcheck . . . . . . . . . . . . . . . . . . . . . . . . . 155 6.13.4 Implementing The Arbitrary Trait . . . . . . . . . . . . . . . . . . . . . . . . 156 6.14 Payload Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 6.14.1 Refactoring With TryFrom . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 6.15 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 7 Reject Invalid Subscribers #2 164 7.1 Confirmation Emails . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 7.1.1 Subscriber Consent . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 7.1.2 The Confirmation User Journey . . . . . . . . . . . . . . . . . . . . . . . . . . . 164 7.1.3 The Implementation Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 7.2 EmailClient, Our Email Delivery Component . . . . . . . . . . . . . . . . . . . . . . 165 7.2.1 How To Send An Email . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 7.2.1.1 Choosing An Email API . . . . . . . . . . . . . . . . . . . . . . . . . 165 7.2.1.2 The Email Client Interface . . . . . . . . . . . . . . . . . . . . . . . . 166 7.2.2 How To Write A REST Client Using reqwest . . . . . . . . . . . . . . . . . . . 167 7.2.2.1 reqwest::Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 7.2.2.2 Connection Pooling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168 7.2.2.3 How To Reuse The Same reqwest::Client In actix-web . . . . . . 169 7.2.2.4 Configuring Our EmailClient . . . . . . . . . . . . . . . . . . . . . . 170 7.2.3 How To Test A REST Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 7.2.3.1 HTTP Mocking With wiremock . . . . . . . . . . . . . . . . . . . . . 173 7.2.3.2 wiremock::MockServer . . . . . . . . . . . . . . . . . . . . . . . . . . 175 7.2.3.3 wiremock::Mock . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 7.2.3.4 The Intent Of A Test Should Be Clear . . . . . . . . . . . . . . . . . . 175 7.2.3.5 Mock expectations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176 7.2.4 First Sketch Of EmailClient::send_email . . . . . . . . . . . . . . . . . . . . 177 7.2.4.1 reqwest::Client::post . . . . . . . . . . . . . . . . . . . . . . . . . 177 7.2.4.2 JSON body . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178 7.2.4.3 Authorization Token . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179 7.2.4.4 Executing The Request . . . . . . . . . . . . . . . . . . . . . . . . . . 182 7.2.5 Tightening Our Happy Path Test . . . . . . . . . . . . . . . . . . . . . . . . . . 183 7.2.5.1 Refactoring: Avoid Unnecessary Memory Allocations . . . . . . . . . 187 7.2.6 Dealing With Failures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188 7.2.6.1 Error Status Codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188 7.2.6.2 Timeouts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 7.2.6.3 Refactoring: Test Helpers . . . . . . . . . . . . . . . . . . . . . . . . . 193 7.2.6.4 Refactoring: Fail fast . . . . . . . . . . . . . . . . . . . . . . . . . . . 194 7.3 Skeleton And Principles For A Maintainable Test Suite . . . . . . . . . . . . . . . . . . 195 7.3.1 Why Do We Write Tests? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196 7.3.2 Why Don’t We Write Tests? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196 7.3.3 Test Code Is Still Code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196 7.3.4 Our Test Suite . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197 7.3.5 Test Discovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198 7.3.6 One Test File, One Crate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198 7.3.7 Sharing Test Helpers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198 7.3.8 Sharing Startup Logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201 7.3.8.1 Extracting Our Startup Code . . . . . . . . . . . . . . . . . . . . . . . 202 7.3.8.2 Testing Hooks In Our Startup Logic . . . . . . . . . . . . . . . . . . . 203 7.3.9 Build An API Client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 7.3.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210 7.4 Refocus . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210 7.5 Zero Downtime Deployments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210 7.5.1 Reliability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210 7.5.2 Deployment Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 4

📄 Page 6

7.5.2.1 Naive Deployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 7.5.2.2 Load Balancers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212 7.5.2.3 Rolling Update Deployments . . . . . . . . . . . . . . . . . . . . . . . 212 7.5.2.4 Digital Ocean App Platform . . . . . . . . . . . . . . . . . . . . . . . 213 7.6 Database Migrations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214 7.6.1 State Is Kept Outside The Application . . . . . . . . . . . . . . . . . . . . . . . 214 7.6.2 Deployments And Migrations . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214 7.6.3 Multi-step Migrations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214 7.6.4 A New Mandatory Column . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215 7.6.4.1 Step 1: Add As Optional . . . . . . . . . . . . . . . . . . . . . . . . . 215 7.6.4.2 Step 2: Start Using The New Column . . . . . . . . . . . . . . . . . . 215 7.6.4.3 Step 3: Backfill And Mark As NOT NULL . . . . . . . . . . . . . . . . . 215 7.6.5 A New Table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216 7.7 Sending A Confirmation Email . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216 7.7.1 A Static Email . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216 7.7.1.1 Red test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217 7.7.1.2 Green test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218 7.7.2 A Static Confirmation Link . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 7.7.2.1 Red Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220 7.7.2.2 Green Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221 7.7.2.3 Refactor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222 7.7.3 Pending Confirmation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 7.7.3.1 Red test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223 7.7.3.2 Green Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 7.7.4 Skeleton of GET /subscriptions/confirm . . . . . . . . . . . . . . . . . . . . 225 7.7.4.1 Red Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 7.7.4.2 Green Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226 7.7.5 Connecting The Dots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227 7.7.5.1 Red Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227 7.7.5.2 Green Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228 7.7.5.3 Refactor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232 7.7.6 Subscription Tokens . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234 7.7.6.1 Red Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234 7.7.6.2 Green Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235 7.8 Database Transactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240 7.8.1 All Or Nothing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240 7.8.2 Transactions In Postgres . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240 7.8.3 Transactions In Sqlx . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 7.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244 8 Error Handling 245 8.1 What Is The Purpose Of Errors? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 8.1.1 Internal Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 8.1.1.1 Enable The Caller To React . . . . . . . . . . . . . . . . . . . . . . . 245 8.1.1.2 Help An Operator To Troubleshoot . . . . . . . . . . . . . . . . . . . 246 8.1.2 Errors At The Edge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247 8.1.2.1 Help A User To Troubleshoot . . . . . . . . . . . . . . . . . . . . . . . 247 8.1.3 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248 8.2 Error Reporting For Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249 8.2.1 Keeping Track Of The Error Root Cause . . . . . . . . . . . . . . . . . . . . . 251 8.2.2 The Error Trait . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255 8.2.2.1 Trait Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256 8.2.2.2 Error::source . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256 8.3 Errors For Control Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258 8.3.1 Layering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258 8.3.2 Modelling Errors as Enums . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259 8.3.3 The Error Type Is Not Enough . . . . . . . . . . . . . . . . . . . . . . . . . . . 260 5

📄 Page 7

8.3.4 Removing The Boilerplate With thiserror . . . . . . . . . . . . . . . . . . . . 263 8.4 Avoid “Ball Of Mud” Error Enums . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264 8.4.1 Using anyhow As Opaque Error Type . . . . . . . . . . . . . . . . . . . . . . . 268 8.4.2 anyhow Or thiserror? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270 8.5 Who Should Log Errors? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270 8.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271 9 Naive Newsletter Delivery 273 9.1 User Stories Are Not Set In Stone . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273 9.2 Do Not Spam Unconfirmed Subscribers . . . . . . . . . . . . . . . . . . . . . . . . . . 273 9.2.1 Set Up State Using The Public API . . . . . . . . . . . . . . . . . . . . . . . . 275 9.2.2 Scoped Mocks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275 9.2.3 Green Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276 9.3 All Confirmed Subscribers Receive New Issues . . . . . . . . . . . . . . . . . . . . . . . 276 9.3.1 Composing Test Helpers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276 9.4 Implementation Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278 9.5 Body Schema . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278 9.5.1 Test Invalid Inputs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279 9.6 Fetch Confirmed Subscribers List . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280 9.7 Send Newsletter Emails . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282 9.7.1 context Vs with_context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283 9.8 Validation Of Stored Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284 9.8.1 Responsibility Boundaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286 9.8.2 Follow The Compiler . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 288 9.8.3 Remove Some Boilerplate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289 9.9 Limitations Of The Naive Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290 9.10 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291 10 Securing Our API 292 10.1 Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292 10.1.1 Drawbacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292 10.1.1.1 Something They Know . . . . . . . . . . . . . . . . . . . . . . . . . . 292 10.1.1.2 Something They Have . . . . . . . . . . . . . . . . . . . . . . . . . . . 292 10.1.1.3 Something They Are . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292 10.1.2 Multi-factor Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293 10.2 Password-based Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293 10.2.1 Basic Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 293 10.2.1.1 Extracting Credentials . . . . . . . . . . . . . . . . . . . . . . . . . . 293 10.2.2 Password Verification - Naive Approach . . . . . . . . . . . . . . . . . . . . . . 297 10.2.3 Password Storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299 10.2.3.1 No Need To Store Raw Passwords . . . . . . . . . . . . . . . . . . . . 299 10.2.3.2 Using A Cryptographic Hash . . . . . . . . . . . . . . . . . . . . . . . 300 10.2.3.3 Preimage Attack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304 10.2.3.4 Naive Dictionary Attack . . . . . . . . . . . . . . . . . . . . . . . . . 304 10.2.3.5 Dictionary Attack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305 10.2.3.6 Argon2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305 10.2.3.7 Salting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307 10.2.3.8 PHC String Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309 10.2.4 Do Not Block The Async Executor . . . . . . . . . . . . . . . . . . . . . . . . . 312 10.2.4.1 Tracing Context Is Thread-Local . . . . . . . . . . . . . . . . . . . . . 316 10.2.5 User Enumeration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317 10.3 Is it safe? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 320 10.3.1 Transport Layer Security (TLS) . . . . . . . . . . . . . . . . . . . . . . . . . . 320 10.3.2 Password Reset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321 10.3.3 Interaction Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321 10.3.4 Machine To Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321 10.3.4.1 Client Credentials via OAuth2 . . . . . . . . . . . . . . . . . . . . . . 321 6

📄 Page 8

10.3.5 Person Via Browser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321 10.3.5.1 Federated Identity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322 10.3.6 Machine to machine, on behalf of a person . . . . . . . . . . . . . . . . . . . . . 322 10.4 Interlude: Next Steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 322 10.5 Login Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323 10.5.1 Serving HTML Pages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323 10.6 Login . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325 10.6.1 HTML Forms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325 10.6.2 Redirect On Success . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327 10.6.3 Processing Form Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328 10.6.3.1 Building An authentication Module . . . . . . . . . . . . . . . . . . 329 10.6.3.2 Rejecting Invalid Credentials . . . . . . . . . . . . . . . . . . . . . . . 332 10.6.4 Contextual Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334 10.6.4.1 Naive Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334 10.6.4.2 Query Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335 10.6.4.3 Cross-Site Scripting (XSS) . . . . . . . . . . . . . . . . . . . . . . . . 337 10.6.4.4 Message Authentication Codes . . . . . . . . . . . . . . . . . . . . . . 338 10.6.4.5 Add An HMAC Tag To Protect Query Parameters . . . . . . . . . . . 338 10.6.4.6 Verifying The HMAC Tag . . . . . . . . . . . . . . . . . . . . . . . . . 342 10.6.4.7 Error Messages Must Be Ephemeral . . . . . . . . . . . . . . . . . . . 344 10.6.4.8 What Is A Cookie? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345 10.6.4.9 An Integration Test For Login Failures . . . . . . . . . . . . . . . . . 345 10.6.4.10 How To Set A Cookie In actix-web . . . . . . . . . . . . . . . . . . . 349 10.6.4.11 An Integration Test For Login Failures - Part 2 . . . . . . . . . . . . . 349 10.6.4.12 How To Read A Cookie In actix-web . . . . . . . . . . . . . . . . . . 351 10.6.4.13 How To Delete A Cookie In actix-web . . . . . . . . . . . . . . . . . 353 10.6.4.14 Cookie Security . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 354 10.6.4.15 actix-web-flash-messages . . . . . . . . . . . . . . . . . . . . . . . 355 10.7 Sessions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 358 10.7.1 Session-based Authentication . . . . . . . . . . . . . . . . . . . . . . . . . . . . 358 10.7.2 Session Store . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 358 10.7.3 Choosing A Session Store . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 358 10.7.3.1 Postgres . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359 10.7.3.2 Redis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359 10.7.4 actix-session . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359 10.7.4.1 Redis In Our Development Setup . . . . . . . . . . . . . . . . . . . . . 361 10.7.4.2 Redis On Digital Ocean . . . . . . . . . . . . . . . . . . . . . . . . . . 362 10.7.5 Admin Dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 362 10.7.5.1 Redirect On Login Success . . . . . . . . . . . . . . . . . . . . . . . . 362 10.7.5.2 Session . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364 10.7.5.3 A Typed Interface To Session . . . . . . . . . . . . . . . . . . . . . . 367 10.7.5.4 Reject Unauthenticated Users . . . . . . . . . . . . . . . . . . . . . . 369 10.8 Seed Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 370 10.8.1 Database Migration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 370 10.8.2 Password Reset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371 10.8.2.1 Form Skeleton . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371 10.8.2.2 Unhappy Path: New Passwords Do Not Match . . . . . . . . . . . . . 375 10.8.2.3 Unhappy Path: The Current Password Is Invalid . . . . . . . . . . . . 377 10.8.2.4 Unhappy Path: The New Password Is Too Short . . . . . . . . . . . . 379 10.8.2.5 Logout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379 10.8.2.6 Happy Path: The Password Was Changed Successfully . . . . . . . . 382 10.9 Refactoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384 10.9.1 How To Write An actix-web Middleware . . . . . . . . . . . . . . . . . . . . . 385 10.10Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 389 11 Fault-tolerant Workflows 391 11.1 POST /admin/newsletters - A Refresher . . . . . . . . . . . . . . . . . . . . . . . . . 391 7

📄 Page 9

11.2 Our Goal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392 11.3 Failure Modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392 11.3.1 Invalid Inputs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 392 11.3.2 Network I/O . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393 11.3.2.1 Postgres . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393 11.3.2.2 Postmark - API Errors . . . . . . . . . . . . . . . . . . . . . . . . . . 393 11.3.3 Application Crashes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393 11.3.4 Author Actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394 11.4 Idempotency: An Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394 11.4.1 Idempotency In Action: Payments . . . . . . . . . . . . . . . . . . . . . . . . . 394 11.4.2 Idempotency Keys . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 395 11.4.3 Concurrent Requests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396 11.5 Requirements As Tests #1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 396 11.6 Implementation Strategies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397 11.6.1 Stateful Idempotency: Save And Replay . . . . . . . . . . . . . . . . . . . . . . 397 11.6.2 Stateless Idempotency: Deterministic Key Generation . . . . . . . . . . . . . . 397 11.6.3 Time Is a Tricky Beast . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398 11.6.4 Making A Choice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398 11.7 Idempotency Store . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398 11.7.1 Which Database Should We Use? . . . . . . . . . . . . . . . . . . . . . . . . . . 398 11.7.2 Schema . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 398 11.8 Save And Replay . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 400 11.8.1 Read Idempotency Key . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 400 11.8.2 Retrieve Saved Responses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402 11.8.3 Save Responses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 405 11.8.3.1 MessageBody and HTTP Streaming . . . . . . . . . . . . . . . . . . . 406 11.8.3.2 Array Of Composite Postgres Types . . . . . . . . . . . . . . . . . . . 408 11.8.3.3 Plug It In . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410 11.9 Concurrent Requests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410 11.9.1 Requirements As Tests #2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410 11.9.2 Synchronization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 411 11.9.2.1 Transaction Isolation Levels . . . . . . . . . . . . . . . . . . . . . . . 415 11.10Dealing With Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 416 11.10.1 Distributed Transactions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 418 11.10.2 Backward Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 418 11.10.3 Forward Recovery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419 11.10.4 Asynchronous Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 419 11.10.4.1 newsletter_issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . 420 11.10.4.2 issue_delivery_queue . . . . . . . . . . . . . . . . . . . . . . . . . . 421 11.10.4.3 POST /admin/newsletters . . . . . . . . . . . . . . . . . . . . . . . . 421 11.10.4.4 Email Processing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 422 11.10.4.5 Worker loop . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425 11.10.4.6 Launching Background Workers . . . . . . . . . . . . . . . . . . . . . 427 11.10.4.7 Updating The Test Suite . . . . . . . . . . . . . . . . . . . . . . . . . 429 11.11Epilogue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 432 8

📄 Page 10

Foreword When you read these lines, Rust has achieved its biggest goal: make an offer to programmers to write their production systems in a different language. By the end of the book, it is still your choice to follow that path, but you have all you need to consider the offer. I’ve been part of the growth process of two widely different languages: Ruby and Rust - by programming them, but also by running events, being part of their project management and running business around them. Through that, I had the privilege of being in touch with many of the creators of those languages and consider some of them friends. Rust has been my one chance in life to see and help a language grow from the experimental stage to adoption in the industry. I’ll let you in on a secret I learned along the way: programming languages are not adopted because of a feature checklist. It’s a complex interplay between good technology, the ability to speak about it and finding enough people willing to take long bets. When I write these lines, over 5000 people have contributed to the Rust project, often for free, in their spare time - because they believe in that bet. But you don’t have to contribute to the compiler or be recorded in a git log to contribute to Rust. Luca’s book is such a contribution: it gives newcomers a perspective on Rust and promotes the good work of those many people. Rust was never intended to be a research platform - it was always meant as a programming language solving real, tangible issues in large codebases. It is no surprise that it comes out of an organization that maintains a very large and complex codebase - Mozilla, creators of Firefox. When I joined Rust, it was just ambition - but the ambition was to industrialize research to make the software of tomorrow better. With all of its theoretical concepts, linear typing, region based memory management, the programming language was always meant for everyone. This reflects in its lingo: Rust uses accessible names like “Ownership” and “Borrowing” for the concepts I just mentioned. Rust is an industry language, through and through. And that reflects in its proponents: I’ve known Luca for years as a community member who knows a ton about Rust. But his deeper interest lies in convincing people that Rust is worth a try by addressing their needs. The title and structure of this book reflects one of the core values of Rust: to find its worth in writing production software that is solid and works. Rust shows its strength in the care and knowledge that went into it to write stable software productively. Such an experience is best found with a guide and Luca is one of the best guides you can find around Rust. Rust doesn’t solve all of your problems, but it has made an effort to eliminate whole categories of mistakes. There’s the view out there that safety features in languages are there because of the incompetence of programmers. I don’t subscribe to this view. Emily Dunham, captured it well in her RustConf 2017 keynote: “safe code allows you to take better risks”. Much of the magic of the Rust community lies in this positive view of its users: whether you are a newcomer or an experienced developer, we trust your experience and your decision-making. In this book, Luca offers a lot of new knowledge that can be applied even outside of Rust, well explained in the context of daily software praxis. I wish you a great time reading, learning and contemplating. Florian Gilcher, Management Director of Ferrous Systems and Co-Founder of the Rust Foundation 9

📄 Page 11

Preface What Is This Book About The world of backend development is vast. The context you operate into has a huge impact on the optimal tools and practices to tackle the problem you are working on. For example, trunk-based development works extremely well to write software that is continuously deployed in a Cloud environment. The very same approach might fit poorly the business model and the challenges faced by a team that sells software that is hosted and run on-premise by their customers - they are more likely to benefit from a Gitflow approach. If you are working alone, you can just push straight to main. There are few absolutes in the field of software development and I feel it’s beneficial to clarify your point of view when evaluating the pros and cons of any technique or approach. Zero To Production will focus on the challenges of writing Cloud-native applications in a team of four or five engineers with different levels of experience and proficiency. Cloud-native applications Defining what Cloud-native application means is, by itself, a topic for a whole new book1. Instead of prescribing what Cloud-native applications should look like, we can lay down what we expect them to do. Paraphrasing Cornelia Davis, we expect Cloud-native applications: • To achieve high-availability while running in fault-prone environments; • To allow us to continuously release new versions with zero downtime; • To handle dynamic workloads (e.g. request volumes). These requirements have a deep impact on the viable solution space for the architecture of our software. High availability implies that our application should be able to serve requests with no downtime even if one or more of our machines suddenly starts failing (a common occurrence in a Cloud environment2). This forces our application to be distributed - there should be multiple instances of it running on multiple machines. The same is true if we want to be able to handle dynamic workloads - we should be able to measure if our system is under load and throw more compute at the problem by spinning up new instances of the application. This also requires our infrastructure to be elastic to avoid overprovisioning and its associated costs. Running a replicated application influences our approach to data persistence - we will avoid using the local filesystem as our primary storage solution, relying instead on databases for our persistence needs. Zero To Production will thus extensively cover topics that might seem tangential to pure backend application development. But Cloud-native software is all about rainbows and DevOps, therefore we will be spending plenty of time on topics traditionally associated with the craft of operating systems. We will cover how to instrument your Rust application to collect logs, traces and metrics to be able to observe our system. 1Like the excellent Cloud-native patterns by Cornelia Davis! 2For example, many companies run their software on AWS Spot Instances to reduce their infrastructure bills. The price of Spot instances is the result of a continuous auction and it can be substantially cheaper than the corresponding full price for On Demand instances (up to 90% cheaper!). There is one gotcha: AWS can decommission your Spot instances at any point in time. Your software must be fault-tolerant to leverage this opportunity. 10

📄 Page 12

We will cover how to set up and evolve your database schema via migrations. We will cover all the material required to use Rust to tackle both day one and day two concerns of a Cloud-native API. Working in a team The impact of those three requirements goes beyond the technical characteristics of our system: it influences how we build our software. To be able to quickly release a new version of our application to our users we need to be sure that our application works. If you are working on a solo project you can rely on your thorough understanding of the whole system: you wrote it, it might be small enough to fit entirely in your head at any point in time.3 If you are working in a team on a commercial project, you will be very often working on code that was neither written or reviewed by you. The original authors might not be around anymore. You will end up being paralysed by fear every time you are about to introduce changes if you are relying on your comprehensive understanding of what the code does to prevent it from breaking. You want automated tests. Running on every commit. On every branch. Keeping main healthy. You want to leverage the type system to make undesirable states difficult or impossible to represent. You want to use every tool at your disposal to empower each member of the team to evolve that piece of software. To contribute fully to the development process even if they might not be as experienced as you or equally familiar with the codebase or the technologies you are using. Zero To Production will therefore put a strong emphasis on test-driven development and continuous integration from the get-go - we will have a CI pipeline set up before we even have a web server up and running! We will be covering techniques such as black-box testing for APIs and HTTP mocking - not wildly popular or well documented in the Rust community yet extremely powerful. We will also borrow terminology and techniques from the Domain Driven Design world, combining them with type-driven design to ensure the correctness of our systems. Our main focus is enterprise software: correct code which is expressive enough to model the domain and supple enough to support its evolution over time. We will thus have a bias for boring and correct solutions, even if they incur a performance overhead that could be optimised away with a more careful and chiseled approach. Get it to run first, optimise it later (if needed). Who Is This Book For The Rust ecosystem has had a remarkable focus on smashing adoption barriers with amazing material geared towards beginners and newcomers, a relentless effort that goes from documentation to the continuous polishing of the compiler diagnostics. There is value in serving the largest possible audience. At the same time, trying to always speak to everybody can have harmful side-effects: material that would be relevant to intermediate and advanced users but definitely too much too soon for beginners ends up being neglected. I struggled with it first-hand when I started to play around with async/await. There was a significant gap between the knowledge I needed to be productive and the knowledge I 3Assuming you wrote it recently. Your past self from one year ago counts as a stranger for all intents and purposes in the world of software development. Pray that your past self wrote comments for your present self if you are about to pick up again an old project of yours. 11

📄 Page 13

had built reading The Rust Book or working in the Rust numerical ecosystem. I wanted to get an answer to a straight-forward question: Can Rust be a productive language for API development? Yes. But it can take some time to figure out how. That’s why I am writing this book. I am writing this book for the seasoned backend developers who have read The Rust Book and are now trying to port over a couple of simple systems. I am writing this book for the new engineers on my team, a trail to help them make sense of the codebases they will contribute to over the coming weeks and months. I am writing this book for a niche whose needs I believe are currently underserved by the articles and resources available in the Rust ecosystem. I am writing this book for myself a year ago. To socialise the knowledge gained during the journey: what does your toolbox look like if you are using Rust for backend development in 2022? What are the design patterns? Where are the pitfalls? If you do not fit this description but you are working towards it I will do my best to help you on the journey: while we won’t be covering a lot of material directly (e.g. most Rust language features) I will try to provide references and links where needed to help you pick up/brush off those concepts along the way. Let’s get started. 12

📄 Page 14

1 Getting Started There is more to a programming language than the language itself: tooling is a key element of the experience of using the language. The same applies to many other technologies (e.g. RPC frameworks like gRPC or Apache Avro) and it often has a disproportionate impact on the uptake (or the demise) of the technology itself. Tooling should therefore be treated as a first-class concern both when designing and teaching the language itself. The Rust community has put tooling at the forefront since its early days: it shows. We are now going to take a brief tour of a set of tools and utilities that are going to be useful in our journey. Some of them are officially supported by the Rust organisation, others are built and maintained by the community. 1.1 Installing The Rust Toolchain There are various ways to install Rust on your system, but we are going to focus on the recommended path: via rustup. Instructions on how to install rustup itself can be found at https://rustup.rs. rustup is more than a Rust installer - its main value proposition is toolchain management. A toolchain is the combination of a compilation target and a release channel. 1.1.1 Compilation Targets The main purpose of the Rust compiler is to convert Rust code into machine code - a set of instruc- tions that your CPU and operating system can understand and execute. Therefore you need a different backend of the Rust compiler for each compilation target, i.e. for each platform (e.g. 64-bit Linux or 64-bit OSX) you want to produce a running executable for. The Rust project strives to support a broad range of compilation targets with various level of guar- antees. Targets are split into tiers, from “guaranteed-to-work” Tier 1 to “best-effort” Tier 3. An exhaustive and up-to-date list can be found here. 1.1.2 Release Channels The Rust compiler itself is a living piece of software: it continuously evolves and improves with the daily contributions of hundreds of volunteers. The Rust project strives for stability without stagnation. Quoting from Rust’s documentation: [..] you should never have to fear upgrading to a new version of stable Rust. Each upgrade should be painless, but should also bring you new features, fewer bugs, and faster compile times. That is why, for application development, you should generally rely on the latest released version of the compiler to run, build and test your software - the so-called stable channel. A new version of the compiler is released on the stable channel every six weeks4 - the latest version at the time of writing is v1.43.15. There are two other release channels: • beta, the candidate for the next release; • nightly, built from the master branch of rust-lang/rust every night, thus the name. 4More details on the release schedule can be found here. 5You can check the next version and its release date at Rust forge. 13

📄 Page 15

Testing your software using the beta compiler is one of the many ways to support the Rust project - it helps catching bugs before the release date6. nightly serves a different purpose: it gives early adopters access to unfinished features7 before they are released (or even on track to be stabilised!). I would invite you to think twice if you are planning to run production software on top of the nightly compiler: it’s called unstable for a reason. 1.1.3 What Toolchains Do We Need? Installing rustup will give you out of the box the latest stable compiler with your host platform as a target. stable is the release channel that we will be using throughout the book to build, test and run our code. You can update your toolchains with rustup update, while rustup toolchain list will give you an overview of what is installed on your system. We will not need (or perform) any cross-compiling - our production workloads will be running in containers, hence we do not need to cross-compile from our development machine to the target host used in our production environment. 1.2 Project Setup A toolchain installation via rustup bundles together various components. One of them is the Rust compiler itself, rustc. You can check it out with rustc --version You will not be spending a lot of quality time working directly with rustc - your main interface for building and testing Rust applications will be cargo, Rust’s build tool. You can double-check everything is up and running with cargo --version Let’s use cargo to create the skeleton of the project we will be working on for the whole book: cargo new zero2prod You should have a new zero2prod folder, with the following file structure: zero2prod/ Cargo.toml .gitignore .git src/ main.rs The project is already a git repository, out of the box. If you are planning on hosting the project on GitHub, you just need to create a new empty repository and run cd zero2prod git add . git commit -am "Project skeleton" git remote add origin git@github.com:YourGitHubNickName/zero2prod.git git push -u origin main We will be using GitHub as a reference given its popularity and the recently released GitHub Actions feature for CI pipelines, but you are of course free to choose any other git hosting solution (or none at all). 6It’s fairly rare for beta releases to contain issues thanks to the CI/CD setup of the Rust project. One of its most interesting components is crater, a tool designed to scrape crates.io and GitHub for Rust projects to build them and run their test suites to identify potential regressions. Pietro Albini gave an awesome overview of the Rust release process in his Shipping a compiler every six weeks talk at RustFest 2019. 7You can check the list of feature flags available on nightly in The Unstable Book. Spoiler: there are loads. 14

📄 Page 16

1.3 IDEs The project skeleton is ready, it is now time to fire up your favourite editor so that we can start messing around with it. Different people have different preferences but I would argue that the bare minimum you want to have, especially if you are starting out with a new programming language, is a setup that supports syntax highlighting, code navigation and code completion. Syntax highlighting gives you immediate feedback on glaring syntax errors, while code navigation and code completion enable “exploratory” programming: jumping in and out of the source of your dependencies, quick access to the available methods on a struct or an enum you imported from a crate without having to continuously switch between your editor and docs.rs. You have two main options for your IDE setup: rust-analyzer and IntelliJ Rust. 1.3.1 Rust-analyzer rust-analyzer8 is an implementation of the Language Server Protocol for Rust. The Language Server Protocol makes it easy to leverage rust-analyzer in many different editors, including but not limited to VS Code, Emacs, Vim/NeoVim and Sublime Text 3. Editor-specific setup instructions can be found here. 1.3.2 IntelliJ Rust IntelliJ Rust provides Rust support to the suite of editors developed by JetBrains. If you don’t have a JetBrains license9, IntelliJ IDEA is available for free and supports IntelliJ Rust. If you have a JetBrains license, CLion is your go-to editor for Rust in JetBrains’ IDE suite. 1.3.3 What Should I Use? As of March 2022, IntelliJ Rust should be preferred. Although rust-analyzer is promising and has shown incredible progress over the last year, it is still quite far from delivering an IDE experience on par with what IntelliJ Rust offers today. On the other hand, IntelliJ Rust forces you to work with a JetBrains’ IDE, which you might or might not be willing to. If you’d like to stick to your editor of choice look for its rust-analyzer integration/plugin. It is worth mentioning that rust-analyzer is part of a larger library-ification effort taking place within the Rust compiler: there is overlap between rust-analyzer and rustc, with a lot of duplicated effort. Evolving the compiler’s codebase into a set of re-usable modules will allow rust-analyzer to leverage an increasingly larger subset of the compiler codebase, unlocking the on-demand analysis capabilities required to offer a top-notch IDE experience. An interesting space to keep an eye on in the future10. 1.4 Inner Development Loop While working on our project, we will be going through the same steps over and over again: • Make a change; • Compile the application; • Run tests; • Run the application. 8rust-analyzer is not the first attempt to implement the LSP for Rust: RLS was its predecessor. RLS took a batch- processing approach: every little change to any of the files in a project would trigger re-compilation of the whole project. This strategy was fundamentally limited and it led to poor performance and responsiveness. RFC2912 formalised the “retirement” of RLS as the blessed LSP implementation for Rust in favour of rust-analyzer. 9Students and teachers can claim a free JetBrains educational license. 10Check their Next Few Years blog post for more details on rust-analyzer’s roadmap and main concerns going forward. 15

📄 Page 17

This is also known as the inner development loop. The speed of your inner development loop is as an upper bound on the number of iterations that you can complete in a unit of time. If it takes 5 minutes to compile and run the application, you can complete at most 12 iterations in an hour. Cut it down to 2 minutes and you can now fit in 30 iterations in the same hour! Rust does not help us here - compilation speed can become a pain point on big projects. Let’s see what we can do to mitigate the issue before moving forward. 1.4.1 Faster Linking When looking at the inner development loop, we are primarily looking at the performance of incre- mental compilation - how long it takes cargo to rebuild our binary after having made a small change to the source code. A sizeable chunk of time is spent in the linking phase - assembling the actual binary given the outputs of the earlier compilation stages. The default linker does a good job, but there are faster alternatives depending on the operating system you are using: • lld on Windows and Linux, a linker developed by the LLVM project; • zld on MacOS. To speed up the linking phase you have to install the alternative linker on your machine and add this configuration file to the project: # .cargo/config.toml # On Windows # ``` # cargo install -f cargo-binutils # rustup component add llvm-tools-preview # ``` [target.x86_64-pc-windows-msvc] rustflags = ["-C", "link-arg=-fuse-ld=lld"] [target.x86_64-pc-windows-gnu] rustflags = ["-C", "link-arg=-fuse-ld=lld"] # On Linux: # - Ubuntu, `sudo apt-get install lld clang` # - Arch, `sudo pacman -S lld clang` [target.x86_64-unknown-linux-gnu] rustflags = ["-C", "linker=clang", "-C", "link-arg=-fuse-ld=lld"] # On MacOS, `brew install michaeleisel/zld/zld` [target.x86_64-apple-darwin] rustflags = ["-C", "link-arg=-fuse-ld=/usr/local/bin/zld"] [target.aarch64-apple-darwin] rustflags = ["-C", "link-arg=-fuse-ld=/usr/local/bin/zld"] There is ongoing work on the Rust compiler to use lld as the default linker where possible - soon enough this custom configuration will not be necessary to achieve higher compilation performance!11 1.4.2 cargo-watch We can also mitigate the impact on our productivity by reducing the perceived compilation time - i.e. the time you spend looking at your terminal waiting for cargo check or cargo run to complete. 11This might not truly be the case though - mold is the newest linker on the block and it looks even faster than lld! It feels a bit early, so we will not be using it as our default linker, but consider checking it out. 16

📄 Page 18

Tooling can help here - let’s install cargo-watch: cargo install cargo-watch cargo-watch monitors your source code to trigger commands every time a file changes. For example: cargo watch -x check will run cargo check after every code change. This reduces the perceived compilation time: • you are still in your IDE, re-reading the code change you just made; • cargo-watch, in the meantime, has already kick-started the compilation process; • once you switch to your terminal, the compiler is already halfway through! cargo-watch supports command chaining as well: cargo watch -x check -x test -x run It will start by running cargo check. If it succeeds, it launches cargo test. If tests pass, it launches the application with cargo run. Our inner development loop, right there! 1.5 Continuous Integration Toolchain, installed. Project skeleton, done. IDE, ready. One last thing to look at before we get into the details of what we will be building: our Continuous Integration (CI) pipeline. In trunk-based development we should be able to deploy our main branch at any point in time. Every member of the team can branch off from main, develop a small feature or fix a bug, merge back into main and release to our users. Continuous Integration empowers each member of the team to integrate their changes into the main branch multiple times a day. This has powerful ripple effects. Some are tangible and easy to spot: it reduces the chances of having to sort out messy merge conflicts due to long-lived branches. Nobody likes merge conflicts. Some are subtler: Continuous Integration tightens the feedback loop. You are less likely to go off on your own and develop for days or weeks just to find out that the approach you have chosen is not endorsed by the rest of the team or it would not integrate well with the rest of the project. It forces you to engage with your teammates earlier than when it feels comfortable, course-correcting if necessary when it is still easy to do so (and nobody is likely to get offended). How do we make it possible? With a collection of automated checks running on every commit - our CI pipeline. If one of the checks fails you cannot merge to main - as simple as that. CI pipelines often go beyond ensuring code health: they are a good place to perform a series of additional important checks - e.g. scanning our dependency tree for known vulnerabilities, linting, formatting, etc. We will run through the different checks that you might want to run as part of the CI pipeline of your Rust projects, introducing the associated tools as we go along. We will then provide a set of ready-made CI pipelines for some of the major CI providers. 17

📄 Page 19

1.5.1 CI Steps 1.5.1.1 Tests If your CI pipeline had a single step, it should be testing. Tests are a first-class concept in the Rust ecosystem and you can leverage cargo to run your unit and integration tests: cargo test cargo test also takes care of building the project before running tests, hence you do not need to run cargo build beforehand (even though most pipelines will invoke cargo build before running tests to cache dependencies). 1.5.1.2 Code Coverage Many articles have been written on the pros and cons of measuring code coverage. While using code coverage as a quality check has several drawbacks I do argue that it is a quick way to collect information and spot if some portions of the codebase have been overlooked over time and are indeed poorly tested. The easiest way to measure code coverage of a Rust project is via cargo tarpaulin, a cargo sub- command developed by xd009642. You can install tarpaulin with # At the time of writing tarpaulin only supports # x86_64 CPU architectures running Linux. cargo install cargo-tarpaulin while cargo tarpaulin --ignore-tests will compute code coverage for your application code, ignoring your test functions. tarpaulin can be used to upload code coverage metrics to popular services like Codecov or Coveralls - instructions can be found in tarpaulin’s README. 1.5.1.3 Linting Writing idiomatic code in any programming language requires time and practice. It is easy at the beginning of your learning journey to end up with fairly convoluted solutions to problems that could otherwise be tackled with a much simpler approach. Static analysis can help: in the same way a compiler steps through your code to ensure it conforms to the language rules and constraints, a linter will try to spot unidiomatic code, overly-complex constructs and common mistakes/inefficiencies. The Rust team maintains clippy, the official Rust linter12. clippy is included in the set of components installed by rustup if you are using the default profile. Often CI environments use rustup’s minimal profile, which does not include clippy. You can easily install it with rustup component add clippy If it is already installed the command is a no-op. You can run clippy on your project with cargo clippy In our CI pipeline we would like to fail the linter check if clippy emits any warnings. We can achieve it with cargo clippy -- -D warnings Static analysis is not infallible: from time to time clippy might suggest changes that you do not believe to be either correct or desirable. You can mute a warning using the #[allow(clippy::lint_name)] attribute on the affected code block or disable the noisy lint altogether for the whole project with a configuration line in clippy.toml 12Yes, clippy is named after the (in)famous paperclip-shaped Microsoft Word assistance. 18

📄 Page 20

or a project-level #![allow(clippy::lint_name)] directive. Details on the available lints and how to tune them for your specific purposes can be found in clippy’s README. 1.5.1.4 Formatting Most organizations have more than one line of defence for the main branch: one is provided by the CI pipeline checks, the other is often a pull request review. A lot can be said on what distinguishes a value-adding PR review process from a soul-sucking one - no need to re-open the whole debate here. I know for sure what should not be the focus of a good PR review: formatting nitpicks - e.g. Can you add a newline here?, I think we have a trailing whitespace there!, etc. Let machines deal with formatting while reviewers focus on architecture, testing thoroughness, re- liability, observability. Automated formatting removes a distraction from the complex equation of the PR review process. You might dislike this or that formatting choice, but the complete erasure of formatting bikeshedding is worth the minor discomfort. rustfmt is the official Rust formatter. Just like clippy, rustfmt is included in the set of default components installed by rustup. If missing, you can easily install it with rustup component add rustfmt You can format your whole project with cargo fmt In our CI pipeline we will add a formatting step cargo fmt -- --check It will fail when a commit contains unformatted code, printing the difference to the console.13 You can tune rustfmt for a project with a configuration file, rustfmt.toml. Details can be found in rustfmt’s README. 1.5.1.5 Security Vulnerabilities cargo makes it very easy to leverage existing crates in the ecosystem to solve the problem at hand. On the flip side, each of those crates might hide an exploitable vulnerability that could compromise the security posture of your software. The Rust Secure Code working group maintains an Advisory Database - an up-to-date collection of reported vulnerabilities for crates published on crates.io. They also provide cargo-audit14, a convenient cargo sub-command to check if vulnerabilities have been reported for any of the crates in the dependency tree of your project. You can install it with cargo install cargo-audit Once installed, run cargo audit to scan your dependency tree. We will be running cargo-audit as part of our CI pipeline, on every commit. We will also run it on a daily schedule to stay on top of new vulnerabilities for dependencies of projects 13It can be annoying to get a fail in CI for a formatting issue. Most IDEs support a “format on save” feature to make the process smoother. Alternatively, you can use a git pre-push hook. 14cargo-deny, developed by Embark Studios, is another cargo sub-command that supports vulnerability scanning of your dependency tree. It also bundles additional checks you might want to perform on your dependencies - it helps you identify unmaintained crates, define rules to restrict the set of allowed software licenses and spot when you have multiple versions of the same crate in your lock file (wasted compilation cycles!). It requires a bit of upfront effort in configuration, but it can be a powerful addition to your CI toolbox. 19