Data Science on AWS Implementing End-to-End, Continuous AI and Machine Learning Pipelines (Chris Fregly, Antje Barth) (z-library.sk, 1lib.sk, z-lib.sk)

Chris Fregly & Antje Barth Data Science on AWS Implementing End-to-End, Continuous AI and Machine Learning Pipelines

Praise for Data Science on AWS “Wow—this book will help you to bring your data science projects from idea all the way to production. Chris and Antje have covered all of the important concepts and the key AWS services, with plenty of real-world examples to get you started on your data science journey." —Jeff Barr, Vice President & Chief Evangelist, Amazon Web Services “It’s very rare to find a book that comprehensively covers the full end-to-end process of model development and deployment! If you’re an ML practitioner, this book is a must!” —Ramine Tinati, Managing Director/Chief Data Scientist Applied Intelligence, Accenture “This book is a great resource for building scalable machine learning solutions on AWS cloud. It includes best practices for all aspects of model building, including training, deployment, security, interpretability, and MLOps.” —Geeta Chauhan, AI/PyTorch Partner Engineering Head, Facebook AI “The landscape of tools on AWS for data scientists and engineers can be absolutely overwhelming. Chris and Antje have done the community a service by providing a map that practitioners can use to orient themselves, find the tools they need to get the job done and build new systems that bring their ideas to life.” —Josh Wills, Author, Advanced Analytics with Spark (O’Reilly)

“Successful data science teams know that data science isn’t just modeling but needs a disciplined approach to data and production deployment. We have an army of tools for all of these at our disposal in major clouds like AWS. Practitioners will appreciate this comprehensive, practical field guide that demonstrates not just how to apply the tools but which ones to use and when.” —Sean Owen, Principal Solutions Architect, Databricks “This is the most extensive resource I know about ML on AWS, unequaled in breadth and depth. While ML literature often focuses on science, Antje and Chris dive deep into the practical architectural concepts needed to serve science in production, such as security, data engineering, monitoring, CI/CD, and costs management. The book is state-of-the-art on the science as well: it presents advanced concepts like Transformer architectures, AutoML, online learning, distillation, compilation, Bayesian model tuning, and bandits. It stands out by providing both a business-friendly description of services and concepts as well as low-level implementation tips and instructions. A must-read for individuals and organizations building ML systems on AWS or improving their knowledge of the AWS AI and machine learning stack.” —Olivier Cruchant, Principal ML Specialist Solutions Architect, Amazon Web Services “This book is a great resource to understand both the end-to-end machine learning workflow in detail and how to build operationally efficient machine learning workloads at scale on AWS. Highly recommend Data Science on AWS for anyone building machine learning workloads on AWS!” —Shelbee Eigenbrode, AI/ML Specialist Solutions Architect, Amazon Web Services “This book is a comprehensive resource for diving into data science on AWS. The authors provide a good balance of theory, discussion, and hands-on examples to guide the reader through implementing all phases of machine learning applications using AWS services. A great resource to not just get started but to scale and secure end-to-end ML applications.” —Sireesha Muppala, PhD, Principal Solutions Architect, AI/ML, Amazon Web Services “Implementing a robust end-to-end machine learning workflow is a daunting challenge, complicated by the wide range of tools and technologies available; the authors do an impressive job of guiding both novice and expert practitioners through this task leveraging the power of AWS services.” —Brent Rabowsky, Data Scientist, Amazon Web Services

“Using real-world examples, Chris and Antje provide indispensable and comprehensive guidance for building and managing ML and AI applications in AWS.” —Dean Wampler, Author, Programming Scala (O’Reilly) "Data Science on AWS is exciting and intimidating due to the vast quantity of services and methodologies available. This book is a welcome guide to getting machine learning into production on the AWS platform, whether you want to do ML with AWS Lambda or with Amazon SageMaker.” —Noah Gift, Duke Faculty and Founder, Pragmatic AI Labs "Data Science on AWS provides an in-depth look at the modern data science stack on AWS. Machine learning practitioners will learn about the services, open source libraries, and infrastructure they can leverage during each phase of the ML pipeline and how to tie it all together using MLOps. This book is a great resource and a definite must-read for anyone looking to level up their ML skills using AWS.” —Kesha Williams, A Cloud Guru “As AWS continues to generate explosive growth, the data science practitioner today needs to know how to operate in the cloud. This book takes the practitioner through key topics in cloud-based data science such as SageMaker, AutoML, Model Deployment, and MLOps cloud security best practices. It’s a bookshelf must-have for those looking to keep pace with machine learning on AWS.” —Josh Patterson, Author, Kubeflow Operations Guide (O’Reilly) “AWS is an extremely powerful tool, a visionary and leader in cloud computing. The variety of available services can be impressive, which is where this book becomes a big deal. Antje and Chris have crafted a complete AWS guide to building ML/AI pipelines complying with best-in-class practices. Allow yourself to keep calm and go to production." —Andy Petrella, CEO and Founder, Kensu

“This book is a must-have for anyone wanting to learn how to organize a data science project in production on AWS. It covers the full journey from research to production and covers the AWS tools and services that could be used for each step along the way.” —Rustem Feyzkhanov, Machine Learning Engineer, Instrumental, Amazon Web Services ML Hero “Chris and Antje manage to compress all of AWS AI in this great book. If you plan to build AI using AWS, this has you covered from beginning to end and more. Well done!” —Francesco Mosconi, Author and Founder, Zero to Deep Learning “Chris and Antje expertly guide ML practitioners through the complex and sometimes overwhelming landscape of managed cloud services on AWS. Because this book serves as a comprehensive atlas of services and their interactions toward the completion of end-to-end data science workflows from data ingestion to predictive application, you’ll quickly find a spot for it on your desk as a vital quick reference!” —Benjamin Bengfort, Rotational Labs “This book covers the different AWS tools for data science as well as how to select the right ones and make them work together.” —Holden Karau, Author, Learning Spark and Kubeflow for Machine Learning (O’Reilly) “The book is easy to read and accompanied by a tested and well-managed code base. I highly recommend it to anyone interested in data science, data engineering, and machine learning engineering at scale.” —Shreenidhi Bharadwaj, Senior Principal, Private Equity/ Venture Capital Advisory (M&A), West Monroe Partners

Chris Fregly and Antje Barth Data Science on AWS Implementing End-to-End, Continuous AI and Machine Learning Pipelines Boston Farnham Sebastopol TokyoBeijing

978-1-492-07939-2 [LSI] Data Science on AWS by Chris Fregly and Antje Barth Copyright © 2021 Antje Barth and Flux Capacitor, LLC. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Jessica Haberman Development Editor: Gary O’Brien Production Editor: Katherine Tozer Copyeditor: Charles Roumeliotis Proofreader: Piper Editorial Consulting, LLC Indexer: Judith McConville Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: O’Reilly Media, Inc. April 2021: First Edition Revision History for the First Edition 2021-04-07: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781492079392 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Data Science on AWS, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the authors, and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii 1. Introduction to Data Science on AWS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Benefits of Cloud Computing 1 Data Science Pipelines and Workflows 4 MLOps Best Practices 7 Amazon AI Services and AutoML with Amazon SageMaker 10 Data Ingestion, Exploration, and Preparation in AWS 13 Model Training and Tuning with Amazon SageMaker 18 Model Deployment with Amazon SageMaker and AWS Lambda Functions 21 Streaming Analytics and Machine Learning on AWS 21 AWS Infrastructure and Custom-Built Hardware 23 Reduce Cost with Tags, Budgets, and Alerts 26 Summary 26 2. Data Science Use Cases. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 Innovation Across Every Industry 29 Personalized Product Recommendations 30 Detect Inappropriate Videos with Amazon Rekognition 36 Demand Forecasting 38 Identify Fake Accounts with Amazon Fraud Detector 42 Enable Privacy-Leak Detection with Amazon Macie 43 Conversational Devices and Voice Assistants 44 Text Analysis and Natural Language Processing 45 Cognitive Search and Natural Language Understanding 50 Intelligent Customer Support Centers 51 Industrial AI Services and Predictive Maintenance 52 Home Automation with AWS IoT and Amazon SageMaker 53 vii

Extract Medical Information from Healthcare Documents 54 Self-Optimizing and Intelligent Cloud Infrastructure 55 Cognitive and Predictive Business Intelligence 56 Educating the Next Generation of AI and ML Developers 60 Program Nature’s Operating System with Quantum Computing 65 Increase Performance and Reduce Cost 70 Summary 73 3. Automated Machine Learning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 Automated Machine Learning with SageMaker Autopilot 76 Track Experiments with SageMaker Autopilot 78 Train and Deploy a Text Classifier with SageMaker Autopilot 78 Automated Machine Learning with Amazon Comprehend 91 Summary 95 4. Ingest Data into the Cloud. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 Data Lakes 98 Query the Amazon S3 Data Lake with Amazon Athena 105 Continuously Ingest New Data with AWS Glue Crawler 109 Build a Lake House with Amazon Redshift Spectrum 111 Choose Between Amazon Athena and Amazon Redshift 118 Reduce Cost and Increase Performance 119 Summary 126 5. Explore the Dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 Tools for Exploring Data in AWS 128 Visualize Our Data Lake with SageMaker Studio 129 Query Our Data Warehouse 142 Create Dashboards with Amazon QuickSight 150 Detect Data-Quality Issues with Amazon SageMaker and Apache Spark 151 Detect Bias in Our Dataset 159 Detect Different Types of Drift with SageMaker Clarify 166 Analyze Our Data with AWS Glue DataBrew 168 Reduce Cost and Increase Performance 170 Summary 172 6. Prepare the Dataset for Model Training. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 Perform Feature Selection and Engineering 173 Scale Feature Engineering with SageMaker Processing Jobs 187 Share Features Through SageMaker Feature Store 194 Ingest and Transform Data with SageMaker Data Wrangler 198 Track Artifact and Experiment Lineage with Amazon SageMaker 199 viii | Table of Contents

Ingest and Transform Data with AWS Glue DataBrew 204 Summary 206 7. Train Your First Model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207 Understand the SageMaker Infrastructure 207 Deploy a Pre-Trained BERT Model with SageMaker JumpStart 212 Develop a SageMaker Model 214 A Brief History of Natural Language Processing 216 BERT Transformer Architecture 219 Training BERT from Scratch 221 Fine Tune a Pre-Trained BERT Model 223 Create the Training Script 226 Launch the Training Script from a SageMaker Notebook 232 Evaluate Models 239 Debug and Profile Model Training with SageMaker Debugger 245 Interpret and Explain Model Predictions 249 Detect Model Bias and Explain Predictions 255 More Training Options for BERT 259 Reduce Cost and Increase Performance 268 Summary 274 8. Train and Optimize Models at Scale. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277 Automatically Find the Best Model Hyper-Parameters 277 Use Warm Start for Additional SageMaker Hyper-Parameter Tuning Jobs 284 Scale Out with SageMaker Distributed Training 288 Reduce Cost and Increase Performance 296 Summary 300 9. Deploy Models to Production. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301 Choose Real-Time or Batch Predictions 301 Real-Time Predictions with SageMaker Endpoints 302 Auto-Scale SageMaker Endpoints Using Amazon CloudWatch 310 Strategies to Deploy New and Updated Models 315 Testing and Comparing New Models 319 Monitor Model Performance and Detect Drift 331 Monitor Data Quality of Deployed SageMaker Endpoints 335 Monitor Model Quality of Deployed SageMaker Endpoints 341 Monitor Bias Drift of Deployed SageMaker Endpoints 345 Monitor Feature Attribution Drift of Deployed SageMaker Endpoints 348 Perform Batch Predictions with SageMaker Batch Transform 351 AWS Lambda Functions and Amazon API Gateway 356 Optimize and Manage Models at the Edge 357 Table of Contents | ix

Deploy a PyTorch Model with TorchServe 357 TensorFlow-BERT Inference with AWS Deep Java Library 360 Reduce Cost and Increase Performance 362 Summary 367 10. Pipelines and MLOps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369 Machine Learning Operations 369 Software Pipelines 371 Machine Learning Pipelines 371 Pipeline Orchestration with SageMaker Pipelines 375 Automation with SageMaker Pipelines 386 More Pipeline Options 391 Human-in-the-Loop Workflows 400 Reduce Cost and Improve Performance 406 Summary 407 11. Streaming Analytics and Machine Learning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 409 Online Learning Versus Offline Learning 410 Streaming Applications 410 Windowed Queries on Streaming Data 411 Streaming Analytics and Machine Learning on AWS 415 Classify Real-Time Product Reviews with Amazon Kinesis, AWS Lambda, and Amazon SageMaker 417 Implement Streaming Data Ingest Using Amazon Kinesis Data Firehose 418 Summarize Real-Time Product Reviews with Streaming Analytics 422 Setting Up Amazon Kinesis Data Analytics 424 Amazon Kinesis Data Analytics Applications 432 Classify Product Reviews with Apache Kafka, AWS Lambda, and Amazon SageMaker 439 Reduce Cost and Improve Performance 440 Summary 442 12. Secure Data Science on AWS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 443 Shared Responsibility Model Between AWS and Customers 443 Applying AWS Identity and Access Management 444 Isolating Compute and Network Environments 452 Securing Amazon S3 Data Access 455 Encryption at Rest 463 Encryption in Transit 467 Securing SageMaker Notebook Instances 469 Securing SageMaker Studio 471 Securing SageMaker Jobs and Models 473 x | Table of Contents

Securing AWS Lake Formation 477 Securing Database Credentials with AWS Secrets Manager 478 Governance 478 Auditability 481 Reduce Cost and Improve Performance 483 Summary 485 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 487 Table of Contents | xi

Preface With this practical book, AI and machine learning (ML) practitioners will learn how to successfully build and deploy data science projects on Amazon Web Services (AWS). The Amazon AI and ML stack unifies data science, data engineering, and application development to help level up your skills. This guide shows you how to build and run pipelines in the cloud, then integrate the results into applications in minutes instead of days. Throughout the book, authors Chris Fregly and Antje Barth demonstrate how to reduce cost and improve performance. • Apply the Amazon AI and ML stack to real-world use cases for natural language processing, computer vision, fraud detection, conversational devices, and more. • Use automated ML (AutoML) to implement a specific subset of use cases with Amazon SageMaker Autopilot. • Dive deep into the complete model development life cycle for a BERT-based nat‐ ural language processing (NLP) use case including data ingestion and analysis, and more. • Tie everything together into a repeatable ML operations (MLOps) pipeline. • Explore real-time ML, anomaly detection, and streaming analytics on real-time data streams with Amazon Kinesis and Amazon Managed Streaming for Apache Kafka (Amazon MSK). • Learn security best practices for data science projects and workflows, including AWS Identity and Access Management (IAM), authentication, authorization, including data ingestion and analysis, model training, and deployment. xiii

Overview of the Chapters Chapter 1 provides an overview of the broad and deep Amazon AI and ML stack, an enormously powerful and diverse set of services, open source libraries, and infra‐ structure to use for data science projects of any complexity and scale. Chapter 2 describes how to apply the Amazon AI and ML stack to real-world use cases for recommendations, computer vision, fraud detection, natural language understanding (NLU), conversational devices, cognitive search, customer support, industrial predictive maintenance, home automation, Internet of Things (IoT), healthcare, and quantum computing. Chapter 3 demonstrates how to use AutoML to implement a specific subset of these use cases with SageMaker Autopilot. Chapters 4–9 dive deep into the complete model development life cycle (MDLC) for a BERT-based NLP use case, including data ingestion and analysis, feature selection and engineering, model training and tuning, and model deployment with Amazon SageMaker, Amazon Athena, Amazon Redshift, Amazon EMR, TensorFlow, PyTorch, and serverless Apache Spark. Chapter 10 ties everything together into repeatable pipelines using MLOps with Sage‐ Maker Pipelines, Kubeflow Pipelines, Apache Airflow, MLflow, and TFX. Chapter 11 demonstrates real-time ML, anomaly detection, and streaming analytics on real-time data streams with Amazon Kinesis and Apache Kafka. Chapter 12 presents a comprehensive set of security best practices for data science projects and workflows, including IAM, authentication, authorization, network isola‐ tion, data encryption at rest, post-quantum network encryption in transit, gover‐ nance, and auditability. Throughout the book, we provide tips to reduce cost and improve performance for data science projects on AWS. Who Should Read This Book This book is for anyone who uses data to make critical business decisions. The guid‐ ance here will help data analysts, data scientists, data engineers, ML engineers, research scientists, application developers, and DevOps engineers broaden their understanding of the modern data science stack and level up their skills in the cloud. The Amazon AI and ML stack unifies data science, data engineering, and application development to help users level up their skills beyond their current roles. We show how to build and run pipelines in the cloud, then integrate the results into applica‐ tions in minutes instead of days. xiv | Preface

Ideally, and to get most out of this book, we suggest readers have the following knowledge: • Basic understanding of cloud computing • Basic programming skills with Python, R, Java/Scala, or SQL • Basic familiarity with data science tools such as Jupyter Notebook, pandas, NumPy, or scikit-learn Other Resources There are many great authors and resources from which this book drew inspiration: • Aurélien Géron’s Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow (O’Reilly) is a great hands-on guide to building intelligent ML sys‐ tems with popular tools such as Python, scikit-learn, and TensorFlow. • Jeremy Howard and Sylvain Gugger’s Deep Learning for Coders with fastai and PyTorch (O’Reilly) is an excellent reference for building deep learning applica‐ tions with PyTorch “without a PhD.” • Hannes Hapke and Catherine Nelson’s Building Machine Learning Pipelines (O’Reilly) is a fantastic and easy-to-read reference for building AutoML pipelines with TensorFlow and TFX. • Eric R. Johnston, Nic Harrigan, and Mercedes Gimeno-Segovia’s Programming Quantum Computers (O’Reilly) is a great introduction to quantum computers with easy-to-understand examples that demonstrate the quantum advantage. • Micha Gorelick and Ian Ozsvald’s High Performance Python (O’Reilly) is an advanced reference that reveals many valuable tips and tricks to profile and opti‐ mize Python code for high-performance data processing, feature engineering, and model training. • Data Science on AWS has a site dedicated to this book that provides advanced workshops, monthly webinars, meetups, videos, and slides related to the content in this book. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Preface | xv

Constant width Used for program listings, as well as within paragraphs to refer to program ele‐ ments such as variable or function names, databases, data types, environment variables, statements, and keywords. Constant width bold Shows commands or other text that should be typed literally by the user. This element signifies a tip or suggestion. This element signifies a general note. Using Code Examples Supplemental material (code examples, exercises, etc.) is available for download at https://github.com/data-science-on-aws. Some of the code examples shown in this book are shortened to highlight a specific implementation. The repo includes addi‐ tional notebooks not covered in this book but useful for readers to review. The note‐ books are organized by book chapter and should be easy to follow along. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Data Science on AWS by Chris Fregly and Antje Barth (O’Reilly). Copyright 2021 Antje Barth and Flux Capacitor, LLC, 978-1-492-07939-2.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. xvi | Preface

O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit http://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/data-science-aws. Email bookquestions@oreilly.com to comment or ask technical questions about this book. For news and information about our books and courses, visit http://oreilly.com. Find us on Facebook: http://facebook.com/oreilly Follow us on Twitter: http://twitter.com/oreillymedia Watch us on YouTube: http://www.youtube.com/oreillymedia The authors regularly share relevant blog posts, conference talks, slides, meetup invites and workshop dates on Twitter or LinkedIn. Follow the authors on Twitter: https://twitter.com/cfregly and https://twitter.com/ anbarth Find the authors on LinkedIn: https://www.linkedin.com/in/cfregly and https:// www.linkedin.com/in/antje-barth Preface | xvii

Acknowledgments We would like to thank our O’Reilly development editor, Gary O’Brien, who helped us navigate the book-authoring process and, more importantly, made us laugh every time we chatted. Thanks, Gary, for letting us include source code and low-level hard‐ ware specifications in Chapter 1! We’d also like to thank Jessica Haberman, senior acquisitions editor, who offered key advice on everything from the initial book pro‐ posal to the final page count. After seven years of submitting book proposals, you helped us raise the bar to the point where the proposal was accepted! Special thanks to Mike Loukides and Nicole Taché from O’Reilly for your thoughtful advice early in the book-writing process, including the chapter outline, introductions, and summaries. We would like to send a warm thank you to book reviewers who tirelessly reviewed— and re-reviewed—every page in this book. The reviewers are listed here in alphabeti‐ cal order by first name: Ali Arsanjani, Andy Petrella, Brent Rabowsky, Dean Wam‐ pler, Francesco Mosconi, Hannah Marlowe, Hannes Hapke, Josh Patterson, Josh Wills, Liam Morrison, Noah Gift, Ramine Tinati, Robert Monarch, Roy Ben-Alta, Rustem Feyzkhanov, Sean Owen, Shelbee Eigenbrode, Sireesha Muppala, Stefan Natu, Ted Dunning, and Tim O’Brien. Your deep technical expertise and thorough feedback has been invaluable not just to this book but to the way we will present technical material in the future. You helped elevate this book from good to great, and we really enjoyed working with you all on this project. Chris I would like to dedicate this book to my late father, Thomas Fregly. Dad: You brought home my first Apple computer when I was 8 years old and forever changed my life. You helped me absorb your university calculus book at age 10 and further solidified my strong interest in mathematics. You taught me how to read voraciously, write suc‐ cinctly, speak effectively, type quickly, and ask questions early. Watching you repair a boat engine while stranded on Lake Michigan, I am continuously inspired to dive deep and understand the hardware that powers my software. While walking around your office at the Chicago Sun-Times, I learned that everybody has an interesting story to tell, including the front-desk person, the CEO, and the maintenance staff. You said “Hello” to everybody equally, asked about their children, listened to their stories, and made them laugh with a funny story of your own. Holding your hand as we walked around your university campus as a child, I learned that it’s OK to leave the sidewalk and carve out my own path through the grass. You said, “Don’t worry, Chris, they’ll eventually pave this path as it’s clearly the shortest path from the engineering building to the cafeteria.” You were right, Dad. Many years later, we walked that newly paved path as we grabbed your favorite drink, Diet Pepsi, from the cafeteria. From you, I learned to carve out my own path through life and not always follow the crowd. xviii | Preface