PyTorch Pocket Reference Building and Deploying Deep Learning Models (Joe Papa) (Z-Library)

Joe Papa PyTorch Building and Deploying Deep Learning Models Pocket Reference

Joe Papa PyTorch Pocket Reference Building and Deploying Deep Learning Models

978-1-492-09000-7 [LSI] PyTorch Pocket Reference by Joe Papa Copyright © 2021 Mobile Insights Technology Group, LLC. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promo‐ tional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Rebecca Novack Development Editor: Jeff Bleiel Production Editor: Beth Kelly Copyeditor: Piper Editorial Consulting, LLC Proofreader: Rachel Head Indexer: Potomac Indexing, LLC Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea May 2021: First Edition Revision History for the First Edition 2021-05-11: First Release See https://oreil.ly/9781492090007 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. PyTorch Pocket Reference, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the author, and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages result‐ ing from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

Table of Contents Preface vii Chapter 1: An Introduction to PyTorch 1 What Is PyTorch? 1 Why Use PyTorch? 2 Getting Started 4 A Fun Example 12 Chapter 2: Tensors 21 What Is a Tensor? 22 Creating Tensors 25 Tensor Operations 34 Chapter 3: Deep Learning Development with PyTorch 51 The Overall Process 52 Data Preparation 54 Model Development 68 Model Deployment 98 iii

Chapter 4: Neural Network Development Reference Designs 103 Image Classification with Transfer Learning 104 Sentiment Analysis with Torchtext 112 Generative Learning—Generating Fashion-MNIST Images with DCGAN 123 Chapter 5: Customizing PyTorch 135 Custom Layers and Activations 136 Custom Model Architectures 143 Custom Loss Functions 145 Custom Optimizer Algorithms 147 Custom Training, Validation, and Test Loops 151 Chapter 6: PyTorch Acceleration and Optimization 155 PyTorch on a TPU 156 PyTorch on Multiple GPUs (Single Machine) 160 Distributed Training (Multiple Machines) 169 Model Optimization 171 Chapter 7: Deploying PyTorch to Production 189 PyTorch Deployment Tools and Libraries 190 Deploying to a Flask App 210 Colab Flask App 213 Deploying to the Cloud with TorchServe 215 Quick Start with Docker 216 Deploying to Mobile and Edge 217 Chapter 8: The PyTorch Ecosystem and Additional Resources 227 The PyTorch Ecosystem 228 Torchvision for Image and Video 237 iv | Table of Contents

Torchtext for NLP 250 TensorBoard for Visualization 259 Papers with Code 270 Additional PyTorch Resources 270 Index 277 Table of Contents | v

Preface We are living in exciting times! Some of us have been fortunate to have lived through huge advances in technology—the inven‐ tion of the personal computer, the dawn of the internet, the proliferation of cell phones, and the advent of social media. And now, major breakthroughs are happening in AI! It’s exciting to watch and be a part of this change. I think we’re just getting started, and it’s amazing to think of how the world might change over the next decade. How great it is that we’re living during these times and can participate in the expansion of AI? PyTorch has, no doubt, enabled some of the finest advances in deep learning and AI. It’s free to download and use, and with it anyone with a computer or internet connection can run AI experiments. In addition to more comprehensive references like this one, there are many free and inexpensive training courses, blog articles, and tutorials that can help you. Anyone can get started using PyTorch for machine learning and AI. Who Should Read This Book This book is written for both beginners and advanced users interested in machine learning and AI. It will help to have some vii

experience writing Python code and a basic understanding of data science and machine learning. If you’re just getting started in machine learning, this book will help you learn the basics of PyTorch and provide some simple examples. If you’ve been using another framework, such as TensorFlow, Caffe2, or MXNet, the book with help you become familiar with the PyTorch API and its programming mindset so you can expand your skillset. If you’ve been using PyTorch for a while, this book will help you expand your knowledge on advanced topics like accelera‐ tion and optimization and provide a quick-reference resource while you use PyTorch for your day-to-day development. Why I Wrote This Book Learning and mastering PyTorch can be very exciting. There’s so much to explore! When I first started learning PyTorch, I wished I had a single resource that would teach me everything. I wanted something that would give me a good high-level look at what PyTorch had to offer, but also would provide examples and enough details when I needed to dig deeper. There are some really good books and courses on PyTorch, but they often focus on tensors and training for deep learning models. The PyTorch online documentation is really good, too, and provides a lot of details and examples; however, I found using it was often cumbersome. I kept having to click around to learn or Google what I needed to know. I needed a book on my desk that I could earmark and reference as I was coding. My goal is that this will be the ultimate PyTorch reference for you. In addition to reading through it to get a high-level under‐ standing of the PyTorch resources available to you, I hope that you earmark the key sections for your development work and keep it on your desk. That way if you forget something, you can get the answer right away. If you prefer ebooks or online books, You can bookmark this book online. However you may use it, I viii | Preface

hope the book helps you create some amazing new technology with PyTorch! Navigating This Book If you’re just beginning to learn PyTorch, you should start at Chapter 1 and read each chapter in sequence. The chapters move from beginner to advanced topics. If you already have some experience with PyTorch, you might want to jump around to the topics that interest you the most. Don’t forget to check out Chapter 8 on the PyTorch Ecosystem. You’re bound to discover something new! This book is roughly organized as follows: • Chapter 1 gives a brief introduction to PyTorch, helps you set up your development environment, and provides a fun example for you to try yourself. • Chapter 2 covers the tensor, PyTorch’s fundamental build‐ ing block. It’s the foundation for everything in PyTorch. • Chapter 3 gives you a comprehensive look at how you can use PyTorch for deep learning, and Chapter 4 provides example reference designs so you can see PyTorch in action. • Chapters 5 and 6 cover more advanced topics. Chapter 5 shows you how you can customize PyTorch components for your own work, while Chapter 6 shows you how to accelerate training and optimize your models. • Chapter 7 shows you how you can deploy PyTorch to pro‐ duction via local machines, cloud servers, and mobile or edge devices. • Chapter 8 guides you in where to go next by introducing the PyTorch Ecosystem, describing popular packages, and listing additional training resources. Preface | ix

Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, state‐ ments, and keywords. Constant width bold Shows commands or other text that should be typed liter‐ ally by the user. Additionally, bold is used for emphasis in functions in tables. Constant width italic Shows text that should be replaced with user-supplied val‐ ues or by values determined by context. Additionally, italic transforms listed in tables are currently not supported by TorchScript. Using Code Examples Supplemental material (code examples, exercises, etc.) is avail‐ able for download at https://github.com/joe-papa/pytorch-book. If you have a technical question or a problem using the code examples, please email bookquestions@oreilly.com. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book x | Preface

and quoting example code does not require permission. Incor‐ porating a significant amount of example code from this book into your product’s documentation does require permission. We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “PyTorch Pocket Reference by Joe Papa (O’Reilly). Copyright 2021 Mobile Insights Technology Group, LLC, 978-1-492-09000-7.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technology and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast col‐ lection of text and video from O’Reilly and 200+ other publish‐ ers. For more information, visit http://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) Preface | xi

707-829-0104 (fax) We have a web page for this book, where we list errata, exam‐ ples, and any additional information. You can access this page at https://oreil.ly/PyTorch-pocket. Email bookquestions@oreilly.com to comment or ask technical questions about this book. For news and information about our books and courses, visit http://oreilly.com. Find us on Facebook: http://facebook.com/oreilly Follow us on Twitter: http://twitter.com/oreillymedia Watch us on YouTube: http://youtube.com/oreillymedia Acknowledgments As a reader, I’m often amazed when reading the acknowledg‐ ments from other authors. Writing a book is no small task, and writing a good book requires the support of many people. Reading the acknowledgments is a constant reminder that we cannot do it alone. I am thankful for the support and encouragement of my friend, Matt Kirk, whom I met at an O’Reilly conference years ago. His shared passion for personal development is an inspiration to create books and courses and help others reach their full poten‐ tial, both personally and professionally. Our weekly Zoom chats and self-help projects during the pandemic definitely hel‐ ped me keep my sanity. Without Matt, this book would not have been possible. I’d like to thank Rebecca Novack for suggesting the project and taking a chance on me, as well as the staff at O’Reilly for mak‐ ing this project happen. Writing a book requires effort, but writing a good book requires dedicated reviewers who care about the readers. I’d like to thank Mike Drob, Axel Sirota, and Jeff Bleiel for taking xii | Preface

the time to review the book and provide countless suggestions. Mike’s suggestions added many practical resources that I would have otherwise overlooked. He made sure that we were using state-of-the-art tools and best practices you won’t find in online documentation. Axel’s attention to detail is incredible. I’m grateful for his encouragement and efforts to review the code and technical details in this book. Jeff is an amazing editor. I’m grateful for his suggestions on the sequencing and flow of the book. He’s significantly helped me become a better author. PyTorch is truly a community project. I am grateful for those at Facebook and the over 1,700 contributors that have developed this machine learning framework. I’d especially like to thank those who have created documentation and tutorials that help others like myself learn PyTorch quickly. Some individuals whose work has helped me the most include Suraj Subramanian, Seth Juarez, Cassie Breviu, Dmitry Soshni‐ kov, Ari Bornstein, Soumith Chintala, Justin Johnson, Jeremy Howard, Rachel Thomas, Francisco Ingham, Sasank Chilam‐ kurthy, Nathan Inkawhich, Sean Robertson, Ben Trevett, Avi‐ nash Sajjanshetty, James Reed, Michael Suo, Michela Paganini, Shen Li, Séb Arnold, Rohan Varma, Pritam Damania, Jeff Tang, and the countless bloggers and YouTubers on the subject of PyTorch. I’m grateful to Manbir Gulati for introducing me to PyTorch and to Rob Miller for giving me the opportunity to lead AI projects with PyTorch. I also appreciated sharing deep learning ideas for this book with my friend Isaac Privitera. Of course, I couldn’t have accomplished anything in life without the hard work and dedication of my mom, Grace, who brought us from humble beginnings to give me and my brother a chance at life. I miss her every day. Special thanks to my brother, Vinnie, who was a big help in completing my home projects, giving me more time to write. I appreciate my stepfather, Lou, for his encouragement while I Preface | xiii

was writing the book. I’d also like to thank my kids, Savannah, Caroline, and George, for being patient and understanding when daddy had to work. Lastly, I’d like to thank my wife, Emily. She has always endlessly supported my ideas and dreams throughout my life. As I tack‐ led the task of writing this book, of course, I relied on her once again. Caring for our three children and taking on new respon‐ sibilities during the pandemic has been a tall order. Still, she has been the support I needed to complete my writing. In fact, while writing this book, we found out that we’re expect‐ ing, and our fourth child is on his way! My wife does it all with a smile and a joke (which are often at my expense), and I love her for it. xiv | Preface

CHAPTER 1 An Introduction to PyTorch PyTorch is one of the most popular deep learning Python libra‐ ries, and it is widely used by the AI research community. Many developers and researchers use PyTorch to accelerate deep learning research experimentation and prototyping. In this chapter, I will give you a brief introduction to what PyTorch is and some of the features that make it popular. I’ll also show you how to install and set up your PyTorch develop‐ ment environment on your local machine and in the cloud. By the end of this chapter, you will be able to verify that PyTorch is properly installed and run a simple PyTorch program. What Is PyTorch? The PyTorch library is primarily developed by Facebook’s AI Research Lab (FAIR) and is free and open source software with over 1,700 contributors. It allows you to easily run array-based calculations, build dynamic neural networks, and perform autodifferentiation in Python with strong graphics processing unit (GPU) acceleration—all important features required for deep learning research. Although some use it for accelerated tensor computing, most use it for deep learning development. 1

PyTorch’s simple and flexible interface enables fast experimen‐ tation. You can load data, apply transforms, and build models with a few lines of code. Then, you have the flexibility to write customized training, validation, and test loops and deploy trained models with ease. It has a strong ecosystem and a large user community, includ‐ ing universities like Stanford and companies such as Uber, NVIDIA, and Salesforce. In 2019, PyTorch dominated machine learning and deep learning conference proceedings: 69% of the Conference on Computer Vision and Pattern Recognition (CVPR) proceedings used PyTorch, over 75% of both the Asso‐ ciation for Computational Linguistics (ACL) and the North American Chapter of the ACL (NAACL) used it, and over 50% of the International Conference on Learning Representations (ICLR) and the International Conference on Machine Learning (ICML) used it as well. There are also over 60,000 repositories on GitHub related to PyTorch. Many developers and researchers use PyTorch to accelerate deep learning research experimentation and prototyping. Its simple Python API, GPU support, and flexibility make it a pop‐ ular choice among academic and commercial research organi‐ zations. Since being open sourced in 2018, PyTorch has reached a stable release and can be easily installed on Windows, Mac, and Linux operating systems. The framework continues to expand rapidly and now facilitates deployment to produc‐ tion environments in the cloud and mobile platforms. Why Use PyTorch? If you’re studying machine learning, conducting deep learning research, or building AI systems, you’ll probably need to use a deep learning framework. A deep learning framework makes it easy to perform common tasks such data loading, preprocess‐ ing, model design, training, and deployment. PyTorch has become very popular with the academic and research com‐ munities due to its simplicity, flexibility, and Python interface. Here are some reasons to learn and use PyTorch: 2 | Chapter 1: An Introduction to PyTorch

PyTorch is popular Many companies and research organizations use PyTorch as their main deep learning framework. In fact, some companies have built their custom machine learning tools on top of PyTorch. As a result, PyTorch skills are in demand. PyTorch is supported by all major cloud platforms, such as Ama‐ zon Web Services (AWS), Google Cloud Platform (GCP), Micro‐ soft Azure, and Alibaba Cloud You can spin up a virtual machine with PyTorch preloaded for frictionless development. You can use prebuilt Docker images, perform large-scale training on cloud GPU plat‐ forms, and run models at production scale. PyTorch is supported by Google Colaboratory and Kaggle Kernels You can run PyTorch code in a browser with no installa‐ tion or configuration needed. You can compete in Kaggle competitions by running PyTorch directly in your kernel. PyTorch is mature and stable PyTorch is regularly maintained and is now beyond release 1.8. PyTorch supports CPU, GPU, TPU, and parallel processing You can accelerate your training and inference using GPUs and TPUs. Tensor processing units (TPUs) are AI- accelerated application-specific integrated circuits (ASIC) chips that were developed by Google to provide an alter‐ native to GPUs for NN hardware acceleration. With paral‐ lel processing, you can apply preprocessing on your CPU while training a model on the GPU or TPU. PyTorch supports distributed training You can train neural networks over multiple GPUs on multiple machines. PyTorch supports deployment to production With the newer TorchScript and TorchServe features, you can easily deploy models to production environments including cloud servers. Why Use PyTorch? | 3

PyTorch is beginning to support mobile deployment Although it’s currently experimental, you can now deploy models to iOS and Android devices. PyTorch has a vast ecosystem and set of open source libraries Libraries such as Torchvision, fastai, and PyTorch Light‐ ning extend capabilities and support specific fields like natural olanguage processing (NLP) and computer vision. PyTorch also has a C++ frontend Although I will focus on the Python interface in this book, PyTorch also supports a frontend C++ interface. If you need to build high-performance, low-latency, or bare- metal applications, you can write them in C++ using the same design and architecture as the Python API. PyTorch supports the Open Neural Network Exchange (ONNX) format natively You can easily export your models to ONNX format and use them with ONNX-compatible platforms, runtimes, or visualizers. PyTorch has a large community of developers and user forums There are more than 38,000 users on the PyTorch forum, and it’s easy to get support or post questions to the com‐ munity by visiting the PyTorch Discussion Forum. Getting Started If you are familiar with PyTorch, you may already have installed it and set up your development environment. If not, I will show you some options to do so in this section. The fastest way to get started is to use Google Colaboratory (or Colab). Google Colab is a free cloud-based development environment similar to Jupyter Notebook and comes with PyTorch already installed. Colab comes with free limited GPU support and interfaces nicely with Google Drive for saving and sharing notebooks. 4 | Chapter 1: An Introduction to PyTorch