AI and Machine Learning for Coders A Programmers Guide to Artificial Intelligence (Laurence Moroney) (Z-Library)

Praise for AI and Machine Learning for Coders “Machine learning should be in the toolbox of every great engineer in this coming decade. For people looking to get started, AI and Machine Learning for Coders by Laurence Moroney is the much-needed practical starting point to dive deep into deep learning, computer vision, and NLP.” —Dominic Monn, Machine Learning at Doist “The book is a great introduction to understand and practice machine learning and artificial intelligence models by using TensorFlow. It covers various deep learning models, and their practical applications, as well as how to utilize TensorFlow framework to develop and deploy ML/AI applications across platforms. I recommend it for anyone who is interested in ML and AI practice.” —Jialin Huang PhD, Data and Applied Scientist at Microsoft “Laurence’s book helped me refresh TensorFlow framework and Coursera Specialization, and motivated me to take the certification provided by Google. If you have time and you are willing to embark to an ML journey, this book is a starting point from the practice side.” —Laura Uzcátegui, Software Engineer “This book is a must-read for developers who would like to get into AI/ML. You will learn a variety of examples by coding instead of math equations.” —Margaret Maynard-Reid, ML Google Developer Expert “A practical handbook to have on your desk for implementing deep learning models.” —Pin-Yu Chen, Research Staff Member at IBM Research AI

“A fun book to read and practice coding for AI and machine learning projects. Intuitive wording and graphs to explain the nonintuitive concepts and algorithms. Cool coding examples to teach you key building blocks for AI and ML. In the end, you can code AI projects for your PC program, Android, iOS and Browser!” —Su Fu, CEO of Alchemist

Laurence Moroney AI and Machine Learning for Coders A Programmer’s Guide to Artificial Intelligence Boston Farnham Sebastopol TokyoBeijing

978-1-492-07819-7 [LSI] AI and Machine Learning for Coders by Laurence Moroney Copyright © 2021 Laurence Moroney. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Rebecca Novack Development Editor: Angela Rufino Production Editor: Katherine Tozer Copyeditor: Rachel Head Proofreader: Piper Editorial, LLC Indexer: Judith McConville Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: O’Reilly Media, Inc. October 2020: First Edition Revision History for the First Edition 2020-10-01: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781492078197 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. AI and Machine Learning for Coders, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the author, and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

Table of Contents Foreword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv Part I. Building Models 1. Introduction to TensorFlow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 What Is Machine Learning? 1 Limitations of Traditional Programming 3 From Programming to Learning 5 What Is TensorFlow? 7 Using TensorFlow 8 Installing TensorFlow in Python 9 Using TensorFlow in PyCharm 9 Using TensorFlow in Google Colab 12 Getting Started with Machine Learning 13 Seeing What the Network Learned 18 Summary 19 2. Introduction to Computer Vision. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Recognizing Clothing Items 21 The Data: Fashion MNIST 22 Neurons for Vision 23 Designing the Neural Network 25 The Complete Code 26 Training the Neural Network 29 v

Exploring the Model Output 29 Training for Longer—Discovering Overfitting 30 Stopping Training 30 Summary 32 3. Going Beyond the Basics: Detecting Features in Images. . . . . . . . . . . . . . . . . . . . . . . . . 33 Convolutions 34 Pooling 35 Implementing Convolutional Neural Networks 37 Exploring the Convolutional Network 39 Building a CNN to Distinguish Between Horses and Humans 42 The Horses or Humans Dataset 42 The Keras ImageDataGenerator 43 CNN Architecture for Horses or Humans 45 Adding Validation to the Horses or Humans Dataset 47 Testing Horse or Human Images 49 Image Augmentation 52 Transfer Learning 55 Multiclass Classification 59 Dropout Regularization 63 Summary 65 4. Using Public Datasets with TensorFlow Datasets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 Getting Started with TFDS 68 Using TFDS with Keras Models 70 Loading Specific Versions 73 Using Mapping Functions for Augmentation 73 Using TensorFlow Addons 74 Using Custom Splits 74 Understanding TFRecord 76 The ETL Process for Managing Data in TensorFlow 78 Optimizing the Load Phase 80 Parallelizing ETL to Improve Training Performance 81 Summary 83 5. Introduction to Natural Language Processing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 Encoding Language into Numbers 85 Getting Started with Tokenization 86 Turning Sentences into Sequences 87 Removing Stopwords and Cleaning Text 91 Working with Real Data Sources 93 vi | Table of Contents

Getting Text from TensorFlow Datasets 93 Getting Text from CSV Files 97 Getting Text from JSON Files 99 Summary 101 6. Making Sentiment Programmable Using Embeddings. . . . . . . . . . . . . . . . . . . . . . . . . 103 Establishing Meaning from Words 103 A Simple Example: Positives and Negatives 104 Going a Little Deeper: Vectors 105 Embeddings in TensorFlow 106 Building a Sarcasm Detector Using Embeddings 106 Reducing Overfitting in Language Models 109 Using the Model to Classify a Sentence 119 Visualizing the Embeddings 120 Using Pretrained Embeddings from TensorFlow Hub 123 Summary 125 7. Recurrent Neural Networks for Natural Language Processing. . . . . . . . . . . . . . . . . . . 127 The Basis of Recurrence 127 Extending Recurrence for Language 130 Creating a Text Classifier with RNNs 132 Stacking LSTMs 134 Using Pretrained Embeddings with RNNs 139 Summary 146 8. Using TensorFlow to Create Text. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 Turning Sequences into Input Sequences 148 Creating the Model 152 Generating Text 154 Predicting the Next Word 154 Compounding Predictions to Generate Text 155 Extending the Dataset 156 Changing the Model Architecture 157 Improving the Data 158 Character-Based Encoding 161 Summary 162 9. Understanding Sequence and Time Series Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 Common Attributes of Time Series 165 Trend 165 Seasonality 165 Table of Contents | vii

Autocorrelation 166 Noise 166 Techniques for Predicting Time Series 167 Naive Prediction to Create a Baseline 167 Measuring Prediction Accuracy 170 Less Naive: Using Moving Average for Prediction 170 Improving the Moving Average Analysis 171 Summary 172 10. Creating ML Models to Predict Sequences. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 Creating a Windowed Dataset 174 Creating a Windowed Version of the Time Series Dataset 176 Creating and Training a DNN to Fit the Sequence Data 178 Evaluating the Results of the DNN 179 Exploring the Overall Prediction 181 Tuning the Learning Rate 183 Exploring Hyperparameter Tuning with Keras Tuner 185 Summary 189 11. Using Convolutional and Recurrent Methods for Sequence Models. . . . . . . . . . . . . . 191 Convolutions for Sequence Data 191 Coding Convolutions 192 Experimenting with the Conv1D Hyperparameters 195 Using NASA Weather Data 198 Reading GISS Data in Python 199 Using RNNs for Sequence Modeling 200 Exploring a Larger Dataset 203 Using Other Recurrent Methods 205 Using Dropout 206 Using Bidirectional RNNs 209 Summary 211 Part II. Using Models 12. An Introduction to TensorFlow Lite. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215 What Is TensorFlow Lite? 215 Walkthrough: Creating and Converting a Model to TensorFlow Lite 217 Step 1. Save the Model 218 Step 2. Convert and Save the Model 219 Step 3. Load the TFLite Model and Allocate Tensors 219 viii | Table of Contents

Step 4. Perform the Prediction 220 Walkthrough: Transfer Learning an Image Classifier and Converting to TensorFlow Lite 222 Step 1. Build and Save the Model 222 Step 2. Convert the Model to TensorFlow Lite 223 Step 3. Optimize the Model 225 Summary 227 13. Using TensorFlow Lite in Android Apps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229 What Is Android Studio? 229 Creating Your First TensorFlow Lite Android App 230 Step 1. Create a New Android Project 230 Step 2. Edit Your Layout File 232 Step 3. Add the TensorFlow Lite Dependencies 234 Step 4. Add Your TensorFlow Lite Model 236 Step 5. Write the Activity Code to Use TensorFlow Lite for Inference 236 Moving Beyond “Hello World”—Processing Images 239 TensorFlow Lite Sample Apps 242 Summary 244 14. Using TensorFlow Lite in iOS Apps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 Creating Your First TensorFlow Lite App with Xcode 245 Step 1. Create a Basic iOS App 245 Step 2. Add TensorFlow Lite to Your Project 247 Step 3. Create the User Interface 248 Step 4. Add and Initialize the Model Inference Class 250 Step 5. Perform the Inference 253 Step 6. Add the Model to Your App 254 Step 7. Add the UI Logic 256 Moving Beyond “Hello World”—Processing Images 258 TensorFlow Lite Sample Apps 261 Summary 261 15. An Introduction to TensorFlow.js. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263 What Is TensorFlow.js? 263 Installing and Using the Brackets IDE 265 Building Your First TensorFlow.js Model 266 Creating an Iris Classifier 270 Summary 274 Table of Contents | ix

16. Coding Techniques for Computer Vision in TensorFlow.js. . . . . . . . . . . . . . . . . . . . . . . 275 JavaScript Considerations for TensorFlow Developers 276 Building a CNN in JavaScript 277 Using Callbacks for Visualization 279 Training with the MNIST Dataset 282 Running Inference on Images in TensorFlow.js 288 Summary 288 17. Reusing and Converting Python Models to JavaScript. . . . . . . . . . . . . . . . . . . . . . . . . . 291 Converting Python-Based Models to JavaScript 291 Using the Converted Models 293 Using Preconverted JavaScript Models 295 Using the Toxicity Text Classifier 295 Using MobileNet for Image Classification in the Browser 298 Using PoseNet 301 Summary 304 18. Transfer Learning in JavaScript. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305 Transfer Learning from MobileNet 306 Step 1. Download MobileNet and Identify the Layers to Use 306 Step 2. Create Your Own Model Architecture with the Outputs from MobileNet as Its Input 308 Step 3. Gather and Format the Data 310 Step 4. Train the Model 316 Step 5. Run Inference with the Model 317 Transfer Learning from TensorFlow Hub 319 Using Models from TensorFlow.org 322 Summary 324 19. Deployment with TensorFlow Serving. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327 What Is TensorFlow Serving? 327 Installing TensorFlow Serving 330 Installing Using Docker 330 Installing Directly on Linux 331 Building and Serving a Model 332 Exploring Server Configuration 335 Summary 338 20. AI Ethics, Fairness, and Privacy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 339 Fairness in Programming 340 Fairness in Machine Learning 343 x | Table of Contents

Tools for Fairness 345 The What-If Tool 345 Facets 346 Federated Learning 349 Step 1. Identify Available Devices for Training 349 Step 2. Identify Suitable Available Devices for Training 350 Step 3. Deploy a Trainable Model to Your Training Set 350 Step 4. Return the Results of the Training to the Server 351 Step 5. Deploy the New Master Model to the Clients 351 Secure Aggregation with Federated Learning 352 Federated Learning with TensorFlow Federated 353 Google’s AI Principles 354 Summary 355 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357 Table of Contents | xi

Foreword Dear Reader, AI is poised to transform every industry, but almost every AI application needs to be customized for its particular use. A system for reading medical records is different from one for finding defects in a factory, which is different from a product recom‐ mendation engine. For AI to reach its full potential, engineers need tools that can help them adapt the amazing capabilities available to the millions of concrete prob‐ lems we wish to solve. When I led the Google Brain team, we started to build the C++ precursor to TensorFlow called DistBelief. We were excited about the potential of harnessing thou‐ sands of CPUs to train a neural network (for instance, using 16,000 CPUs to train a cat detector on unlabeled YouTube videos). How far deep learning has come since then! What was once cutting-edge can now be done for around $3,000 of cloud com‐ puting credits, and Google routinely trains neural networks using TPUs and GPUs at a scale that was unimaginable just years ago. TensorFlow, too, has come a long way. It is far more usable than what we had in the early days, and has rich features ranging from modeling, to using pretrained models, to deploying on low-compute edge devices. It is today empowering hundreds of thou‐ sands of developers to build their own deep learning models. Laurence Moroney, as Google’s lead AI Advocate, has been a major force in building TensorFlow into one of the world’s leading AI frameworks. I was privileged to sup‐ port his teaching TensorFlow with deeplearning.ai and Coursera. These courses have reached over 80,000 learners and received numerous glowing reviews. One unexpected aspect of friendship with Laurence is that he is also a free source of Irish poetry. He once Slacked me: xiii

Andrew sang a sad old song fainted through miss milliner invitation hoops fainted fainted [...] He had trained an LSTM on lyrics of traditional Irish songs and it generated these lines. If AI opens the door to fun like that, how could anyone not want to get involved? You can (i) work on exciting projects that move humanity forward, (ii) advance your career, and (iii) get free Irish poetry. I wish you the best in your journey learning TensorFlow. With Laurence as a teacher, great adventures await you. Keep learning, — Andrew Ng Founder, deeplearning.ai xiv | Foreword

Preface Welcome to AI and Machine Learning for Coders, a book that I’ve been wanting to write for many years but that has only really become possible due to recent advances in machine learning (ML) and, in particular, TensorFlow. The goal of this book is to prepare you, as a coder, for many of the scenarios that you can address with machine learning, with the aim of equipping you to be an ML and AI developer without need‐ ing a PhD! I hope that you’ll find it useful, and that it will empower you with the con‐ fidence to get started on this wonderful and rewarding journey. Who Should Read This Book If you’re interested in AI and ML, and you want to get up and running quickly with building models that learn from data, this book is for you. If you’re interested in get‐ ting started with common AI and ML concepts—computer vision, natural language processing, sequence modeling, and more—and want to see how neural networks can be trained to solve problems in these spaces, I think you’ll enjoy this book. And if you have models that you’ve trained and want to get them into the hands of users on mobile, in the browser, or via the cloud, then this book is also for you! Most of all, if you’ve been put off entering this valuable area of computer science because of perceived difficulty, in particular believing that you’ll need to dust off your old calculus books, then fear not: this book takes a code-first approach that shows you just how easy it is to get started in the world of machine learning and artificial intelligence using Python and TensorFlow. Why I Wrote This Book I first got seriously involved with artificial intelligence in the spring of 1992. A freshly minted physics graduate living in London in the midst of a terrible recession, I had been unemployed for six months. The British government started a program to train 20 people in AI technology, and put out a call for applicants. I was the first xv

participant selected. Three months later, the program failed miserably, because while there was plenty of theoretical work that could be done with AI, there was no easy way to do it practically. One could write simple inference in a language called Prolog, and perform list processing in a language called Lisp, but there was no clear path to deploying them in industry. The famous “AI winter” followed. Then, in 2016, while I was working at Google on a product called Firebase, the com‐ pany offered machine learning training to all engineers. I sat in a room with a num‐ ber of other people and listened to lectures about calculus and gradient descent. I couldn’t quite match this to a practical implementation of ML, and I was suddenly transported back to 1992. I brought feedback about this, and about how we should educate people in ML, to the TensorFlow team—and they hired me in 2017. With the release of TensorFlow 2.0 in 2018, and in particular the emphasis on high-level APIs that made it easy for developers to get started, I realized the need was there for a book that took advantage of this, and widened access to ML so that it wasn’t just for mathe‐ maticians or PhDs anymore. I believe that more people using this technology and deploying it to end users will lead to an explosion in AI and ML that will prevent another AI winter, and change the world very much for the better. I’m already seeing the impact of this, from the work done by Google on diabetic retinopathy, through Penn State University and PlantVillage building an ML model for mobile that helps farmers diagnose cassava disease, Médecins Sans Frontières using TensorFlow models to help diagnose antibi‐ otic resistance, and much, much more! Navigating This Book The book is written in two main parts. Part I (Chapters 1–11) talks about how to use TensorFlow to build machine learning models for a variety of scenarios. It takes you from first principles—building a model with a neural network containing only one neuron—through computer vision, natural language processing, and sequence mod‐ eling. Part II (Chapters 12–20) then walks you through scenarios for putting your models in people’s hands on Android and iOS, in browsers with JavaScript, and serv‐ ing via the cloud. Most chapters are standalone, so you can drop in and learn some‐ thing new, or, of course, you could just read the book cover to cover. Technology You Need to Understand The goal of the first half of the book is to help you learn how to use TensorFlow to build models with a variety of architectures. The only real prerequisite to this is understanding Python, and in particular Python notation for data and array process‐ ing. You might also want to explore Numpy, a Python library for numeric calculations. If you have no familiarity with these, they are quite easy to learn, and xvi | Preface

you can probably pick up what you need as you go along (although some of the array notation might be a bit hard to grasp). For the second half of the book, I generally will not teach the languages that are shown, but instead show how TensorFlow models can be used in them. So, for exam‐ ple, in the Android chapter (Chapter 13) you’ll explore building apps in Kotlin with Android studio, and in the iOS chapter (Chapter 14) you’ll explore building apps in Swift with Xcode. I won’t be teaching the syntax of these languages, so if you aren’t familiar with them, you may need a primer—Learning Swift by Jonathan Manning, Paris Buttfield-Addison, and Tim Nugent (O’Reilly) is a great choice. Online Resources A variety of online resources are used by, and supported in, this book. At the very least I would recommend that you keep an eye on TensorFlow and its associated You‐ Tube channel for any updates and breaking changes to technologies discussed in the book. The code for this book is available at https://github.com/lmoroney/tfbook, and I will keep it up to date there as the platform evolves. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program ele‐ ments such as variable or function names, data types, environment variables, statements, and keywords. Constant width bold Used for emphasis in code snippets. This element signifies a note. Preface | xvii

Using Code Examples This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “AI and Machine Learning for Coders, by Laurence Moroney. Copyright 2021 Laurence Moroney, 978-1-492-07819-7.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit http://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) xviii | Preface