Page
1
Gant Laborde Foreword by Laurence Moroney Learning TensorFlow.js Powerful Machine Learning in JavaScript
Page
2
(This page has no text content)
Page
3
Praise for Learning TensorFlow.js What Gant has done with this book is to cut to the chase, and teach you the important stuff you need to know while keeping you firmly within the web developer role, using JavaScript and the Browser. —Laurence Moroney, Lead AI Advocate, Google Machine learning has the potential to influence every industry out there. This book enables you to take your first steps with TensorFlow.js, allowing any JavaScript developer to gain superpowers in their next web application and beyond. This book is a great introduction to machine learning using TensorFlow.js that also applies what you learn with real world examples that are easy to digest. —Jason Mayes, Senior Developer Relations Engineer for TensorFlow.js at Google Gant’s ability to navigate explaining the complexities of machine learning while avoiding the pitfalls of complicated mathematics is uncanny, and you’d be hard-pressed to find a better introduction to data science using JavaScript. —Lee Warrick, Fullstack JavaScript Developer I’m delighted to have read Learning TensorFlow.js. It is without doubt a good way to get out of my comfort zone of backend engineering and try out building some exciting frontend applications, leveraging the power of the book’s content about Tensorflow.js as the go-to framework for ML web applications. —Laura Uzcátegui, Software Engineer, Microsoft
Page
4
This book serves as the right introduction to building small deep learning models fit for web and mobile based applications. The examples in the book along with the detailed explanation will make your learning smooth and fun. —Vishwesh Ravi Shrimali, Engineer, Mercedes Benz R&D India I wish that I’d had this book myself to learn neural networks and TensorFlow.js in the past! Astonishingly simple and beautifully written, it goes from zero to doing a full capstone project in 12 short chapters. A must-have in every library. —Axel Sirota, Machine Learning Research Engineer This is a much-needed introduction to TensorFlow.js, with great examples, amazing illustrations, and insightful quotes at the beginning of each chapter. A must-read for anyone who’s serious about doing AI with JavaScript. —Alexey Grigorev, Founder of DataTalks.Club Machine learning for the web is still in its infancy, and books such as the one you’re holding right now are super important. As a machine learning engineer working on ML tools for the JavaScript environment, my top recommendation for web developers seeking to add ML to their projects is definitely LearningTensorFlow.js. —Rising Odegua, Cocreator of Danfo.js
Page
5
Gant Laborde Learning TensorFlow.js Powerful Machine Learning in JavaScript Boston Farnham Sebastopol TokyoBeijing
Page
6
978-1-492-09079-3 [LSI] Learning TensorFlow.js by Gant Laborde Copyright © 2021 Gant Laborde. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Jennifer Pollock Development Editor: Michele Cronin Production Editor: Caitlin Ghegan Copyeditor: Kim Wimpsett Proofreader: JM Olejarz Indexer: Ellen Troutman-Zaig Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea May 2021: First Edition Revision History for the First Edition 2021-05-07: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781492090793 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Learning TensorFlow.js, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the author, and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.
Page
7
This book is dedicated to the most infectious & gentle smile. To the irrepressible spark & endless joy of my heart. To my loving daughter, I love you, Mila.
Page
8
(This page has no text content)
Page
9
Table of Contents Foreword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii 1. AI Is Magic. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 The Path of AI in JavaScript 2 What Is Intelligence? 3 The History of AI 4 The Neural Network 6 Today’s AI 8 Why TensorFlow.js? 9 Significant Support 9 Online Ready 10 Offline Ready 10 Privacy 10 Diversity 10 Types of Machine Learning 11 Quick Definition: Supervised Learning 11 Quick Definition: Unsupervised Learning 12 Quick Definition: Semisupervised Learning 12 Quick Definition: Reinforcement Learning 13 Information Overload 13 AI Is Everywhere 14 A Tour of What Frameworks Provide 14 What Is a Model? 16 In This Book 17 Associated Code 18 vii
Page
10
Chapter Sections 20 Common AI/ML Terminology 20 Chapter Review 24 Review Questions 24 2. Introducing TensorFlow.js. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Hello, TensorFlow.js 25 Leveraging TensorFlow.js 27 Let’s Get TensorFlow.js Ready 28 Getting Set Up with TensorFlow.js in the Browser 29 Using NPM 29 Including a Script Tag 29 Getting Set Up with TensorFlow.js Node 30 Verifying TensorFlow.js Is Working 32 Download and Run These Examples 32 Let’s Use Some Real TensorFlow.js 34 The Toxicity Classifier 35 Loading the Model 40 Classifying 42 Try It Yourself 42 Chapter Review 42 Chapter Challenge: Truck Alert! 43 Review Questions 44 3. Introducing Tensors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Why Tensors? 45 Hello, Tensors 46 Creating Tensors 47 Tensors for Data Exercises 50 Tensors on Tour 53 Tensors Provide Speed 53 Tensors Provide Direct Access 53 Tensors Batch Data 54 Tensors in Memory 54 Deallocating Tensors 54 Automatic Tensor Cleanup 55 Tensors Come Home 57 Retrieving Tensor Data 58 Tensor Manipulation 60 Tensors and Mathematics 60 Recommending Tensors 61 viii | Table of Contents
Page
11
Chapter Review 66 Chapter Challenge: What Makes You So Special? 66 Review Questions 67 4. Image Tensors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 Visual Tensors 70 Quick Image Tensors 72 JPGs and PNGs and GIFs, Oh My! 76 Browser: Tensor to Image 76 Browser: Image to Tensor 77 Node: Tensor to Image 80 Node: Image to Tensor 83 Common Image Modifications 85 Mirroring Image Tensors 85 Resizing Image Tensors 89 Cropping Image Tensors 91 New Image Tools 93 Chapter Review 93 Chapter Challenge: Sorting Chaos 93 Review Questions 94 5. Introducing Models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 Loading Models 98 Loading Models Via Public URL 98 Loading Models from Other Locations 101 Our First Consumed Model 101 Loading, Encoding, and Asking a Model 102 Interpreting the Results 105 Cleaning the Board After 107 Our First TensorFlow Hub Model 107 Exploring TFHub 107 Wiring Up Inception v3 108 Our First Overlayed Model 110 The Localization Model 111 Labeling the Detection 113 Chapter Review 116 Chapter Challenge: Cute Faces 116 Review Questions 117 6. Advanced Models and UI. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119 MobileNet Again 120 Table of Contents | ix
Page
12
SSD MobileNet 122 Bounding Outputs 124 Reading Model Outputs 124 Displaying All Outputs 126 Detection Cleanup 127 Quality Checking 128 IoUs and NMS 129 Adding Text Overlays 134 Solving Low Contrast 134 Solving Draw Order 136 Connecting to a Webcam 138 Moving from Image to Video 139 Activating a Webcam 140 Drawing Detections 141 Chapter Review 142 Chapter Challenge: Top Detective 142 Review Questions 143 7. Model-Making Resources. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145 Out-of-Network Model Shopping 146 Model Zoos 146 Converting Models 146 Your First Customized Model 149 Meet Teachable Machine 150 Use Teachable Machine 151 Gathering Data and Training 152 Verifying the Model 155 Machine Learning Gotchas 157 Small Amounts of Data 157 Poor Data 157 Data Bias 158 Overfitting 158 Underfitting 158 Datasets Shopping 159 The Popular Datasets 161 Chapter Review 162 Chapter Challenge: R.I.P. You Will Be MNIST 162 Review Questions 163 8. Training Models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 Training 101 166 x | Table of Contents
Page
13
Data Prep 167 Design a Model 167 Identify Learning Metrics 169 Task the Model with Training 171 Put It All Together 171 Nonlinear Training 101 174 Gathering the Data 175 Adding Activations to Neurons 175 Watching Training 178 Improving Training 180 Chapter Review 185 Chapter Challenge: The Model Architect 185 Review Questions 186 9. Classification Models and Data Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 Classification Models 188 The Titanic 190 Titanic Dataset 190 Danfo.js 191 Preparing for the Titanic 192 Training on Titanic Data 197 Feature Engineering 199 Dnotebook 200 Titanic Visuals 201 Creating Features (aka Preprocessing) 204 Feature Engineered Training Results 207 Reviewing Results 207 Chapter Review 207 Chapter Challenge: Ship Happens 208 Review Questions 209 10. Image Training. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 Understanding Convolutions 212 Convolutions Quick Summary 213 Adding Convolution Layers 215 Understanding Max Pooling 216 Max Pooling Quick Summary 216 Adding Max Pooling Layers 218 Training Image Classification 218 Handling Image Data 220 The Sorting Hat 220 Table of Contents | xi
Page
14
Getting Started 222 Converting Folders of Images 224 The CNN Model 227 Training and Saving 231 Testing the Model 232 Building a Sketchpad 232 Reading the Sketchpad 233 Chapter Review 236 Chapter Challenge: Saving the Magic 236 Review Questions 237 11. Transfer Learning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239 How Does Transfer Learning Work? 240 Transfer Learning Neural Networks 241 Easy MobileNet Transfer Learning 242 TensorFlow Hub Check, Mate! 244 Utilizing Layers Models for Transfer Learning 248 Shaving Layers on MobileNet 249 Layers Feature Model 250 A Unified Model 251 No Training Needed 251 Easy KNN: Bunnies Versus Sports Cars 253 Chapter Review 256 Chapter Challenge: Warp-Speed Learning 256 Review Questions 257 12. Dicify: Capstone Project. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259 A Dicey Challenge 260 The Plan 261 The Data 261 The Training 263 The Website 263 Generating Training Data 263 Training 268 The Site Interface 269 Cut into Dice 270 Reconstruct the Image 272 Chapter Review 274 Chapter Challenge: Easy as 01, 10, 11 275 Review Questions 276 xii | Table of Contents
Page
15
Afterword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277 A. Chapter Review Answers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281 B. Chapter Challenge Answers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289 C. Rights and Licenses. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303 Table of Contents | xiii
Page
16
(This page has no text content)
Page
17
Foreword AI and machine learning are revolutionary technologies that can change the world, but they can only do that if there are developers using good APIs to take advantage of the advancements these technologies bring. One such advancement is the ability to run machine learning models in the browser, empowering applications that act intelligently. The rise of TensorFlow.js tells me that AI has arrived. It’s no longer exclusively in the realm of data scientists with supercomputers; it’s now accessible to the millions of developers who code in JavaScript daily. But there’s a gap. The tools and techniques for building models are still very much in the hands of those who know the mysteries of Python, NumPy, graphics processing units (GPUs), data science, feature modeling, supervised learning, tensors, and many more weird and wonderful terms that you probably aren’t familiar with! What Gant has done with this book is to cut to the chase, teaching you the important stuff you need to know while keeping you firmly within the web developer role, using JavaScript and the browser. He’ll introduce you to the concepts of AI and machine learning with a clear focus on how they can be used in the platform you care about. Often, I hear developers ask, when wanting to use machine learning, “Where can I find stuff that I can reuse? I don’t want to learn to be an ML engineer just to figure out if this stuff will work for me!” Gant answers that question in this book. You’ll discover premade models that you can take lock, stock, and barrel from TensorFlow Hub. You will also learn how to stand on the shoulders of giants by taking selected portions of models built using millions of items of data and many thousands of hours of training, and see how you can transfer learn from them to your own model. Then, just drop it into your page and have Java‐ Script do the rest! Developers ask, “How can I use machine learning on the platform I care about without extensive retraining?” xv
Page
18
This book goes deeply into that—showing you how to bridge the gap between Java‐ Script and models that were trained using TensorFlow. From data conversion between primitives and tensors to parsing output probabilities into text, this book guides you through the steps to integrate tightly with your site. Developers ask me, “I want to go beyond other people’s work and simple prototypes. Can I do that as a web developer?” Again, yes. By the time you’ve finished this book, not only will you be familiar with using models, but Gant will give you all the details you need to create them yourself. You’ll learn how to train complex models such as convolutional neural networks to recognize the contents of images, and you’ll do it all in JavaScript. A survey in October 2020 showed that there were 12.4 million JavaScript developers in the world. Other surveys showed that there are about 300,000 AI practitioners globally. With the technology of TensorFlow.js and the skills in this book, you, dear JavaScript developer, can be a part of making AI matter. And this book is a wonder‐ ful, wonderful place to start. Enjoy the journey! — Laurence Moroney March 2021 xvi | Foreword
Page
19
Preface “If you choose not to decide, you still have made a choice.” —Geddy Lee (Rush) Let’s Do This Hindsight is always 20/20. “I should have bought some bitcoin when it was at X” or “If only I had applied at startup Y before they became famous.” The world is replete with moments that define us for better or worse. Time never goes backward, but it echoes the lessons of our younger choices as we go forward. You’re lucky enough to have this book and this moment to decide. The foundations of the software industry are changing due to artificial intelligence. The changes will ultimately be decided by those who grab hold and shape the world into what it will be tomorrow. Machine learning is an adventure into new possibili‐ ties, and when it is unified with the broad exposure of JavaScript, the limits drift away. As I like to tell my audience in my talks on AI, “You didn’t come this far in creating software only to come this far.” So let’s get started and see where our imagination takes us. Why TensorFlow.js? TensorFlow is one of the most popular machine learning frameworks on the market. It’s supported by Google’s top minds and is responsible for powering many of the world’s most influential companies. TensorFlow.js is the indomitable JavaScript framework of TensorFlow and is better than all the competitors. In short, if you want the power of a framework in JavaScript, there’s only one choice that can do it all. xvii
Page
20
Who Should Read This Book? Two primary demographics will enjoy and benefit from the contents of this book: The JavaScript developer If you’re familiar with JavaScript, but you’ve never touched machine learning before, this book will be your guide. It leans into the framework to keep you active in pragmatic and exciting creations. You’ll comprehend the basics of machine learning with hands-on experience through the construction of all kinds of projects. While we won’t shy away from math or deeper concepts, we also won’t overly complicate the experience with them. Read this book if you’re build‐ ing websites in JavaScript and want to gain a new superpower. The AI specialist If you’re familiar with TensorFlow or even the fundamental principles of linear algebra, this book will supply you with countless examples of how to bring your skills to JavaScript. Here, you’ll find various core concepts illustrated, displayed, and portrayed in the TensorFlow.js framework. This will allow you to apply your vast knowledge to a medium that can exist efficiently on edge devices like client browsers or the Internet of Things (IoT). Read this book and learn how to bring your creations to countless devices with rich interactive experiences. This book requires a moderate amount of comfort in reading and understanding modern JavaScript. Book Overview When outlining this book, I realized I’d have to make a choice. Either I could create a whirlwind adventure into a variety of applications of machine learning and touch on each with small, tangible examples, or I could choose a single path that tells an ever- growing story of the concepts. After polling my friends and followers, it was clear that the latter was needed. To keep this book sane and under a thousand pages, I chose to remove any JavaScript frameworks and to focus on a singular pragmatic journey into the visual aspects of AI. Each chapter ends with questions and a particular challenge for you to test your resolve. The Chapter Challenge sections have been carefully constructed to solidify lessons into your TensorFlow.js muscle memory. The Chapters Chapters 1 and 2 begin with core concepts and a concrete example. This yin-and- yang approach reflects the teaching style of the book. Each chapter builds on the les‐ sons, vocabulary, and functions mentioned in previous chapters. xviii | Preface