Name: Machine Learning for Business Using Amazon SageMaker and Jupyter (Doug Hudgeon, Richard Nichol)（Z-Library）
Author: Unknown Author

Page 1

M A N N I N G Doug Hudgeon Richard Nichol Using Amazon SageMaker and Jupyter

Page 2

Each scenario chapter covers an operational area of a typical company. Chapters 3 and 4 (retention and support) deal with customers. Chapters 2 and 5 (purchase approval and invoice audit) deal with suppliers. And chapters 6 and 7 deal with facilities management (power consumption forecasting). Your company Customers Support (ch04) Retention (ch03) Suppliers Facilities Purchase approval (ch02) Invoice audit (ch05) Power consumption forecasting (ch06 & ch07)

Page 3

Machine Learning for Business USING AMAZON SAGEMAKER AND JUPYTER DOUG HUDGEON AND RICHARD NICHOL M A N N I N G SHELTER ISLAND

Page 4

For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact Special Sales Department Manning Publications Co. 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 Email: orders@manning.com ©2020 by Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps. Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine. Development editor: Toni Arritola Technical development editor: Arthur Zubarev Manning Publications Co. Review editor: Ivan Martinović 20 Baldwin Road Production editor: Deirdre Hiam PO Box 761 Copy editor: Frances Buran Shelter Island, NY 11964 Proofreader: Katie Tennant Technical proofreader: Karsten Strøbæk Typesetter: Dennis Dalinnik Cover designer: Marija Tudor ISBN: 9781617295836 Printed in the United States of America

Page 5

brief contents PART 1 MACHINE LEARNING FOR BUSINESS ................................1 1 ■ How machine learning applies to your business 3 PART 2 SIX SCENARIOS: MACHINE LEARNING FOR BUSINESS ......23 2 ■ Should you send a purchase order to a technical approver? 25 3 ■ Should you call a customer because they are at risk of churning? 49 4 ■ Should an incident be escalated to your support team? 76 5 ■ Should you question an invoice sent by a supplier? 99 6 ■ Forecasting your company’s monthly power usage 128 7 ■ Improving your company’s monthly power usage forecast 161 PART 3 MOVING MACHINE LEARNING INTO PRODUCTION.......185 8 ■ Serving predictions over the web 187 9 ■ Case studies 211iii

Page 6

(This page has no text content)

Page 7

contents preface xiii acknowledgments xv about this book xvii about the authors xx about the cover illustration xxi PART 1 MACHINE LEARNING FOR BUSINESS ......................1 1 How machine learning applies to your business 3 1.1 Why are our business systems so terrible? 4 1.2 Why is automation important now? 8 What is productivity? 9 ■ How will machine learning improve productivity? 9 1.3 How do machines make decisions? 10 People: Rules-based or not? 10 ■ Can you trust a pattern-based answer? 11 ■ How can machine learning improve your business systems? 12 1.4 Can a machine help Karen make decisions? 12 Target variables 13 ■ Features 13 1.5 How does a machine learn? 14v

Page 8

CONTENTSvi1.6 Getting approval in your company to use machine learning to make decisions 17 1.7 The tools 18 What are AWS and SageMaker, and how can they help you? 18 What is a Jupyter notebook? 19 1.8 Setting up SageMaker in preparation for tackling the scenarios in chapters 2 through 7 19 1.9 The time to act is now 20 PART 2 SIX SCENARIOS: MACHINE LEARNING FOR BUSINESS....................................................23 2 Should you send a purchase order to a technical approver? 25 2.1 The decision 26 2.2 The data 27 2.3 Putting on your training wheels 28 2.4 Running the Jupyter notebook and making predictions 29 Part 1: Loading and examining the data 32 ■ Part 2: Getting the data into the right shape 36 ■ Part 3: Creating training, validation, and test datasets 39 ■ Part 4: Training the model 41 Part 5: Hosting the model 43 ■ Part 6: Testing the model 44 2.5 Deleting the endpoint and shutting down your notebook instance 46 Deleting the endpoint 46 ■ Shutting down the notebook instance 47 3 Should you call a customer because they are at risk of churning? 49 3.1 What are you making decisions about? 50 3.2 The process flow 50 3.3 Preparing the dataset 52 Transformation 1: Normalizing the data 53 ■ Transformation 2: Calculating the change from week to week 54 3.4 XGBoost primer 54 How XGBoost works 54 ■ How the machine learning model determines whether the function is getting better or getting worse AUC 57

Page 9

CONTENTS vii3.5 Getting ready to build the model 58 Uploading a dataset to S3 59 ■ Setting up a notebook on SageMaker 60 3.6 Building the model 61 Part 1: Loading and examining the data 62 ■ Part 2: Getting the data into the right shape 65 ■ Part 3: Creating training, validation, and test datasets 65 ■ Part 4: Training the model 67 ■ Part 5: Hosting the model 70 ■ Part 6: Testing the model 70 3.7 Deleting the endpoint and shutting down your notebook instance 73 Deleting the endpoint 73 ■ Shutting down the notebook instance 74 3.8 Checking to make sure the endpoint is deleted 74 4 Should an incident be escalated to your support team? 76 4.1 What are you making decisions about? 77 4.2 The process flow 77 4.3 Preparing the dataset 78 4.4 NLP (natural language processing) 79 Creating word vectors 80 ■ Deciding how many words to include in each group 82 4.5 What is BlazingText and how does it work? 83 4.6 Getting ready to build the model 84 Uploading a dataset to S3 85 ■ Setting up a notebook on SageMaker 86 4.7 Building the model 86 Part 1: Loading and examining the data 87 ■ Part 2: Getting the data into the right shape 90 ■ Part 3: Creating training and validation datasets 93 ■ Part 4: Training the model 93 Part 5: Hosting the model 95 ■ Part 6: Testing the model 96 4.8 Deleting the endpoint and shutting down your notebook instance 97 Deleting the endpoint 97 ■ Shutting down the notebook instance 97 4.9 Checking to make sure the endpoint is deleted 97

Page 10

CONTENTSviii5 Should you question an invoice sent by a supplier? 99 5.1 What are you making decisions about? 100 5.2 The process flow 101 5.3 Preparing the dataset 103 5.4 What are anomalies 104 5.5 Supervised vs. unsupervised machine learning 105 5.6 What is Random Cut Forest and how does it work? 106 Sample 1 106 ■ Sample 2 109 5.7 Getting ready to build the model 114 Uploading a dataset to S3 114 ■ Setting up a notebook on SageMaker 115 5.8 Building the model 115 Part 1: Loading and examining the data 116 ■ Part 2: Getting the data into the right shape 120 ■ Part 3: Creating training and validation datasets 121 ■ Part 4: Training the model 121 Part 5: Hosting the model 122 ■ Part 6: Testing the model 123 5.9 Deleting the endpoint and shutting down your notebook instance 126 Deleting the endpoint 126 ■ Shutting down the notebook instance 126 5.10 Checking to make sure the endpoint is deleted 126 6 Forecasting your company’s monthly power usage 128 6.1 What are you making decisions about? 129 Introduction to time-series data 130 ■ Kiara’s time-series data: Daily power consumption 132 6.2 Loading the Jupyter notebook for working with time-series data 133 6.3 Preparing the dataset: Charting time-series data 134 Displaying columns of data with a loop 137 ■ Creating multiple charts 138 6.4 What is a neural network? 139 6.5 Getting ready to build the model 140 Uploading a dataset to S3 141 ■ Setting up a notebook on SageMaker 141

Page 11

CONTENTS ix6.6 Building the model 141 Part 1: Loading and examining the data 142 ■ Part 2: Getting the data into the right shape 144 ■ Part 3: Creating training and testing datasets 147 ■ Part 4: Training the model 150 Part 5: Hosting the model 152 ■ Part 6: Making predictions and plotting results 153 6.7 Deleting the endpoint and shutting down your notebook instance 158 Deleting the endpoint 158 ■ Shutting down the notebook instance 158 6.8 Checking to make sure the endpoint is deleted 159 7 Improving your company’s monthly power usage forecast 161 7.1 DeepAR’s ability to pick up periodic events 161 7.2 DeepAR’s greatest strength: Incorporating related time series 163 7.3 Incorporating additional datasets into Kiara’s power consumption model 164 7.4 Getting ready to build the model 165 Downloading the notebook we prepared 165 ■ Setting up the folder on SageMaker 166 ■ Uploading the notebook to SageMaker 166 Downloading the datasets from the S3 bucket 166 ■ Setting up a folder on S3 to hold your data 167 ■ Uploading the datasets to your AWS bucket 167 7.5 Building the model 168 Part 1: Setting up the notebook 168 ■ Part 2: Importing the datasets 169 ■ Part 3: Getting the data into the right shape 170 Part 4: Creating training and test datasets 172 ■ Part 5: Configuring the model and setting up the server to build the model 175 ■ Part 6: Making predictions and plotting results 179 7.6 Deleting the endpoint and shutting down your notebook instance 182 Deleting the endpoint 182 ■ Shutting down the notebook instance 183 7.7 Checking to make sure the endpoint is deleted 183

Page 12

CONTENTSxPART 3 MOVING MACHINE LEARNING INTO PRODUCTION ..................................................185 8 Serving predictions over the web 187 8.1 Why is serving decisions and predictions over the web so difficult? 188 8.2 Overview of steps for this chapter 189 8.3 The SageMaker endpoint 189 8.4 Setting up the SageMaker endpoint 190 Uploading the notebook 191 ■ Uploading the data 193 Running the notebook and creating the endpoint 194 8.5 Setting up the serverless API endpoint 197 Setting up your AWS credentials on your AWS account 198 Setting up your AWS credentials on your local computer 199 Configuring your credentials 200 8.6 Creating the web endpoint 201 Installing Chalice 202 ■ Creating a Hello World API 204 Adding the code that serves the SageMaker endpoint 205 Configuring permissions 207 ■ Updating requirements.txt 207 Deploying Chalice 208 8.7 Serving decisions 208 9 Case studies 211 9.1 Case study 1: WorkPac 212 Designing the project 214 ■ Stage 1: Preparing and testing the model 215 ■ Stage 2: Implementing proof of concept (POC) 216 Stage 3: Embedding the process into the company’s operations 216 Next steps 217 ■ Lessons learned 217 9.2 Case study 2: Faethm 217 AI at the core 217 ■ Using machine learning to improve processes at Faethm 217 ■ Stage 1: Getting the data 219 ■ Stage 2: Identifying the features 220 ■ Stage 3: Validating the results 220 ■ Stage 4: Implementing in production 220 9.3 Conclusion 220 Perspective 1: Building trust 221 ■ Perspective 2: Geting the data right 221 ■ Perspective 3: Designing your operating model to make the most of your machine learning capability 221 ■ Perspective 4: What does your company look like once you are using machine learning everywhere? 221

Page 13

CONTENTS xiappendix A Signing up for Amazon AWS 222 appendix B Setting up and using S3 to store files 229 appendix C Setting up and using AWS SageMaker to build a machine learning system 238 appendix D Shutting it all down 243 appendix E Installing Python 247 index 249

Page 14

(This page has no text content)

Page 15

preface This book shows you how to apply machine learning in your company to make your business processes faster and more resilient to change. This book is for people begin- ning their journey in machine learning or for those who are more experienced with machine learning but want to see how it can be applied in practice. Based on our experiences with automating business processes and implementing machine learning applications, we wanted to write a book that would allow anyone to start using machine learning in their company. The caveat to anyone isn’t that you need to have a certain technical background, it’s that you’re willing to put in the time when you run the code to understand what’s happening and why. We look at a variety of different functions within various companies ranging across accounts payable (supplier invoices), facilities management (power consumption forecasting), customer support (support tickets), and sales (customer retention). The intent is that this will give you some insight into the range and scale of potential appli- cations of machine learning and encourage you to discover new business applications on your own. A secondary focus of this book is to demonstrate how you can use the Amazon SageMaker cloud service to rapidly and cost effectively bring your business ideas to life. Most of the ideas we present can be implemented using other services (such as Google Cloud or Microsoft Azure); however, the differences are significant enough that to cover multiple providers would be beyond the scope of this book. xiii

Page 16

PREFACExiv We hope you enjoy our book and that you’re able to dramatically improve the productivity of your company by applying the techniques inside. Please hit us up in liveBook if you have questions, comments, suggestions, or examples of how you’ve tackled certain problems. See page xxi for access to the liveBook site. We’d love to hear from you.

Page 17

acknowledgments Writing this book was a lot of work but would have been a lot more without Richie cranking out the notebook code and contributing to chapter ideas. My advice to any- one looking to write a technical book is to find a coauthor and break up the work. Richie and I have different coding styles, and I learned to appreciate his way of tack- ling certain problems during my documentation of his code. I’d like to acknowledge the team at Manning for their help and guidance through the process, and Toni Arritola, in particular, for accommodating the different time zones and having the flexibility to deal with two very busy people in putting this book together. Thank you to everyone at Manning: Deirdre Hiam, our production editor, Frances Buran, our copy editor, Katie Tennant, our proofreader, Arthur Zubarev, our technical development editor, Ivan Martinović, our review editor, and Karsten Strøbæk, our tech- nical proofreader. To all of our reviewers—Aleksandr Novomlinov, Arpit Khandelwal, Burkhard Nestmann, Clemens Baader, Conor Redmond, Dana Arritola, Dary Merckens, Dhivya Sivasubramanian, Dinesh Ghanta, Gerd Klevesaat, James Black, James Nyika, Jeff Smith, Jobinesh Purushothaman Manakkattil, John Bassil, Jorge Ezequiel Bo, Kevin Kraus, Laurens Meulman, Madhavan Ramani, Mark Poler, Muhammad Sohaib Arif, Nikos Kanakaris, Paulo Nuin, Richard Tobias, Ryan Kramer, Sergio Fernandez Gonzalez, Shawn Eion Smith, Syed Nouman Hasany, and Taylor Delehanty—thank you, your suggestions helped make this a better book. And, of course, I’d like to thank my spouse and family for their patience and understanding. —Doug Hudgeonxv

Page 18

ACKNOWLEDGMENTSxviI’m very grateful to Doug for asking me to join him as coauthor in writing this book, but also for his creativity, positivity, friendship, and sense of humor. Although it was a lot of work, it was also a pleasure. I’d also like to offer my special thanks to my parents, family, and friends for putting up with the long hours and lost weekends. Most of all, I’d like to thank my wife, Xenie, who could not have been more supportive and understanding during the years I com- pleted my studies as well as this book. No husband could hope for a better wife, and I can’t believe how lucky I am to be spending my days beside her. —Richard Nichol

Page 19

about this book Companies are on the cusp of a massive leap in productivity. Today, thousands of peo- ple are involved in process work, where they take information from one source and put it into another place. For example, take procurement and accounts payable: ■ Procurement staff help a customer create a purchase order, and then send it to a supplier. ■ The supplier’s order-processing staff then take the purchase order and enter it into the order-processing system, where it’s fulfilled and shipped to the customer that placed the order. ■ Staff on the customer’s loading dock receive the goods, and the finance staff enters the invoice into the customer’s finance system. Over the next decade, all of these processes will be completely automated in almost every company, and machine learning will play a big part in automating the decision points at each stage of the process. It will help businesses make the following decisions: ■ Does the person approving the order have the authority to do so? ■ Is it OK to substitute a product for an out-of-stock item? ■ If a supplier has substituted a product, will the receiver accept it? ■ Is the invoice OK to pay as is or should it be queried? The real benefit of machine learning for business is that it allows you to build decision- making applications that are resilient to change. Instead of programming dozens or hundreds of rules into your systems, you feed in past examples of good and badxvii

Page 20

ABOUT THIS BOOKxviiidecisions, and then let the machine make a determination based on how similar the current scenario is to past examples. The benefit of this is that the system doesn’t break when it comes across novel input. The challenge is that it takes a different mindset and approach to deliver a machine learning project than it does to deliver a normal IT project. In a normal IT project, you can test each of the rules to ensure they work. In a machine learning project, you can only test to see whether the algorithm has responded appropriately to the test scenarios. And you don’t know how it will react to novel input. Trusting in safeguards that catch it when it’s not reacting appropriately requires you and your stakeholders to be comfortable with this uncertainty. Who should read this book This book is targeted at people who may be more comfortable using Excel than using a programming language such as Python. Each chapter contains a fully work- ing Jupyter notebook that creates the machine learning model, deploys it, and runs it against a dataset prepared for the chapter. You don’t need to do any coding to see the code in action. Each chapter then takes you through the code step by step so that you understand how it works. With minor modifications, you can apply the code directly to your own data. By the end of the book, you should be able to tackle a wide range of machine learning projects within your company. How this book is organized: A roadmap This book has three parts: Part 1 starts with a description of why businesses need to become a lot more pro- ductive to remain competitive, and it explains how effective decision-making plays a role in this. You’ll then move on to why machine learning is a good way to make busi- ness decisions, and how, using open source tools and tools from AWS, you can start applying machine learning to making decisions in your business. In part 2, you’ll then work through six scenarios (one scenario per chapter) that show how machine learning can be used to make decisions in your business. The sce- narios focus on how an ordinary company can use machine learning, rather than on how Facebook or Google or Amazon use machine learning. Finally, in part 3, you’ll learn how to set up and share your machine learning mod- els on the web so your company can make decisions using machine learning. You’ll then go through some case studies that show how companies manage the change that comes along with using machine learning to make decisions. About the code In each chapter in part 2, we provide you a Jupyter notebook and one or more sample datasets that you can upload to AWS SageMaker and run. In part 3, we provide the code to set up a serverless API to serve your predictions to users across the web.

Machine Learning for Business Using Amazon SageMaker and Jupyter (Doug Hudgeon, Richard Nichol)（Z-Library）

Machine Learning for Business Using Amazon SageMaker and Jupyter (Doug Hudgeon, Richard Nichol)（Z-Library）

Registered users can read the full content for free

AI Reading Assistant

Passage locations

Support Author

Recommended for You