AI at the Edge (Daniel Situnayake, Jenny Plunkett) (Z-Library)

📄 File Format: PDF

💾 File Size: 11.3 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

(This page has no text content)

📄 Page 2

Praise for AI at the Edge AI at the Edge introduces the new and fast-growing field of edge AI in a practical, easy-to- follow way. It demystifies jargon and highlights real challenges that you are likely to encounter when building edge AI applications. The book offers an essential guide to going from concept to deployment—a must-read for getting started in the field. —Wiebke Hutiri, Delft University of Technology I really love the writing style which makes complex technical topics approachable and digestible. I can imagine it being used as a reference book, returning to it time and time again —which I will certainly be doing! —Fran Baker, Director of Sustainability and Social Impact, Arm What a wonderfully accessible and thorough introduction to the emerging field of edge AI! It covers an impressive breadth of topics, from the core concepts to the latest hardware and software tools, it’s full of actionable advice, and includes several end-to-end examples. Anyone joining this exciting new field will benefit from the deep insights and clarity of thought this book provides. —Aurélien Geron, former lead of YouTube’s automatic video classification team and best-selling author This is the guide to creating smarter devices: AI at the Edge provides an excellent introduction on combining modern AI techniques and embedded systems. —Elecia White, author of Making Embedded Systems and host of the Embedded podcast

📄 Page 3

AI at the Edge Solving Real-World Problems with Embedded Machine Learning Daniel Situnayake and Jenny Plunkett

📄 Page 4

AI at the Edge by Daniel Situnayake and Jenny Plunkett Copyright © 2023 Daniel Situnayake and Jenny Plunkett. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Nicole Butterfield Development Editor: Angela Rufino Production Editor: Elizabeth Faerm Copyeditor: nSight, Inc. Proofreader: Charles Roumeliotis Indexer: WordCo Indexing Services, Inc. Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea January 2023: First Edition Revision History for the First Edition 2023-01-10: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781098120207 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. AI at the Edge, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the authors, and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology

📄 Page 5

this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. 978-1-098-12020-7 [LSI]

📄 Page 6

Dedication Jenny would like to dedicate this book to every woman who is currently pursuing, or is interested in pursuing, an engineering degree—you can do anything you put your mind to. Dan would like to dedicate this book to the Situnayake family. It’s been a tough few years, but we always make it through as a team.

📄 Page 7

Foreword In 2022, GitHub CEO Thomas Dohmke said, “I think the shift to the cloud will happen at such a rapid rate, that in just a few years I predict there will be no more code on your local computer.” This book does a great job of explaining why I and a lot of other people in the emerging field of edge ML think he’s dead wrong. We’re starting to see the emergence of many practical applications like high-quality voice recognition, forest fire prevention, and smart home controls that are only possible because local devices are now capable of running advanced machine learning algorithms. Jenny and Dan have put together a wonderful book that not only explains why adding intelligence to edge applications is so crucial to solving important problems, but also walks the reader through the steps required to design, implement, and test these kinds of applications. It can feel pretty intimidating when you first start looking at a machine learning project on the edge. The field involves a lot of jargon, is changing rapidly, and requires knowledge from domains like embedded systems and artificial intelligence that have traditionally not been well integrated. What the authors have achieved is a gentle but thorough introduction to everything you need to know to work effectively on an application. They’ve also managed to make it accessible to a wide range of readers thanks to their emphasis on examples from the real world and use of plain English instead of math or code to explain even complex topics. This makes the book easy to recommend to product managers, executives, and designers, as well as engineers. They’ve managed to take a lot of hard-won knowledge gained from experience and distill it down to lessons that will give any team working on these kinds of applications a big head start. They also manage to explore beyond the practical concerns of how to build an edge ML application, and will help you understand how to avoid causing harm with your work. The ethical concerns around AI can seem overwhelming, but the authors manage to break them down into questions you can apply in a straightforward way as part of the project planning and testing process. This will help all the stakeholders on your project collaborate, and hopefully avoid a lot of the potential dangers involved in giving computers more decision-making power over our lives. I’ve been working on edge ML applications for over ten years now, first at a startup, then as a tech lead at Google—and now as the founder of another startup, and I will be asking everyone who joins our team to read this book. If you have any interest at all in this area, whether as a coder, designer, manager, or just someone who cares about this new technology that’s emerging in our world, I can’t recommend this book highly enough. I guarantee that reading it will introduce you to a lot of fascinating ideas, as well as help you build the next generation of smart devices. Pete Warden, CEO at Useful Sensors Inc., creator of TensorFlow Lite for Microcontrollers 1 From GitHub’s Twitter account. 1

📄 Page 8

Preface Over the past few years, a growing community of engineers and researchers have quietly rewritten the rules for how computers interact with the physical world. The result, a technology known as “edge artificial intelligence,” promises to upend a century of computer history and touch the lives of every human being. With a tiny software update, edge AI technology can grant cheap, energy-efficient processors— already inside everything from dishwashers to thermostats—the ability to perceive and understand the world. We can empower everyday objects with their own intelligence, no longer dependent on data-hungry centralized servers. And next-generation tools put this magic in reach of everyone, from high school students to conservation researchers. There are already many edge AI products out there in the world. Here are some that we’ll meet in the pages of this book: Smart devices that help prevent forest fires caused by electricity transmission, by mounting to electricity pylons and predicting when a fault may occur Wearable bands that keep firefighters safe by warning when they’re at risk from heat strain and overexertion Voice user interfaces that provide hands-free control of technology, no internet connection required Smart collars that monitor the movements of wild elephants, helping researchers understand their behavior and protect them from conflict Wildlife cameras that identify specific animal species and help scientists understand their behavior The technology of edge AI is still fresh and new, and these existing applications are just a glimpse of what is possible. As more people learn how to work with edge AI, they’ll create applications that solve problems across every avenue of human activity. The goal of this book is to empower you to be one of them. We want to help you create successful edge AI products based on your own unique perspectives. About This Book This book is designed for the engineers, scientists, product managers, and decision makers who will drive this revolution. It’s a high-level guide to the entire space, providing a workflow and a framework for solving real-world problems using edge AI. Among other things, we hope to teach you:

📄 Page 9

The opportunities, limitations, and risks inherent to various edge AI technologies A framework for analyzing problems and designing solutions using AI and embedded machine learning An end-to-end practical workflow for successfully developing edge AI applications In the first part of the book, the initial chapters will introduce and discuss the key concepts, helping you understand the lay of the land. The next few will take you through the practical processes that will help you design and implement your own applications. In the second part of the book, starting in Chapter 11, we’ll use three end-to-end walkthroughs to demonstrate how to apply your knowledge to solve real problems in scientific, industrial, and consumer projects. By the end of the book, you’ll feel confident in viewing the world through the lens of edge AI, and you’ll have a solid set of tools you can use to help build effective solutions. NOTE This book covers a lot of topics! For an overview of everything we’ve included, take a quick look at the table of contents. What to Expect This isn’t a programming book or a tutorial for a particular set of tools, so don’t expect a ton of line-by-line code explanations or step-by-step guides to using specific software. Instead, you’ll learn how to apply general frameworks to solve problems using whichever tools are best suited to the job. That said, this is a topic that benefits greatly from tangible, interactive examples that can be explored, customized, and built upon. In the course of the book, we’ll provide all sorts of artifacts you can explore—from Git repositories to free online datasets and example training pipelines. Many of these will be hosted in Edge Impulse, which is an engineering tool for building edge AI applications. It’s built on open source technologies and standard best practices, so you’ll be able to understand the principles even if you do your own work on a different platform. The book’s authors are both big fans of Edge Impulse—but they may be biased, since they are part of the team that built it! NOTE To guarantee portability, all the artifacts of the machine learning pipeline can be exported from Edge Impulse in open formats, including the datasets, machine learning models, and C++ implementations of any signal processing code. 1

📄 Page 10

What You Need to Know Already This book is about building software that runs on edge devices, so some familiarity with the high-level concepts of embedded development will be helpful. This could be on either resource- constrained devices such as microcontrollers or digital signal processors (DSPs), or on general- purpose devices such as embedded Linux computers. That said, if you’re just getting started with embedded software, you should have no trouble keeping up! We’ll keep things simple and introduce new topics as they come up. Beyond that, no particular knowledge is assumed. Since the goal of this book is to provide a practical road map for an entire field of engineering, we’ll cover a lot of topics at a high level. If you’re interested in digging deeper into anything we mention—from the fundamentals of machine learning to the essentials of ML application design—we’ll provide lots of resources that we’ve found useful in our own learning. Responsible, Ethical, and Effective AI The most important part of building any kind of application is ensuring that it works correctly in the real world. Unfortunately, AI applications are especially vulnerable to a class of issues that make them appear to work well when in reality they are failing—often in very harmful ways. Avoiding this class of problems will be a core theme—if not the core theme—of this book. Because modern AI development is an iterative process, it isn’t enough to test your system at the end of the workflow to see whether it works. Instead, you need to be thinking about the potential pitfalls at every step along the way. You’ll have to understand where the risks lie, critically review your intermediate results, and make informed decisions that take the needs of your stakeholders into account. Over the course of the book, we’ll introduce a strong framework that will help you understand, reason, measure performance, and make decisions based on an awareness of the things that can go wrong when building AI applications. It will be the foundation for our entire development process and will shape the way we design our applications. This process begins at the very inception of a project. To build effective applications, it’s critical to understand that there are certain use cases for which our current approach to artificial intelligence is simply not an appropriate tool. In many situations, the risk of causing harm— physical, financial, or societal—outweighs the potential benefit of deploying AI. This book will teach you how to identify these risks and take them into account when exploring the feasibility of a project. As domain experts, we have the responsibility to make sure the technology we create is used appropriately. Nobody else is better positioned to do this work, so it falls on us to do it well. This book will help you make the right decisions and create applications that perform well, avoid harm, and benefit the wider world.

📄 Page 11

Further Resources A book that covered all of embedded AI, from low-level implementation to high-level design patterns, would be the size of an entire bookshelf! Instead of trying to squeeze everything into one volume, the book you’re reading will provide a detailed but high-level road map of the whole space. To zoom in on the minutiae that are relevant for your particular project, “Learning Edge AI Skills” recommends plenty of further resources. REACHING OUT The authors would love to hear from you; get in touch at hello@edgeaibook.com. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords. Constant width bold Shows commands or other text that should be typed literally by the user. Constant width italic Shows text that should be replaced with user-supplied values or by values determined by context. TIP This element signifies a tip or suggestion. NOTE This element signifies a general note.

📄 Page 12

WARNING This element indicates a warning or caution. Using Code Examples Supplemental material (code examples, exercises, etc.) is available for download at https://github.com/ai-at-the-edge. If you have a technical question or a problem using the code examples, please send email to bookquestions@oreilly.com. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “AI at the Edge by Daniel Situnayake and Jenny Plunkett (O’Reilly). Copyright 2023 Daniel Situnayake and Jenny Plunkett, 978-1-098-12020- 7.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. O’Reilly Online Learning NOTE For more than 40 years, O’Reilly Media has provided technology and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit http://oreilly.com.

📄 Page 13

How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/ai-at-the-edge. Email bookquestions@oreilly.com to comment or ask technical questions about this book. For news and information about our books and courses, visit https://oreilly.com. Find us on LinkedIn: https://linkedin.com/company/oreilly-media. Follow us on Twitter: https://twitter.com/oreillymedia. Watch us on YouTube: https://youtube.com/oreillymedia. Acknowledgments This book wouldn’t have been possible without the hard work and support of a large number of people to whom we are very grateful. We’ve been honored with a foreword by the one and only Pete Warden, who—beyond being a visionary technologist who deserves much of the credit for launching this field—is a wonderful human being and a great friend. Thank you so much for your support, Pete! We extend our deep gratitude to Wiebke (Toussaint) Hutiri, who went truly above and beyond in helping shape and inform the responsible AI content in this book, including contributing a fantastic introduction to “Responsible Design and AI Ethics”. You are a star in your field. We are indebted to our incredible panel of technical reviewers and advisors whose wisdom and insight has had such a huge impact on this book. Their names are Alex Elium, Aurélien Geron, Carlos Roberto Lacerda, David J. Groom, Elecia White, Fran Baker, Jen Fox, Leonardo Cavagnis, Mat Kelcey, Pete Warden, Vijay Janapa Reddi, and Wiebke (Toussaint) Hutiri. An additional big thanks to Benjamin Cabé for allowing us to feature his artificial nose project. Any inaccuracies are entirely the responsibility of the authors.

📄 Page 14

We’d also like to thank the amazing team at O’Reilly, especially Angela Rufino, who has shepherded us through the writing process with the utmost understanding and care. Major gratitude to Elizabeth Faerm, Kristen Brown, Mike Loukides, Nicole Taché, and Rebecca Novack. This book would not exist without the support of our team at Edge Impulse, an all-star cast of absolute heroes. Special thanks to the founders, Zach Shelby and Jan Jongboom, for believing in our vision for this book, supporting us in making it happen, and creating a space where ideas can bloom. Much love to the entire team, which at the time of writing includes: Adam Benzion, Alessandro Grande, Alex Elium, Amir Sherman, Arjan Kamphuis, Artie Beavis, Arun Rajasekaran, Ashvin Roharia, Aurelien Lequertier, Carl Ward, Clinton Oduor, David Schwarz, David Tischler, Dimi Tomov, Dmitry Maslov, Emile Bosch, Eoin Jordan, Evan Rust, Fernando Jiménez Moreno, Francesco Varani, Jed Huang, Jim Edson, Jim van der Voort, Jodie Lane, John Pura, Jorge Silva, Joshua Buck, Juliette Okel, Keelin Murphy, Kirtana Moorthy, Louis Moreau, Louise Paul, Maggi Yang, Mat Kelcey, Mateusz Majchrzycki, Mathijs Baaijens, Mihajlo Raljic, Mike Senese, Mikey Beavis, MJ Lee, Nabil Koroghli, Nick Famighetti, Omar Shrit, Othman Mekhannene, Paige Holvik, Raul James, Raul Vergara, RJ Vissers, Ross Lowe, Sally Atkinson, Saniea Akhtar, Sara Olsson, Sergi Mansilla, Shams Mansoor, Shawn Hanscom, Shawn Hymel, Sheena Patel, Tyler Hoyle, Vojislav Milivojevic, William DeLey, Yan Li, Yana Vibe, and Zin Kyaw. You make magic happen. Jenny would like to thank her Texas family and friends for being super supportive over the years, and her cats Blue Gene and Beatrice for being the best coworkers. She especially would like to thank her dad, Michael Plunkett, who encouraged her to pursue electrical engineering at The University of Texas at Austin, and who inspired her lifelong curiosity in new technologies. Dan would like to thank his family and friends for being supportive of every big adventure. He’s deeply grateful to Lauren Ward for her love and partnership throughout all of our journeys. And he thanks Minicat for her calming feline presence—and permission to use her photographs in this book. 1 Edge Impulse is described in the academic paper “Edge Impulse: An MLOps Platform for Tiny Machine Learning” (S. Hymel et. al, 2022).

📄 Page 15

Chapter 1. A Brief Introduction to Edge AI Welcome on board! In this chapter, we’ll be taking a comprehensive tour of the edge AI world. We’ll define the key terms, learn what makes “edge AI” different from other AI, and explore some of the most important use cases. Our goal for this chapter is to answer these two important questions: What is edge AI, anyway? Why would I ever need it? Defining Key Terms Each area of technology has its own taxonomy of buzzwords, and edge AI is no different. In fact, the term edge AI is a union of two buzzwords, fused together into one mighty term. It’s often heard alongside its siblings, embedded machine learning and TinyML. Before we move on, we better spend some time defining these terms and understanding what they mean. Since we’re dealing with compound buzzwords, let’s deal with the most fundamental parts first. Embedded What is “embedded”? Depending on your background, this may be the most familiar of all the terms we’re trying to describe. Embedded systems are the computers that control the electronics of all sorts of physical devices, from Bluetooth headphones to the engine control unit of a modern car. Embedded software is software that runs on them. Figure 1-1 shows a few places where embedded systems can be found.

📄 Page 16

Figure 1-1. Embedded systems are present in every part of our world, including the home and the workplace Embedded systems can be tiny and simple, like the microcontroller that controls a digital watch, or large and sophisticated, like the embedded Linux computer inside a smart TV. In contrast to general-purpose computers, like a laptop or smartphone, embedded systems are usually meant to perform one specific, dedicated task. Since they power much of our modern technology, embedded systems are extraordinarily widespread. In fact, there were over 28 billion microcontrollers shipped in the year 2020 —just one type of embedded processor. They’re in our homes, our vehicles, our factories, and our city streets. It’s likely you are never more than a few feet from an embedded system. It’s common for embedded systems to reflect the constraints of the environments into which they are deployed. For example, many embedded systems are required to run on battery power, so they’re designed with energy efficiency in mind—perhaps with limited memory or an extremely slow clock rate. Programming embedded systems is the art of navigating these constraints, writing software that performs the task required while making the most out of limited resources. This can be incredibly difficult. Embedded systems engineers are the unsung heroes of the modern world. If you happen to be one, thank you for your hard work! 1

📄 Page 17

The Edge (and the Internet of Things) The history of computer networks has been a gigantic tug of war. In the first systems—individual computers the size of a room—computation was inherently centralized. There was one machine, and that one machine did all the work. Eventually, however, computers were connected to terminals (as shown in Figure 1-2) that took over some of their responsibilities. Most of the computation was happening in the central mainframe, but some simple tasks—like figuring out how to render letters onto a cathode-ray tube screen—were done by the terminal’s electronics. Figure 1-2. Mainframe computers performed the bulk of the computation, while simple terminals processed input, printed output, and rendered basic graphics Over time, terminals became more and more sophisticated, taking over more and more functions that were previously the job of the central computer. The tug-of-war had begun! Once the personal computer was invented, small computers could do useful work without even being connected to another machine. The rope had been pulled to the opposite extreme—from the center of the network to the edge. The growth of the internet, along with web applications and services, made it possible to do some really cool stuff—from streaming video to social networking. All of this depends on computers being connected to servers, which have gradually taken over more and more of the work. Over the past decade, most of our computing has become centralized again—this time in the “cloud.” When the internet goes down, our modern computers aren’t much use! But the computers we use for work and play are not our only connected devices. In fact, it is estimated that in 2021 there were 12.2 billion assorted items connected to the internet, creating and consuming data. This vast network of objects is called the Internet of Things (IoT), and it includes everything you can think of: industrial sensors, smart refrigerators, internet-connected security cameras, personal automobiles, shipping containers, fitness trackers, and coffee machines. TIP 2

📄 Page 18

The first ever IoT device was created in 1982. Students at Carnegie Mellon University connected a Coke vending machine to the ARPANET—an early precursor to the internet—so they could check whether it was empty without leaving their lab. All of these devices are embedded systems containing microprocessors that run software written by embedded software engineers. Since they’re at the edge of the network, we can also call them edge devices. Performing computation on edge devices is known as edge computing. The edge isn’t a single place; it’s more like a broad region. Devices at the edge of the network can communicate with each other, and they can communicate with remote servers, too. There are even servers that live at the edge of the network. Figure 1-3 shows how this looks. Figure 1-3. Devices at the edge of the network can communicate with the cloud, with edge infrastructure, and with each other; edge applications generally span multiple locations within this map (for example, data might be sent from a sensor-equipped IoT device to a local edge server for processing) There are some major benefits to being at the edge of the network. For one, it’s where all the data comes from! Edge devices are our link between the internet and the physical world. They can use sensors to collect data based on what is going on around them, be that the heart rate of a runner

📄 Page 19

or the temperature of a cold drink. They can make decisions on that data locally and send it to other locations. Edge devices have access to data that nobody else does. ARE MOBILE PHONES AND TABLETS EDGE DEVICES? As portable computers that live at the edge of the network, mobile phones, tablets, and even personal computers are all edge devices. Mobile phones were one of the first platforms to feature edge AI: modern mobile phones use it for many purposes, from voice activation to smart photography. We’ll come back to edge devices later (since they’re the focus of this book). Until then, let’s continue to define some terms. Artificial Intelligence Phew! This is a big one. Artificial intelligence (AI) is a very big idea, and it’s terribly hard to define. Since the dawn of time, humans have dreamed of creating intelligent entities that can help us in our struggle to survive. In the modern world we dream of robot sidekicks who assist with our adventures: hyperintelligent, synthetic minds that will solve all of our problems, and miraculous enterprise products that will optimize our business processes and guarantee us rapid promotion. But to define AI, we have to define intelligence—which turns out to be particularly tough. What does it mean to be intelligent? Does it mean that we can talk, or think? Clearly not—just ask the slime mold (see Figure 1-4), a simple organism with no central nervous system that is capable of solving a maze. 3

📄 Page 20

Figure 1-4. Slime molds are single-celled organisms that have been documented as being able to solve mazes in order to locate food, via a process of biological computation—as shown in “Slime Mould Solves Maze in One Pass Assisted by Gradient of Chemo-Attractants” (Andrew Adamatzky, arXiv, 2011) Since this isn’t a philosophy book, we don’t have the time to fully explore the topic of intelligence. Instead, we want to suggest a quick-and-dirty definition: Intelligence means knowing the right thing to do at the right time. This probably doesn’t stand up to academic debate, but that’s fine with us. It gives us a tool to explore the subject. Here are some tasks that require intelligence, according to our definition: Taking a photo when an animal is in the frame Applying the brakes when a driver is about to crash Informing an operator when a machine sounds broken Answering a question with relevant information Creating an accompaniment to a musical performance Turning on a faucet when someone wants to wash their hands