The Developer’s Playbook for Large Language Model Security (for True Epub) (Steve Wilson) (Z-Library)

📄 File Format: PDF

💾 File Size: 2.4 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

(This page has no text content)

📄 Page 2

Praise for The Developer’s Playbook for Large Language Model Security Steve Wilson’s playbook is essential for AI developers and red teamers. It transforms the enormous risks into manageable challenges, providing the expertise to secure customer-facing and internal LLM-based apps. —Marten Mickos, CEO, HackerOne A must-read for innovators, delivered by the father of LLM Security, Steve Wilson. Essential for leaders, this book delivers crucial insights into securing LLM technologies. —Sherri Douville, CEO, Medigram Steve Wilson’s invaluable industry expertise, paired with his unique dynamic approach to a rapidly shifting landscape, makes this a must- read. Drawing from my experience in AI red teaming, I wholeheartedly advocate for this book’s pinnacle full-stack approach and rigorous, multi-faceted insights. —Ads Dawson, senior security engineer, Cohere

📄 Page 3

The Developer’s Playbook for Large Language Model Security is a critical and comprehensive guide for the security industry as we race to keep pace with the rapid adoption of GenAI and LLMs and ensure secure organizational outcomes. —Chris Hughes, president, Aquia & founder, Resilient Cyber This book is insightful, clear, crisp, and succinct, yet detailed. It explores the spectrum of crucial topics, including LLM architectures, trust boundaries, RAG, prompt injection, and excessive agency. If you are working with LLMs, you need to read and understand this book. —Krishna Sankar, Distinguished AI engineer & NIST AI Safety Institute principal investigator In The Developer’s Playbook for Large Language Model Security, readers embark on an entertaining and exciting journey to the LLM security frontier. Steve Wilson provides a compass to navigate LLM security, where the thrill of innovation meets high stakes and real- world consequences. —Sandy Dunn, CISO, Brand Engagement Networks

📄 Page 4

The Developer’s Playbook for Large Language Model Security Building Secure AI Applications Steve Wilson

📄 Page 5

The Developer’s Playbook for Large Language Model Security by Steve Wilson Copyright © 2024 Stephen Wilson. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisition Editor: Nicole Butterfield Development Editor: Jeff Bleiel Production Editor: Aleeya Rahman Copyeditor: Penelope Perkins Proofreader: Piper Editorial Consulting, LLC Indexer: WordCo Indexing Services, Inc.

📄 Page 6

Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea September 2024: First Edition Revision History for the First Edition 2024-09-03: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781098162207 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. The Developer’s Playbook for Large Language Model Security, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the author and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at

📄 Page 7

your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. 978-1-098-16220-7 [LSI]

📄 Page 8

Preface Everywhere in the world, we’re riding the large language model (LLM) wave, and it’s exhilarating! When ChatGPT burst onto the scene, it didn’t just walk into the record books; it smashed them, becoming the fastest- adopted application in history. Now, it’s as if every software vendor on the planet is racing to embed generative AI and LLM technologies into their stack, pushing us into uncharted territories. The buzz is real, the hype is justified, and the possibilities seem limitless. But hold on because there’s a twist. As we marvel at these technological wonders, their security scaffolding is, to put it mildly, a work in progress. The hard truth? Many developers are stepping into this new era without a map, largely unaware of the security and safety quicksand beneath the surface. It’s almost routine now: every week, we’re hit with another headline screaming about an LLM hiccup. The fallout from these individual incidents has been moderate so far, but make no mistake—we’re flirting with disaster. The risks aren’t just hypothetical; they’re as real as it gets, and the clock is ticking. Without a deep dive into the murky waters of LLM security risks and how to navigate them, we’re not just risking minor glitches; we’re courting major catastrophes. It’s time for developers to gear up, get informed, and get ahead of the curve. Fast!

📄 Page 9

Who Should Read This Book The primary audience for this book is development teams that are building custom applications that embed LLM technologies. Through my recent work in this area, I’ve come to understand that these teams are often large and their members include an incredibly diverse set of backgrounds. These include software developers skilled in “web app” technologies who are taking their first steps with AI. These teams may also consist of AI experts who are bringing their craft out of the back office for the first time and into the limelight, where the security risks are much different. They also include application security pros and data science specialists. Beyond that core audience, I’ve learned that others have found much of this information useful. This includes the extended teams involved in these projects, who want to understand the underpinnings of the technologies to help mitigate the critical risks of adopting these new technologies. These include software development executives, chief information security officers (CISOs), quality engineers, and security operations teams. Why I Wrote This Book I’ve always been fascinated by artificial intelligence. As a preteen, I fondly remember writing video games on my Atari 400 home computer. Circa 1980, this little machine had only 8 kilobytes of RAM. But I still managed

📄 Page 10

to cram a complete clone of the Tron Lightcycles game onto that machine, complete with a simple but effective AI to drive one of the cycles when you were playing in single-player mode. In my professional career, I’ve been involved with several AI-related projects. After college, my best friend Tom Santos and I started an AI software company based on a few thousand lines of handcrafted C++ code that solved seemingly intractable problems with genetic algorithms. I’d later help build a large-scale machine learning system at Citrix with my friends Kedar Poduri and Ebenezer Schubert. But when I saw ChatGPT for the first time, I knew everything had changed. When I first encountered LLMs, I worked at a company that built cybersecurity software. My job was helping large companies find and track vulnerabilities in their software. It quickly became apparent that LLMs offered unique and serious security vulnerabilities. Over the next few months, I retooled my career to go after this disruption. I started a popular open source project around LLM security, which you’ll hear more about later. I even switched jobs to join Exabeam, a company that works at the intersection of AI and cybersecurity. When an editor from O’Reilly approached me about writing a book on this topic, I knew I had to jump at the chance.

📄 Page 11

Navigating This Book This book has 12 chapters that are divided into three logical sections. I’ll sketch out each section and chapter here to give you an idea of the approach and so you’ll know what’s coming as you read.

📄 Page 12

Section 1: Laying the Foundation (Chapters 1–3) The initial chapters of this book establish the groundwork for understanding the security posture of LLM-based applications. They should give you the grounding you can use to confidently unpack the issues facing the development of apps using LLMs: Chapter 1, “Chatbots Breaking Bad”, walks through a real-world case study whereby amateur hackers destroyed an expensive and promising chatbot project from one of the world’s largest software companies. This will set the stage for your forthcoming battles in this arena. Chapter 2, “The OWASP Top 10 for LLM Applications”, introduces a project I founded in 2023 that aims to identify and address the unique security challenges posed by LLMs. The knowledge gained working on that directly led to my writing this book. Chapter 3, “Architectures and Trust Boundaries”, explores the structure of applications using LLMs, emphasizing the importance of controlling the various data flows within the application.

📄 Page 13

Section 2: Risks, Vulnerabilities, and Remediations (Chapters 4–9) In these chapters, we’ll break down the significant risk areas you face when developing LLM applications. These risks include issues with flavors familiar to any application security practitioner, such as injection attacks, sensitive information leakage, and software supply chain risk. You’ll also be introduced to classes of vulnerabilities well known to machine learning aficionados but less familiar in web development, such as training data poisoning. Along the way, you’ll also learn about all-new security and safety concerns plaguing these new generative AI systems, such as hallucinations, overreliance, and excessive agency. I’ll walk you through real-world case studies to help you understand the risks and implications and advise you on how to prevent or mitigate these risks on a case-by-case basis: Chapter 4, “Prompt Injection”, explores how attackers can manipulate LLMs by crafting specific inputs that cause them to perform unintended actions. Chapter 5, “Can Your LLM Know Too Much?”, dives into the risks of sensitive information leakage, showcasing how LLMs can inadvertently expose data they’ve been trained on and how to safeguard against this vulnerability.

📄 Page 14

Chapter 6, “Do Language Models Dream of Electric Sheep?”, examines the unique phenomenon of “hallucinations” in LLMs—instances where models generate false or misleading information. Chapter 7, “Trust No One”, focuses on the principle of zero trust, explaining the importance of not taking any output at face value and ensuring rigorous validation processes are in place to handle LLM outputs. Chapter 8, “Don’t Lose Your Wallet”, tackles the economic risks of deploying LLM technologies, focusing on denial-of-service (DoS), denial-of-wallet (DoW), and model cloning attacks. These threats exploit similar vulnerabilities to impose financial burdens, disrupt services, or steal intellectual property. Chapter 9, “Find the Weakest Link”, highlights the vulnerabilities within the software supply chain and the critical steps needed to secure it from potential breaches that could compromise the entire application. By understanding and addressing these risks, developers can better secure their applications against an evolving landscape of threats.

📄 Page 15

Section 3: Building a Security Process and Preparing for the Future (Chapters 10–12) The chapters in Section 2 will give you the tools you need to understand and address the various individual threats you’ll see in this space. This last section is about bringing it all together: In Chapter 10, “Learning from Future History”, I’ll use some famous science fiction anecdotes to illustrate how multiple weaknesses and design issues can stitch together to spell disaster. By explaining these futuristic case studies, I hope to help you prevent a future like this from ever occurring. In Chapter 11, “Trust the Process”, we’ll get down to the serious business of building LLM-savvy security practices into your software factory—without this, I do not believe you can successfully secure this type of software at scale. Finally, in Chapter 12, “A Practical Framework for Responsible AI Security”, we’ll examine the trajectory of LLM and AI technologies to see where they’re taking us and the likely implications to security and safety requirements. I’ll also introduce you to the Responsible Artificial Intelligence Software Engineering (RAISE) framework that will give you a simple, checklist-based approach to ensuring you’re putting into practice the most important tools and lessons to keep your software safe and secure.

📄 Page 16

Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords. Constant width bold Shows commands or other text that should be typed literally by the user. Constant width italic Shows text that should be replaced with user-supplied values or by values determined by context. TIP This element signifies a tip or suggestion.

📄 Page 17

NOTE This element signifies a general note. WARNING This element indicates a warning or caution. O’Reilly Online Learning NOTE For more than 40 years, O’Reilly Media has provided technology and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit https://oreilly.com.

📄 Page 18

How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-889-8969 (in the United States or Canada) 707-827-7019 (international or local) 707-829-0104 (fax) support@oreilly.com https://www.oreilly.com/about/contact.html We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/the- developers-playbook. For news and information about our books and courses, visit https://oreilly.com. Find us on LinkedIn: https://linkedin.com/company/oreilly-media.

📄 Page 19

Watch us on YouTube: https://youtube.com/oreillymedia. Acknowledgments I’d like to thank all the friends, family, and colleagues who encouraged me or provided feedback on various aspects of the project: Will Chilcutt, Fabrizio Cilli, Ads Dawson, Ron Del Rosario, Sherri Douville, Sandy Dunn, Ken Huang, Gavin Klondike, Marko Lihter, Marten Mickos, Eugene Neelou, Chase Peterson, Karla Roland, Jason Ross, Tom Santos, Robert Simonoff, Yuvraj Singh, Rachit Sood, Seth Summersett, Darcie Tuuri, Ashish Verma, Jeff Williams, Alexa Wilson, Dave Wilson, and Zoe Wilson. I want to thank the team at O’Reilly for supporting and guiding me on this project. I also owe a tremendous debt of gratitude to Nicole Butterfield, who approached me with the idea for this book and guided me through the proposal phase. I also want to express my appreciation for Jeff Bleiel, my editor, whose patience, skills, and expertise significantly impacted the book. Special thanks to our technical reviewers: Pamela Isom, Chenta Lee, Thomas Nield, and Matteo Dora.

📄 Page 20

Chapter 1. Chatbots Breaking Bad Large language models and generative AI jumped to the forefront of public consciousness with the release of ChatGPT on November 30, 2022. Within five days, it went viral on social media and attracted its first million users. By January, ChatGPT surpassed one hundred million users, making it the fastest-growing internet service in history. However, a steady stream of security concerns emerged in the following months. These included privacy and security issues that caused companies like Samsung and countries like Italy to ban its usage. In this book, we’ll explore what underlies these concerns and how you can mitigate these issues. However, to best understand what’s going on here and why these problems are so challenging to solve, in this chapter, we will briefly rewind further in time. In doing so, we’ll see these types of issues aren’t new and understand why they will be so hard to fix permanently. Let’s Talk About Tay In March 2016, Microsoft announced a new project called Tay. Microsoft intended Tay to be “a chatbot created for 18- to 24-year-olds in the U.S. for entertainment purposes.” It was a cute name for a fluffy, early experiment in AI. Tay was designed to mimic a 19-year-old American girl’s language patterns and learn from interacting with human users of Twitter, Snapchat,