AI Essentials for Tech Executives (for Raymond Rhine) (Hamel Husain, Greg Ceccarelli) (Z-Library)

AI Essentials for Tech Executives A Practical Guide to Unlocking the Competitive Potential of AI Hamel Husain and Greg Ceccarelli

AI Essentials for Tech Executives by Hamel Husain and Greg Ceccarelli Copyright © 2025 O’Reilly Media, Inc. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Editor: Mike Loukides Production Editor: Kristen Brown Copyeditor: Miah Sandvik Interior Designer: David Futato Cover Designer: Ellie Volckhausen Illustrator: Kate Dullea

February 2025: First Edition Revision History for the First Edition 2025-02-27: First Release The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. AI Essentials for Tech Executives, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the authors and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. 979-8-341-62245-6 [LSI]

Preface Why We Wrote This Book There’s a significant disconnect between the people building AI products and the executives making decisions about them, leading to misguided strategies and wasted resources. For nonpractitioners, navigating the AI space is overwhelming. There’s a sea of advice on how to create effective AI products, but much of it comes from sources that aren’t hands-on with the technology. Without practical experience, it’s challenging to separate fact from fiction and make informed decisions. On the flip side, AI practitioners in the trenches often struggle to communicate their knowledge in a way that resonates with executives. They tend to think and speak in technical terms, which can be off-putting or confusing for those without a deep technical background. This is where we come in. I’m Hamel Husain, an AI practitioner deeply involved in the technical aspects of artificial intelligence. My partner is Greg Ceccarelli, who brings extensive executive leadership experience as a Director and Chief Product Officer at large organizations like Pluralsight and GitHub.

With the combination of my hands-on AI experience and Greg’s executive leadership background, you can be sure that everything you read from us is both technically accurate and accessible to executives like you. In this report, you’ll receive practical and actionable insights on unlocking the true potential of AI to create competitive advantage. What to Expect Each chapter is concise, yet packed with insights you can put into action right away. We’re committed to delivering high-value content that respects your time. With our help, you’ll be well on your way to: Distinguishing between industry hype and reality Developing an AI roadmap aligned with your business goals Navigating the complexities of AI implementation with confidence Fostering a culture of AI innovation within your organization As a bonus, we’ll also be providing you with cheat sheets and templates you can use to put these lessons into practice right away.

Chapter 1. 99% of Executives Are Misled by AI Advice As an executive, you’re bombarded with articles and advice on building AI products. The problem is, a lot of this “advice” comes from other executives who rarely interact with the practitioners actually working with AI. This disconnect leads to misunderstandings, misconceptions, and wasted resources. A Case Study in Misleading AI Advice An example of this disconnect in action comes from an interview with Jake Heller, CEO of Casetext. During the interview, Jake made a statement about AI testing that was widely shared: One of the things we learned is that after it passes 100 tests, the odds that it will pass a random distribution of 100k user inputs with 100% accuracy is very high. (emphasis added)

This claim was then amplified by influential figures like Jared Friedman and Garry Tan of Y Combinator, reaching countless founders and executives:

The morning after this advice was shared, I received numerous emails from founders asking if they should aim for 100% test-pass rates. If you’re not hands-on with AI, this advice might sound reasonable. But any practitioner would know it’s deeply flawed. “Perfect” Is Flawed In AI, a perfect score is a red flag. This happens when a model has inadvertently been trained on data or prompts that are too similar to tests. Like a student who was given the answers before an exam, the model will look good on paper but be unlikely to perform well in the real world. If you are sure your data is clean but you’re still getting 100% accuracy, chances are your test is too weak or not measuring what matters. Tests that always pass don’t help you improve; they’re just giving you a false sense of security. Most importantly, when all your models have perfect scores, you lose the ability to differentiate between them. You won’t be able to identify why one model is better than another, or strategize about how to make further improvements. The goal of evaluations isn’t to pat yourself on the back for a perfect score.

It’s to uncover areas for improvement and ensure your AI is truly solving the problems it’s meant to address. By focusing on real-world performance and continuous improvement, you’ll be much better positioned to create AI that delivers genuine value. Evals are a big topic, and we’ll dive into them more in a future chapter. Moving Forward When you’re not hands-on with AI, it’s hard to separate hype from reality. Here are some key takeaways to keep in mind: Be skeptical of advice or metrics that sound too good to be true. Focus on real-world performance and continuous improvement. Seek advice from experienced AI practitioners who can communicate effectively with executives. (You’ve come to the right place!) We’ll dive deeper into how to test AI, along with a data review toolkit in a future chapter. First, we’ll look at the biggest mistake executives make when investing in AI.

Chapter 2. The #1 Mistake Companies Make with AI One of the first questions I ask tech leaders is how they plan to improve AI reliability, performance, or user satisfaction. If the answer is “We just bought XYZ tool for that, so we’re good,” I know they’re headed for trouble. Focusing on tools over processes is a red flag and the biggest mistake I see executives make when it comes to AI. Improvement Requires Process Assuming that buying a tool will solve your AI problems is like joining a gym but not actually going. You’re not going to see improvement by just throwing money at the problem. Tools are only the first step; the real work comes after. For example, the metrics that come built-in to many tools rarely correlate with what you actually care about. Instead, you need to design metrics that are specific to your business, along with tests to evaluate your AI’s performance. The data you get from these tests should also be reviewed regularly to make sure you’re on track. No matter what area of AI you’re working on—model evaluation, retrieval-augmented generation (RAG), or prompting strategies —the process is what matters most. Of course, there’s more to making

improvements than just relying on tools and metrics. You also need to develop and follow processes. Rechat’s Success Story Rechat is a great example of how focusing on processes can lead to real improvements. The company decided to build an AI agent for real estate agents to help with a large variety of tasks related to different aspects of the job. However, they were struggling with consistency. When the agent worked, it was great, but when it didn’t, it was a disaster. The team would make a change to address a failure mode in one place but end up causing issues in other areas. They were stuck in a cycle of whack-a-mole. They didn’t have visibility into their AI’s performance beyond “vibe checks,” and their prompts were becoming increasingly unwieldy. When I came in to help, the first thing I did was apply a systematic approach that is illustrated in Figure 2-1.

Figure 2-1. The virtuous cycle This is a virtuous cycle for systematically improving large language models (LLMs). The key insight is that you need both quantitative and qualitative 1

feedback loops that are fast. You start with LLM invocations (both synthetic and human-generated), then simultaneously: Run unit tests to catch regressions and verify expected behaviors. Collect detailed logging traces to understand model behavior. These feed into evaluation and curation (which needs to be increasingly automated over time). The eval process combines: Human review Model-based evaluation A/B testing The results then inform two parallel streams: Fine-tuning with carefully curated data Prompt engineering improvements These both feed into model improvements, which starts the cycle again. The dashed line around the edge emphasizes this as a continuous, iterative process—you keep cycling through faster and faster to drive continuous improvement. By focusing on the processes outlined in this diagram, Rechat was able to reduce its error rate by over 50% without investing in new tools!

Check out this ~15-minute video on how we implemented this process-first approach at Rechat. Avoid the Red Flags Instead of asking which tools you should invest in, you should be asking your team: What are our failure rates for different features or use cases? What categories of errors are we seeing? Does the AI have the proper context to help users? How is this being measured? What is the impact of recent changes to the AI? The answers to each of these questions should involve appropriate metrics and a systematic process for measuring, reviewing, and improving them. If your team struggles to answer these questions with data and metrics, you are in danger of going off the rails! Avoiding Jargon Is Critical We’ve talked about why focusing on processes is better than just buying tools. But there’s one more thing that’s just as important: how we talk about AI. Using the wrong words can hide real problems and slow down progress.

To focus on processes, we need to use clear language and ask good questions. That’s why we provide an AI communication cheat sheet for executives in Chapter 3. That chapter helps you: Understand what AI can and can’t do Ask questions that lead to real improvements Ensure that everyone on your team can participate Using this cheat sheet will help you talk about processes, not just tools. It’s not about knowing every tech word. It’s about asking the right questions to understand how well your AI is working and how to make it better. Diagram adapted from my blog post, “Your AI Product Needs Evals”.1

Chapter 3. AI Communication Cheat Sheet for Executives Why Plain Language Matters in AI As an executive, using simple language helps your team understand AI concepts better. This cheat sheet will show you how to avoid jargon and speak plainly about AI. This way, everyone on your team can work together more effectively. At the end of this chapter, you’ll find a helpful glossary. It explains common AI terms in plain language. Helps Your Team Understand and Work Together Using simple words breaks down barriers. It makes sure everyone—no matter their technical skills—can join the conversation about AI projects. When people understand, they feel more involved and responsible. They are more likely to share ideas and spot problems when they know what’s going on. Improves Problem-Solving and Decision Making

Focusing on actions instead of fancy tools helps your team tackle real challenges. When we remove confusing words, it’s easier to agree on goals and make good plans. Clear talk leads to better problem-solving because everyone can pitch in without feeling left out. Reframing AI Jargon into Plain Language Here’s how to translate common technical terms into everyday language that anyone can understand. Examples of Common Terms, Translated Changing technical terms into everyday words makes AI easy to understand. The following table shows how to say things more simply:

Instead of saying… Say… “We’re implementing a RAG approach.” “We’re making sure the AI always has the right information to answer questions well.” “We’ll use few-shot prompting and chain-of- thought reasoning.” “We’ll give examples and encourage the AI to think before it answers.” “Our model suffers from hallucination issues.” “Sometimes, the AI makes things up, so we need to check its answers.” “Let’s adjust the hyperparameters to optimize performance.” “We can tweak the settings to make the AI work better.” “We need to prevent prompt injection attacks.” “We should make sure users can’t trick the AI into ignoring our rules.”