Page
1
Building AI Systems with Python Practical Machine Learning and Agentic Workflows with Python and PyTorch — Martin Hander
Page
2
Building AI Systems with Python Practical Machine Learning and Agentic Workflows with Python and PyTorch Martin Hander
Page
3
Building AI Systems with Python: Practical Machine Learning and Agentic Workflows with Python and PyTorch ISBN-13 (pbk): 979-8-8688-2757-0 ISBN-13 (electronic): 979-8-8688-2758-7 https://doi.org/10.1007/979-8-8688-2758-7 Copyright © 2026 by Martin Hander This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights. While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein. Managing Director, Apress Media LLC: Welmoed Spahr Acquisitions Editor: Celestin Suresh John Editorial Assistant: Gryffin Winkler Cover designed by eStudioCalamar Cover image designed by Freepik (www.freepik.com) Distributed to the book trade worldwide by Springer Science+Business Media New York, 1 New York Plaza, New York, NY 10004. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail orders-ny@springer-sbm.com, or visit www.springeronline.com. Apress Media, LLC is a Delaware LLC and the sole member (owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation. For information on translations, please e-mail booktranslations@springernature.com; for reprint, paperback, or audio rights, please e-mail bookpermissions@springernature.com. Apress titles may be purchased in bulk for academic, corporate, or promotional use. eBook versions and licenses are also available for most titles. For more information, reference our Print and eBook Bulk Sales web page at http://www.apress.com/bulk-sales. Any source code or other supplementary material referenced by the author in this book is available to readers on GitHub. For more detailed information, please visit https://www.apress.com/gp/services/ source-code. If disposing of this product, please recycle the paper Martin Hander Gablingen, Germany
Page
4
For Adrian, You reminded me, over and over, that the point of building things isn’t the cleverness inside them, but the people they serve. This book is about turning ideas into systems that work in the real world: honest about what they know, careful with what they don’t, and useful even on messy days. I hope it helps you build that way too, with curiosity, patience, and a good sense of responsibility. May you always stay brave enough to try, humble enough to learn, and kind enough to use your skills well. With all my love, Dad
Page
5
v Table of Contents About the Author xxiii About the Technical Reviewer xxv Acknowledgments xxvii Introduction xxix Part I: Foundations That Ship (with Python) 1 Chapter 1: ML for Builders 3 1.1 Python Version and Packages ................................................................................................. 3 1.2 NumPy: The Muscle Underneath ............................................................................................. 3 1.3 Pandas: Tables You Can Reason About .................................................................................... 4 1.4 A Scikit-Learn Baseline You Can Trust .................................................................................... 5 1.5 Bias and Variance, Without the Folklore .................................................................................. 7 1.6 Data-Centric Work Is Where Most Wins Live ........................................................................... 7 1.7 Hygiene That Makes Results Stick .......................................................................................... 8 1.8 Where We’re Going Next .......................................................................................................... 9 Chapter 2: Data Engineering for ML 11 2.1 Python Version, Packages, and the Running Example .......................................................... 11 2.2 From Tables to Tensors Without Chaos ................................................................................. 12 2.3 Dataset and DataLoader: The Contract with Your Model ....................................................... 14 2.4 Streaming and Sharding with torchdata ............................................................................... 15 2.5 Images, Audio, and Text Without Reinventing Wheels ........................................................... 16 2.6 Weak Supervision When Gold Labels Are Scarce .................................................................. 19 2.7 Synthetic Data That Helps Instead of Hurting ....................................................................... 21
Page
6
vi 2.8 Putting It Together: Arrow → Dataset → Loader ................................................................... 23 2.9 A Few Guardrails That Pay for Themselves ........................................................................... 24 2.10 Where We’re Going Next ...................................................................................................... 25 Chapter 3: Evaluation, Testing, and Measurement 27 3.1 Metrics That Live with Your Model (Torchmetrics) ................................................................ 27 3.2 Tests That Keep You from Backsliding (Pytest) ..................................................................... 29 3.3 Properties That Should Always Hold (Hypothesis) ................................................................. 31 3.4 Golden Sets: Small, Human-Curated Truth ............................................................................ 32 3.5 Offline vs. Online: Designing an A/B Test You Can Trust ........................................................ 33 3.6 Putting It Together ................................................................................................................. 34 Chapter 4: ML Software Engineering 35 4.1 What This Chapter Is Doing ................................................................................................... 35 4.2 A Project Structure That Survives Growth ............................................................................. 35 4.3 pyproject.toml: One File to Rule Them All ............................................................................. 37 4.4 Fast Feedback with ruff and Safer Code with mypy ............................................................. 40 4.5 A Thin CLI over a Clean SDK (Typer) ...................................................................................... 41 4.6 Notebooks and Repos: A Healthy Relationship...................................................................... 42 4.7 Reproducibility Without Heroics: uv, Conda, and Lockfiles .................................................... 43 4.8 A Word on Configs and Artifacts ............................................................................................ 46 4.9 A Few Terms Used in This Chapter ........................................................................................ 46 4.10 Bringing It Together ............................................................................................................. 47 Chapter 5: Optimization 101 in PyTorch 49 5.1 A Tiny Runnable Setup for the Examples .............................................................................. 49 5.2 Autograd Without Magic ........................................................................................................ 50 5.3 Custom Losses That Stay Numerically Sane ......................................................................... 51 5.4 Schedulers That Earn Their Keep .......................................................................................... 53 5.5 Regularization You Can Feel .................................................................................................. 55 5.6 Mixed Precision with torch.autocast ..................................................................................... 55 Table of ConTenTs
Page
7
vii 5.7 Early Stopping Without Self-Deception ................................................................................. 57 5.8 A Tidy Loop That Puts It All Together ..................................................................................... 59 5.9 When Each Optimization Tool Is Worth Reaching For ............................................................ 62 Part II: Modern Architectures (PyTorch-First) 65 Chapter 6: Transformers in PyTorch 67 6.1 What to Watch for in This Chapter ......................................................................................... 67 6.2 Attention by Hand (and the Fast Path)................................................................................... 67 6.3 nn.MultiheadAttention and a Minimal Transformer Block ..................................................... 69 6.4 Hugging Face Transformers in Practice ................................................................................ 70 6.5 KV Cache: What It Is and How to Use It ................................................................................. 72 6.6 Long-Context: Making Attention Reach Farther Without Melting VRAM ................................ 73 6.7 Mixture-of-Experts (MoE) at a Glance ................................................................................... 75 6.8 The Ladder of Abstraction in This Chapter ............................................................................ 76 6.9 Putting It Together ................................................................................................................. 77 Chapter 7: Diffusion and Generative Media 79 7.1 A Note on Runtime and Hardware ......................................................................................... 79 7.2 Getting Productive with diffusers .......................................................................................... 79 7.3 Conditioning with ControlNet ................................................................................................ 81 7.4 Making Diffusion Fast: Consistency Distillation in Practice .................................................. 82 7.5 Safety Filters and Content Moderation That Actually Run ..................................................... 83 7.6 Training, Fine-Tuning, and Adapters (the Short Version)........................................................ 85 7.7 Performance and Stability Notes You’ll Actually Use ............................................................ 85 7.8 What to Ship .......................................................................................................................... 86 Chapter 8: Multimodal Learning 87 8.1 Vision–Language Alignment (CLIP-Style) .............................................................................. 87 8.2 torchvision Transforms and Ops You’ll Actually Use .............................................................. 89 8.3 torchaudio: I/O, Features, and Batching That Won’t Bite ....................................................... 91 8.4 ASR: From Waveforms to Words ............................................................................................ 92 8.5 TTS: From Text to Waveforms ................................................................................................ 93 Table of ConTenTs
Page
8
viii 8.6 Wiring It Together: A Speech–Vision Loop ............................................................................. 94 8.7 Training Notes That Matter .................................................................................................... 94 8.8 What to Ship .......................................................................................................................... 95 Chapter 9: Classical Models That Still Deliver 97 9.1 Recommenders (Two-Tower Retrieval) ................................................................................. 97 9.2 Time Series with PyTorch Forecasting .................................................................................. 99 9.3 Tabular: LightGBM with Torch Embeddings ......................................................................... 101 9.4 Anomaly Detection That Holds Up ....................................................................................... 104 9.5 What to Ship ........................................................................................................................ 106 Chapter 10: Reinforcement Learning in Practice 107 10.1 RLHF and RLAIF: Preferences into Gradients .................................................................... 107 10.2 Offline RL: Learning When You Can’t Act ........................................................................... 109 10.3 Environments at Speed: Gymnasium and envpool ............................................................ 110 10.4 A Compact PPO Spine (PyTorch) ........................................................................................ 112 10.5 A Quick Tour of PyTorch RL Libraries ................................................................................. 114 10.6 Evaluation Without Self-Delusion ...................................................................................... 114 10.7 What to Ship ...................................................................................................................... 115 Part III: LLMs and GenAI in Practice 117 Chapter 11: LLM Fundamentals 119 11.1 Tokenization: Where Cost and Meaning Meet ................................................................... 119 11.2 Pretraining Objectives: What the Loss Is Asking For ......................................................... 120 11.3 Scaling Laws: Spending Compute with Taste ................................................................... 121 11.4 Context Windows and Retrieval: Fitting More into the Model’s Head ................................ 122 11.5 Closing Thoughts ............................................................................................................... 123 Chapter 12: Adapting Models Efficiently 125 12.1 LoRA and QLoRA in practice (with peft) ............................................................................ 125 12.2 Quantization with bitsandbytes: What Each Bit Buys You ................................................. 128 12.3 Adapters Beyond LoRA ...................................................................................................... 129 12.4 Distillation: Teaching a Smaller (or Faster) Student .......................................................... 129 Table of ConTenTs
Page
9
ix 12.5 8/4/3/2-Bit: A Pragmatic Buyer’s Guide ............................................................................ 130 12.6 Putting It Together: A Tidy Adaptation Workflow ................................................................ 131 12.7 Common Gotchas (and Easy Fixes) ................................................................................... 131 12.8 What to Ship ...................................................................................................................... 132 Chapter 13: Prompt Engineering That Lasts 133 13.1 Where These Prompts Actually Go .................................................................................... 133 13.2 A Minimal Pattern for Assembling Prompts ...................................................................... 134 13.3 System Prompts: Your Contract with the Model ................................................................ 135 13.4 Few-Shot: Show, Don’t Tell ............................................................................................... 136 13.5 Structured Outputs That Parse Every Time ....................................................................... 136 Define the Contract in Python .............................................................................................. 137 Prompt with the Schema (Shortened) and Strict Instructions ............................................. 137 Parse Robustly ..................................................................................................................... 138 13.6 Tool-Augmented Prompting (Function Calling) .................................................................. 138 Define Tools and a Contract ................................................................................................. 139 Planner Prompt (Model Decides) ......................................................................................... 139 Dispatcher Loop ................................................................................................................... 140 13.7 Patterns That Survive Model Upgrades ............................................................................. 140 13.8 Quick Tests You Should Actually Write .............................................................................. 141 13.9 End-to-End Example: Putting the Pieces into One LLM Call .............................................. 141 13.10 What to Ship .................................................................................................................... 142 Chapter 14: Retrieval-Augmented Generation 143 14.1 Data → Chunks → Embeddings ....................................................................................... 143 14.2 Indexing with FAISS (and Friends) .................................................................................... 144 14.3 Retrieval → Reranking (Dense + Cross-Encoder) ............................................................. 146 14.4 Prompt Composition with Citations (and a Budget) .......................................................... 146 14.5 Freshness: Recency Filters, Time Decay, and Updates ..................................................... 147 14.6 End-to-End: A Tiny RAG Function ...................................................................................... 148 14.7 Storing and Serving: FAISS vs. Vector DBs (Quick Notes) ................................................. 149 14.8 Evaluation: Retrieval, Grounding, and Answers ................................................................. 149 Table of ConTenTs
Page
10
x 14.9 Common Failure Modes (and Simple Fixes) ...................................................................... 150 14.10 What to Ship .................................................................................................................... 151 Chapter 15: Tool Using LLMs 153 15.1 Function Schemas: Typed Tools the Model Can Understand ............................................. 153 15.2 Tool Routing: Plan → Validate → Execute → Synthesize ................................................. 155 15.3 Calling External APIs: FastAPI Clients Done Right ............................................................. 157 15.4 Sandboxed Code Execution: Safe(-ish) Patterns ............................................................... 159 15.5 Patterns for Reliability and Safety .................................................................................... 161 15.6 Minimal End-to-End Example ........................................................................................... 162 15.7 What to Ship ...................................................................................................................... 162 Chapter 16: LLM Evaluation and Observability 163 16.1 What to Measure (and Why) .............................................................................................. 163 16.2 Rubric Evals (LLM-As-Judge or Humans) ......................................................................... 163 16.3 Pairwise Evals: A/B via Wins, Not Vibes ............................................................................ 165 16.4 Prompt/Response Traces (Spans You Can Replay) ......................................................... 166 16.5 Guardrail Tests (As Code) .................................................................................................. 168 16.6 Dashboards (W&B/MLflow/Custom) .................................................................................. 169 Weights and Biases (W&B) .................................................................................................. 169 MLflow ................................................................................................................................. 170 Custom (JSONL + Notebook) ............................................................................................... 171 16.7 Putting It Together: A Tiny Eval Pipeline ............................................................................ 171 16.8 Common Pitfalls (and Fixes) ............................................................................................. 172 16.9 What to Ship ...................................................................................................................... 173 Chapter 17: Responsible AI Foundations 175 17.1 Threat Model (in One Page) ............................................................................................... 175 17.2 Prompt-Injection and Tool-Use Defenses .......................................................................... 176 Planner → Dispatcher Contract (Constrained JSON) ........................................................... 176 Context Isolation for RAG ..................................................................................................... 176 Planning Guard (Injection Heuristic) .................................................................................... 176 Table of ConTenTs
Page
11
xi Sandboxes and Allow-Lists ................................................................................................. 177 Red-Team in CI .................................................................................................................... 177 17.3 PII Detection and Redaction .............................................................................................. 177 Fast Regex + NER Baseline (Online) .................................................................................... 178 Log Minimization and Reversible Detokenization (with Controls) ....................................... 179 Dataset Scrubbing ............................................................................................................... 179 17.4 GDPR (and Friends) in Practice + Differential Privacy ...................................................... 179 Differential Privacy (DP-SGD with Opacus) .......................................................................... 180 17.5 Model Specs and Policy Guards ........................................................................................ 181 Model Spec (YAML) .............................................................................................................. 181 Policy Guard Engine ............................................................................................................. 182 17.6 Governance: Making It Real .............................................................................................. 183 17.7 Checklists.......................................................................................................................... 183 17.8 What to Ship ...................................................................................................................... 184 Part IV: Agentic AI and AI Agents (Python Ecosystem) 185 Chapter 18: Agent Architectures 187 18.1 Building Blocks (Schemas, Tools, Controller) .................................................................... 187 18.2 ReAct: Reason + Act in a Single Loop ............................................................................... 189 Prompt Shape (Conceptual) ................................................................................................. 190 Loop ..................................................................................................................................... 190 18.3 Reflexion: Learn from Your Own Mistakes ........................................................................ 191 18.4 Tree-of-Thought (ToT): Branch, Score, and Search ............................................................ 192 18.5 Graph-of-Thought (GoT): DAGs of Subproblems ................................................................ 193 18.6 Controllers: Budgets, Safety, and Selection ...................................................................... 195 18.7 Putting It Together: A Pluggable Agent .............................................................................. 196 18.8 When to Use Which Pattern ............................................................................................... 197 18.9 Pitfalls and Guardrails ....................................................................................................... 198 18.10 What to Ship .................................................................................................................... 198 Table of ConTenTs
Page
12
xii Chapter 19: Memory and State 199 19.1 Three Kinds of Memory ..................................................................................................... 199 19.2 A Tiny Memory Schema (Pydantic) .................................................................................... 199 19.3 Vector Store for Semantic Memory (FAISS) ....................................................................... 200 19.4 Retrieval That Mixes Recency and Relevance ................................................................. 201 19.5 Summarization Loops (“Compress or Die”) ...................................................................... 202 19.6 Procedural Memory (State That Drives Tools) ................................................................... 203 19.7 Putting It Together: Memory-Aware Context Builder ......................................................... 203 19.8 TTL and Compaction (the Hygiene) ................................................................................... 204 TTL (Time to Live) ................................................................................................................ 204 Compaction ......................................................................................................................... 205 19.9 Privacy Hooks (Quick Tie-In) ............................................................................................. 205 19.10 Tests That Catch Regressions ......................................................................................... 205 19.11 Minimal End-to-End “Memory Service” .......................................................................... 206 19.12 Pitfalls and Fixes ............................................................................................................. 206 19.13 What to Ship .................................................................................................................... 207 Chapter 20: Tools and Environments for Agents 209 20.1 Design Goals (and a Tiny Threat Model) ............................................................................ 209 20.2 Action Catalog: One Source of Truth for Tools ................................................................... 209 YAML Spec ........................................................................................................................... 210 Loader, Registry, and Enforcement ...................................................................................... 211 20.3 Browser Tool (Safe Fetch, Readable Text, Allow-List)........................................................ 213 20.4 File and OS Tools (Workspace-Scoped, No Shell Tricks) ................................................... 214 Safe Workspace Paths ......................................................................................................... 214 Allowed OS Commands (No Shell, Hard Allow-List) ............................................................. 215 20.5 Code Execution Sandboxes (Out-of-Process, Resource-Capped) ..................................... 216 20.6 Safety Rails (Auth, Approvals, Policy Guards, Redaction) .................................................. 217 20.7 Controller Hooks: Budgets, Traces, and Cancellations ...................................................... 218 20.8 Tests You Should Actually Write ........................................................................................ 219 20.9 Putting It Together: A Minimal Agent Loop ........................................................................ 219 Table of ConTenTs
Page
13
xiii 20.10 Pitfalls and Fixes ............................................................................................................. 220 20.11 What to Ship .................................................................................................................... 221 Chapter 21: Multi-agent Systems 223 21.1 Architectural Patterns You’ll Actually Use ......................................................................... 223 21.2 A Tiny Substrate: Messages, Roles, and Controller ........................................................... 223 21.3 Role Assignment (Skills ↔ Tasks) ..................................................................................... 225 21.4 Collaboration: Committee and Debate (Propose → Critique → Revise → Judge) ............ 226 21.5 Auctions and contract-net: Allocating Work by Bids ......................................................... 228 21.6 Blackboard with Compaction (Avoid Memory Bloat) ......................................................... 229 21.7 Coordination Failures and How to Mitigate Them ............................................................. 229 21.8 Observability and Evaluation (per-Agent and per-Team) ................................................... 230 21.9 Minimal Runnable Examples ............................................................................................. 231 Two-Agent Debate with a Stub Judge ................................................................................. 231 Auction for Heterogeneous Workers .................................................................................... 231 21.10 Safety Rails in Multi-agent Settings ............................................................................... 232 21.11 What to Ship .................................................................................................................... 232 Chapter 22: Reliability and Determinism 233 22.1 Determinism Knobs You’ll Actually Use ............................................................................. 233 22.2 Idempotence (Do It Twice, Get the Same World) ............................................................... 234 Patterns ............................................................................................................................... 234 Minimal Idempotency Store ................................................................................................. 234 22.3 Retries That Don’t Wreck Things ....................................................................................... 235 When to Retry ...................................................................................................................... 235 Exponential Backoff with Decorrelated Jitter ...................................................................... 236 22.4 Constrained Decoding (Always-Parse Outputs) ................................................................ 236 Tier A: “Parse or Fail” (Cheap, Baseline) .............................................................................. 236 Tier B: Grammar/JSON-Schema Guided Decoding .............................................................. 237 Tier C: Function/Tool Calling with Strict Schemas ............................................................... 238 22.5 Transactional Tool Use (Do Many Things or None)............................................................. 238 Unit-of-Work (Single Database) ........................................................................................... 238 Table of ConTenTs
Page
14
xiv Saga (Multi-step, Cross-Service) ......................................................................................... 240 Outbox Pattern (Exactly-Once Side Effects) ........................................................................ 241 22.6 Putting It Together: A Reliable Tool Call ............................................................................. 241 22.7 Tests You Should Actually Write ........................................................................................ 242 22.8 Checklist (Pre-prod) .......................................................................................................... 242 22.9 What to Ship ...................................................................................................................... 243 Chapter 23: Agent Evals and Benchmarks 245 23.1 What an “Agent Eval” Actually Measures .......................................................................... 245 23.2 Task Suites: Format and Slices ......................................................................................... 246 JSONL Schema (Minimal) .................................................................................................... 246 23.3 A Tiny, Reusable Eval Harness ........................................................................................... 246 Scoring Functions (Exact, Fuzzy, Rubric) ............................................................................. 247 Batch Evaluator with Slices and Budgets ............................................................................ 247 23.4 Autonomy Levels (How Hands-Off Are We?) ...................................................................... 248 23.5 Latency and Cost Trade-Offs (Pareto, Budgets, Knobs) ..................................................... 249 23.6 Robustness Probes ........................................................................................................... 250 23.7 Incident Post-Mortems (Make Them Boring, Repeatable) ................................................. 250 Template (Markdown) .......................................................................................................... 251 Helper: Pull a Trace and Render a Snippet .......................................................................... 251 23.8 Putting It Together: A Runnable Sketch ............................................................................. 252 23.9 Common Pitfalls (and Fixes) ............................................................................................. 253 23.10 Checklists........................................................................................................................ 253 23.11 What to Ship .................................................................................................................... 254 Part V: MLOps and LLMOps (PyTorch in Production) 255 Chapter 24: From Experiment to Production 257 24.1 Experiment Tracking You’ll Actually Use............................................................................ 257 W&B: Runs + Artifacts (Datasets, Models) ........................................................................... 257 MLflow: Runs + Model Registry .......................................................................................... 258 24.2 Model Registries: Versioned, Staged, and Reproducible .................................................. 259 Promotion Gate (CI) .............................................................................................................. 260 Table of ConTenTs
Page
15
xv 24.3 Lineage: Know Your Parents (and Children) ..................................................................... 260 Minimal Lineage Record ...................................................................................................... 260 24.4 Feature Stores: Offline ↔ Online Consistency ................................................................. 261 Toy Feature Store (Interface) ............................................................................................... 261 24.5 Embedding Stores: Text/Image → Vectors You Can Update ............................................. 263 Minimal Upsert/Delete with FAISS ....................................................................................... 263 24.6 Packaging Inference: Signatures, Validators, and Rollouts ............................................... 264 24.7 Putting It Together: An End-to-End Flow ........................................................................... 265 24.8 Tests and Checks That Prevent Pain ................................................................................. 265 24.9 What to Ship ...................................................................................................................... 265 Chapter 25: Serving and Inference (Rewrite with Code) 267 25.1 A Quick Hygiene Pass Before Anything Else ..................................................................... 267 25.2 Packaging PyTorch Models You’ll Maintain ....................................................................... 268 25.3 Streaming with FastAPI Without Getting Fancy ................................................................. 271 25.4 LLM Back Ends That Do the Heavy Lifting ......................................................................... 271 25.5 When “Serverless” Is the Right Hammer .......................................................................... 272 25.6 Batching and First-Token Time Live Together ................................................................... 273 25.7 What to Ship (Minimum Viable Serving) ............................................................................ 273 Chapter 26: Performance Engineering 275 26.1 torch.compile (Inductor): Free Speed (Mostly) ................................................................. 275 26.2 Automatic Mixed Precision (AMP) ..................................................................................... 276 26.3 CUDA Graphs: Flatten Launch Overhead ........................................................................... 277 26.4 KV-Cache (Incremental Attention for LLMs) ...................................................................... 278 High-Level (HF) .................................................................................................................... 278 Lower-Level (Manual Loop) ................................................................................................. 278 26.5 Quantization ..................................................................................................................... 279 torch.ao.quantization (PyTorch) .......................................................................................... 279 bitsandbytes (LLMs on GPU) ............................................................................................... 280 Table of ConTenTs
Page
16
xvi 26.6 CPU/GPU/NPU Quick Wins ................................................................................................ 280 CPU ...................................................................................................................................... 280 GPU ...................................................................................................................................... 281 NPU/Apple Silicon/Vendor Accelerators............................................................................... 281 26.7 Dataloader and IO Plumbing ............................................................................................. 281 26.8 Profiling: Measure Before You Tweak ................................................................................ 282 PyTorch Profiler ................................................................................................................... 282 Microbenchmarks ................................................................................................................ 282 26.9 Transformer-Specific Knobs .............................................................................................. 283 26.10 A Practical Acceleration Recipe ...................................................................................... 283 26.11 Gotchas and Fixes ........................................................................................................... 283 26.12 Ready-to-Ship Checklist ................................................................................................. 284 Chapter 27: Monitoring, Drift, and Feedback Loops 285 27.1 What to Watch (and Where It Shows Up) ........................................................................... 285 27.2 Data Drift: KS/χ2 and PSI in 50 Lines ................................................................................ 286 Numeric: KS + Population Stability Index (PSI) ................................................................... 286 Categorical: χ2 .................................................................................................................... 286 27.3 Embedding Drift: Cosine/Fréchet in Practice .................................................................... 287 27.4 Prompt Drift: Versioning + Distribution Checks................................................................. 288 27.5 Safety Drift: Rolling Rates and EWMA Alerts ..................................................................... 288 27.6 A Tiny Monitor Service (Rolling Windows) ......................................................................... 289 27.7 Human Feedback Pipelines (HF Loops You’ll Maintain) .................................................... 290 Minimal Feedback API (FastAPI Sketch) .............................................................................. 291 27.8 OpenTelemetry Hooks: Traces, Metrics, and Logs ............................................................ 291 Basic Setup (OTLP Exporter) ................................................................................................ 292 Wrap LLM/Tool Calls with Spans and Attributes .................................................................. 292 FastAPI Auto-instrumentation (Optional) ............................................................................. 293 27.9 Stitching It Together (Periodic Job) ................................................................................... 293 27.10 Pitfalls and Fixes ............................................................................................................. 294 27.11 What to Ship .................................................................................................................... 294 Table of ConTenTs
Page
17
xvii Chapter 28: Cost Management 295 28.1 The Cost Anatomy (What You Actually Pay For) ................................................................. 295 28.2 Token Budgets (Shape the Request, Cap the Spend) ........................................................ 295 28.3 Hybrid Routing (Small-First, Escalate Only When Needed) ............................................... 297 28.4 Caching (Exact, Then Semantic) ........................................................................................ 298 Exact-Match, Parameter-Aware Cache (Must-Have) ........................................................... 298 Semantic Cache (When Exact Keys Miss)............................................................................ 299 28.5 Distillation (Big Teacher → Small Student) ....................................................................... 299 28.6 Spot Fleets and Autoscaling (Cheap Capacity, Graceful Risk) ........................................... 301 A Tiny Autoscaler (Queue-Depth Target) .............................................................................. 301 Spot-Aware Placement (Pseudo-strategy) .......................................................................... 302 28.7 Cost Telemetry and Gating (Treat $ Like ms) ..................................................................... 302 28.8 A Simple “Cheap First” End-to-End .................................................................................. 303 28.9 Pitfalls and Fixes ............................................................................................................... 303 28.10 Ship List .......................................................................................................................... 304 Chapter 29: Compliance and Auditability 305 29.1 What “Good” Looks Like .................................................................................................... 305 29.2 Data Retention (Logs, RAG Docs, Embeddings) ................................................................. 306 Retention Policy Config + Sweeper ..................................................................................... 306 RAG Doc and Embedding Deletion (DSR-Friendly) ............................................................... 307 29.3 Audit Trail: Spans You Can Replay ..................................................................................... 307 “Evidence Pack” Exporter (Zip Everything About a Trace) ................................................... 309 29.4 Explainability Notes (Tabular, Deep, and LLM-Safe) .......................................................... 309 Tabular (SHAP or Linear Weights) ........................................................................................ 310 Deep Nets (Integrated Gradients, Summarized) .................................................................. 310 LLM Outputs (No CoT; Cite Evidence and Parameters) ........................................................ 311 29.5 Red-Team Logs: Adversarial Tests As Data ....................................................................... 311 29.6 Playbooks (Incident and DSR Templates) .......................................................................... 313 Incident Runbook (Markdown Template) ............................................................................. 313 DSR (Data Subject Request) Checklist................................................................................. 314 Table of ConTenTs
Page
18
xviii 29.7 Model Cards (Truthful, Versioned) ..................................................................................... 314 29.8 Checklists.......................................................................................................................... 316 29.9 What to Ship ...................................................................................................................... 317 Part VI: Edge, On-Device, and Enterprise Integration (Python-Centric) 319 Chapter 30: On-Device and Edge AI 321 30.1 Design Constraints at the Edge (What Drives Your Choices) ............................................. 321 30.2 Graph Capture in PyTorch: TorchScript vs. torch.export .................................................... 321 TorchScript (Stable, Legacy-Friendly) .................................................................................. 322 torch.export (Future-Facing, Back-End-Agnostic) .............................................................. 322 30.3 ExecuTorch Overview (PyTorch-on-Mobile Runtime) ........................................................ 323 Export Pipeline (Python, High Level) .................................................................................... 323 30.4 ONNX Runtime Path (Portable, Optimized Back Ends) ...................................................... 324 Export to ONNX .................................................................................................................... 324 Inference with ONNX Runtime (CPU/CUDA) ......................................................................... 324 30.5 TensorRT Path (Max GPU Perf on NVIDIA) .......................................................................... 325 ONNX → TensorRT Engine (Python) ..................................................................................... 325 PyTorch → TensorRT in One Step (torch-tensorrt) ............................................................... 326 30.6 Size and Latency: Quantization, AMP, and Memory ........................................................... 328 30.7 Browser: WebGPU and WASM (Python → Web Handoff) ................................................... 329 What You Ship from Python ................................................................................................. 329 Minimal JS (for Your Front-End Team) ................................................................................. 329 30.8 Accuracy and Drift Testing Across Back Ends ................................................................... 330 30.9 Packaging and CI for Edge ................................................................................................ 331 30.10 Quick Recipes ................................................................................................................. 331 1) Android/iOS (CPU) with ExecuTorch ................................................................................. 331 2) Jetson (GPU) with TensorRT ............................................................................................. 331 3) Desktop App (CPU/GPU) with ONNX Runtime .................................................................. 331 4) Browser Inference (WebGPU) .......................................................................................... 332 30.11 Troubleshooting Cheatsheet ............................................................................................ 332 30.12 What to Ship .................................................................................................................... 332 Table of ConTenTs
Page
19
xix Chapter 31: Enterprise Systems and Knowledge 333 31.1 Reality Check: Constraints That Drive Design ................................................................... 333 31.2 Connectors (Warehouses and Doc Stores) ........................................................................ 334 31.3 Ingestion: Normalize → Chunk → Tag .............................................................................. 336 31.4 Embeddings and Indices (pgvector or FAISS) ................................................................... 337 FAISS (Simple, Fast, File-Backed) ........................................................................................ 337 pgvector (SQL-Friendly) ....................................................................................................... 337 31.5 Query-Time Routing: SQL vs. Docs (and Both) .................................................................. 339 31.6 Lineage and Governance Integration ................................................................................ 340 31.7 Governance and Safety: Enforce at Retrieval, Not Just at Answer Time ........................... 341 31.8 Observability and Evaluation (Enterprise-Flavored) .......................................................... 342 31.9 What to Ship ...................................................................................................................... 342 Chapter 32: Security for GenAI Systems 345 32.1 Threat Modeling That Fits in Your Repo ............................................................................. 345 32.2 Supply-Chain Defenses You Can Prove ............................................................................. 347 32.3 Hardening Inference Without Killing Velocity .................................................................... 351 32.4 Provenance Without Drama: Receipts and C2PA ............................................................... 352 32.5 Closing the Loop: Incidents, Egress, and Steady Habits.................................................... 355 Part VII: Patterns and Playbooks (End-to-End, with Code) 357 Chapter 33: Conversational and Helpdesk Bots 359 33.1 A Small but Complete Service ........................................................................................... 359 33.2 Reasonable Defaults That Keep You Out of Trouble ........................................................... 367 33.3 Escalation That Feels Respectful ...................................................................................... 368 33.4 Where to Tighten Things Next ........................................................................................... 368 Chapter 34: Code and DevOps Assistants 369 34.1 Repo Context: Ingest Once, Answer Fast ........................................................................... 369 34.2 From Plan to Multi-file Edits ............................................................................................. 372 34.3 Test Generation That Isn’t Magical .................................................................................... 373 34.4 CI Rehearsal: Run What Your CI Will Run ........................................................................... 375 Table of ConTenTs
Page
20
xx 34.5 Safe Apply: Branch, Write, Test, Commit, and PR .............................................................. 376 34.6 Wiring It Together: A Tiny CLI ............................................................................................. 378 34.7 CI Integration: Write the Workflow the Bot Expects ........................................................... 380 34.8 How the Pieces Play Together ........................................................................................... 381 Chapter 35: Analytics Copilots 383 35.1 A Minimal Semantic Layer You Can Code-Review............................................................. 383 35.2 Declare a Semantic Model over a Real Table .................................................................... 386 35.3 From a Question to a Plan: Intent → Semantic Ask → SQL .............................................. 387 35.4 Governance Hooks: Allow-Lists, Lineage, and Policy Gates .............................................. 389 35.5 Chart Specs the Front End Can Render ............................................................................. 390 35.6 A Tiny End-to-End Service (FastAPI + SQLite Demo) ........................................................ 391 35.7 Where This Grows Inside an Enterprise ............................................................................ 394 Chapter 36: Ecommerce and Marketing 395 36.1 A Lightweight Ecommerce Spine ...................................................................................... 395 36.2 Personalization with RAG .................................................................................................. 398 36.3 Recommenders + LLM Hybrids ......................................................................................... 400 36.4 Synthetic Campaigns That Learn Fast .............................................................................. 402 36.5 Stitching It into a Service .................................................................................................. 405 36.6 A Few Closing Notes ......................................................................................................... 407 Chapter 37: Autonomous Ops and RPA 20 409 37.1 Data Model: Tickets, Runbooks, and Runs ........................................................................ 409 37.2 Tools with Guardrails ......................................................................................................... 412 37.3 Runbooks with Human Gates ............................................................................................ 413 37.4 The Agent Loop (State Machine) ....................................................................................... 414 37.5 FastAPI Edge with Approval Gates .................................................................................... 416 37.6 Adding an LLM Without Giving It the Keys ......................................................................... 420 37.7 Why This Pattern Works .................................................................................................... 420 Table of ConTenTs