Building Machine Learning Systems with a Feature Store Batch, Real-Time, and LLM Systems (Jim Dowling)（Z-Library） (1)

【One-Line Pitch】 A practical, architecture-first guide for data scientists and ML engineers who want to build production-grade batch, real-time, and LLM-powered systems by decomposing them into feature, training, and inference pipelines connected by a feature store. 【Book Arc】 - **Opening (~0%–10%)**: Introduces the core Feature-Training-Inference (FTI) architecture, positioning feature stores as the solution for managing context and history in stateful ML systems. Establishes the book's central promise: any AI system can be modularized into independent pipelines connected by a shared data layer. - **Early (~10%–25%)**: Distinguishes batch, real-time, and agentic ML systems using concrete examples (Spotify's weekly recommendations vs. TikTok's real-time feed). Explains why feature stores emerged (Uber's Michelangelo, 2017) and how they enable low-latency retrieval of precomputed features for online models. - **Early (~25%–35%)**: Dives into ML pipeline engineering—writing modular, testable Python code (functions/classes, DRY principles) and introduces a taxonomy of data transformations: model-independent (feature pipelines), model-dependent (training/inference), and on-demand (real-time inference). Includes first hands-on example: Titanic survival prediction as a full ML system. - **Middle (~35%–50%)**: Walks through a complete air quality forecasting service, covering data exploration, missing-data handling, feature group creation, and pipeline orchestration. Transitions into Part II on feature stores, explaining point-in-time correctness, temporal joins, and data leakage prevention in time-series data. - **Middle (~50%–65%)**: Details feature store architectures (offline/online stores, vector indexes, batch vs. stream APIs) and builds a real-time credit card fraud detection system, showing how to design a data model mixing stream, batch, and on-demand features. - **Late (~65%–100%)**: Excerpts do not cover the later parts in detail, but based on the book's structure, this section covers data transformations with frameworks (Pandas, Polars, Spark, Flink), real-time ML via shift-left/shift-right feature computation, and agentic LLM systems with RAG and tools. 【Key Takeaways】 - **The FTI architecture is the unifying pattern for all AI systems** (Early): Decompose any ML system into feature, training, and inference pipelines connected by a shared data layer. This modularity lets teams develop each pipeline independently as long as they respect data contracts (schemas, validation rules, SLOs). - **Feature stores solve the stateful online ML problem** (Early): When clients can't send all context/history in a request, precompute features and store them in a feature store for low-latency retrieval. This pattern, pioneered by Uber's Michelangelo, is essential for personalized real-time predictions. - **Data transformation taxonomy clarifies pipeline responsibilities** (Early): Model-independent transformations (MITs) belong in feature pipelines; model-dependent transformations (MDTs) happen in training/inference; on-demand transformations (ODTs) compute real-time features at inference. Knowing which goes where prevents architecture mistakes. - **Point-in-time correctness prevents data leakage** (Middle): When building training data from time-series across multiple tables, temporal joins are required to avoid future data leakage or stale features. Feature stores provide APIs for consistent snapshots, making this hard problem tractable. - **Feature store architecture separates offline and online paths** (Middle): Batch APIs write to offline stores (lakehouse tables), while stream APIs (Flink, Spark Structured Streaming) feed online stores and vector indexes. Understanding this dual-path design is critical for real-time systems. - **Start with a minimal viable ML system, then iterate** (Early): The book's approach is to build an MVPS (minimal viable prediction system) with a UI/dashboard, then incrementally add testing, validation, and automation. This mirrors how students in the author's course build working systems in 2–3 weeks. - **Modular code is non-negotiable for ML pipelines** (Early): Refactor notebooks into tested functions/classes following DRY principles. Spaghetti notebooks are untestable and unmaintainable—treat pipeline code like production software engineering. 【Reading Tips】 - **Skim the taxonomy sections in Chapters 1–2** if you're already familiar with ML pipelines; the MIT/MDT/ODT distinction is worth internalizing, but the surrounding examples are illustrative. - **Deep-read the Titanic and air quality examples** (Early–Middle): These are complete, runnable systems using Hopsworks Serverless. Follow along with the code to understand the full pipeline lifecycle—they're the book's best teaching devices. - **Pay special attention to the fraud detection data model** (Middle ~47–53%): This is where the book shows how to combine stream, batch, and on-demand features into a coherent feature group design. It's the most complex example and worth studying carefully. - **If you're focused on LLM/agentic systems**, note that the excerpts don't cover Part VI in detail—you may want to skim earlier parts for the FTI architecture and jump ahead, but be aware you'll miss the data-modeling foundations. - **Install Hopsworks and follow along** (Early): The book is hands-on; you'll need a free serverless account and API key to run examples. Don't just read—execute the notebooks to internalize the patterns. 【Coverage Limits】 This guide is based on excerpts covering roughly the first half of the book (Parts I–II and into Part III). Later sections on advanced data transformations (Spark/Flink), shift-left/shift-right real-time computation, and agentic LLM systems with RAG are not covered in detail here.