Stream Processing with Apache Spark Mastering Structured Streaming and Spark Streaming (Gerard Maas, Francois Garillot)（Z-Library）

【One-Line Pitch】 A practical guide for Spark developers who want to process data in real time, comparing the original Spark Streaming library with the newer Structured Streaming API so you can pick the right tool and build reliable streaming jobs. Best for engineers already comfortable with Spark batch programming who now need to handle unbounded, continuously arriving data. 【Book Arc】 - **Opening (~0%–10%)**: Frames the core problem—extracting insight from infinite event streams—and introduces the two Spark streaming APIs the book contrasts. Establishes foundational concepts like event time vs. processing time and the uncertainty of unbounded input. - **Early (~10%–30%)**: Covers stream-processing fundamentals: timelines, windows (tumbling/sliding), the effect of time on aggregation, and architectural models (Lambda vs. Kappa). Positions Spark's abstraction layers (Core, SQL, libraries) as the platform for streaming. - **Early–Middle (~30%–45%)**: Explains Spark's fault tolerance and resilience—RDDs, driver/executor architecture, shuffle service, and high-availability modes—then moves into the Structured Streaming programming model: sources, transformations, sinks, output modes, and the lazy start() lifecycle. - **Middle (~45%–65%)**: Deepens Structured Streaming with practical detail: source and sink configuration (Kafka, socket, file, memory, console, foreach), event-time handling, windows, watermarks, and stateful operations. - **Late (~65%–90%)**: Shifts to Spark Streaming (the original DStream API), its integration with other Spark APIs, and advanced techniques including approximation algorithms and machine learning on streams. - **Ending (~90%–100%)**: Compares Spark to other stream processors (Storm, Flink, Kafka Streams) and points to community resources for staying current. 【Key Takeaways】 - **Event time vs. processing time is the central distinction** (Early): the book stresses that correlating, ordering, and aggregating events depends on which timeline you use, and that late/out-of-order data is a first-class concern. - **Unbounded input introduces uncertainty** (Early): streaming systems can't assume regular arrival, so peak-load planning and resource matching become core design challenges. - **Lambda vs. Kappa architectures are a real trade-off** (Early): Kappa is lighter and simpler, but the book argues Lambda may still be needed in some cases; batch datasets also serve as benchmarks for streaming iterations. - **Spark's fault tolerance is foundational, not streaming-specific** (Early–Middle): RDDs, driver/executor design, shuffle service, and HA modes underpin long-running streaming jobs. - **Structured Streaming reuses batch APIs but with key differences** (Middle): most Dataset/DataFrame transformations carry over, yet operations like a full count have no defined meaning on a stream and require writeStream.start(). - **Sources, sinks, and output modes form the streaming contract** (Middle): the book details Kafka, socket, file, memory, console, and foreach sinks, plus how output modes relate to aggregations. - **Typed Datasets can be slower than DataFrames** (Middle): closures in Dataset operations are opaque to the query planner, so the DataFrame API may optimize better. - **Spark Streaming (DStreams) remains relevant** (Late): the book covers its operation, integration with other Spark APIs, and advanced approximation/ML techniques alongside Structured Streaming. 【Reading Tips】 - Deep-read the early chapters on event time, windows, and architectures—these concepts recur throughout and are the hardest to retrofit later. - Skim the source/sink configuration catalogs (Kafka, file, foreach) on first pass; return to them as reference when implementing. - Pay close attention to the Structured Streaming programming model chapter—the lazy evaluation and start() lifecycle trip up newcomers. - If you only need Structured Streaming, treat the Spark Streaming (DStream) chapters as comparative context rather than required reading. - Use the final comparison chapter to decide whether Spark fits your use case versus Flink or Kafka Streams. 【Coverage Limits】 The excerpts are heavily weighted toward the table of contents and early conceptual chapters; the later Spark Streaming, advanced techniques, and comparison chapters are only partially represented, so this guide's late-stage coverage is inferred from chapter titles and brief fragments rather than full detail.