Build a DeepSeek Model (From Scratch) (MEAP) (Raj Abhijit Dandekar, Rajat Dandekar etc.)（Z-Library）

【One-Line Pitch】 A hands-on guide to rebuilding the architectural tricks that made DeepSeek cheap and powerful—Multi-Head Latent Attention, Mixture-of-Experts, Multi-Token Prediction, and reasoning-focused post-training—at a scale you can actually run. Best for practitioners who already know basic Transformer/LLM mechanics and want first-principles engineering intuition rather than API-level usage. 【Book Arc】 - **Opening (~0%–10%)**: Frames the problem: why open models lagged closed ones, and how DeepSeek's blend of MoE, latent attention, MTP, distillation, and parallelism closed the gap. Sets up a four-stage roadmap and treats each innovation as a case study (limitation → advanced replacement). - **Early (~10%–35%)**: Stage 1, the KV Cache foundation. Walks through pre-training vs. inference, autoregressive generation, and a step-by-step trace of the Transformer block to prove that naive inference repeats huge amounts of computation—motivating the cache. - **Middle (~35%–50%)**: The KV cache as a memory-for-compute trade-off, including a size formula and its scaling with model size. Then the first-generation fixes: Multi-Query Attention (memory-first, but hurts head specialization) and Grouped-Query Attention as a middle ground, with PyTorch implementations. - **Late (~50%–80%)**: (Excerpts do not cover this range in detail.) Per the roadmap, this is where MLA, MoE, MTP, FP8 quantization, and DualPipe-style parallelism are built—the core DeepSeek-specific architecture and training pipeline. - **Ending (~80%–100%)**: (Excerpts do not cover this range.) Post-training—supervised fine-tuning, RL for reasoning, and distillation into smaller deployable models—plus assembling everything into a cohesive mini-DeepSeek. 【Key Takeaways】 - **Inference, not training, is where the KV cache matters** (Early): the book separates the two phases and focuses entirely on the fixed-parameter usage phase, where generation happens token by token. - **Autoregressive generation is provably redundant** (Early): because only the last token's context vector drives the next prediction, re-processing the growing sequence repeats identical Q/K/V projections—the mathematical setup for caching. - **Cache Keys and Values, never Queries** (Early): prior K/V vectors don't change, so they're stored and appended to; only the new token's projections are computed each step. - **Caching trades memory for compute** (Middle): the KV cache size follows a concrete formula and grows with model size, becoming the new bottleneck that caps batch size and context length. - **MQA is a memory-first compromise** (Middle): sharing one K/V across all heads slashes cache size but forces heads to lose specialization, degrading the model's ability to capture diverse relationships. - **GQA is the balanced middle ground** (Middle): grouping heads so K/V are shared only within groups recovers much of the lost capacity while keeping memory savings—implemented directly in PyTorch. - **Each DeepSeek innovation targets a specific bottleneck** (Opening): MLA, MoE, MTP, FP8, and parallelism are framed as targeted answers to computational complexity, memory bandwidth, or parameter scaling—not arbitrary tricks. - **Build-it-yourself is the pedagogical bet** (Opening): the authors argue that implementing these systems yields ownership of the knowledge that reading papers or calling APIs cannot. 【Reading Tips】 - Deep-read the Early chapters on autoregressive redundancy and the KV cache derivation—everything later (MLA especially) builds on this intuition, and the math is the payoff. - Skim the MQA/GQA code listings if you already know attention; focus on the *why* (head specialization trade-off) rather than the boilerplate. - Keep the four-stage roadmap visible as you read; each chapter is a case study, so track which bottleneck each component solves. - Treat the PyTorch implementations as templates to run and modify—this is a build book, not a survey. - Note that the excerpts here are thin past the midpoint; expect MLA, MoE, MTP, and post-training to carry the heaviest conceptual load. 【Coverage Limits】 This guide is based on stratified excerpts covering roughly the first half of the book (fundamentals through GQA); the DeepSeek-specific innovations (MLA, MoE, MTP, FP8, DualPipe, RL post-training, distillation) are described only via the book's stated roadmap, not from detailed excerpt content.