Post-Training AI A Practical Guide to Fine-Tuning and Reinforcement Learning (Early Release) (Ben Burtenshaw)（Z-Library）

【One-Line Pitch】 A hands-on field guide for engineers who want to turn open-weight base models into reliable, domain-specific systems through supervised fine-tuning, preference learning, and reinforcement learning—without depending on frontier APIs. Best suited to practitioners with some ML background who learn by building. 【Book Arc】 - **Opening (~0%–10%)**: Defines post-training as the second training phase after pre-training, separating what it can and cannot change, and framing why owning model weights matters for cost and control. - **Early (~10%–35%)**: Surveys the open-model wave (Llama, Gemma, Qwen, DeepSeek, Olmo) and contrasts prompting/RAG with weight-level behavior change; introduces agentic tasks as the newest frontier. - **Middle (~35%–55%)**: Walks through the core techniques in order—SFT and chat templates, then RLHF/PPO and its operational burden, then DPO and variants, then verifiable-reward RL and GRPO. - **Late (~55%–80%)**: Excerpts indicate coverage of learning-signal design (environments, rewards, data) and production pipelines with PEFT and distributed scaling, though the sampled text does not detail these chapters. - **Ending (~80%–100%)**: Promised material on reasoning, superhuman performance, and production realities; the excerpts do not cover these chapters, so their content cannot be summarized here. 【Key Takeaways】 - **Post-training shapes behavior, not knowledge** (Opening): It teaches format, preferences, and reasoning on checkable tasks, but cannot inject facts or capabilities absent from the base weights—data quality and reward design set the ceiling. - **Ownership changes the economics and the risk profile** (Early): Self-hosted post-trained models can undercut per-token API costs at volume and remove provider dependency, letting teams iterate on their own schedule. - **SFT is fast but only teaches imitation** (Middle): A small, clean, consistently formatted dataset beats a large noisy one; the model learns what to say, not how to judge quality. - **Preference learning supplies the quality signal** (Middle): RLHF's three-stage pipeline (SFT → reward model → PPO) showed a smaller aligned model can beat a much larger raw one, at the cost of four models in memory and fragile tuning. - **DPO is the pragmatic default** (Middle): By optimizing directly on chosen/rejected pairs, it cuts memory and instability, which is why DPO and variants (IPO, KTO, ORPO) dominate open pipelines. - **Verifiable rewards unlock reasoning and agents** (Middle): GRPO-style training on math, code, and tool-use tasks rewards outcomes a verifier can check, extending the DeepSeek-R1 loop to multi-step trajectories. - **Agents raise new training problems** (Middle): Longer episodes make credit assignment hard, action spaces widen, and you need sandboxed environments—areas the book flags as still evolving. - **Match technique to task** (Early): Extraction models may need only SFT; general assistants benefit from preference learning; math/coding models justify verifiable-reward RL. 【Reading Tips】 - Read Chapters 1–2 closely for the conceptual map and a first-principles speed run; they anchor everything later. - Skim the model-family survey (Llama/Qwen/DeepSeek recipes) for pattern recognition, then return when choosing a base model. - Treat the RLHF-vs-DPO-vs-GRPO comparison as the decision core—note memory, stability, and data requirements for each. - Since this is an Early Release, expect gaps and unedited prose; verify chapter availability and check the promised GitHub repo before relying on code. - Pair each technique with a small single-GPU experiment before attempting distributed pipelines. 【Coverage Limits】 This guide reflects only the opening and early-to-middle chapters available in the sampled excerpts; later chapters on environments, production pipelines, reasoning, and production realities are listed but not covered, so their specifics are unknown.