AI Systems Performance Engineering (Chris Fregly)（Z-Library）

【One-Line Pitch】 A hands-on engineering guide to making large-model training and inference actually run fast on real GPU clusters, from the silicon and OS layers up through CUDA, PyTorch, and multinode serving. Best for ML platform engineers, performance-minded practitioners, and infrastructure teams who already know deep learning basics but need to squeeze latency and throughput out of GPU systems. 【Book Arc】 - **Opening (~0%–10%)**: Frames the whole problem — why AI workloads hit memory, bandwidth, and interconnect walls — and introduces the hardware substrate (Grace CPU, Blackwell dual-die GPU, superchip memory architecture) that later optimizations depend on. - **Early (~10%–32%)**: Moves down the stack into OS, Docker, and Kubernetes tuning: NUMA awareness, CPU pinning, MPS, cgroup I/O isolation, and networking choices (RDMA vs TCP, jumbo frames, sysctl tuning) that decide whether GPUs stay fed. - **Middle (~32%–55%)**: Covers distributed communication and storage I/O — DataParallel mechanics and the GIL bottleneck, NIXL for direct GPU-to-GPU KV cache transfer, GDS, and how to tell communication-bound from compute-bound workloads with profilers. - **Late (~55%–80%)**: Shifts to inference at scale: disaggregated prefill/decode, parallelism strategies for massive MoE models (tensor, pipeline, expert, context, hybrid), and speculative/parallel decoding techniques. - **Ending (~80%–100%)**: Closes with profiling, debugging, and tuning inference in production, plus Triton pipelining/double buffering and the PyTorch XLA backend — turning the earlier concepts into a repeatable tuning workflow. 【Key Takeaways】 - **Locality is the master constraint** (Early): Keep data and compute on the same NUMA node, bind CPU threads to the GPU's local node, and prefer NVMe/local storage — cross-node memory access can nearly double latency. - **The CPU must never starve the GPU** (Early): Grace's design philosophy — huge memory bandwidth, large L3, low-latency NVLink-C2C — exists so preprocessing, tokenization, and data feeding don't become the bottleneck. - **MPS shares compute, not memory** (Early): It helps when jobs leave slack, but one process can OOM the whole GPU, and clients typically must share a Unix user — so it's a QoS tool, not a partitioning tool. - **Interconnect choice dominates distributed training** (Early–Middle): RDMA/InfiniBand can be 5–10× lower latency than TCP, and large-message throughput matters more than tiny-message latency for gradient all-reduce. - **NIXL is the inference-era answer to KV cache movement** (Middle): It picks the fastest lane automatically — NVLink, NVSwitch, InfiniBand/RoCE, even NVMe — and overlaps transfer with computation so destination GPUs keep decoding. - **Diagnose before you tune** (Middle): Vary the compute-to-communication ratio and watch NIC throughput to determine whether you're network-bound or GPU-bound; combine Nsight Systems, Nsight Compute, and NCCL/PyTorch logging. - **Parallelism is a menu, not a default** (Late): Tensor, pipeline, expert, data, context, and hybrid parallelism each fit different model shapes — massive MoE serving in particular needs expert parallelism plus routing and load-balancing care. - **Decoding is where inference latency lives** (Late): Speculative decoding (draft-model, EAGLE, Medusa), interleaved multi-request decode, and disaggregated prefill/decode directly target TTFT and TPOT. 【Reading Tips】 - Deep-read the hardware and OS chapters (Opening–Early) even if you're a software person — most later optimizations only make sense once NUMA, NVLink, and memory tiers are clear. - Skim the parallelism taxonomy in the Late section on a first pass, then return when you actually need to serve a specific model size; the trade-offs are easier to absorb with a concrete workload in mind. - Treat the profiling chapters as a workflow to practice, not just read: set up Nsight Systems + Nsight Compute + NCCL logging on your own cluster and reproduce one diagnosis. - Keep a running checklist of tunables (numactl bindings, MPS thread percentage, MTU 9000, sysctl buffers, congestion control) — the book's value is in the cumulative stack, not any single knob. - Don't skip the "Key Takeaways" boxes at chapter ends; they compress each layer into the few decisions that actually move performance. 【Coverage Limits】 This guide is built from stratified excerpts covering the front matter, hardware, OS/container tuning, distributed communication, and inference-serving chapters; the excerpts do not cover the full depth of the Triton, XLA, and late-stage profiling material, so specific implementation details there are summarized rather than detailed.