CUDA Programming with Python：From Basics to Expert Proficiency (--)（Z-Library）

【One-Line Pitch】 A practical, Python-first route into GPU computing: learn CUDA's execution and memory model, then write and optimize kernels through Numba and related Python tooling. Best for Python-fluent researchers, data scientists, and engineers who want real speedups without abandoning their existing stack. 【Book Arc】 - **Opening (~0%–10%)**: Orients you in CUDA and GPU computing — what CUDA is, its history, GPU vs CPU architecture, and the parallel-processing mindset — then walks through environment setup (CUDA Toolkit, VS Code, Anaconda/Python, Numba, Jupyter) and verification. - **Early (~10%–30%)**: Builds the Python foundation (NumPy, JIT compilation, Numba basics) and the CUDA architecture/memory model — SMs, CUDA cores, the memory hierarchy, coalescing, latency/bandwidth — before introducing core programming concepts: kernels, thread hierarchy, grid/block dimensions, host–device transfers, synchronization, and error handling. - **Middle (~30%–55%)**: Shifts to parallel programming theory (data vs task parallelism, Amdahl's Law, synchronization, load balancing, scalability) and then to hands-on Numba-CUDA: writing, compiling, and running kernels, managing memory, and optimizing. - **Late (~55%–80%)**: Moves into advanced territory — streams and concurrent execution, asynchronous transfers, dynamic parallelism, CUDA Graphs, Thrust, GPU API interoperability, peer-to-peer memory — plus debugging and profiling with Nsight, CUDA-GDB, and the Visual Profiler. - **Ending (~80%–100%)**: Consolidates optimization strategy: performance metrics, kernel launch configuration, shared-memory efficiency, divergence reduction, streams/events, and case studies, closing with best practices. 【Key Takeaways】 - **CUDA is a host/device split, not a single program** (Early): CPU code controls flow and data movement; GPU code runs the compute-heavy kernels. Understanding this division is the mental model everything else builds on. - **The thread hierarchy is the core abstraction** (Early): threads → blocks → grids, with shared memory within blocks and independent blocks across the grid. This structure is what makes GPU scaling possible. - **Memory behavior dominates performance** (Middle): global vs shared vs constant/texture vs registers, plus coalescing and access patterns, matter more than raw arithmetic. Most optimization is memory optimization. - **Python access comes primarily through Numba** (Middle): Numba's JIT and CUDA support let you write GPU kernels in Python, trading some low-level control for far faster iteration. PyCUDA appears as an alternative for direct CUDA C kernels. - **Parallelism has laws and limits** (Middle): Amdahl's Law, load balancing, and partitioning determine realistic speedups; not every problem parallelizes well. - **Advanced concurrency unlocks throughput** (Late): streams, async transfers, dynamic parallelism, and CUDA Graphs let you overlap work and hide latency. - **Debugging and profiling are first-class skills** (Late): Nsight, CUDA-GDB, and the Visual Profiler turn guesswork into measured optimization; race conditions and memory errors are the usual culprits. - **Optimization is systematic, not heroic** (Ending): measure metrics, tune launch configs, exploit shared memory, reduce divergence, and iterate — supported by case studies and best practices. 【Reading Tips】 - **Deep-read the memory model and optimization chapters** (Middle and Ending); skim the history/evolution material in the Opening if you already know GPU background. - **Set up the environment early and actually run the sample kernels** — the book's value is hands-on, and installation troubleshooting is covered so you can unblock yourself. - **Treat debugging/profiling as a skill to practice, not read**: pick one kernel and profile it with Nsight before moving on. - **If you're new to parallel thinking**, slow down on Amdahl's Law and data vs task parallelism — these frame why some kernels never get faster. - **Keep NumPy fluency handy**; the Python-side chapters assume comfort with array programming. 【Coverage Limits】 This guide is synthesized from stratified excerpts covering the table of contents and the opening/middle chapters; later chapters (advanced techniques, debugging, optimization) are represented mainly by their section structure, so specific examples, code, and case-study details in those parts are not covered here.