High Performance Python Practical Performant Programming for Humans, 2nd Edition (Micha Gorelick, Ian Ozsvald)（Z-Library）

📄 File Format: PDF

💾 File Size: 10.6 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

Second Edition Micha Gorelick & Ian Ozsvald High Performance Python Practical Performant Programming for Humans

📄 Page 2

(This page has no text content)

📄 Page 3

Micha Gorelick and Ian Ozsvald High Performance Python Practical Performant Programming for Humans SECOND EDITION Boston Farnham Sebastopol TokyoBeijing

📄 Page 4

978-1-492-05502-0 [LSI] High Performance Python by Micha Gorelick and Ian Ozsvald Copyright © 2020 Micha Gorelick and Ian Ozsvald. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institu‐ tional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Tyler Ortman Indexer: Potomac Indexing, LLC Development Editor: Sarah Grey Interior Designer: David Futato Production Editor: Christopher Faucher Cover Designer: Karen Montgomery Copyeditor: Arthur Johnson Illustrator: Rebecca Demarest Proofreader: Sharon Wilkey September 2014: First Edition May 2020: Second Edition Revision History for the Second Edition 2020-04-30: First release See http://oreilly.com/catalog/errata.csp?isbn=9781492055020 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. High Performance Python, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the authors, and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. High Performance Python is available under the Creative Commons Attribution-NonCommercial- NoDerivs 3.0 International License.

📄 Page 5

Table of Contents Foreword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii 1. Understanding Performant Python. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 The Fundamental Computer System 2 Computing Units 2 Memory Units 5 Communications Layers 8 Putting the Fundamental Elements Together 10 Idealized Computing Versus the Python Virtual Machine 10 So Why Use Python? 14 How to Be a Highly Performant Programmer 16 Good Working Practices 17 Some Thoughts on Good Notebook Practice 19 Getting the Joy Back into Your Work 20 2. Profiling to Find Bottlenecks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Profiling Efficiently 22 Introducing the Julia Set 23 Calculating the Full Julia Set 26 Simple Approaches to Timing—print and a Decorator 30 Simple Timing Using the Unix time Command 33 Using the cProfile Module 35 Visualizing cProfile Output with SnakeViz 39 Using line_profiler for Line-by-Line Measurements 40 Using memory_profiler to Diagnose Memory Usage 46 Introspecting an Existing Process with PySpy 54 iii

📄 Page 6

Bytecode: Under the Hood 55 Using the dis Module to Examine CPython Bytecode 55 Different Approaches, Different Complexity 57 Unit Testing During Optimization to Maintain Correctness 59 No-op @profile Decorator 60 Strategies to Profile Your Code Successfully 62 Wrap-Up 64 3. Lists and Tuples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 A More Efficient Search 68 Lists Versus Tuples 71 Lists as Dynamic Arrays 72 Tuples as Static Arrays 76 Wrap-Up 77 4. Dictionaries and Sets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 How Do Dictionaries and Sets Work? 83 Inserting and Retrieving 83 Deletion 87 Resizing 87 Hash Functions and Entropy 88 Dictionaries and Namespaces 92 Wrap-Up 95 5. Iterators and Generators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 Iterators for Infinite Series 101 Lazy Generator Evaluation 103 Wrap-Up 107 6. Matrix and Vector Computation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 Introduction to the Problem 110 Aren’t Python Lists Good Enough? 115 Problems with Allocating Too Much 117 Memory Fragmentation 120 Understanding perf 122 Making Decisions with perf’s Output 125 Enter numpy 126 Applying numpy to the Diffusion Problem 129 Memory Allocations and In-Place Operations 133 Selective Optimizations: Finding What Needs to Be Fixed 137 numexpr: Making In-Place Operations Faster and Easier 140 A Cautionary Tale: Verify “Optimizations” (scipy) 142 iv | Table of Contents

📄 Page 7

Lessons from Matrix Optimizations 143 Pandas 146 Pandas’s Internal Model 146 Applying a Function to Many Rows of Data 148 Building DataFrames and Series from Partial Results Rather than Concatenating 156 There’s More Than One (and Possibly a Faster) Way to Do a Job 157 Advice for Effective Pandas Development 159 Wrap-Up 160 7. Compiling to C. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 What Sort of Speed Gains Are Possible? 162 JIT Versus AOT Compilers 164 Why Does Type Information Help the Code Run Faster? 164 Using a C Compiler 165 Reviewing the Julia Set Example 166 Cython 167 Compiling a Pure Python Version Using Cython 167 pyximport 169 Cython Annotations to Analyze a Block of Code 170 Adding Some Type Annotations 172 Cython and numpy 176 Parallelizing the Solution with OpenMP on One Machine 178 Numba 180 Numba to Compile NumPy for Pandas 182 PyPy 183 Garbage Collection Differences 184 Running PyPy and Installing Modules 185 A Summary of Speed Improvements 186 When to Use Each Technology 187 Other Upcoming Projects 188 Graphics Processing Units (GPUs) 189 Dynamic Graphs: PyTorch 190 Basic GPU Profiling 193 Performance Considerations of GPUs 194 When to Use GPUs 196 Foreign Function Interfaces 197 ctypes 199 cffi 201 f2py 204 CPython Module 207 Wrap-Up 211 Table of Contents | v

📄 Page 8

8. Asynchronous I/O. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213 Introduction to Asynchronous Programming 215 How Does async/await Work? 218 Serial Crawler 219 Gevent 221 tornado 226 aiohttp 229 Shared CPU–I/O Workload 233 Serial 233 Batched Results 235 Full Async 238 Wrap-Up 243 9. The multiprocessing Module. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 An Overview of the multiprocessing Module 248 Estimating Pi Using the Monte Carlo Method 250 Estimating Pi Using Processes and Threads 251 Using Python Objects 252 Replacing multiprocessing with Joblib 260 Random Numbers in Parallel Systems 263 Using numpy 264 Finding Prime Numbers 267 Queues of Work 273 Verifying Primes Using Interprocess Communication 278 Serial Solution 283 Naive Pool Solution 284 A Less Naive Pool Solution 285 Using Manager.Value as a Flag 286 Using Redis as a Flag 288 Using RawValue as a Flag 290 Using mmap as a Flag 291 Using mmap as a Flag Redux 293 Sharing numpy Data with multiprocessing 295 Synchronizing File and Variable Access 301 File Locking 302 Locking a Value 305 Wrap-Up 308 10. Clusters and Job Queues. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311 Benefits of Clustering 312 Drawbacks of Clustering 313 $462 Million Wall Street Loss Through Poor Cluster Upgrade Strategy 315 vi | Table of Contents

📄 Page 9

Skype’s 24-Hour Global Outage 315 Common Cluster Designs 316 How to Start a Clustered Solution 317 Ways to Avoid Pain When Using Clusters 318 Two Clustering Solutions 319 Using IPython Parallel to Support Research 319 Parallel Pandas with Dask 322 NSQ for Robust Production Clustering 326 Queues 327 Pub/sub 328 Distributed Prime Calculation 330 Other Clustering Tools to Look At 334 Docker 335 Docker’s Performance 335 Advantages of Docker 339 Wrap-Up 340 11. Using Less RAM. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341 Objects for Primitives Are Expensive 342 The array Module Stores Many Primitive Objects Cheaply 344 Using Less RAM in NumPy with NumExpr 346 Understanding the RAM Used in a Collection 350 Bytes Versus Unicode 352 Efficiently Storing Lots of Text in RAM 353 Trying These Approaches on 11 Million Tokens 354 Modeling More Text with Scikit-Learn’s FeatureHasher 362 Introducing DictVectorizer and FeatureHasher 362 Comparing DictVectorizer and FeatureHasher on a Real Problem 365 SciPy’s Sparse Matrices 366 Tips for Using Less RAM 370 Probabilistic Data Structures 371 Very Approximate Counting with a 1-Byte Morris Counter 372 K-Minimum Values 375 Bloom Filters 379 LogLog Counter 385 Real-World Example 389 12. Lessons from the Field. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393 Streamlining Feature Engineering Pipelines with Feature-engine 394 Feature Engineering for Machine Learning 394 The Hard Task of Deploying Feature Engineering Pipelines 395 Leveraging the Power of Open Source Python Libraries 395 Table of Contents | vii

📄 Page 10

Feature-engine Smooths Building and Deployment of Feature Engineering Pipelines 396 Helping with the Adoption of a New Open Source Package 397 Developing, Maintaining, and Encouraging Contribution to Open Source Libraries 398 Highly Performant Data Science Teams 400 How Long Will It Take? 400 Discovery and Planning 401 Managing Expectations and Delivery 402 Numba 403 A Simple Example 404 Best Practices and Recommendations 405 Getting Help 409 Optimizing Versus Thinking 409 Adaptive Lab’s Social Media Analytics (2014) 412 Python at Adaptive Lab 413 SoMA’s Design 413 Our Development Methodology 414 Maintaining SoMA 414 Advice for Fellow Engineers 415 Making Deep Learning Fly with RadimRehurek.com (2014) 415 The Sweet Spot 416 Lessons in Optimizing 417 Conclusion 420 Large-Scale Productionized Machine Learning at Lyst.com (2014) 420 Cluster Design 420 Code Evolution in a Fast-Moving Start-Up 421 Building the Recommendation Engine 421 Reporting and Monitoring 422 Some Advice 422 Large-Scale Social Media Analysis at Smesh (2014) 422 Python’s Role at Smesh 423 The Platform 423 High Performance Real-Time String Matching 424 Reporting, Monitoring, Debugging, and Deployment 425 PyPy for Successful Web and Data Processing Systems (2014) 426 Prerequisites 427 The Database 428 The Web Application 428 OCR and Translation 429 Task Distribution and Workers 429 Conclusion 429 viii | Table of Contents

📄 Page 11

Task Queues at Lanyrd.com (2014) 430 Python’s Role at Lanyrd 430 Making the Task Queue Performant 431 Reporting, Monitoring, Debugging, and Deployment 431 Advice to a Fellow Developer 431 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433 Table of Contents | ix

📄 Page 12

(This page has no text content)

📄 Page 13

Foreword When you think about high performance computing, you might imagine giant clus‐ ters of machines modeling complex weather phenomena or trying to understand sig‐ nals in data collected about far-off stars. It’s easy to assume that only people building specialized systems should worry about the performance characteristics of their code. By picking up this book, you’ve taken a step toward learning the theory and practices you’ll need to write highly performant code. Every programmer can benefit from understanding how to build performant systems. There are an obvious set of applications that are just on the edge of possible, and you won’t be able to approach them without writing optimally performant code. If that’s your practice, you’re in the right place. But there is a much broader set of applica‐ tions that can benefit from performant code. We often think that new technical capabilities are what drives innovation, but I’m equally fond of capabilities that increase the accessibility of technology by orders of magnitude. When something becomes ten times cheaper in time or compute costs, suddenly the set of applications you can address is wider than you imagined. The first time this principle manifested in my own work was over a decade ago, when I was working at a social media company, and we ran an analysis over multiple tera‐ bytes of data to determine whether people clicked on more photos of cats or dogs on social media. It was dogs, of course. Cats just have better branding. This was an outstandingly frivolous use of compute time and infrastructure at the time! Gaining the ability to apply techniques that had previously been restricted to sufficiently high-value applications, such as fraud detection, to a seemingly trivial question opened up a new world of now-accessible possibilities. We were able to take what we learned from these experiments and build a whole new set of products in search and content discovery. Foreword | xi

📄 Page 14

For an example that you might encounter today, consider a machine-learning system that recognizes unexpected animals or people in security video footage. A sufficiently performant system could allow you to embed that model into the camera itself, improving privacy or, even if running in the cloud, using significantly less compute and power—benefiting the environment and reducing your operating costs. This can free up resources for you to look at adjacent problems, potentially building a more valuable system. We all desire to create systems that are effective, easy to understand, and performant. Unfortunately, it often feels like we have to pick two (or one) out of the three! High Performance Python is a handbook for people who want to make things that are capa‐ ble of all three. This book stands apart from other texts on the subject in three ways. First, it’s written for us—humans who write code. You’ll find all of the context you need to understand why you might make certain choices. Second, Gorelick and Ozsvald do a wonderful job of curating and explaining the necessary theory to support that context. Finally, in this updated edition, you’ll learn the specific quirks of the most useful libraries for implementing these approaches today. This is one of a rare class of programming books that will change the way you think about the practice of programming. I’ve given this book to many people who could benefit from the additional tools it provides. The ideas that you’ll explore in its pages will make you a better programmer, no matter what language or environment you choose to work in. Enjoy the adventure. — Hilary Mason, Data Scientist in Residence at Accel xii | Foreword

📄 Page 15

Preface Python is easy to learn. You’re probably here because now that your code runs cor‐ rectly, you need it to run faster. You like the fact that your code is easy to modify and you can iterate with ideas quickly. The trade-off between easy to develop and runs as quickly as I need is a well-understood and often-bemoaned phenomenon. There are solutions. Some people have serial processes that have to run faster. Others have problems that could take advantage of multicore architectures, clusters, or graphics processing units. Some need scalable systems that can process more or less as expediency and funds allow, without losing reliability. Others will realize that their coding techni‐ ques, often borrowed from other languages, perhaps aren’t as natural as examples they see from others. In this book we will cover all of these topics, giving practical guidance for under‐ standing bottlenecks and producing faster and more scalable solutions. We also include some war stories from those who went ahead of you, who took the knocks so you don’t have to. Python is well suited for rapid development, production deployments, and scalable systems. The ecosystem is full of people who are working to make it scale on your behalf, leaving you more time to focus on the more challenging tasks around you. Who This Book Is For You’ve used Python for long enough to have an idea about why certain things are slow and to have seen technologies like Cython, numpy, and PyPy being discussed as possible solutions. You might also have programmed with other languages and so know that there’s more than one way to solve a performance problem. While this book is primarily aimed at people with CPU-bound problems, we also look at data transfer and memory-bound solutions. Typically, these problems are faced by scientists, engineers, quants, and academics. Preface | xiii

📄 Page 16

We also look at problems that a web developer might face, including the movement of data and the use of just-in-time (JIT) compilers like PyPy and asynchronous I/O for easy-win performance gains. It might help if you have a background in C (or C++, or maybe Java), but it isn’t a prerequisite. Python’s most common interpreter (CPython—the standard you nor‐ mally get if you type python at the command line) is written in C, and so the hooks and libraries all expose the gory inner C machinery. There are lots of other techniques that we cover that don’t assume any knowledge of C. You might also have a lower-level knowledge of the CPU, memory architecture, and data buses, but again, that’s not strictly necessary. Who This Book Is Not For This book is meant for intermediate to advanced Python programmers. Motivated novice Python programmers may be able to follow along as well, but we recommend having a solid Python foundation. We don’t cover storage-system optimization. If you have a SQL or NoSQL bottle‐ neck, then this book probably won’t help you. What You’ll Learn Your authors have been working with large volumes of data, a requirement for I want the answers faster! and a need for scalable architectures, for many years in both industry and academia. We’ll try to impart our hard-won experience to save you from making the mistakes that we’ve made. At the start of each chapter, we’ll list questions that the following text should answer. (If it doesn’t, tell us and we’ll fix it in the next revision!) We cover the following topics: • Background on the machinery of a computer so you know what’s happening behind the scenes • Lists and tuples—the subtle semantic and speed differences in these fundamental data structures • Dictionaries and sets—memory allocation strategies and access algorithms in these important data structures • Iterators—how to write in a more Pythonic way and open the door to infinite data streams using iteration • Pure Python approaches—how to use Python and its modules effectively xiv | Preface

📄 Page 17

• Matrices with numpy—how to use the beloved numpy library like a beast • Compilation and just-in-time computing—processing faster by compiling down to machine code, making sure you’re guided by the results of profiling • Concurrency—ways to move data efficiently • multiprocessing—various ways to use the built-in multiprocessing library for parallel computing and to efficiently share numpy matrices, and some costs and benefits of interprocess communication (IPC) • Cluster computing—convert your multiprocessing code to run on a local or remote cluster for both research and production systems • Using less RAM—approaches to solving large problems without buying a humungous computer • Lessons from the field—lessons encoded in war stories from those who took the blows so you don’t have to Python 3 Python 3 is the standard version of Python as of 2020, with Python 2.7 deprecated after a 10-year migration process. If you’re still on Python 2.7, you’re doing it wrong —many libraries are no longer supported for your line of Python, and support will become more expensive over time. Please do the community a favor and migrate to Python 3, and make sure that all new projects use Python 3. In this book, we use 64-bit Python. Whilst 32-bit Python is supported, it is far less common for scientific work. We’d expect all the libraries to work as usual, but numeric precision, which depends on the number of bits available for counting, is likely to change. 64-bit is dominant in this field, along with *nix environments (often Linux or Mac). 64-bit lets you address larger amounts of RAM. *nix lets you build applications that can be deployed and configured in well-understood ways with well- understood behaviors. If you’re a Windows user, you’ll have to buckle up. Most of what we show will work just fine, but some things are OS-specific, and you’ll have to research a Windows sol‐ ution. The biggest difficulty a Windows user might face is the installation of modules: research in sites like Stack Overflow should give you the solutions you need. If you’re on Windows, having a virtual machine (e.g., using VirtualBox) with a running Linux installation might help you to experiment more freely. Windows users should definitely look at a packaged solution like those available through Anaconda, Canopy, Python(x,y), or Sage. These same distributions will make the lives of Linux and Mac users far simpler too. Preface | xv

📄 Page 18

Changes from Python 2.7 If you’ve upgraded from Python 2.7, you might not be aware of a few relevant changes: • / meant integer division in Python 2.7, whereas it performs float division in Python 3. • str and unicode were used to represent text data in Python 2.7; in Python 3, everything is a str, and these are always Unicode. For clarity, a bytes type is used if we’re using unencoded byte sequences. If you’re in the process of upgrading your code, two good guides are “Porting Python 2 Code to Python 3” and “Supporting Python 3: An in-depth guide”. With a distribu‐ tion like Anaconda or Canopy, you can run both Python 2 and Python 3 simultane‐ ously—this will simplify your porting. License This book is licensed under Creative Commons Attribution-NonCommercial- NoDerivs 3.0. You’re welcome to use this book for noncommercial purposes, including for noncommercial teaching. The license allows only for complete reproductions; for partial reproductions, please contact O’Reilly (see “How to Contact Us” on page xviii). Please attribute the book as noted in the following section. We negotiated that the book should have a Creative Commons license so the con‐ tents could spread further around the world. We’d be quite happy to receive a beer if this decision has helped you. We suspect that the O’Reilly staff would feel similarly about the beer. How to Make an Attribution The Creative Commons license requires that you attribute your use of a part of this book. Attribution just means that you should write something that someone else can follow to find this book. The following would be sensible: “High Performance Python, 2nd ed., by Micha Gorelick and Ian Ozsvald (O’Reilly). Copyright 2020 Micha Gore‐ lick and Ian Ozsvald, 978-1-492-05502-0.” xvi | Preface

📄 Page 19

Errata and Feedback We encourage you to review this book on public sites like Amazon—please help oth‐ ers understand if they would benefit from this book! You can also email us at feedback@highperformancepython.com. We’re particularly keen to hear about errors in the book, successful use cases where the book has helped you, and high performance techniques that we should cover in the next edition. You can access the web page for this book at https://oreil.ly/high- performance-python-2e. Complaints are welcomed through the instant-complaint-transmission-service > /dev/null. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program ele‐ ments such as variable or function names, databases, datatypes, environment variables, statements, and keywords. Constant width bold Shows commands or other text that should be typed literally by the user. Constant width italic Shows text that should be replaced with user-supplied values or by values deter‐ mined by context. This element signifies a tip, suggestion, or critical thinking ques‐ tion. This element signifies a general note. Preface | xvii

📄 Page 20

This element indicates a warning or caution. Using Code Examples Supplemental material (code examples, exercises, etc.) is available for download at https://github.com/mynameisfiber/high_performance_python_2e. If you have a technical question or a problem using the code examples, please send email to bookquestions@oreilly.com. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit http://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: xviii | Preface