Expert Python Programming, 2nd Edition Become an ace Python programmer by learning best coding practices and advance-level… (Michal Jaworski, Tarek Ziade)（Z-Library）

📄 File Format: PDF

💾 File Size: 41.4 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

www.it-ebooks.info

📄 Page 2

Expert Python Programming Second Edition Become an ace Python programmer by learning best coding practices and advance-level concepts with Python 3.5 Michał Jaworski Tarek Ziadé BIRMINGHAM - MUMBAI www.it-ebooks.info

📄 Page 3

Expert Python Programming Second Edition Copyright © 2016 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. First published: September 2008 Second edition: May 2016 Production reference: 1160516 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-78588-685-0 www.packtpub.com www.it-ebooks.info

📄 Page 4

Credits Authors Michał Jaworski Tarek Ziadé Reviewer Facundo Batista Commissioning Editor Kunal Parikh Acquisition Editor Meeta Rajani Technical Editor Pankaj Kadam Copy Editor Laxmi Subramanian Proofreader Safis Editing Indexer Rekha Nair Graphics Jason Monteiro Production Coordinator Aparna Bhagat Cover Work Aparna Bhagat www.it-ebooks.info

📄 Page 5

About the Authors Michał Jaworski has 7 years of experience in Python. He is also the creator of graceful, which is a REST framework built on top of falcon. He has been in various roles at different companies: from an ordinary full-stack developer through software architect to VP of engineering in a fast-paced start-up company. He is currently a lead backend engineer in TV Store team at Opera Software. He is highly experienced in designing high-performance distributed services. He is also an active contributor to some of the popular Python open source projects. Tarek Ziadé is an engineering manager at Mozilla, working with a team specialized in building web services in Python at scale for Firefox. He's contributed to the Python packaging effort and has worked with a lot of different Python web frameworks since Zope in the early days. Tarek has also created Afpy, the French Python User Group, and has written two books on Python in French. He has delivered numerous talks and tutorials in French at international events such as Solutions Linux, PyCon, OSCON, and EuroPython. www.it-ebooks.info

📄 Page 6

About the Reviewer Facundo Batista is a specialist in the Python programming language, with more than 15 years of experience with it. He is a core developer of the language, and a member by merit of the Python Software Foundation. He also received the 2009 Community Service Award for organizing PyCon Argentina and the Argentinian Python community as well as contributions to the standard library and work in translating the Python documentation. He delivers talks in the main Python conferences in Argentina and other countries (The United States and Europe). In general, he has strong distributed collaborative experience from being involved in FLOSS development and working with people around the globe for more than 10 years. He worked as a telecommunication engineer at Telefónica Móviles and Ericsson, and as a Python expert at Cyclelogic (developer in chief) and Canonical (senior software developer, his current position). He also loves playing tennis, and is a father of two wonderful children. www.it-ebooks.info

📄 Page 7

www.PacktPub.com eBooks, discount offers, and more Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at customercare@packtpub.com for more details. At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks. TM https://www2.packtpub.com/books/subscription/packtlib Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can search, access, and read Packt's entire library of books. Why subscribe? • Fully searchable across every book published by Packt • Copy and paste, print, and bookmark content • On demand and accessible via a web browser www.it-ebooks.info

📄 Page 8

[ i ] Table of Contents Preface xi Chapter 1: Current Status of Python 1 Where are we now and where we are going? 2 Why and how does Python change? 2 Getting up to date with changes – PEP documents 3 Python 3 adoption at the time of writing this book 4 The main differences between Python 3 and Python 2 5 Why should I care? 5 The main syntax differences and common pitfalls 5 Syntax changes 6 Changes in the standard library 7 Changes in datatypes and collections 8 The popular tools and techniques used for maintaining cross-version compatibility 8 Not only CPython 12 Why should I care? 13 Stackless Python 13 Jython 14 IronPython 14 PyPy 15 Modern approaches to Python development 16 Application-level isolation of Python environments 17 Why isolation? 19 Popular solutions 21 virtualenv 21 venv 23 buildout 24 Which one to choose? 24 www.it-ebooks.info

📄 Page 9

Table of Contents [ ii ] System-level environment isolation 25 Virtual development environments using Vagrant 26 Containerization versus virtualization 27 Popular productivity tools 28 Custom Python shells – IPython, bpython, ptpython, and so on 29 Setting up the PYTHONSTARTUP environment variable 30 IPython 30 bpython 30 ptpython 31 Interactive debuggers 31 Useful resources 32 Summary 33 Chapter 2: Syntax Best Practices – below the Class Level 35 Python's built-in types 36 Strings and bytes 36 Implementation details 38 String concatenation 39 Collections 40 Lists and tuples 40 Dictionaries 45 Sets 49 Beyond basic collections – the collections module 50 Advanced syntax 51 Iterators 51 The yield statement 52 Decorators 56 General syntax and possible implementations 57 Usage and useful examples 61 Context managers – the with statement 68 General syntax and possible implementations 69 Other syntax elements you may not know yet 72 The for … else … statement 73 Function annotations 73 The general syntax 74 The possible uses 74 Summary 75 Chapter 3: Syntax Best Practices – above the Class Level 77 Subclassing built-in types 78 Accessing methods from superclasses 80 Old-style classes and super in Python 2 82 Understanding Python's Method Resolution Order 83 www.it-ebooks.info

📄 Page 10

Table of Contents [ iii ] super pitfalls 87 Mixing super and explicit class calls 87 Heterogeneous arguments 89 Best practices 90 Advanced attribute access patterns 91 Descriptors 92 Real-life example – lazily evaluated attributes 95 Properties 98 Slots 101 Metaprogramming 102 Decorators – a method of metaprogramming 103 Class decorators 103 Using the __new__() method to override instance creation process 105 Metaclasses 108 The general syntax 109 New Python 3 syntax for metaclasses 112 Metaclass usage 115 Metaclass pitfalls 115 Some tips on code generation 116 exec, eval, and compile 117 Abstract Syntax Tree 118 Projects using code generation patterns 120 Summary 123 Chapter 4: Choosing Good Names 125 PEP 8 and naming best practices 125 Why and when to follow PEP 8? 126 Beyond PEP 8 – team-specific style guidelines 126 Naming styles 127 Variables 127 Constants 128 Naming and usage 129 Public and private variables 130 Functions and methods 131 The private controversy 132 Special methods 134 Arguments 134 Properties 134 Classes 135 Modules and packages 135 The naming guide 136 Using the has or is prefix for Boolean elements 136 Using plurals for variables that are collections 136 Using explicit names for dictionaries 136 www.it-ebooks.info

📄 Page 11

Table of Contents [ iv ] Avoiding generic names 136 Avoiding existing names 138 Best practices for arguments 138 Building arguments by iterative design 139 Trust the arguments and your tests 139 Using *args and **kwargs magic arguments carefully 141 Class names 143 Module and package names 143 Useful tools 144 Pylint 144 pep8 and flake8 146 Summary 147 Chapter 5: Writing a Package 149 Creating a package 149 The confusing state of Python packaging tools 150 The current landscape of Python packaging thanks to PyPA 150 Tool recommendations 151 Project configuration 152 setup.py 152 setup.cfg 153 MANIFEST.in 154 Most important metadata 154 Trove classifiers 155 Common patterns 156 The custom setup command 161 Working with packages during development 161 setup.py install 162 Uninstalling packages 162 setup.py develop or pip -e 162 Namespace packages 163 Why is it useful? 163 PEP 420 – implicit namespace packages 166 Namespace packages in previous Python versions 167 Uploading a package 168 PyPI – Python Package Index 169 Uploading to PyPI – or other package index 169 .pypirc 170 Source packages versus built packages 171 sdist 171 bdist and wheels 172 Standalone executables 174 When are standalone executables useful? 175 www.it-ebooks.info

📄 Page 12

Table of Contents [ v ] Popular tools 176 PyInstaller 177 cx_Freeze 181 py2exe and py2app 183 Security of Python code in executable packages 184 Making decompilation harder 185 Summary 186 Chapter 6: Deploying Code 187 The Twelve-Factor App 188 Deployment automation using Fabric 189 Your own package index or index mirror 195 PyPI mirroring 196 Deployment using a package 197 Common conventions and practices 207 The filesystem hierarchy 207 Isolation 207 Using process supervision tools 208 Application code should be run in user space 210 Using reverse HTTP proxies 210 Reloading processes gracefully 211 Code instrumentation and monitoring 212 Logging errors – sentry/raven 213 Monitoring system and application metrics 215 Dealing with application logs 218 Basic low-level log practices 218 Tools for log processing 220 Summary 223 Chapter 7: Python Extensions in Other Languages 225 Different language means – C or C++ 226 How do extensions in C or C++ work 226 Why you might want to use extensions 228 Improving performance in critical code sections 228 Integrating existing code written in different languages 229 Integrating third-party dynamic libraries 230 Creating custom datatypes 230 Writing extensions 230 Pure C extensions 231 A closer look at Python/C API 235 Calling and binding conventions 240 Exception handling 242 Releasing GIL 244 Reference counting 246 www.it-ebooks.info

📄 Page 13

Table of Contents [ vi ] Cython 248 Cython as a source to source compiler 248 Cython as a language 250 Challenges 253 Additional complexity 253 Debugging 254 Interfacing with dynamic libraries without extensions 255 ctypes 255 Loading libraries 255 Calling C functions using ctypes 257 Passing Python functions as C callbacks 258 CFFI 262 Summary 263 Chapter 8: Managing Code 265 Version control systems 265 Centralized systems 266 Distributed systems 268 Distributed strategies 270 Centralized or distributed? 271 Use Git if you can 271 Git flow and GitHub flow 272 Continuous development processes 276 Continuous integration 277 Testing every commit 278 Merge testing through CI 279 Matrix testing 280 Continuous delivery 280 Continuous deployment 281 Popular tools for continuous integration 282 Jenkins 282 Buildbot 286 Travis CI 288 GitLab CI 290 Choosing the right tool and common pitfalls 290 Problem 1 – too complex build strategies 291 Problem 2 – too long building time 291 Problem 3 – external job definitions 292 Problem 4 – lack of isolation 293 Summary 294 Chapter 9: Documenting Your Project 295 The seven rules of technical writing 295 Write in two steps 296 Target the readership 297 www.it-ebooks.info

📄 Page 14

Table of Contents [ vii ] Use a simple style 298 Limit the scope of information 299 Use realistic code examples 299 Use a light but sufficient approach 300 Use templates 301 A reStructuredText primer 301 Section structure 303 Lists 305 Inline markup 306 Literal block 306 Links 307 Building the documentation 308 Building the portfolio 308 Design 309 Usage 310 Operations 315 Making your own portfolio 315 Building the landscape 316 Producer's layout 317 Consumer's layout 318 Documentation building and continuous integration 322 Summary 323 Chapter 10: Test-Driven Development 325 I don't test 325 Test-driven development principles 326 Preventing software regression 328 Improving code quality 329 Providing the best developer documentation 329 Producing robust code faster 330 What kind of tests? 330 Acceptance tests 330 Unit tests 331 Functional tests 331 Integration tests 332 Load and performance testing 332 Code quality testing 333 Python standard test tools 333 unittest 333 doctest 337 I do test 339 unittest pitfalls 339 unittest alternatives 340 nose 340 py.test 344 www.it-ebooks.info

📄 Page 15

Table of Contents [ viii ] Testing coverage 348 Fakes and mocks 351 Building a fake 351 Using mocks 356 Testing environment and dependency compatibility 358 Dependency matrix testing 358 Document-driven development 362 Writing a story 362 Summary 364 Chapter 11: Optimization – General Principles and Profiling Techniques 365 The three rules of optimization 365 Make it work first 366 Work from the user's point of view 367 Keep the code readable and maintainable 367 Optimization strategy 368 Find another culprit 368 Scale the hardware 368 Writing a speed test 369 Finding bottlenecks 370 Profiling CPU usage 370 Macro-profiling 371 Micro-profiling 375 Measuring Pystones 378 Profiling memory usage 379 How Python deals with memory 379 Profiling memory 382 C code memory leaks 390 Profiling network usage 391 Summary 392 Chapter 12: Optimization – Some Powerful Techniques 393 Reducing the complexity 394 Cyclomatic complexity 396 The big O notation 396 Simplifying 399 Searching in a list 399 Using a set instead of a list 400 Using collections 401 deque 401 defaultdict 403 namedtuple 404 www.it-ebooks.info

📄 Page 16

Table of Contents [ ix ] Using architectural trade-offs 405 Using heuristics and approximation algorithms 405 Using task queues and delayed processing 406 Using probabilistic data structures 410 Caching 411 Deterministic caching 412 Nondeterministic caching 415 Cache services 416 Memcached 416 Summary 419 Chapter 13: Concurrency 421 Why concurrency? 422 Multithreading 423 What is multithreading? 424 How Python deals with threads 425 When should threading be used? 426 Building responsive interfaces 426 Delegating work 426 Multiuser applications 427 An example of a threaded application 428 Multiprocessing 442 The built-in multiprocessing module 445 Using process pools 449 Using multiprocessing.dummy as a multithreading interface 450 Asynchronous programming 451 Cooperative multitasking and asynchronous I/O 452 Python async and await keywords 453 asyncio in older versions of Python 457 A practical example of asynchronous programming 458 Integrating nonasynchronous code with async using futures 461 Executors and futures 462 Using executors in an event loop 464 Summary 465 Chapter 14: Useful Design Patterns 467 Creational patterns 467 Singleton 468 Structural patterns 471 Adapter 472 Interfaces 474 Proxy 487 Facade 489 www.it-ebooks.info

📄 Page 17

Table of Contents [ x ] Behavioral patterns 490 Observer 490 Visitor 492 Template 495 Summary 498 Index 499 www.it-ebooks.info

📄 Page 18

[ xi ] Preface Python rocks! From the earliest version in the late 1980s to the current version, it has evolved with the same philosophy: providing a multiparadigm programming language with readability and productivity in mind. People used to see Python as yet another scripting language and wouldn't feel right about using it to build large systems. However, over the years and thanks to some pioneer companies, it became obvious that Python could be used to build almost any kind of system. In fact, many developers that come from another language are charmed by Python and make it their language of choice. This is something you are probably aware of if you have bought this book, so there's no need to convince you about the merits of the language any further. This book is written to express many years of experience of building all kinds of applications with Python, from small system scripts done in a couple of hours to very large applications written by dozens of developers over several years. It describes the best practices used by developers when working with Python. This book covers some topics that do not focus on the language itself but rather on the tools and techniques used to work with it. In other words, this book describes how an advanced Python developer works every day. www.it-ebooks.info

📄 Page 19

Preface [ xii ] What this book covers Chapter 1, Current Status of Python, showcases the current state of the Python language and its community. It shows how Python is constantly changing, why it is changing, and also why these facts are important for anyone who wants to call themselves a Python professional. This chapter also features the most popular and canonical ways of working in Python—popular productivity tools and conventions that are de facto standards now. Chapter 2, Syntax Best Practices – below the Class Level, presents iterators, generators, descriptors, and so on, in an advanced way. It also covers useful notes about Python idioms and internal CPython types implementations with their computational complexities as a rationale for showcased idioms. Chapter 3, Syntax Best Practices – above the Class Level, explains syntax best practices, but focuses above the class level. It covers more advanced object-oriented concepts and mechanisms available in Python. This knowledge is required in order to understand the last section of the chapter, which presents different approaches to metaprogramming in Python. Chapter 4, Choosing Good Names, involves choosing good names. It is an extension to PEP 8 with naming best practices, but also gives tips on designing good APIs. Chapter 5, Writing a Package, explains how to create the Python package and which tools to use in order to properly distribute it on the official Python Package Index or any other package repository. Information about packages is supplemented with a brief review of the tools that allow you to create standalone executables from Python sources. Chapter 6, Deploying Code, aims mostly at Python web developers and backend engineers, because it deals with code deployments. It explains how Python applications should be built in order to be easily deployed to remote servers and what tools you can use in order to automate that process. This chapter dovetails with Chapter 5, Writing a Package, because it shows how packages and private package repositories can be used to streamline your application deployments. Chapter 7, Python Extensions in Other Languages, explains why writing C extensions for Python might be a good solution sometimes. It also shows that it is not as hard as it seems to be as long as the proper tools are used. Chapter 8, Managing Code, gives some insight into how a project code base can be managed and explains how to set up various continuous development processes. Chapter 9, Documenting Your Project, covers documentation and provides tips on technical writing and how Python projects should be documented. www.it-ebooks.info

📄 Page 20

Preface [ xiii ] Chapter 10, Test-Driven Development, explains the basic principles of test-driven development and the tools that can be used in this development methodology. Chapter 11, Optimization – General Principles and Profiling Techniques, explains optimization. It provides profiling techniques and an optimization strategy guideline. Chapter 12, Optimization – Some Powerful Techniques, extends Chapter 11, Optimization – General Principles and Profiling Techniques, by providing some common solutions to the performance problems that are often found in Python programs. Chapter 13, Concurrency, introduces the vast topic of concurrency in Python. It explains what concurrency is, when it might be necessary to write concurrent applications, and what are the main approaches to concurrency for Python programmers. Chapter 14, Useful Design Patterns, concludes the book with a set of useful design patterns and example implementations in Python. What you need for this book This book is written for developers who work under any operating system for which Python 3 is available. This is not a book for beginners, so I assume you have Python installed in your environment or know how to install it. Anyway, this book takes into account the fact that not everyone needs to be fully aware of the latest Python features or officially recommended tools. This is why the first chapter provides a recap of common utilities (such as virtual environments and pip) that are now considered standard tools of professional Python developers. Who this book is for This book is written for Python developers who wish to go further in mastering Python. And by developers I mean mostly professionals, so programmers who write software in Python for a living. This is because it focuses mostly on tools and practices that are crucial for creating performant, reliable, and maintainable software in Python. It does not mean that hobbyists won't find anything interesting. This book should be great for anyone who is interested in learning advance-level concepts with Python. Anyone who has basic Python skills should be able to follow the content of the book, although it might require some additional effort from less experienced programmers. It should also be a good introduction to Python 3.5 for those who are still a bit behind and continue to use Python in version 2.7 or older. www.it-ebooks.info