Expert Python Programming, Fourth Edition (Michał Jaworski, Tarek Ziadé) (z-library.sk, 1lib.sk, z-lib.sk)

📄 File Format: PDF

💾 File Size: 4.3 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

(This page has no text content)

📄 Page 2

Expert Python Programming Fourth Edition Master Python by learning the best coding practices and advanced programming concepts Michał Jaworski Tarek Ziadé BIRMINGHAM—MUMBAI "Python" and the Python Logo are trademarks of the Python Software Foundation.

📄 Page 3

Expert Python Programming Fourth Edition Copyright © 2021 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. Producer: Shailesh Jain Acquisition Editor – Peer Reviews: Saby D'silva Project Editor: Rianna Rodrigues Content Development Editor: Edward Doxey Copy Editor: Safis Editor Technical Editor: Aditya Sawant Proofreader: Safis Editor Indexer: Pratik Shirodkar Presentation Designer: Pranit Padwal

📄 Page 4

First published: September 2008 Second edition: May 2016 Third edition: April 2019 Fourth edition: May 2021 Production reference: 2291221 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-80107-110-9 www.packt.com

📄 Page 5

Contributors About the authors Michał Jaworski has more than 10 years of professional experience in writing software using various programming languages. Michał has spent most of his career writing high-performance and distributed backend services for web applications. He has served in various roles at multiple companies: from an ordinary software engineer to lead software architect. His beloved language of choice has always been Python. I want to thank my wife for giving me constant support. Oliwia, you're the only person who knew from the very beginning that I was lying to myself when I said that this will be a light project that won't take much of my (our) time. I don't know why, but you encouraged me to do it anyway. Tarek Ziadé is a software engineer, located in Burgundy, France. He works at Elastic, building tools for developers. Before Elastic, he worked at Mozilla for 10 years, and he founded a French Python User group, called AFPy. Tarek has also written several articles about Python for various magazines, and a few books in French and English. I would like to thank Freya, Suki, Milo, and Amina for being so supportive of all my book projects.

📄 Page 6

About the reviewer Tal Einat has been developing software for nearly 20 years, of which Python has always been a major element. He's been a core developer of the Python language since 2010. Tal holds a B.Sc. in Math & Physics from Tel Aviv University. He likes hiking, computer games, philosophical sci-fi, and spending time with his family. For the past eight years, Tal has been developing educational technology, first at Compedia where he built a group developing VR and AR education apps, and later at the startup FullProof of which he was a co-founder. Tal currently works at Rhino Health, a startup working to the enable development and use of medical AI models with patient data from across the globe while preserving patient privacy. I dedicate my part of the work on this book to my grandfather Jacob "Yanek" Friedman who passed away recently — you live on in our thoughts, making us strong and putting a smile on our faces. I'd like to thank my wife, children, brother, parents and extended family, for being ever supportive and filling my life with so much joy.

📄 Page 7

Contents   1. Preface 1. Who this book is for 2. What this book covers 3. To get the most out of this book 4. Get in touch 1. Current Status of Python 1. Where are we now and where are we going? 2. What to do with Python 2 3. Keeping up to date 1. PEP documents 2. Active communities 3. Other resources 4. Summary 2. Modern Python Development Environments 1. Technical requirements 2. Python's packaging ecosystem 1. Installing Python packages using pip 3. Isolating the runtime environment 1. Application-level isolation versus system-level isolation 4. Application-level environment isolation 1. Poetry as a dependency management system 5. System-level environment isolation 1. Containerization versus virtualization 2. Virtual environments using Docker 1. Writing your first Dockerfile 2. Running containers 3. Setting up complex environments 4. Useful Docker and Docker Compose recipes for Python 3. Virtual development environments using Vagrant 6. Popular productivity tools 1. Custom Python shells 2. Using IPython 3. Incorporating shells in your own scripts and programs 4. Interactive debuggers 5. Other productivity tools 7. Summary

📄 Page 8

3. New Things in Python 1. Technical requirements 2. Recent language additions 1. Dictionary merge and update operators 1. Alternative – Dictionary unpacking 2. Alternative – ChainMap from the collections module 2. Assignment expressions 3. Type-hinting generics 4. Positional-only parameters 5. zoneinfo module 6. graphlib module 3. Not that new, but still shiny 1. breakpoint() function 2. Development mode 3. Module-level __getattr__() and __dir__() functions 4. Formatting strings with f-strings 5. Underscores in numeric literals 6. secrets module 4. What may come in the future? 1. Union types with the | operator 2. Structural pattern matching 5. Summary 4. Python in Comparison with Other Languages 1. Technical requirements 2. Class model and object-oriented programming 1. Accessing super-classes 2. Multiple inheritance and Method Resolution Order 3. Class instance initialization 4. Attribute access patterns 5. Descriptors 1. Real-life example – lazily evaluated attributes 6. Properties 3. Dynamic polymorphism 1. Operator overloading 1. Dunder methods (language protocols) 2. Comparison to C++ 2. Function and method overloading 1. Single-dispatch functions 4. Data classes 5. Functional programming

📄 Page 9

1. Lambda functions 2. The map(), filter(), and reduce() functions 3. Partial objects and partial functions 4. Generators 5. Generator expressions 6. Decorators 6. Enumerations 7. Summary 5. Interfaces, Patterns, and Modularity 1. Technical requirements 2. Interfaces 1. A bit of history: zope.interface 2. Using function annotations and abstract base classes 1. Using collections.abc 3. Interfaces through type annotations 3. Inversion of control and dependency injection 1. Inversion of control in applications 2. Using dependency injection frameworks 4. Summary 6. Concurrency 1. Technical requirements 2. What is concurrency? 3. Multithreading 1. What is multithreading? 2. How Python deals with threads 3. When should we use multithreading? 1. Application responsiveness 2. Multiuser applications 3. Work delegation and background processing 4. An example of a multithreaded application 1. Using one thread per item 2. Using a thread pool 3. Using two-way queues 4. Dealing with errors in threads 5. Throttling 4. Multiprocessing 1. The built-in multiprocessing module 2. Using process pools 3. Using multiprocessing.dummy as the multithreading interface 5. Asynchronous programming

📄 Page 10

1. Cooperative multitasking and asynchronous I/O 2. Python async and await keywords 3. A practical example of asynchronous programming 4. Integrating non-asynchronous code with async using futures 1. Executors and futures 2. Using executors in an event loop 6. Summary 7. Event-Driven Programming 1. Technical requirements 2. What exactly is event-driven programming? 1. Event-driven != asynchronous 2. Event-driven programming in GUIs 3. Event-driven communication 3. Various styles of event-driven programming 1. Callback-based style 2. Subject-based style 3. Topic-based style 4. Event-driven architectures 1. Event and message queues 5. Summary 8. Elements of Metaprogramming 1. Technical requirements 2. What is metaprogramming? 3. Using decorators to modify function behavior before use 1. One step deeper: class decorators 4. Intercepting the class instance creation process 5. Metaclasses 1. The general syntax 2. Metaclass usage 3. Metaclass pitfalls 4. Using the __init__subclass__() method as an alternative to metaclasses 6. Code generation 1. exec, eval, and compile 2. The abstract syntax tree 3. Import hooks 4. Notable examples of code generation in Python 1. Falcon's compiled router 2. Hy 7. Summary

📄 Page 11

9. Bridging Python with C and C++ 1. Technical requirements 2. C and C++ as the core of Python extensibility 3. Compiling and loading Python C extensions 4. The need to use extensions 1. Improving performance in critical code sections 2. Integrating existing code written in different languages 3. Integrating third-party dynamic libraries 4. Creating efficient custom datatypes 5. Writing extensions 1. Pure C extensions 1. A closer look at the Python/C API 2. Calling and binding conventions 3. Exception handling 4. Releasing GIL 5. Reference counting 2. Writing extensions with Cython 1. Cython as a source-to-source compiler 2. Cython as a language 6. Downsides of using extensions 1. Additional complexity 2. Harder debugging 7. Interfacing with dynamic libraries without extensions 1. The ctypes module 1. Loading libraries 2. Calling C functions using ctypes 3. Passing Python functions as C callbacks 2. CFFI 8. Summary 10. Testing and Quality Automation 1. Technical requirements 2. The principles of test-driven development 3. Writing tests with pytest 1. Test parameterization 2. pytest's fixtures 3. Using fakes 4. Mocks and the unittest.mock module 4. Quality automation 1. Test coverage 2. Style fixers and code linters

📄 Page 12

3. Static type analysis 5. Mutation testing 6. Useful testing utilities 1. Faking realistic data values 2. Faking time values 7. Summary 11. Packaging and Distributing Python Code 1. Technical requirements 2. Packaging and distributing libraries 1. The anatomy of a Python package 1. setup.py 2. setup.cfg 3. MANIFEST.in 4. Essential package metadata 5. Trove classifiers 2. Types of package distributions 1. sdist distributions 2. bdist and wheel distributions 3. Registering and publishing packages 4. Package versioning and dependency management 1. The SemVer standard for semantic versioning 2. CalVer for calendar versioning 5. Installing your own packages 1. Installing packages directly from sources 2. Installing packages in editable mode 6. Namespace packages 7. Package scripts and entry points 3. Packaging applications and services for the web 1. The Twelve-Factor App manifesto 2. Leveraging Docker 3. Handling environment variables 4. The role of environment variables in application frameworks 4. Creating standalone executables 1. When standalone executables are useful 2. Popular tools 1. PyInstaller 2. cx_Freeze 3. py2exe and py2app 3. Security of Python code in executable packages 5. Summary

📄 Page 13

12. Observing Application Behavior and Performance 1. Technical requirements 2. Capturing errors and logs 1. Python logging essentials 1. Logging system components 2. Logging configuration 2. Good logging practices 3. Distributed logging 4. Capturing errors for later review 3. Instrumenting code with custom metrics 1. Using Prometheus 4. Distributed application tracing 1. Distributed tracing with Jaeger 5. Summary 13. Code Optimization 1. Technical requirements 2. Common culprits for bad performance 1. Code complexity 1. Cyclomatic complexity 2. The big O notation 2. Excessive resource allocation and leaks 3. Excessive I/O and blocking operations 3. Code profiling 1. Profiling CPU usage 1. Macro-profiling 2. Micro-profiling 2. Profiling memory usage 1. Using the objgraph module 2. C code memory leaks 4. Reducing complexity by choosing appropriate data structures 1. Searching in a list 2. Using sets 3. Using the collections module 1. deque 2. defaultdict 3. namedtuple 5. Leveraging architectural trade-offs 1. Using heuristics and approximation algorithms 2. Using task queues and delayed processing 3. Using probabilistic data structures

📄 Page 14

4. Caching 1. Deterministic caching 2. Non-deterministic caching 6. Summary 14. Other Books You May Enjoy 15. Index Landmarks   1. Cover 2. Index

📄 Page 15

Preface Python rocks! From the earliest version in the late 1980s to the current 3.9 version, Python has evolved with the same philosophy: providing a multi-paradigm programming language with readability and productivity in mind. Initially, people used to see Python as yet another scripting language. Many of them didn't believe it could be used to build large and complex systems. But over the years, and thanks to some pioneer companies, it became obvious that Python could be used to build almost any kind of a software. Although writing Python code is easy, making it readable, reusable, and easy to maintain is challenging. You can achieve those qualities only through good software artistry and technique, which you will build gradually by constantly learning and gaining experience. This book was written to express many years of professional experience in building all kinds of applications with Python, from small system scripts done in a couple of hours to very large applications written by dozens of developers over several years. This book is divided into three parts:   1. Knowing your tools: Chapters 1 to 4 focus on basic elements of Python programmer's toolbelt. From productivity tools, through modern environments, to the newest syntax elements introduced in the latest Python releases. It also offers a safe landing zone for programmers who have experience with other programming languages and are just starting to learn more advanced Python. 2. Building applications with Python: Chapters 5 to 9 are all about design patterns, programming paradigms, and metaprogramming techniques. We will try to build some small but useful programs and will be often taking a deeper look into application architecture. We will also go a bit beyond Python and see how we can integrate code written using other programming languages.

📄 Page 16

3. Maintaining Python applications: Chapters 10 to 13 will be discussing all the things that usually happen after the application "goes live". We will showcase tools and techniques that help to keep applications easily maintainable and show how to approach common problems with packaging, deployment, monitoring, and performance optimization.

📄 Page 17

Who this book is for The Python programming book is intended for expert programmers who want to learn about Python's advanced-level concepts and latest features. This book is written for Python developers who wish to go further in mastering Python. And by developers, I mean mostly professional programmers who write Python software for a living. This is because it focuses mostly on tools and practices that are crucial for creating performant, reliable, and maintainable software in Python. However, this does not mean that hobbyists won't find anything interesting. This book is great for anyone who is interested in learning advanced-level concepts with Python. Anyone who has basic Python skills should be able to follow the content of the book, although it might require some additional effort from less experienced programmers. It should also be a good introduction to the newest releases of Python for those who are still a bit behind and continue to use older versions of Python.

📄 Page 18

What this book covers Chapter 1, Current Status of Python, showcases the current state of the Python language and its community. We will see how Python is constantly changing and why it is changing. We will learn what to do with old Python 2 code and how to be constantly up to date with what is currently happening in the Python community. Chapter 2, Modern Python Development Environments, describes modern ways of setting up repeatable and consistent development environments for Python programmers. We will learn differences between application-level and system- level isolation. We will concentrate on two popular tools for environment isolation, virtualenv-type environments and Docker containers, but will also review other alternatives. At the end of the chapter, we will discuss common productivity tools that are extremely useful during development. Chapter 3, New Things in Python, showcases recent Python language additions. We will review the most important Python syntax changes that happened in the last four releases of Python. We will also take a look at exciting changes that are scheduled for the next major Python release—Python 3.10. Chapter 4, Python in Comparison with Other Languages, shows how Python compares to other languages. We will learn what programming idioms are and how to recognize them in code. We will take a deeper look into key elements of Python's object-oriented programming model and how it is different from other object-oriented programming languages but will also discuss other popular programming language features like descriptors, decorators, and dataclasses. This chapter should allow programmers with experience in other languages to safely land in the Python ecosystem. Chapter 5, Interfaces, Patterns, and Modularity, discusses elements of Python that allow for implementing various reusable design patterns. If focuses on the concept of class interfaces and how they can be implemented in Python. It also discusses inversion of control and dependency injection—two extremely useful but not necessarily popular programming techniques.

📄 Page 19

Chapter 6, Concurrency, explains how to implement concurrency in Python using different approaches and libraries. It features three major concurrency models: multithreading, multiprocessing and asynchronous programming. In this chapter we will learn key differences between those models and how to use them effectively. Chapter 7, Event-Driven Programming, describes what event-driven programming is and how it relates to asynchronous programming and different concurrency models. We will present various approaches to event-driven programming along with useful libraries. Chapter 8, Elements of Metaprogramming, presents an overview of common approaches to metaprogramming available to Python programmers. We will learn about common metaprogramming techniques like decorators, as well as metaclasses and code generation patterns. Chapter 9, Bridging Python with C and C++, explains how to integrate code written in different languages in your Python application. We will learn when extensions in C can be useful and how to create them. Chapter 10, Testing and Quality Automation, is about providing automated testing and quality processes. We will learn about a popular testing framework — Pytest —and many useful testing techniques. We will also cover tools that can be used to assess code quality metrics and improve code style in fully automated way. Chapter 11, Packaging and Distributing Python Code, describes the current state of Python packaging and best practices for creating packages that are to be distributed as open source code in the Python Package Index (PyPI). We will also cover the topics of packaging applications for web development and creating standalone Python executables for desktop applications. Chapter 12, Observing Application Behavior and Performance, discusses the topic of application observability. We will learn about Python logging systems, how to monitor application metrics and perform distributed transaction tracing. We will also learn how to scale simple observability practices to large-scale distributed systems. Chapter 13, Code Optimization, discusses the basic rules of optimization that every developer should be aware of. We will learn how to identify application performance bottlenecks and how to use common profiling tools. We will also

📄 Page 20

learn common optimization techniques and strategies that can be easily applied in many situations once you know where the bottleneck is. To get the most out of this book This book is written for developers who work under any operating system for which Python 3 is available. This is not a book for beginners, so I assume you have Python installed in your environment or know how to install it. Anyway, this book takes into account the fact that not everyone needs to be fully aware of the latest Python features or officially recommended tools. This is why Chapter 2, Modern Python Development Environments provides an overview of recommended techniques and tools (such as virtual environments and  pip ) for setting up development environments. Download the example code files The code bundle for the book is also hosted on GitHub at https://github.com/PacktPublishing/Expert-Python-Programming-Fourth- Edition. In case there's an update to the code, it will be updated on the existing GitHub repository. We also have other code bundles from our rich catalog of books and videos available at https://github.com/PacktPublishing/. Check them out! Download the color images We also provide a PDF file that has color images of the screenshots/diagrams used in this book. You can download it here: https://static.packt- cdn.com/downloads/9781801071109_ColorImages.pdf.