Functional Python Programming Discover the power of functional programming, generator functions, lazy evaluation, the built-in… (Steven F. Lott) (Z-Library)

📄 File Format: PDF

💾 File Size: 2.5 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

(This page has no text content)

📄 Page 2

Functional Python Programming Second Edition Steven F. Lott BIRMINGHAM - MUMBAI

📄 Page 3

Functional Python Programming Second Edition Copyright 2018 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. Commissioning Editor: Merint Methew Acquisition Editor: Sandeep Mishra Content Development Editor: Priyanka Sawant Technical Editor: Ketan Kamble Copy Editor: Safis Editing Project Coordinator: Vaidehi Sawant Proofreader: Safis Editing Indexer: Mariammal Chettiyar Graphics: Jason Monteiro Production Coordinator: Deepika Naik First published: January 2015 Second edition: April 2018 Production reference: 1120418 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-78862-706-1

📄 Page 4

Mapt is an online digital library that gives you full access to over 5,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website. Why subscribe? Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals Improve your learning with Skill Plans built especially for you Get a free eBook or video every month Mapt is fully searchable Copy and paste, print, and bookmark content PacktPub.com Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at for more details. At , you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks.

📄 Page 5

About the author Steven F. Lott has been programming since the '70s, when computers were large, expensive, and rare. He's been using Python to solve business problems for over 10 years. His other titles with Packt Publishing include Python Essentials, Mastering Object-Oriented Python, Functional Python Programming, and Python for Secret Agents. Steven is currently a technomad who lives in city along the east coast of the U.S. You can follow his technology blog (slott-softwarearchitect).

📄 Page 6

About the reviewer Yogendra Sharma is a developer with experience in architecture, design, and development of scalable and distributed applications. He was awarded a bachelor s degree from the Rajasthan Technical University in computer science. With a core interest in microservices and Spring, he also has hands-on experience in technologies such as AWS Cloud, Python, J2EE, NodeJS, JavaScript, Angular, MongoDB, and Docker. Currently, he works as an IoT and Cloud Architect at Intelizign Engineering Services Pune. Packt is searching for authors like you If you're interested in becoming an author for Packt, please visit and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea.

📄 Page 7

Table of Contents Copyright and Credits 3 Preface 1 Chapter 1: Understanding Functional Programming 8 Identifying a paradigm 9 Subdividing the procedural paradigm 10 Using the functional paradigm 11 Using a functional hybrid 14 Looking at object creation 15 The stack of turtles 16 A classic example of functional programming 17 Exploratory data analysis 20 Summary 21 Chapter 2: Introducing Essential Functional Concepts 22 First-class functions 23 Pure functions 23 Higher-order functions 24 Immutable data 25 Strict and non-strict evaluation 27 Recursion instead of an explicit loop state 29 Functional type systems 33 Familiar territory 33 Learning some advanced concepts 34 Summary 35 Chapter 3: Functions, Iterators, and Generators 36 Writing pure functions 37 Functions as first-class objects 39 Using strings 41 Using tuples and named tuples 42 Using generator expressions 44 Exploring the limitations of generators 46 Combining generator expressions 48 Cleaning raw data with generator functions 49 Using lists, dicts, and sets 50 Using stateful mappings 54 Using the bisect module to create a mapping 56 Using stateful sets 58

📄 Page 8

Table of Contents [ ii ] Summary 58 Chapter 4: Working with Collections 60 An overview of function varieties 61 Working with iterables 61 Parsing an XML file 63 Parsing a file at a higher level 65 Pairing up items from a sequence 67 Using the iter() function explicitly 70 Extending a simple loop 71 Applying generator expressions to scalar functions 74 Using any() and all() as reductions 76 Using len() and sum() 79 Using sums and counts for statistics 79 Using zip() to structure and flatten sequences 82 Unzipping a zipped sequence 84 Flattening sequences 84 Structuring flat sequences 86 Structuring flat sequences – an alternative approach 88 Using reversed() to change the order 89 Using enumerate() to include a sequence number 90 Summary 91 Chapter 5: Higher-Order Functions 92 Using max() and min() to find extrema 93 Using Python lambda forms 97 Lambdas and the lambda calculus 98 Using the map() function to apply a function to a collection 99 Working with lambda forms and map() 100 Using map() with multiple sequences 101 Using the filter() function to pass or reject data 103 Using filter() to identify outliers 104 The iter() function with a sentinel value 105 Using sorted() to put data in order 106 Writing higher-order functions 108 Writing higher-order mappings and filters 108 Unwrapping data while mapping 110 Wrapping additional data while mapping 112 Flattening data while mapping 114 Structuring data while filtering 116 Writing generator functions 117 Building higher-order functions with callables 120 Assuring good functional design 121 Review of some design patterns 123

📄 Page 9

Table of Contents [ iii ] Summary 124 Chapter 6: Recursions and Reductions 126 Simple numerical recursions 127 Implementing tail-call optimization 128 Leaving recursion in place 129 Handling difficult tail-call optimization 130 Processing collections through recursion 131 Tail-call optimization for collections 132 Reductions and folding a collection from many items to one 134 Group-by reduction from many items to fewer 136 Building a mapping with Counter 137 Building a mapping by sorting 138 Grouping or partitioning data by key values 140 Writing more general group-by reductions 143 Writing higher-order reductions 144 Writing file parsers 146 Parsing CSV files 148 Parsing plain text files with headers 150 Summary 153 Chapter 7: Additional Tuple Techniques 154 Using tuples to collect data 155 Using named tuples to collect data 157 Building named tuples with functional constructors 160 Avoiding stateful classes by using families of tuples 161 Assigning statistical ranks 165 Wrapping instead of state changing 167 Rewrapping instead of state changing 168 Computing Spearman rank-order correlation 170 Polymorphism and type-pattern matching 171 Summary 178 Chapter 8: The Itertools Module 179 Working with the infinite iterators 180 Counting with count() 181 Counting with float arguments 182 Re-iterating a cycle with cycle() 184 Repeating a single value with repeat() 186 Using the finite iterators 187 Assigning numbers with enumerate() 188 Running totals with accumulate() 190 Combining iterators with chain() 191 Partitioning an iterator with groupby() 192 Merging iterables with zip_longest() and zip() 194 Filtering with compress() 194

📄 Page 10

Table of Contents [ iv ] Picking subsets with islice() 196 Stateful filtering with dropwhile() and takewhile() 197 Two approaches to filtering with filterfalse() and filter() 198 Applying a function to data via starmap() and map() 199 Cloning iterators with tee() 201 The itertools recipes 201 Summary 203 Chapter 9: More Itertools Techniques 205 Enumerating the Cartesian product 206 Reducing a product 206 Computing distances 208 Getting all pixels and all colors 210 Performance analysis 212 Rearranging the problem 214 Combining two transformations 215 Permuting a collection of values 216 Generating all combinations 218 Recipes 220 Summary 221 Chapter 10: The Functools Module 222 Function tools 223 Memoizing previous results with lru_cache 223 Defining classes with total ordering 225 Defining number classes 228 Applying partial arguments with partial() 230 Reducing sets of data with the reduce() function 231 Combining map() and reduce() 232 Using the reduce() and partial() functions 234 Using the map() and reduce() functions to sanitize raw data 235 Using the groupby() and reduce() functions 236 Summary 239 Chapter 11: Decorator Design Techniques 241 Decorators as higher-order functions 241 Using the functools update_wrapper() functions 246 Cross-cutting concerns 246 Composite design 247 Preprocessing bad data 249 Adding a parameter to a decorator 251 Implementing more complex decorators 253 Complex design considerations 254 Summary 258

📄 Page 11

Table of Contents [ v ] Chapter 12: The Multiprocessing and Threading Modules 259 Functional programming and concurrency 260 What concurrency really means 261 The boundary conditions 261 Sharing resources with process or threads 262 Where benefits will accrue 263 Using multiprocessing pools and tasks 264 Processing many large files 264 Parsing log files – gathering the rows 266 Parsing log lines into namedtuples 267 Parsing additional fields of an Access object 270 Filtering the access details 273 Analyzing the access details 275 The complete analysis process 276 Using a multiprocessing pool for concurrent processing 277 Using apply() to make a single request 280 Using the map_async(), starmap_async(), and apply_async() functions 280 More complex multiprocessing architectures 281 Using the concurrent.futures module 282 Using concurrent.futures thread pools 282 Using the threading and queue modules 283 Designing concurrent processing 284 Summary 286 Chapter 13: Conditional Expressions and the Operator Module 287 Evaluating conditional expressions 288 Exploiting non-strict dictionary rules 289 Filtering true conditional expressions 291 Finding a matching pattern 292 Using the operator module instead of lambdas 293 Getting named attributes when using higher-order functions 295 Starmapping with operators 296 Reducing with operator module functions 298 Summary 299 Chapter 14: The PyMonad Library 301 Downloading and installing 301 Functional composition and currying 302 Using curried higher-order functions 304 Currying the hard way 306 Functional composition and the PyMonad * operator 307 Functors and applicative functors 308 Using the lazy List() functor 310 Monad bind() function and the >> operator 313 Implementing simulation with monads 314

📄 Page 12

Table of Contents [ vi ] Additional PyMonad features 318 Summary 319 Chapter 15: A Functional Approach to Web Services 320 The HTTP request-response model 321 Injecting state through cookies 323 Considering a server with a functional design 324 Looking more deeply into the functional view 324 Nesting the services 325 The WSGI standard 326 Throwing exceptions during WSGI processing 329 Pragmatic WSGI applications 331 Defining web services as functions 331 Creating the WSGI application 332 Getting raw data 335 Applying a filter 337 Serializing the results 337 Serializing data into JSON or CSV formats 339 Serializing data into XML 340 Serializing data into HTML 341 Tracking usage 343 Summary 344 Chapter 16: Optimizations and Improvements 346 Memoization and caching 347 Specializing memoization 348 Tail recursion optimizations 350 Optimizing storage 352 Optimizing accuracy 353 Reducing accuracy based on audience requirements 353 Case study–making a chi-squared decision 354 Filtering and reducing the raw data with a Counter object 355 Reading summarized data 357 Computing sums with a Counter object 358 Computing probabilities from Counter objects 360 Computing expected values and displaying a contingency table 361 Computing the chi-squared value 363 Computing the chi-squared threshold 364 Computing the incomplete gamma function 365 Computing the complete gamma function 368 Computing the odds of a distribution being random 369 Functional programming design patterns 371 Summary 373 Other Books You May Enjoy 375

📄 Page 13

Table of Contents [ vii ] Index 378

📄 Page 14

Preface Functional programming offers a variety of techniques for creating succinct and expressive software. While Python is not a purely functional programming language, we can do a great deal of functional programming in Python. Python has a core set of functional programming features. This lets us borrow many design patterns and techniques from other functional languages. These borrowed concepts can lead us to create succinct and elegant programs. Python's generator expressions, in particular, negate the need to create large in-memory data structures, leading to programs that may execute more quickly because they use fewer resources. We can t easily create purely functional programs in Python. Python lacks a number of features that would be required for this. We don t have unlimited recursion, for example, we don t have lazy evaluation of all expressions, and we don t have an optimizing compiler. There are several key features of functional programming languages that are available in Python. One of the most important ones is the idea of functions being first-class objects. Python also offers a number of higher-order functions. The built-in , , and functions are widely used in this role, and less- obvious are functions such as , , and . We ll look at the core features of functional programming from a Python point of view. Our objective is to borrow good ideas from functional programming languages and use those ideas to create expressive and succinct applications in Python. Who this book is for This book is for programmers who want to create succinct, expressive Python programs by borrowing techniques and design patterns from functional programming languages. Some algorithms can be expressed elegantly in a functional style; we can and should adapt this to make Python programs more readable and maintainable. In some cases, a functional approach to a problem will also lead to extremely high- performance algorithms. Python makes it too easy to create large intermediate data structures, tying up memory (and processor time.) With functional programming design patterns, we can often replace large lists with generator expressions that are equally expressive but take up much less memory and run much more quickly.

📄 Page 15

Preface [ 2 ] What this book covers , Understanding Functional Programming, introduces some of the techniques that characterize functional programming. We ll identify some of the ways to map those features to Python. Finally, we ll also address some ways that the benefits of functional programming accrue when we use these design patterns to build Python applications. , Introducing Essential Functional Concepts, delves into six central features of the functional programming paradigm. We ll look at each in some detail to see how they re implemented in Python. We ll also point out some features of functional languages that don t apply well to Python. In particular, many functional languages have complex type- matching rules required to support compiling and optimizing. , Functions, Iterators, and Generators, will show how to leverage immutable Python objects, and generator expressions adapt functional programming concepts to the Python language. We ll look at some of the built-in Python collections and how we can leverage them without departing too far from functional programming concepts. , Working with Collections, shows how you can use a number of built-in Python functions to operate on collections of data. This chapter will focus on a number of relatively simple functions, such as and , which will reduce a collection of values to a single result. , Higher-Order Functions, examines the commonly-used higher-order functions such as and . It also shows a number of other functions that are also higher-order functions as well as how we can create our own higher-order functions. , Recursions and Reductions, teaches how to design an algorithm using recursion and then optimize it into a high-performance loop. We ll also look at some other reductions that are widely used, including . , Additional Tuple Techniques, showcases a number of ways that we can use immutable tuples (and namedtuples) instead of stateful objects. Immutable objects have a much simpler interface we never have to worry about abusing an attribute and setting an object into some inconsistent or invalid state. , The Itertools Module, examines a number of functions in this standard library module. This collection of functions simplifies writing programs that deal with collections or generator functions.

📄 Page 16

Preface [ 3 ] , More Itertools Techniques, covers the combinatoric functions in the itertools module. These functions are somewhat less useful. This chapter includes some examples that illustrate ill-considered use of these functions and the consequences of combinatoric explosion. , The Functools Module, focuses on how to use some of the functions in this module for functional programming. A few functions in this module are more appropriate for building decorators, and they are left for , Decorator Design Techniques. The other functions, however, provide several more ways to design and implement function programs. , Decorator Design Techniques, looks at how you can look at a decorator as a way to build a composite function. While there is considerable flexibility here, there are also some conceptual limitations: we ll look at ways that overly-complex decorators can become confusing rather than helpful. , The Multiprocessing and Threading Modules, points out an important consequence of good functional design: we can distribute the processing workload. Using immutable objects means that we can t corrupt an object because of poorly-synchronized write operations. , Conditional Expressions and the Operator Module, lists some ways to break out of Python s strict order of evaluation. There are limitations to what we can achieve here. We ll also look at the operator module and how this can lead to slight clarification of some simple kinds of processing. , The PyMonad Library, examines some of the features of the PyMonad library. This provides some additional functional programming features. It also provides a way to learn more about monads. In some functional languages, monads are an important way to force a particular order for operations that might get optimized into an undesirable order. Since Python already has strict ordering of f expressions and statements, the monad feature is more instructive than practical. , A Functional Approach to Web Services, shows how we can think of web services as a nested collection of functions that transform a request into a reply. We ll see ways to leverage functional programming concepts for building responsive, dynamic web content. , Optimizations and Improvements, includes some additional tips on performance and optimization. We ll emphasize techniques such as memoization, because they re easy to implement and can in the right context yield dramatic performance improvements.

📄 Page 17

Preface [ 4 ] To get the most out of this book This book presumes some familiarity with Python 3 and general concepts of application development. We won t look deeply at subtle or complex features of Python; we ll avoid much consideration of the internals of the language. We ll presume some familiarity with functional programming. Since Python is not a functional programming language, we can t dig deeply into functional concepts. We ll pick and choose the aspects of functional programming that fit well with Python and leverage just those that seem useful. Some of the examples use exploratory data analysis (EDA) as a problem domain to show the value of functional programming. Some familiarity with basic probability and statistics will help with this. There are only a few examples that move into more serious data science. You ll need to have Python 3.6 installed and running. For more information on Python, visit . The examples all make extensive use of type hints, which means that the latest version of mypy must be installed as well. Check out for the latest version of mypy. Examples in , More Itertools Techniques, use PIL and Beautiful Soup 4. The Pillow fork of the original PIL library works nicely; refer to and . Examples in , The PyMonad Library, use PyMonad; check out . All of these packages should be installed using the following: $ pip install pillow beautifulsoup4 PyMonad Download the example code files You can download the example code files for this book from your account at . If you purchased this book elsewhere, you can visit and register to have the files emailed directly to you.

📄 Page 18

Preface [ 5 ] You can download the code files by following these steps: Log in or register at .1. Select the SUPPORT tab.2. Click on Code Downloads & Errata.3. Enter the name of the book in the Search box and follow the onscreen4. instructions. Once the file is downloaded, please make sure that you unzip or extract the folder using the latest version of: WinRAR/7-Zip for Windows Zipeg/iZip/UnRarX for Mac 7-Zip/PeaZip for Linux The code bundle for the book is also hosted on GitHub at . We also have other code bundles from our rich catalog of books and videos available at . Check them out! Conventions used There are a number of text conventions used throughout this book. : Indicates code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is an example: "Python has other statements, such as or , which modify the rules for variables in a particular namespace."

📄 Page 19

Preface [ 6 ] A block of code is set as follows: When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold: s += n Any command-line input or output is written as follows: $ pip install pillow beautifulsoup4 PyMonad Bold: Indicates a new term, an important word, or words that you see onscreen. For example, words in menus or dialog boxes appear in the text like this. Here is an example: "For our purposes, we will distinguish between only two of the many paradigms: functional programming and imperative programming." Warnings or important notes appear like this. Tips and tricks appear like this. Get in touch Feedback from our readers is always welcome. General feedback: Email and mention the book title in the subject of your message. If you have questions about any aspect of this book, please email us at .

📄 Page 20

Preface [ 7 ] Errata: Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you have found a mistake in this book, we would be grateful if you would report this to us. Please visit , selecting your book, clicking on the Errata Submission Form link, and entering the details. Piracy: If you come across any illegal copies of our works in any form on the Internet, we would be grateful if you would provide us with the location address or website name. Please contact us at with a link to the material. If you are interested in becoming an author: If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, please visit . Reviews Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions, we at Packt can understand what you think about our products, and our authors can see your feedback on their book. Thank you! For more information about Packt, please visit .