Page
1
M A N N I N G Boris Paskhaver
Page
4
Pandas in Action BORIS PASKHAVER M A N N I N G SHELTER ISLAND
Page
5
For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact Special Sales Department Manning Publications Co. 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 Email: orders@manning.com ©2021 by Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps. Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine. Manning Publications Co. Development editor: Sarah Miller 20 Baldwin Road Technical development editor: Al Krinker PO Box 761 Review editor: Aleks Dragosavljević Shelter Island, NY 11964 Production editor: Deirdre S. Hiam Copy editor: Keir Simpson Proofreader: Jason Everett Technical proofreader: Mathijs Affourtit Typesetter and cover designer: Marija Tudor ISBN 9781617297434 Printed in the United States of America
Page
6
For Meredith Edwards, my ray of sunshine
Page
8
contents preface xiii acknowledgments xv about this book xvii about the author xx about the cover illustration xxi PART 1 CORE PANDAS .................................................. 1 1 Introducing pandas 3 1.1 Data in the 21st century 4 1.2 Introducing pandas 4 Pandas vs. graphical spreadsheet applications 6 ■ Pandas vs. its competitors 8 1.3 A tour of pandas 9 Importing a data set 9 ■ Manipulating a DataFrame 11 Counting values in a Series 14 ■ Filtering a column by one or more criteria 15 ■ Grouping data 18 2 The Series object 22 2.1 Overview of a Series 23 Classes and instances 24 ■ Populating the Series with values 24 Customizing the Series index 26 ■ Creating a Series with missing values 29vii
Page
9
CONTENTSviii2.2 Creating a Series from Python objects 30 2.3 Series attributes 32 2.4 Retrieving the first and last rows 34 2.5 Mathematical operations 36 Statistical operations 36 ■ Arithmetic operations 43 Broadcasting 45 2.6 Passing the Series to Python’s built-in functions 48 2.7 Coding challenge 50 Problems 50 ■ Solutions 50 3 Series methods 54 3.1 Importing a data set with the read_csv function 55 3.2 Sorting a Series 60 Sorting by values with the sort_values method 60 ■ Sorting by index with the sort_index method 62 ■ Retrieving the smallest and largest values with the nsmallest and nlargest methods 64 3.3 Overwriting a Series with the inplace parameter 65 3.4 Counting values with the value_counts method 66 3.5 Invoking a function on every Series value with the apply method 72 3.6 Coding challenge 75 Problems 75 ■ Solutions 76 4 The DataFrame object 79 4.1 Overview of a DataFrame 80 Creating a DataFrame from a dictionary 80 ■ Creating a DataFrame from a NumPy ndarray 81 4.2 Similarities between Series and DataFrames 83 Importing a DataFrame with the read_csv function 83 Shared and exclusive attributes of Series and DataFrames 84 Shared methods of Series and DataFrames 87 4.3 Sorting a DataFrame 90 Sorting by a single column 90 ■ Sorting by multiple columns 92 4.4 Sorting by index 94 Sorting by row index 94 ■ Sorting by column index 95 4.5 Setting a new index 95
Page
10
CONTENTS ix4.6 Selecting columns and rows from a DataFrame 96 Selecting a single column from a DataFrame 96 ■ Selecting multiple columns from a DataFrame 97 4.7 Selecting rows from a DataFrame 99 Extracting rows by index label 99 ■ Extracting rows by index position 101 ■ Extracting values from specific columns 103 4.8 Extracting values from Series 106 4.9 Renaming columns or rows 106 4.10 Resetting an index 108 4.11 Coding challenge 109 Problems 109 ■ Solutions 109 5 Filtering a DataFrame 113 5.1 Optimizing a data set for memory use 114 Converting data types with the astype method 116 5.2 Filtering by a single condition 120 5.3 Filtering by multiple conditions 124 The AND condition 124 ■ The OR condition 125 Inversion with ~ 126 ■ Methods for Booleans 127 5.4 Filtering by condition 127 The isin method 127 ■ The between method 128 The isnull and notnull methods 130 ■ Dealing with null values 132 5.5 Dealing with duplicates 134 The duplicated method 134 ■ The drop_duplicates method 136 5.6 Coding challenge 139 Problems 139 ■ Solutions 140 PART 2 APPLIED PANDAS .......................................... 145 6 Working with text data 147 6.1 Letter casing and whitespace 148 6.2 String slicing 151 6.3 String slicing and character replacement 153 6.4 Boolean methods 155
Page
11
CONTENTSx6.5 Splitting strings 157 6.6 Coding challenge 162 Problems 162 ■ Solutions 162 6.7 A note on regular expressions 163 7 MultiIndex DataFrames 165 7.1 The MultiIndex object 166 7.2 MultiIndex DataFrames 170 7.3 Sorting a MultiIndex 175 7.4 Selecting with a MultiIndex 179 Extracting one or more columns 179 ■ Extracting one or more rows with loc 182 ■ Extracting one or more rows with iloc 186 7.5 Cross-sections 188 7.6 Manipulating the Index 189 Resetting the index 189 ■ Setting the index 193 7.7 Coding challenge 194 Problems 194 ■ Solutions 195 8 Reshaping and pivoting 198 8.1 Wide vs. narrow data 199 8.2 Creating a pivot table from a DataFrame 200 The pivot_table method 201 ■ Additional options for pivot tables 205 8.3 Stacking and unstacking index levels 207 8.4 Melting a data set 209 8.5 Exploding a list of values 213 8.6 Coding challenge 214 Problems 214 ■ Solutions 215 9 The GroupBy object 220 9.1 Creating a GroupBy object from scratch 221 9.2 Creating a GroupBy object from a data set 222 9.3 Attributes and methods of a GroupBy object 225 9.4 Aggregate operations 229 9.5 Applying a custom operation to all groups 232 9.6 Grouping by multiple columns 233
Page
12
CONTENTS xi9.7 Coding challenge 235 Problems 235 ■ Solutions 235 10 Merging, joining, and concatenating 239 10.1 Introducing the data sets 241 10.2 Concatenating data sets 243 10.3 Missing values in concatenated DataFrames 245 10.4 Left joins 247 10.5 Inner joins 249 10.6 Outer joins 251 10.7 Merging on index labels 253 10.8 Coding challenge 254 Problems 256 ■ Solutions 256 11 Working with dates and times 260 11.1 Introducing the Timestamp object 261 How Python works with datetimes 261 ■ How pandas works with datetimes 264 11.2 Storing multiple timestamps in a DatetimeIndex 266 11.3 Converting column or index values to datetimes 268 11.4 Using the DatetimeProperties object 269 11.5 Adding and subtracting durations of time 273 11.6 Date offsets 275 11.7 The Timedelta object 277 11.8 Coding challenge 282 Problems 282 ■ Solutions 283 12 Imports and exports 289 12.1 Reading from and writing to JSON files 290 Loading a JSON file Into a DataFrame 292 ■ Exporting a DataFrame to a JSON file 298 12.2 Reading from and writing to CSV files 299 12.3 Reading from and writing to Excel workbooks 301 Installing the xlrd and openpyxl libraries in an Anaconda environment 301 ■ Importing Excel workbooks 302 Exporting Excel workbooks 305
Page
13
CONTENTSxii12.4 Coding challenge 306 Problems 307 ■ Solutions 307 13 Configuring pandas 310 13.1 Getting and setting pandas options 311 13.2 Precision 315 13.3 Maximum column width 316 13.4 Chop threshold 316 13.5 Option context 317 14 Visualization 319 14.1 Installing matplotlib 320 14.2 Line charts 320 14.3 Bar graphs 325 14.4 Pie charts 327 appendix A Installation and setup 329 appendix B Python crash course 347 appendix C NumPy crash course 383 appendix D Generating fake data with Faker 391 appendix E Regular expressions 397 index 409
Page
14
preface Truth be told, I discovered pandas entirely by luck. In 2015, I interviewed for a data operations analyst position at Indeed.com, the world’s largest jobs site. For my final technical challenge, I was asked to derive insights from an internal data set, using the Microsoft Excel spreadsheet software. Eager to impress, I pulled out as many tricks as I could from my data analysis toolbox: column sorts, text manipulations, pivot tables, and of course the iconic VLOOKUP function. (OK, maybe iconic is a bit of an exaggeration.) Strange as it may sound, at the time I didn’t realize that there were any tools for data analysis besides Excel. Excel was ubiquitous: my parents used it, my teachers used it, and my colleagues used it. It felt like an established standard. So when I received a job offer, I immediately bought about $100 worth of Excel books and started studying. It was time to become a spreadsheet specialist! I showed up for my first day of work with a printout of the 50 most-used Excel func- tions. Barely after I finished logging into my work computer, my manager pulled me into a conference room and informed me that priorities had shifted. The team’s data sets had ballooned to a size that Excel could no longer support. My teammates were also looking for ways to automate the redundant steps in their daily and weekly reports. Luckily, my manager had figured out a solution to both problems. He asked me whether I’d heard of pandas. “The furry animal?” I asked, perplexed. “No,” he said. “The Python data analysis library.” After all my prep, it was time to learn a new technology from scratch. I was a little nervous; I’d never written a line of code before. I was an Excel guy, wasn’t I? Was I capable of doing this? There was only one way to find out. I started diving into the offi-xiii
Page
15
PREFACExivcial pandas documentation, into YouTube videos, books, workshops, Stack Overflow questions, and whatever data sets I could get my hands on. I was relieved to discover how easy and joyful it was to get started with pandas. The code felt intuitive and straightforward. The library was fast. The features were well-developed and expansive. With pandas, I could accomplish a lot of data manipulation with a little code. Stories like mine are common in the Python community. The language’s astro- nomical growth over the past decade is often attributed to the ease with which new developers can pick it up. I am confident that if you’re in a position similar to mine, you can learn pandas just as well. If you’re looking to expand your data analysis skills beyond Excel spreadsheets, this book is your invitation. When I felt comfortable with pandas, I continued to explore Python and then other programming languages. In many ways, pandas spearheaded my transition into full-time software engineering. I owe a lot to this powerful library, and I’m excited to pass on the torch of knowledge to you. I hope that you discover the magic of what code can do for you.
Page
16
acknowledgments It took a lot to get Pandas in Action to the finish line, and I want to express my utmost gratitude to the people who supported me in its two-year writing process. First and foremost, a warm thank you to my wonderful girlfriend, Meredith. From the first sentence, she was steadfast in her support. She’s a vivacious, funny, and kind soul who always picked me up when the going got tough. This book is better because of her. Thank you, Merbear. Thank you to my parents, Irina and Dmitriy, for providing a welcoming home where I can always find respite. Thank you to my twin sisters, Mary and Alexandra. They’re remarkably clever, inquisitive, and hard-working for their age, and I couldn’t be prouder of them. Good luck at college! Thanks to Watson, our golden retriever. He’s not much of a Python expert, but he makes up for it with his entertaining and friendly demeanor. A big thank you to my editor, Sarah Miller, who was an absolute joy to work with. I am grateful for her patience and insights throughout the process. She was the true captain of the ship, and she kept everything sailing smoothly. I would not be a software engineer without the opportunities I was given at Indeed. I want to offer my former manager, Srdjan Bodruzic, a hearty thank you for his gener- osity and mentorship (and for hiring me!). Thanks to my CX teammates—Tommy Winschel, Danny Moncada, JP Schultz, and Travis Wright—for their wisdom and humor. Thanks to other Indeedians who offered a helping hand during my tenure: Matthew Morin, Chris Hatton, Chip Borsi, Nicole Saglimbene, Danielle Scoli, Blairr Swayne, and George Improglou. Thanks to anybody I’ve shared a dinner with at Sophie’s Cuban Cuisine!xv
Page
17
ACKNOWLEDGMENTSxvi I started writing this book as a software engineer at Stride Consulting. I want to thank many Striders for their support throughout the process: David “The Domina- tor” DiPanfilo, Min Kwak, Ben Blair, Kirsten Nordine, Michael “Bobby” Nunez, Jay Lee, James Yoo, Ray Veliz, Nathan Riemer, Julia Berchem, Dan Plain, Nick Char, Grant Ziolkowski, Melissa Wahnish, Dave Anderson, Chris Aporta, Michael Carlson, John Galioto, Sean Marzug-McCarthy, Travis Vander Hoop, Steve Solomon, and Jan Mlčoch. Thank you to the friendly faces I’ve had the opportunity to work with as a software engineer and consultant: Francis Hwang, Inhak Kim, Liana Lim, Matt Bambach, Bren- ton Morris, Ian McNally, Josh Philips, Artem Kochnev, Andrew Kang, Andrew Fader, Karl Smith, Bradley Whitwell, Brad Popiolek, Eddie Wharton, Jen Kwok, and my favor- ite coffee crew: Adam McAmis and Andy Fritz. Thank you to the following people for all they add to my life: Nick Bianco, Cam Stier, Keith David, Michael Cheung, Thomas Philippeau, Nicole DiAndrea, and James Rokeach. Thanks to my favorite band, New Found Glory, for providing the soundtrack to many writing sessions. Pop punk’s not dead! Thank you to the Manning staff who shepherded the project to completion and helped with marketing efforts: Jennifer Houle, Aleksandar Dragosavljević, Radmila Ercegovac, Candace Gillhoolley, Stjepan Jureković, and Lucas Weber. Thanks also to the Manning staff who oversaw the content: Sarah Miller, my developmental editor; Deirdre Hiam, my product manager; Keir Simpson, my copyeditor; and Jason Everett, my proofreader. Thanks to the technical reviewers who helped me iron out the kinks: Al Pezewski, Alberto Ciarlanti, Ben McNamara, Björn Neuhaus, Christopher Kottmyer, Dan Sheikh, Dragos Manailoiu, Erico Lendzian, Jeff Smith, Jérôme Bâton, Joaquin Bel- tran, Jonathan Sharley, Jose Apablaza, Ken W. Alger, Martin Czygan, Mathijs Affourtit, Matthias Busch, Mike Cuddy, Monica E. Guimaraes, Ninoslav Cerkez, Rick Prins, Syed Hasany, Viton Vitanis, and Vybhavreddy Kammireddy Changalreddy. I am a better writer and educator thanks to your efforts. Finally, to the city of Hoboken, my home for the past six years. I wrote many parts of this manuscript in its public library, local cafes, and bubble tea shops. I made many forward strides in my life in this town, and it is forever etched into my history. Thank you, Hoboken!
Page
18
about this book Who should read this book Pandas in Action is a comprehensive introduction to the pandas library for data analy- sis. Pandas enables you to perform a multitude of data manipulations with ease: sort- ing, joining, pivoting, cleaning, deduping, aggregating, and more. The book approaches the subject matter incrementally. It introduces pandas one piece at a time, starting with its smaller building blocks and proceeding to its larger data structures. Pandas in Action is written for data analysts who have intermediate experience with spreadsheet software (such as Microsoft Excel, Google Sheets, and Apple Numbers) and/or alternative data analysis tools (such as R and SAS). It is also a fitting title for Python developers who are curious to learn more about data analysis. How this book is organized: A road map Pandas in Action consists of 14 chapters spread across two parts. Part 1, “Core pandas,” introduces the base mechanics of the pandas library in an incremental manner: Chapter 1 analyzes a sample dataset with pandas to present a big-picture over- view of what the library is capable of. Chapter 2 introduces the Series object, a core pandas data structure that stores a collection of ordered data. Chapter 3 dives into the Series object in greater depth. We explore various Series operations, including sorting values, dropping duplicates, extracting minimums and maximums, and more. Chapter 4 introduces the DataFrame, a two-dimensional table of data. We apply concepts from the previous chapters to the new data structure and introduce additional manipulations.xvii
Page
19
ABOUT THIS BOOKxviii Chapter 5 shows you how to filter subsets of rows from a DataFrame by using var- ious logical conditions: equality, inequality, comparison, inclusion, exclusion, and more. Part 2, “Applied pandas,” focuses on more-advanced pandas features and the prob- lems they solve in real-world datasets: Chapter 6 teaches you how to work with imperfect text data in pandas. We dis- cuss how to solve issues such as removing whitespace, fixing character casing, and extracting multiple values from a single column. Chapter 7 discusses the MultiIndex, which allows us to combine multiple col- umn values into a single identifier for a row of data. Chapter 8 describes how to aggregate our data in a pivot table, shift headers from the row axis to the column axis, and convert our data from wide format to narrow format. Chapter 9 explores how to group rows into buckets and aggregate the resulting collections via the GroupBy object. Chapter 10 walks you through combining multiple data sets into a single one by using various joins. Chapter 11 demonstrates how to work with dates and times in pandas. It covers topics such as sorting dates, calculating durations, and determining whether a date falls at the start of a month or quarter. Chapter 12 shows you how to import additional file types into pandas, including Excel and JSON. We also learn how to export data from pandas. Chapter 13 focuses on configuring the library’s settings. We dive into how to modify the number of displayed rows, alter the precision of floating-point num- bers, round values below a threshold, and more. Chapter 14 explores data visualization using the matplotlib library. We see how to use pandas data to create line charts, bar graphs, pie charts, and more. Each chapter builds upon the preceding one. For those who are learning pandas from scratch, I recommend proceeding through the chapters in linear order. Simultane- ously, to ensure that the book is helpful as a reference guide, I’ve written each chapter as an independent tutorial with its own data sets. We start writing our code from scratch at the beginning of each chapter, so you can start with any chapter you like. Most chapters conclude with a coding challenge that allows you to practice its con- cepts. I strongly recommend taking a shot at these exercises. Pandas is built on the Python programing language, and basic knowledge of the lan- guage’s mechanics is recommended before you get started. For those who have limited experience in Python, appendix B offers a hearty introduction to the language. About the code This book contains many examples of source code, which is formatted in a fixed-width font like this to separate it from ordinary text.
Page
20
ABOUT THIS BOOK xix The source code for the book’s examples is available at the following GitHub repository: https://github.com/paskhaver/pandas-in-action. For those who are new to Git and GitHub, look for a Download Zip button on the repository page. Those who are experienced with Git and GitHub are welcome to clone the repo from the command line. The repository also includes the complete data sets for the text. When I was learn- ing pandas, one of my biggest frustrations was that tutorials loved to rely on randomly generated data. There was no consistency, no context, no story, no fun. In this book, we’ll work with many real-world data sets that cover everything from basketball play- ers’ salaries to Pokémon types to restaurant health inspections. Data is everywhere around us, and pandas is one of the best tools available today to make sense of it. I hope that you enjoy the casual focus of the data sets. liveBook discussion forum Purchase of Pandas in Action includes free access to a private web forum run by Man- ning Publications where you can make comments about the book, ask technical ques- tions, and receive help from the author and from other users. To access the forum, go to https://livebook.manning.com/#!/book/pandas-in-action/discussion. You can also learn more about Manning’s forums and the rules of conduct at https://live book.manning.com/#!/discussion. Manning’s commitment to our readers is to provide a venue where meaningful dialogue between individual readers and between readers and the author can take place. It is not a commitment to any specific amount of participation on the part of the author, whose contribution to the forum remains voluntary (and unpaid). We sug- gest that you try asking the author some challenging questions lest their interest stray! The forum and the archives of previous discussions will be accessible from the pub- lisher’s website as long as the book is in print. Other online resources The official pandas documentation is available at https://pandas.pydata.org /docs. In my spare time, I create technical video courses on Udemy. You can find the courses at https://www.udemy.com/user/borispaskhaver; they include a 20- hour pandas course and a 60-hour Python course. Feel free to reach out to me via Twitter (https://twitter.com/borispaskhaver) or LinkedIn (https://www.linkedin.com/in/boris-paskhaver).