Python Polars The Definitive Guide Transforming, Analyzing, and Visualizing Data with a Fast and Expressive DataFrame API (Jeroen Janssens, Thijs Nieuwdorp) (z-library.sk, 1lib.sk, z-lib.sk)

Page 1

Jeroen Janssens & Thijs Nieuwdorp Foreword by Ritchie Vink, Creator of Polars Python Polars The Definitive Guide Transforming, Analyzing, and Visualizing Data with a Fast and Expressive DataFrame API

Page 2

ISBN: 978-1-098-15608-4 US $79.99 CAN $99.99 DATA Jeroen Janssens is a senior developer relations engineer at Posit PBC. He’s passionate about open source and sharing his knowledge. He is the author of Data Science at the Command Line (O’Reilly). Jeroen holds a PhD in machine learning from Tilburg University and an MSc in artificial intelligence from Maastricht University. Thijs Nieuwdorp is the lead data scientist at Xomnia. He’s an early adopter of and contributor to Polars and still uses it on a daily basis. Thijs enjoys building innovative workflow automation to maximize efficiency, identifying insights for strategic decisions, and sharing lessons learned to help peers avoid pitfalls and elevate their craft. Unlock the power of Polars, a Python package for transforming, analyzing, and visualizing data. In this hands-on guide, Jeroen Janssens and Thijs Nieuwdorp walk you through every feature of Polars, showing you how to use it for real-world tasks like data wrangling, exploratory data analysis, building pipelines, and more. Whether you’re a seasoned data professional or new to data science, you’ll quickly master Polars’ expressive API and its underlying concepts. You don’t need to have experience with pandas, but if you do, this book will help you make a seamless transition. The many practical examples and real-world datasets are available on GitHub, so you can easily follow along. • Process data from CSV, Parquet, spreadsheets, databases, and the cloud • Get a solid understanding of expressions, the building blocks of every query • Handle complex data types, including text, time, and nested structures • Use both eager and lazy APIs, and know when to use each • Visualize your data with Altair, hvPlot, plotnine, and Great Tables • Extend Polars with your own Python functions and Rust plugins • Leverage GPU acceleration to boost performance even further Python Polars: The Definitive Guide “Polars has become a rising star in the Python data ecosystem, showing what’s possible in a next-generation data frame library. Jeroen and Thijs have written a timely and essential resource to help you take advantage of everything Polars has to offer.” Wes McKinney Creator of pandas and principal architect at Posit PBC

Page 3

Praise for Python Polars: The Definitive Guide Jeroen and Thijs have done an excellent job—not only teaching you the ins and outs of Polars but also helping you unlearn habits from other tools like pandas. They really bring out the power of expressions, which are key to using Polars effectively, guiding you toward a more declarative, functional approach to data processing. As you work through this book, I’m sure you’ll gain a deep understanding of Polars and discover fresh ways to approach data processing. —Ritchie Vink, Creator of Polars (excerpt from the Foreword) Polars has become a rising star in the Python data ecosystem, showing what’s possible in a next-generation data frame library. Jeroen and Thijs have written a timely and essential resource to help you take advantage of everything Polars has to offer. —Wes McKinney, Creator of pandas, Principal Architect, Posit PBC Polars has brought a ton of much-needed innovation to the data frame world with its much more streamlined API and efficient implementation. As a result, the capabilities of data analysis in Python are pushed to new heights. We also greatly enjoy Ritchie and team as a part of the Amsterdam data ecosystem. I greatly respect Jeroen’s commitment to teaching data science in an accessible way, whether it be on the command line or elsewhere. His and Thijs’s book is a testament to this commitment, and I recommend it to the data science community. —Hannes Mühleisen, Cocreator of DuckDB

Page 4

As a client working closely with Thijs and Jeroen on migrating a data pipeline to Polars, we were initially skeptical, but we soon experienced the speed and intuitiveness of Polars and its API. While Jeroen and Thijs worked late hours to make progress with their book, we directly benefited from the improvements in our pipeline. We hope this book helps you along the way and that you find all the little gems Polars has to offer—while being lazy, of course! —Marnix van Lieshout and Bram Timmers, Data Scientists at Alliander This book will change how you think about data analysis. Jeroen and Thijs have done a phenomenal job including all kinds of comparisons, diagrams, and examples. Polars has an incredible amount of functionality, and it’s clear they’ve put great care into organizing and breaking all the pieces down. I appreciate their focus on data visualization throughout the book and the inclusion of table styling! —Michael Chow, Principal Software Engineer at Posit PBC, Co-maintainer of Great Tables This book cleverly demystifies Polars’ powerful ecosystem. Thijs and Jeroen seamlessly guide you through the theoretical foundations and hands-on examples, making complex concepts accessible without sacrificing depth. Whether you’re migrating from pandas or starting fresh with Polars, this guide provides the roadmap you need to realize your data workflows could be running at a far more bearable speed. —Hella Haanstra, Machine Learning Engineer at Xomnia When I first interacted with Polars, it was so early days that I made the PR for the .pipe() method on DataFrames. I was pleased with the speedups that Polars gave me, but I was massively impressed by the API. It just felt like such good taste and a great direction for the future. Fast-forward a few years, and today Polars has become an established tool with so many features that the ecosystem was in dire need of a guide. This book gives us just that. It is a guide, but also a reference! —Vincent D. Warmerdam, Data Person, Cofounder of Calmcode Python Polars: The Definitive Guide manages to offer a comprehensive overview of everything Polars has to offer while also providing a great learning experience in the form of excellent code examples. Truly a great resource! —Stijn de Gooijer, Core Contributor to Polars

Page 5

Polars is emerging as one of the leading data frameworks in Python, especially for time series analysis and forecasting. It is now fully integrated with libraries like Nixtla’s MLForecast and StatsForecast, allowing for the creation of forecasts at scale with high performance. Jeroen and Thijs have done an excellent job of establishing a solid foundation for both new practitioners who wish to learn how to process data with Python and experienced users looking to transition from pandas to Polars. —Rami Krispin, Senior Manager, Data Science and Engineering at Apple The depth that Jeroen and Thijs went into in order to produce this phenomenally good book is impressive. There are some pretty good Polars books out there, but this is the best one. They don’t just repeat what’s in the user guide—they go above and beyond: in-depth explanations of expressions, (friendly) comparisons with other tools, an example of how to go beyond what Polars offers out-of-the-box with a geocoding plugin! Whether you’re new to Polars or want to improve your understanding of it, I wholeheartedly recommend this book. —Marco Gorelli, Senior Software Engineer at Quansight, Core Contributor to Polars and pandas, Creator of Narwhals

Page 6

(This page has no text content)

Page 7

Jeroen Janssens and Thijs Nieuwdorp Python Polars: The Definitive Guide Transforming, Analyzing, and Visualizing Data with a Fast and Expressive DataFrame API

Page 8

978-1-098-15608-4 [LSI] Python Polars: The Definitive Guide by Jeroen Janssens and Thijs Nieuwdorp Copyright © 2025 Jeroen Janssens and Thijs Nieuwdorp. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Aaron Black Development Editor: Sarah Grey Production Editor: Jonathon Owen Copyeditor: Sonia Saruba Proofreader: Miah Sandvik Indexer: WordCo Indexing Services, Inc. Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea February 2025: First Edition Revision History for the First Edition 2025-02-19: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781098156084 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Python Polars: The Definitive Guide, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the authors and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

Page 9

Table of Contents Foreword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi Part I. Begin 1. Introducing Polars. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 What Is This Thing Called Polars? 4 Key Features 4 Key Concepts 4 Advantages 5 Why You Should Use Polars 5 Performance 6 Usability 6 Popularity 7 Sustainability 8 Polars Compared to Other Data Processing Packages 8 Why We Focus on Python Polars 10 How This Book Is Organized 10 An ETL Showcase 11 Extract 12 Bonus: Visualizing Neighborhoods and Stations 17 Transform 21 Bonus: Visualizing Daily Trips per Borough 26 Load 28 Bonus: Becoming Faster by Being Lazy 29 Takeaways 32 vii

Page 10

2. Getting Started. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Setting Up Your Environment 33 Downloading the Project 34 Installing uv 35 Installing the Project 35 Working with the Virtual Environment 35 Verifying Your Installation 36 Crash Course in JupyterLab 37 Keyboard Shortcuts 38 Installing Polars on Other Projects 39 All Optional Dependencies 40 Optional Dependencies for Interoperability 40 Optional Dependencies for Working with Spreadsheets 40 Optional Dependencies for Working with Databases 41 Optional Dependencies for Working with Remote Filesystems 41 Optional Dependencies for Other I/O Formats 41 Optional Dependencies for Extra Functionality 42 Installing Optional Dependencies 42 Configuring Polars 42 Temporary Configuration Using a Context Manager 43 Local Configuration Using a Decorator 46 Compiling Polars from Scratch 46 Edge Case: Very Large Datasets 47 Edge Case: Processors Lacking AVX Support 48 Takeaways 48 3. Moving from pandas to Polars. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 Animals 50 Similarities to Recognize 50 Appearances to Appreciate 51 Differences in Code 51 Differences in Display 52 Concepts to Unlearn 57 Index 57 Axes 58 Indexing and Slicing 59 Eagerness 61 Relaxedness 63 Syntax to Forget 64 Common Operations Side By Side 65 To and From pandas 69 Takeaways 70 viii | Table of Contents

Page 11

Part II. Form 4. Data Structures and Data Types. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 Series, DataFrames, and LazyFrames 73 Data Types 75 Nested Data Types 77 Missing Values 79 Data Type Conversion 84 Takeaways 86 5. Eager and Lazy APIs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 Eager API: DataFrame 87 Lazy API: LazyFrame 89 Performance Differences 90 Functionality Differences 91 Attributes 92 Aggregation Methods 92 Computation Methods 93 Descriptive Methods 93 GroupBy Methods 94 Exporting Methods 94 Manipulation and Selection Methods 95 Miscellaneous Methods 97 Tips and Tricks 98 Going from LazyFrame to DataFrame and Vice Versa 98 Joining a DataFrame with a LazyFrame 99 Caching Intermittent Results 100 Takeaways 101 6. Reading and Writing Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 Format Overview 104 Reading CSV Files 105 Parsing Missing Values Correctly 107 Reading Files with Encodings Other Than UTF-8 108 Reading Excel Spreadsheets 110 Working with Multiple Files 111 Reading Parquet 114 Reading JSON and NDJSON 115 JSON 115 NDJSON 118 Other File Formats 120 Querying Databases 121 Table of Contents | ix

Page 12

Writing Data 123 CSV Format 123 Excel Format 124 Parquet Format 124 Other Considerations 125 Takeaways 125 Part III. Express 7. Beginning Expressions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 Methods and Namespaces 131 Expressions by Example 131 Selecting Columns with Expressions 132 Creating New Columns with Expressions 133 Filtering Rows with Expressions 135 Aggregating with Expressions 135 Sorting Rows with Expressions 136 The Definition of an Expression 137 Properties of Expressions 139 Creating Expressions 141 From Existing Columns 142 From Literal Values 143 From Ranges 145 Other Functions to Create Expressions 146 Renaming Expressions 147 Expressions Are Idiomatic 149 Takeaways 151 8. Continuing Expressions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 Types of Operations 154 Example A: Element-Wise Operations 155 Example B: Operations That Summarize to One 155 Example C: Operations That Summarize to One or More 156 Example D: Operations That Extend 156 Element-Wise Operations 157 Operations That Perform Mathematical Transformations 157 Operations Related to Trigonometry 159 Operations That Round and Categorize 160 Operations for Missing or Infinite Values 161 Other Operations 163 Nonreducing Series-Wise Operations 164 x | Table of Contents

Page 13

Operations That Accumulate 164 Operations That Fill and Shift 166 Operations Related to Duplicate Values 167 Operations That Compute Rolling Statistics 168 Operations That Sort 170 Other Operations 171 Series-Wise Operations That Summarize to One 172 Operations That Are Quantifiers 173 Operations That Compute Statistics 174 Operations That Count 176 Other Operations 178 Series-Wise Operations That Summarize to One or More 179 Operations Related to Unique Values 179 Operations That Select 180 Operations That Drop Missing Values 181 Other Operations 182 Series-Wise Operations That Extend 185 Takeaways 185 9. Combining Expressions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 Inline Operators Versus Methods 188 Arithmetic Operations 190 Comparison Operations 191 Boolean Algebra Operations 195 Bitwise Operations 197 Using Functions 199 When, Then, Otherwise 202 Takeaways 204 Part IV. Transform 10. Selecting and Creating Columns. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209 Selecting Columns 211 Introducing Selectors 212 Selecting Based on Name 213 Selecting Based on Data Type 214 Selecting Based on Position 216 Combining Selectors 218 Creating Columns 220 Related Column Operations 225 Dropping 225 Table of Contents | xi

Page 14

Renaming 225 Stacking 226 Adding Row Indices 227 Takeaways 227 11. Filtering and Sorting Rows. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229 Filtering Rows 230 Filtering Based on Expressions 230 Filtering Based on Column Names 231 Filtering Based on Constraints 232 Sorting Rows 233 Sorting Based on a Single Column 234 Sorting in Reverse 235 Sorting Based on Multiple Columns 235 Sorting Based on Expressions 236 Sorting Nested Data Types 237 Related Row Operations 239 Filtering Missing Values 239 Slicing 240 Top and Bottom 241 Sampling 241 Semi-Joins 241 Takeaways 242 12. Working with Textual, Temporal, and Nested Data Types. . . . . . . . . . . . . . . . . . . . . . 245 String 246 String Methods 246 String Examples 248 Categorical 252 Categorical Methods 253 Categorical Examples 253 Enum 256 Temporal 257 Temporal Methods 257 Temporal Examples 259 List 263 List Methods 263 List Examples 265 Array 267 Array Methods 267 Array Examples 268 Struct 270 xii | Table of Contents

Page 15

Struct Methods 270 Struct Examples 271 Takeaways 274 13. Summarizing and Aggregating. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275 Split, Apply, and Combine 276 GroupBy Context 276 The Descriptives 279 Advanced Methods 284 Row-Wise Aggregations 289 Window Functions in Selection Context 291 Dynamic Grouping 293 Rolling Aggregations 294 Upsampling 297 Takeaways 299 14. Joining and Concatenating. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301 Joining 301 Join Strategies 302 Joining on Multiple Columns 306 Validation 306 Inexact Joining 308 Inexact Join Strategies 310 Additional Fine-Tuning 312 Use Case: Marketing Campaign Attribution 312 Vertical and Horizontal Concatenation 316 Vertical 317 Horizontal 318 Diagonal 318 Align 319 Relaxed 322 Stacking 323 Appending 324 Extending 324 Takeaways 325 15. Reshaping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327 Wide Versus Long DataFrames 327 Pivot to a Wider DataFrame 330 Unpivot to a Longer DataFrame 335 Transposing 337 Exploding 339 Table of Contents | xiii

Page 16

Partition into Multiple DataFrames 342 Takeaways 345 Part V. Advance 16. Visualizing Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349 NYC Bike Trips 351 Built-In Plotting with Altair 353 Introducing Altair 353 Methods in the Plot Namespaces 354 Plotting DataFrames 355 Too Large to Handle 357 Plotting Series 359 pandas-Like Plotting with hvPlot 363 Introducing hvPlot 363 A First Plot 364 Methods in the hvPlot Namespace 365 pandas as Backup 366 Manual Transformations 367 Changing the Plotting Backend 368 Plotting Points on a Map 369 Composing Plots 369 Adding Interactive Widgets 371 Publication-Quality Graphics with plotnine 372 Introducing plotnine 373 Plots for Exploration 373 Plots for Communication 377 Styling DataFrames With Great Tables 381 Takeaways 386 17. Extending Polars. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387 User-Defined Functions in Python 387 Applying a Function to Elements 388 Applying a Function to a Series 390 Applying a Function to Groups 391 Applying a Function to an Expression 394 Applying a Function to a DataFrame or LazyFrame 395 Registering Your Own Namespace 396 Polars Plugins in Rust 397 Prerequisites 398 The Anatomy of a Plugin Project 398 xiv | Table of Contents

Page 17

The Plugin 398 Compiling the Plugin 401 Performance Benchmark 401 Register Arguments 402 Using a Rust Crate 405 Use Case: geo 405 Takeaways 416 18. Polars Internals. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 417 Polars’ Architecture 417 Arrow 419 Multithreaded Computations and SIMD Operations 421 The String Data Type in Memory 422 ChunkedArrays in Series 423 Query Optimization 424 LazyFrame Scan-Level Optimizations 425 Other Optimizations 427 Checking Your Expressions 429 meta Namespace Overview 429 meta Namespace Examples 430 Profiling Polars 432 Tests in Polars 434 Comparing DataFrames and Series 435 Common Antipatterns 437 Using Brackets for Column Selection 437 Misusing Collect 437 Using Python Code in your Polars Queries 438 Takeaways 439 Appendix: Accelerating Polars with the GPU. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 461 Table of Contents | xv

Page 18

(This page has no text content)

Page 19

Foreword It was never meant to be this serious. In December 2019, I became a dad, and then the pandemic hit. This left me dazed; juggling life with a newborn and trying to hold on to a sense of “me.” In a burst of new-dad sleep deprivation, I figured I’d take on a new project. Honestly, it started as nothing more than a hobby. At work, I had to join two CSV files while programming in Rust, and it felt like way more hassle than it should have been. I wondered if there were easier ways to get it done than setting up SQLite. So, as one often does in software development, I decided to try my hand at creating my own join algorithm. At the time, I was pretty new to Rust and didn’t know much about optimizing performance. So as a proud writer of my first join algorithm, I learned that my implementation was much slower than pandas. This unsatisfying result planted the seed of what would later become Polars. This led me down a path of researching database engines, learning Rust, and a lot of trial and error over the next year and a half. As I learned more about databases, algorithms, performance, memory, unsafe code, etc., my goals shifted. I went from just wanting to make a faster join than pandas to building a DataFrame package for Rust, and eventually, a high-performance query engine that could rival the state of the art in the Python landscape. I drew inspiration from pandas’ strengths and weaknesses, the declarative approaches of SQL and PySpark, functional programming principles, and the rigor of Rust’s type system. At first, I even thought I’d model it directly on the pandas API. But I quickly realized that limiting myself that way hurt my creative drive and would only lead to a less effective tool. When I decided to let go of this constraint and merged the lazy and eager APIs into one expression-based API, the project was molding into something people might now recognize as Polars. xvii

Page 20

On March 15, 2021, I released this as a research project on PyPI. I was able to evolve this pet project into a full-fledged DataFrame processing package called Polars. Eventually, Polars’ success enabled me to secure funding and start my own company, Polars Inc. Today, Polars is its own company—something I’m proud of—and it’s a constant source of energy for me. Finally, a lot of the ideas I have can come to fruition (not all of them, but hey, I’m ambitious). With Polars, my hope is to create one DataFrame API that works across the board, whether you’re dealing with small or big data, all on the same tech stack. Together with a dedicated team at Polars Inc. and a growing, enthusiastic commu‐ nity of contributors, we’re tackling many interesting technical challenges. Each scale comes with its own set of challenges, and finding solutions for these tough problems so that millions of users have a better experience is incredibly rewarding. One chal‐ lenge involves accurately handling time zones. This may sound trivial, but dealing with time zone conversions and ensuring consistency across different systems is notoriously difficult. A big technical challenge we’re currently facing is translating the DataFrame API into a full streaming model. This transition requires rethinking how operations—such as computing the mean or accessing the last row—will work in a streaming context, as opposed to in-memory batch processing. I’m confident that we’ll rise to these challenges. Over time, Polars has grown beyond my initial expectations. It’s been incredibly rewarding to see it used in surprising scenarios, such as finite element method simu‐ lations involving 1,500 joins. Users have also employed Polars for metaprogramming tasks, generating complex queries that would be impractical to write by hand. Looking ahead, the future of Polars is exciting. We’re focused on extending its stream‐ ing capabilities, allowing you to process huge datasets on your own laptop. We’re also working on creating a distributed cloud environment to do fast distributed comput‐ ing on massive datasets. Another key goal is to support extension types, paving the way, for example, for geospatial data types. These developments aim to make Polars a go-to tool for data processing—from tiny datasets that fit in memory to massive datasets that require distributed computing. On top of that, we’ll keep investing in the user experience through improvements like error messages that inform you up front when a query will eventually fail. I’m really glad you’re holding Python Polars: The Definitive Guide by Jeroen Janssens and Thijs Nieuwdorp. I know Jeroen and Thijs well from our days at Xomnia, where we shared a lot of training sessions, company trips, drinks, ideas—the whole mix. When Jeroen first suggested I write a book, I had to thank him—my energy was all going into Polars—but I wholeheartedly supported him when he decided to take up the challenge with Thijs. Then, on a company trip to Jordan, both Jeroen and xviii | Foreword