Beautiful Visualization Looking at Data through the Eyes of Experts (Julie Steele, Noah Iliinsky) (z-library.sk, 1lib.sk, z-lib.sk)

Beautiful Visualization

Beautiful Visualization Edited by Julie Steele and Noah Iliinsky Beijing · Cambridge · Farnham · Köln · Sebastopol · Taipei · Tokyo

Beautiful Visualization Edited by Julie Steele and Noah Iliinsky Copyright © 2010 O’Reilly Media, Inc. All rights reserved. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://my.safaribooksonline.com). For more information, contact our corporate/institutional sales department: (800) 998-9938 or corporate@oreilly.com. Editor: Julie Steele Production Editor: Rachel Monaghan Copyeditor: Rachel Head Proofreader: Rachel Monaghan Indexer: Julie Hawks Cover Designer: Karen Montgomery Interior Designer: Ron Bilodeau Illustrator: Robert Romano The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Beautiful Visualization, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O’Reilly Media, Inc. was aware of a trademark claim, the designations have been printed in caps or initial caps. While every precaution has been taken in the preparation of this book, the publisher and au- thors assume no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein. ISBN: 978-1-449-37987-2

v C o n t e n t s Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi 1  On Beauty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Noah Iliinsky What Is Beauty? 1 Learning from the Classics 3 How Do We Achieve Beauty? 6 Putting It Into Practice 11 Conclusion 13 2  Once Upon a Stacked Time Series . . . . . . . . . . . . . . 15 Matthias Shapiro Question + Visual Data + Context = Story 16 Steps for Creating an Effective Visualization 18 Hands-on Visualization Creation 26 Conclusion 36 3  Wordle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 Jonathan Feinberg Wordle’s Origins 38 How Wordle Works 46 Is Wordle Good Information Visualization? 54 How Wordle Is Actually Used 57 Conclusion 58 Acknowledgments 58 References 58 4  Color: The Cinderella of Data Visualization . . . . . . . . . 59 Michael Driscoll Why Use Color in Data Graphics? 59 Luminosity As a Means of Recovering Local Density 64 Looking Forward: What About Animation? 65 Methods 65 Conclusion 67 References and Further Reading 67

vi ConTenTS 5  Mapping Information: Redesigning the New York City Subway Map . . . . . . . . . . . . . . . . . . . . . . . . . . . 69 Eddie Jabbour, as told to Julie Steele The Need for a Better Tool 69 London Calling 71 New York Blues 72 Better Tools Allow for Better Tools 73 Size Is Only One Factor 73 Looking Back to Look Forward 75 New York’s Unique Complexity 77 Geography Is About Relationships 79 Sweat the Small Stuff 85 Conclusion 89 6  Flight Patterns: A Deep Dive . . . . . . . . . . . . . . . . . 91 Aaron Koblin with Valdean Klump Techniques and Data 94 Color 95 Motion 98 Anomalies and Errors 99 Conclusion 101 Acknowledgments 102 7  Your Choices Reveal Who You Are: Mining and Visualizing Social Patterns . . . . . . . . . . . 103 Valdis Krebs Early Social Graphs 103 Social Graphs of Amazon Book Purchasing Data 111 Conclusion 121 References 122 8  Visualizing the U.S. Senate Social Graph (1991–2009) . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 Andrew Odewahn Building the Visualization 124 The Story That Emerged 131 What Makes It Beautiful? 136 And What Makes It Ugly? 137 Conclusion 141 References 142

viiConTenTS 9  The Big Picture: Search and Discovery . . . . . . . . . . . 143 Todd Holloway The Visualization Technique 144 YELLOWPAGES.COM 144 The Netflix Prize 151 Creating Your Own 156 Conclusion 156 References 156 10  Finding Beautiful Insights in the Chaos of Social Network Visualizations . . . . . . . . . . . . . . . 157 Adam Perer Visualizing Social Networks 157 Who Wants to Visualize Social Networks? 160 The Design of SocialAction 162 Case Studies: From Chaos to Beauty 166 References 173 11  Beautiful History: Visualizing Wikipedia . . . . . . . . . . . 175 Martin Wattenberg and Fernanda Viégas Depicting Group Editing 175 History Flow in Action 184 Chromogram: Visualizing One Person at a Time 186 Conclusion 191 12  Turning a Table into a Tree: Growing Parallel Sets into a Purposeful Project . . . . . . . . . . . . . . . . . . . . 193 Robert Kosara Categorical Data 194 Parallel Sets 195 Visual Redesign 197 A New Data Model 199 The Database Model 200 Growing the Tree 202 Parallel Sets in the Real World 203 Conclusion 204 References 204

viii ConTenTS 13  The Design of “X by Y” . . . . . . . . . . . . . . . . . . . . . 205 Moritz Stefaner Briefing and Conceptual Directions 205 Understanding the Data Situation 207 Exploring the Data 208 First Visual Drafts 211 The Final Product 216 Conclusion 223 Acknowledgments 225 References 225 14  Revealing Matrices . . . . . . . . . . . . . . . . . . . . . . . 227 Maximilian Schich The More, the Better? 228 Databases As Networks 230 Data Model Definition Plus Emergence 231 Network Dimensionality 233 The Matrix Macroscope 235 Reducing for Complexity 239 Further Matrix Operations 246 The Refined Matrix 247 Scaling Up 247 Further Applications 249 Conclusion 250 Acknowledgments 250 References 250 15  This Was 1994: Data Exploration with the NYTimes Article Search API . . . . . . . . . . . . 255 Jer Thorp Getting Data: The Article Search API 255 Managing Data: Using Processing 257 Three Easy Steps 262 Faceted Searching 263 Making Connections 265 Conclusion 270

ixConTenTS 16  A Day in the Life of the New York Times . . . . . . . . . . 271 Michael Young and Nick Bilton Collecting Some Data 272 Let’s Clean ’Em First 273 Python, Map/Reduce, and Hadoop 274 The First Pass at the Visualization 274 Scene 1, Take 1 277 Scene 1, Take 2 279 The Second Pass at the Visualization 280 Visual Scale and Other Visualization Optimizations 284 Getting the Time Lapse Working 285 So, What Do We Do with This Thing? 287 Conclusion 287 Acknowledgments 290 17  Immersed in Unfolding Complex Systems . . . . . . . . . 291 Lance Putnam, Graham Wakefield, Haru Ji, Basak Alper, Dennis Adderton, and Professor JoAnn Kuchera-Morin Our Multimodal Arena 291 Our Roadmap to Creative Thinking 293 Project Discussion 296 Conclusion 309 References 309 18  Postmortem Visualization: The Real Gold Standard . . . 311 Anders Persson Background 312 Impact on Forensic Work 312 The Virtual Autopsy Procedure 315 The Future for Virtual Autopsies 325 Conclusion 327 References and Suggested Reading 327

x ConTenTS 19  Animation for Visualization: Opportunities and Drawbacks . . . . . . . . . . . . . . . . 329 Danyel Fisher Principles of Animation 330 Animation in Scientific Visualization 331 Learning from Cartooning 331 Presentation Is Not Exploration 338 Types of Animation 339 Staging Animations with DynaVis 344 Principles of Animation 348 Conclusion: Animate or Not? 349 Further Reading 350 Acknowledgments 350 References 351 20  Visualization: Indexed. . . . . . . . . . . . . . . . . . . . . . 353 Jessica Hagy Visualization: It’s an Elephant. 353 Visualization: It’s Art. 355 Visualization: It’s Business. 356 Visualization: It’s Timeless. 357 Visualization: It’s Right Now. 359 Visualization: It’s Coded. 360 Visualization: It’s Clear. 361 Visualization: It’s Learnable. 363 Visualization: It’s a Buzzword. 365 Visualization: It’s an Opportunity. 366 Contributors . . . . . . . . . . . . . . . . . . . . . . . . . . . 369 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375

xi Preface THIS BOOk FOUND ITS BEGINNINGS as a natural outgrowth of Toby Segaran and Jeff Hammerbacher’s Beautiful Data (O’Reilly), which explores everything from data gathering to data storage and organization and data analysis. While working on that project, it became clear to us that visualization—the practice of presenting informa- tion for consumption as art—was a topic deep and wide enough to warrant a separate examination. When done beautifully, successful visualizations are deceptive in their simplicity, offering the viewer insight and new understanding at a glance. We hoped to help those new to this growing field uncover the methods and decision-making processes experts use to achieve this end. Particularly intriguing when assembling a list of potential contributors was how many ways the word beautiful can be interpreted. The book that founded this series, Andy Oram and Greg Wilson’s Beautiful Code (O’Reilly), defined beauty as a simple and elegant solution to some kind of problem. But visualization—as a combination of information and art—naturally combines both problem solving and aesthetics, allowing us to consider beauty in both the intellectual and classic senses. We hope you will be as delighted as we are by the diversity of backgrounds, projects, and approaches represented in this book. Different as they are, the chapters do offer some themes to the thoughtful and observant. Look for ideas about storytelling, color use, levels of granularity in the data, and user exploration woven throughout the book. Tug on these threads, and see where they take you in your own work.

xii preFaCe The royalties for this book are being donated to Architecture for Humanity (http://www. architectureforhumanity.org), an organization dedicated to making the world better by bringing design, construction, and development services to the places where they are most critically needed. We hope you’ll consider how your own design processes shape the world. How This Book Is Organized Here’s a preview of what you’ll find in this book: Chapter 1, On Beauty, by Noah Iliinsky, offers an examination of what we mean by beauty in the context of visualization, why it’s a worthy goal to pursue, and how to get there. Chapter 2, Once Upon a Stacked Time Series: The Importance of Storytelling in Information Visualization, by Matthias Shapiro, explains the importance of storytelling to visualiza- tion and walks readers through the creation of a simple visualization project they can do on their own. Chapter 3, Wordle, by Jonathan Feinberg, explains the inner workings of his popu- lar method for visualizing a body of text, discussing both the technical and aesthetic choices the author made along the way. Chapter 4, Color: The Cinderella of Data Visualization, by Michael Driscoll, shows how color can be used effectively to convey additional dimensions of data that our brains are able to recognize before we’re aware of it. Chapter 5, Mapping Information: Redesigning the New York City Subway Map, by Eddie Jabbour, explores the humble subway map as a basic visualization tool for understand- ing complex systems. Chapter 6, Flight Patterns: A Deep Dive, by Aaron Koblin with Valdean Klump, visualizes civilian air traffic in the United States and Canada to reveal a method to the madness of air travel. Chapter 7, Your Choices Reveal Who You Are: Mining and Visualizing Social Patterns, by Valdis Krebs, digs into behavioral data to show how the books we buy and the people we associate with reveal clues about our deeper selves. Chapter 8, Visualizing the U.S. Senate Social Graph (1991–2009), by Andrew Odewahn, uses quantitative evidence to evaluate a qualitative story about voting coalitions in the United States Senate. Chapter 9, The Big Picture: Search and Discovery, by Todd Holloway, uses a proximity graphing technique to explore the dynamics of search and discovery as they apply to YELLOWPAGES.COM and the Netflix Prize.

xiiipreFaCe Chapter 10, Finding Beautiful Insights in the Chaos of Social Network Visualizations, by Adam Perer, empowers users to dig into chaotic social network visualizations with interactive techniques that integrate visualization and statistics. Chapter 11, Beautiful History: Visualizing Wikipedia, by Martin Wattenberg and Fernanda Viégas, takes readers through the process of exploring an unknown phenomenon through visualization, from initial sketches to published scientific papers. Chapter 12, Turning a Table into a Tree: Growing Parallel Sets into a Purposeful Project, by Robert Kosara, emphasizes the relationship between the visual representation of data and the underlying data structure or database design. Chapter 13, The Design of “X by Y”: An Information-Aesthetic Exploration of the Ars Electronica Archives, by Moritz Stefaner, describes the process of striving to find a repre- sentation of information that is not only useable and informative but also sensual and evocative. Chapter 14, Revealing Matrices, by Maximilian Schich, uncovers nonintuitive structures in curated databases arising from local activity by the curators and the heterogeneity of the source data. Chapter 15, This Was 1994: Data Exploration with the NYTimes Article Search API, by Jer Thorp, guides readers through using the API to explore and visualize data from the New York Times archives. Chapter 16, A Day in the Life of the New York Times, by Michael Young and Nick Bilton, relates how the New York Times R&D group is using Python and Map/Reduce to exam- ine web and mobile site traffic data across the country and around the world. Chapter 17, Immersed in Unfolding Complex Systems, by Lance Putnam, Graham Wakefield, Haru Ji, Basak Alper, Dennis Adderton, and Professor JoAnn Kuchera-Morin, describes the remarkable scientific exploration made possible by cutting-edge visualization and sonification techniques at the AlloSphere. Chapter 18, Postmortem Visualization: The Real Gold Standard, by Anders Persson, exam- ines new imaging technologies being used to collect and analyze data on human and animal cadavers. Chapter 19, Animation for Visualization: Opportunities and Drawbacks, by Danyel Fisher, attempts to work out a framework for designing animated visualizations. Chapter 20, Visualization: Indexed., by Jessica Hagy, provides insight into various aspects of the “elephant” we call visualization such that we come away with a better idea of the big picture.

xiv preFaCe Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Also used for emphasis in the text. Constant width Used for program listings, as well as within paragraphs to refer to program ele- ments such as variable or function names, databases, data types, environment variables, statements, and keywords. Constant width bold Used for emphasis within code listings. Constant width italic Shows text that should be replaced with user-supplied values or by values deter- mined by context. Using Code Examples This book is here to help you get your job done. In general, you may use the code in this book in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Beautiful Visualization, edited by Julie Steele and Noah Iliinsky. Copyright 2010 O’Reilly Media, Inc., 978-1-449-37987-2.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com.

xvpreFaCe How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at: http://www.oreilly.com/catalog/0636920000617 To comment or ask technical questions about this book, send email to: bookquestions@oreilly.com For more information about our books, conferences, Resource Centers, and the O’Reilly Network, see our website at: http://www.oreilly.com Safari® Books Online Safari Books Online is an on-demand digital library that lets you easily search over 7,500 technology and creative reference books and videos to find the answers you need quickly. With a subscription, you can read any page and watch any video from our library online. Read books on your cell phone and mobile devices. Access new titles before they are available for print, and get exclusive access to manuscripts in development and post feedback for the authors. Copy and paste code samples, organize your favor- ites, download chapters, bookmark key sections, create notes, print out pages, and benefit from tons of other time-saving features. O’Reilly Media has uploaded this book to the Safari Books Online service. To have full digital access to this book and others on similar topics from O’Reilly and other publish- ers, sign up for free at http://my.safaribooksonline.com.

xvi preFaCe Acknowledgments First and foremost, we both wish to thank the contributors who gave of their time and expertise to share their wisdom with us. Their collective vision and experience is impressive, and has been an inspiration in our own work. From Julie: Thanks to my family—Guy, Barbara, Pete, and Matt—for your constant support, and for being the first encouragers of my curiosity about the world. And Martin, for your companionship and never-ending flow of ideas; you inspire me. From Noah: Thanks to everyone who has supported me in this particular line of inquiry over the years, especially my teachers, colleagues, and family, and everyone who has asked good questions and made me think.

1 C h a p t e r o n e On Beauty Noah Iliinsky THIS CHAPTER IS AN EXAMINATION OF WHAT WE MEAN BY BEAUTY in the context of visualization, why it’s a worthy goal to pursue, and how to get there. We’ll start with a discussion of the elements of beauty, look at some examples and counterexamples, and then focus on the critical steps to realize a beautiful visualization.* What Is Beauty? What do we mean when we say a visual is beautiful? Is it an aesthetic judgment, in the traditional sense of the word? It can be, but when we’re discussing visuals in this context, beauty can be considered to have four key elements, of which aesthetic judg- ment is only one. For a visual to qualify as beautiful, it must be aesthetically pleasing, yes, but it must also be novel, informative, and efficient. Novel For a visual to truly be beautiful, it must go beyond merely being a conduit for infor- mation and offer some novelty: a fresh look at the data or a format that gives readers a spark of excitement and results in a new level of understanding. Well-understood for- mats (e.g., scatterplots) may be accessible and effective, but for the most part they no longer have the ability to surprise or delight us. Most often, designs that delight us do * I use the words visualization and visual interchangeably in this chapter, to refer to all types of struc- tured representation of information. This encompasses graphs, charts, diagrams, maps, storyboards, and less formally structured illustrations.

2 BeauTiFul ViSualizaTion so not because they were designed to be novel, but because they were designed to be effective; their novelty is a byproduct of effectively revealing some new insight about the world. Informative The key to the success of any visual, beautiful or not, is providing access to informa- tion so that the user may gain knowledge. A visual that does not achieve this goal has failed. Because it is the most important factor in determining overall success, the abil- ity to convey information must be the primary driver of the design of a visual. There are dozens of contextual, perceptive, and cognitive considerations that come into play in making an effective visual. Though many of these are largely outside the scope of this chapter, we can focus on two particulars: the intended message and the con- text of use. Keen attention to these two factors, in addition to the data itself, will go far toward making a data visualization effective, successful, and beautiful; we will look at them more closely a little later. Efficient A beautiful visualization has a clear goal, a message, or a particular perspective on the information that it is designed to convey. Access to this information should be as straightforward as possible, without sacrificing any necessary, relevant complexity. A visual must not include too much off-topic content or information. Putting more information on the page may (or may not) result in conveying more information to the reader. However, presenting more information necessarily means that it will take the reader longer to find any desired subset of that information. Irrelevant data is the same thing as noise. If it’s not helping, it’s probably getting in the way. Aesthetic The graphical construction—consisting of axes and layout, shape, colors, lines, and typography—is a necessary, but not solely sufficient, ingredient in achieving beauty. Appropriate usage of these elements is essential for guiding the reader, communicat- ing meaning, revealing relationships, and highlighting conclusions, as well as for visual appeal. The graphical aspects of design must primarily serve the goal of presenting informa- tion. Any facet of the graphical treatment that does not aid in the presentation of information is a potential obstacle: it may reduce the efficiency and inhibit the suc- cess of a visualization. As with the data presented, less is usually more in the graphics department. If it’s not helping, it’s probably getting in the way.