Digital Library

Think Like a Data Scientist Tackle the Data Science Process Step-by-Step (Brian Godsey) (z-library.sk, 1lib.sk, z-lib.sk)

science

Language English

Data collected from customers, scientific measurements, IoT sensors, and so on is valuable only if you understand it. Data scientists revel in the interesting and rewarding challenge of observing, exploring, analyzing, and interpreting this data. Getting started with data science means more than mastering analytic tools and techniques, however; the real magic happens when you begin to think like a data scientist. This book will get you there. Think Like a Data Scientist teaches you a step-by-step approach to solving real-world data-centric problems. By breaking down carefully crafted examples, you’ll learn to combine analytic, programming, and business perspectives into a repeatable process for extracting real knowledge from data. As you read, you'll discover (or remember) valuable statistical techniques and explore powerful data science software. More importantly, you’ll put this knowledge together using a structured process for data science. When you've finished, you'll have a strong foundation for a lifetime of data science learning and practice. What’s Inside • The data science process, step-by-step • How to anticipate problems • Dealing with uncertainty • Best practices in software and scientific thinking Readers need beginner programming skills and knowledge of basic statistics. Brian Godsey has worked in software, academia, finance, and defense and has launched several data-centric start-ups.

Format PDF

Size 5.2 MB

Views

Downloads

0.00

Total Donations

(First 20 pages)

Registered users can read the full content for free

Page 1

M A N N I N G Brian Godsey Tackle the data science process step-by-step

Page 2

The lifecycle of a data science project This book is organized around the three phases of a data science project: The first phase is preparation—time and effort spent gathering information at the beginning of a project can spare big headaches later. The second phase is building the product, from planning through execution, using what you learned during the preparation phase and all the tools that statistics and software can provide. The third and final phase is finishing—delivering the product, getting feedback, making revisions, supporting the product, and wrapping up the project. Finish Build Prepare Assess Deliver Plan Analyze Engineer Optimize Execute Explore Wrap up Wrangle Revise Set goals Data science process

Page 3

Think Like a Data Scientist TACKLE THE DATA SCIENCE PROCESS STEP-BY-STEP BRIAN GODSEY M A N N I N G SHELTER ISLAND

Page 4

For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact Special Sales Department Manning Publications Co. 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 Email: orders@manning.com © 2017 by Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps. Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine. Manning Publications Co. Development editor: Karen Miller 20 Baldwin Road Review editor: Aleksandar Dragosavljević PO Box 761 Technical development editor: Mike Shepard Shelter Island, NY 11964 Project editor: Kevin Sullivan Copy editor: Linda Recktenwald Proofreader: Corbin Collins Typesetter: Dennis Dalinnik Cover designer: Marija Tudor ISBN: 9781633430273 Printed in the United States of America 1 2 3 4 5 6 7 8 9 10 – EBM – 22 21 20 19 18 17

Page 5

To all thoughtful, deliberate problem-solvers who consider themselves scientists first and builders second For everyone everywhere who ever taught me anything

Page 6

(This page has no text content)

Page 7

brief contents PART 1 PREPARING AND GATHERING DATA AND KNOWLEDGE .....1 1 ■ Philosophies of data science 3 2 ■ Setting goals by asking good questions 19 3 ■ Data all around us: the virtual wilderness 37 4 ■ Data wrangling: from capture to domestication 67 5 ■ Data assessment: poking and prodding 84 PART 2 BUILDING A PRODUCT WITH SOFTWARE AND STATISTICS ........................................................105 6 ■ Developing a plan 107 7 ■ Statistics and modeling: concepts and foundations 129 8 ■ Software: statistics in action 166 9 ■ Supplementary software: bigger, faster, more efficient 201 10 ■ Plan execution: putting it all together 215 PART 3 FINISHING OFF THE PRODUCT AND WRAPPING UP ......237 11 ■ Delivering a product 239 12 ■ After product delivery: problems and revisions 256 13 ■ Wrapping up: putting the project away 274v

Page 8

(This page has no text content)

Page 9

contents preface xv acknowledgments xvi about this book xvii about the cover illustration xxi PART 1 PREPARING AND GATHERING DATA AND KNOWLEDGE ................................................1 1 Philosophies of data science 3 1.1 Data science and this book 5 1.2 Awareness is valuable 7 1.3 Developer vs. data scientist 8 1.4 Do I need to be a software developer? 10 1.5 Do I need to know statistics? 11 1.6 Priorities: knowledge first, technology second, opinions third 12 1.7 Best practices 13 Documentation 14 ■ Code repositories and versioning 14 Code organization 15 ■ Ask questions 16 ■ Stay close to the data 17 1.8 Reading this book: how I discuss concepts 17vii

Page 10

CONTENTSviii2 Setting goals by asking good questions 19 2.1 Listening to the customer 20 Resolving wishes and pragmatism 20 ■ The customer is probably not a data scientist 22 ■ Asking specific questions to uncover fact, not opinions 23 ■ Suggesting deliverables: guess and check 24 Iterate your ideas based on knowledge, not wishes 25 2.2 Ask good questions—of the data 26 Good questions are concrete in their assumptions 27 Good answers: measurable success without too much cost 29 2.3 Answering the question using data 30 Is the data relevant and sufficient? 31 ■ Has someone done this before? 32 ■ Figuring out what data and software you could use 32 ■ Anticipate obstacles to getting everything you want 33 2.4 Setting goals 34 What is possible? 34 ■ What is valuable? 34 ■ What is efficient? 35 2.5 Planning: be flexible 35 3 Data all around us: the virtual wilderness 37 3.1 Data as the object of study 37 The users of computers and the internet became data generators 38 Data for its own sake 40 ■ Data scientist as explorer 41 3.2 Where data might live, and how to interact with it 44 Flat files 45 ■ HTML 47 ■ XML 48 ■ JSON 49 Relational databases 50 ■ Non-relational databases 52 APIs 52 ■ Common bad formats 54 ■ Unusual formats 55 Deciding which format to use 55 3.3 Scouting for data 56 First step: Google search 57 ■ Copyright and licensing 57 The data you have: is it enough? 58 ■ Combining data sources 59 ■ Web scraping 60 ■ Measuring or collecting things yourself 61 3.4 Example: microRNA and gene expression 62 4 Data wrangling: from capture to domestication 67 4.1 Case study: best all-time performances in track and field 68 Common heuristic comparisons 68 ■ IAAF Scoring Tables 69 Comparing performances using all data available 70

Page 11

CONTENTS ix4.2 Getting ready to wrangle 70 Some types of messy data 71 ■ Pretend you’re an algorithm 71 Keep imagining: what are the possible obstacles and uncertainties? 73 ■ Look at the end of the data and the file 74 Make a plan 75 4.3 Techniques and tools 76 File format converters 76 ■ Proprietary data wranglers 77 Scripting: use the plan, but then guess and check 77 4.4 Common pitfalls 78 Watch out for Windows/Mac/Linux problems 78 Escape characters 79 ■ The outliers 82 Horror stories around the wranglers’ campfire 82 5 Data assessment: poking and prodding 84 5.1 Example: the Enron email data set 85 5.2 Descriptive statistics 86 Stay close to the data 87 ■ Common descriptive statistics 88 Choosing specific statistics to calculate 89 ■ Make tables or graphs where appropriate 91 5.3 Check assumptions about the data 92 Assumptions about the contents of the data 92 ■ Assumptions about the distribution of the data 92 ■ A handy trick for uncovering your assumptions 94 5.4 Looking for something specific 95 Find a few examples 95 ■ Characterize the examples: what makes them different? 96 ■ Data snooping (or not) 98 5.5 Rough statistical analysis 99 Dumb it down 99 ■ Take a subset of the data 102 Increasing sophistication: does it improve results? 103 PART 2 BUILDING A PRODUCT WITH SOFTWARE AND STATISTICS ..............................................105 6 Developing a plan 107 6.1 What have you learned? 109 Examples 109 ■ Evaluating what you’ve learned 112 6.2 Reconsidering expectations and goals 113 Unexpected new information 114 ■ Adjusting goals 116 Consider more exploratory work 117

Page 12

CONTENTSx6.3 Planning 117 Examples 118 6.4 Communicating new goals 127 7 Statistics and modeling: concepts and foundations 129 7.1 How I think about statistics 130 7.2 Statistics: the field as it relates to data science 131 What statistics is 131 ■ What statistics is not 132 7.3 Mathematics 134 Example: long division 134 ■ Mathematical models 137 Mathematics vs. statistics 140 7.4 Statistical modeling and inference 141 Defining a statistical model 142 ■ Latent variables 143 Quantifying uncertainty: randomness, variance, and error terms 144 ■ Fitting a model 148 ■ Bayesian vs. frequentist statistics 153 ■ Drawing conclusions from models 156 7.5 Miscellaneous statistical methods 159 Clustering 159 ■ Component analysis 160 ■ Machine learning and black box methods 162 8 Software: statistics in action 166 8.1 Spreadsheets and GUI-based applications 167 Spreadsheets 167 ■ Other GUI-based statistical applications 171 ■ Data science for the masses 172 8.2 Programming 173 Getting started with programming 174 ■ Languages 182 8.3 Choosing statistical software tools 191 Does the tool have an implementation of the methods? 191 Flexibility is good 192 ■ Informative is good 192 Common is good 192 ■ Well documented is good 193 Purpose-built is good 193 ■ Interoperability is good 194 Permissive licenses are good 194 ■ Knowledge and familiarity are good 195 8.4 Translating statistics into software 195 Using built-in methods 195 ■ Writing your own methods 199

Page 13

CONTENTS xi9 Supplementary software: bigger, faster, more efficient 201 9.1 Databases 202 Types of databases 202 ■ Benefits of databases 204 How to use databases 206 ■ When to use databases 206 9.2 High-performance computing 207 Types of HPC 207 ■ Benefits of HPC 208 ■ How to use HPC 208 ■ When to use HPC 209 9.3 Cloud services 209 Types of cloud services 209 ■ Benefits of cloud services 210 How to use cloud services 210 ■ When to use cloud services 210 9.4 Big data technologies 211 Types of big data technologies 212 ■ Benefits of big data technologies 213 ■ How to use big data technologies 213 When to use big data technologies 213 9.5 Anything as a service 213 10 Plan execution: putting it all together 215 10.1 Tips for executing the plan 216 If you’re a statistician 216 ■ If you’re a software engineer 218 If you’re a beginner 219 ■ If you’re a member of a team 219 If you’re leading a team 220 10.2 Modifying the plan in progress 221 Sometimes the goals change 222 ■ Something might be more difficult than you thought 222 ■ Sometimes you realize you made a bad choice 223 10.3 Results: knowing when they’re good enough 223 Statistical significance 223 ■ Practical usefulness 224 Reevaluating your original accuracy and significance goals 225 10.4 Case study: protocols for measurement of gene activity 227 The project 227 ■ What I knew 228 ■ What I needed to learn 228 ■ The resources 229 ■ The statistical model 229 The software 232 ■ The plan 232 ■ The results 233 Submitting for publication and feedback 234 ■ How it ended 235

Page 14

CONTENTSxiiPART 3 FINISHING OFF THE PRODUCT AND WRAPPING UP .................................................237 11 Delivering a product 239 11.1 Understanding your customer 240 Who is the entire audience for the results? 240 ■ What will be done with the results? 241 11.2 Delivery media 242 Report or white paper 242 ■ Analytical tool 243 ■ Interactive graphical application 245 ■ Instructions for how to redo the analysis 247 ■ Other types of products 248 11.3 Content 249 Make important, conclusive results prominent 249 Don’t include results that are virtually inconclusive 249 Include obvious disclaimers for less significant results 250 User experience 250 11.4 Example: analyzing video game play 253 12 After product delivery: problems and revisions 256 12.1 Problems with the product and its use 257 Customers not using the product correctly 257 UX problems 259 ■ Software bugs 261 The product doesn’t solve real problems 262 12.2 Feedback 264 Feedback means someone is using your product 264 Feedback is not disapproval 264 ■ Read between the lines 265 Ask for feedback if you must 267 12.3 Product revisions 268 Uncertainty can make revisions necessary 268 ■ Designing revisions 269 ■ Engineering revisions 270 ■ Deciding which revisions to make 272

Page 15

CONTENTS xiii13 Wrapping up: putting the project away 274 13.1 Putting the project away neatly 275 Documentation 276 ■ Storage 278 ■ Thinking ahead to future scenarios 281 ■ Best practices 283 13.2 Learning from the project 284 Project postmortem 284 13.3 Looking toward the future 287 exercises: examples and answers 290 index 299

Page 16

(This page has no text content)

Page 17

preface In 2012, an article in the Harvard Business Review named the role of data scientist “the sexiest job of the 21st century.” With 87 years left in the century, it’s fair to say they might yet change their minds. Nevertheless, at the moment, data scientists are getting a lot of attention, and as a result, books about data science are proliferating. There would be no sense in adding another book to the pile if it merely repeated or repackaged text that is easily found elsewhere. But, while surveying new data science literature, it became clear to me that most authors would rather explain how to use all the latest tools and technol- ogies than discuss the nuanced problem-solving nature of the data science process. Armed with several books and the latest knowledge of algorithms and data stores, many aspiring data scientists were still asking the question: Where do I start? And so, here is another book on data science. This one, however, attempts to lead you through the data science process as a path with many forks and potentially unknown destinations. The book warns you of what may be ahead, tells you how to prepare for it, and suggests how to react to surprises. It discusses what tools might be the most useful, and why, but the main objective is always to navigate the path—the data science process—intelligently, efficiently, and successfully, to arrive at practical solutions to real-life data-centric problems.xv

Page 18

acknowledgments I would like to thank everyone at Manning who helped to make this book a reality, and Marjan Bace, Manning’s publisher, for giving me this opportunity. I’d also like to thank Mike Shepard for evaluating the technical aspects of the book, and the reviewers who contributed helpful feedback during development of the manuscript. Those reviewers include Casimir Saternos, Clemens Baader, David Krief, Gavin Whyte, Ian Stirk, Jenice Tom, Łukasz Bonenberg, Martin Perry, Nicolas Boulet- Lavoie, Pouria Amirian, Ran Volkovich, Shobha Iyer, and Valmiky Arquissandas. Finally, I extend special thanks to my teammates, current and former, at Unoceros and Panopticon Labs for providing ample fodder for this book in many forms: experi- ences and knowledge in software development and data science, fruitful conversa- tions, crazy ideas, funny stories, awkward mistakes, and most importantly, willingness to indulge my curiosity.xvi

Page 19

about this book Data science still carries the aura of a new field. Most of its components—statistics, software development, evidence-based problem solving, and so on—descend directly from well-established, even old, fields, but data science seems to be a fresh assemblage of these pieces into something that is new, or at least feels new in the context of current public discourse. Like many new fields, data science hasn’t quite found its footing. The lines between it and other related fields—as far as those lines matter—are still blurry. Data science may rely on, but is not equivalent to, database architecture and administra- tion, big data engineering, machine learning, or high-performance computing, to name a few. The core of data science doesn’t concern itself with specific database implementa- tions or programming languages, even if these are indispensable to practitioners. The core is the interplay between data content, the goals of a given project, and the data- analytic methods used to achieve those goals. The data scientist, of course, must man- age these using any software necessary, but which software and how to implement it are details that I like to imagine have been abstracted away, as if in some distant future reality. This book attempts to foresee that future in which the most common, rote, mechanical tasks of data science are stripped away, and we are left with only the core: applying the scientific method to data sets in order to achieve a project’s goals. This, the process of data science, involves software as a necessary set of tools, just as a traditional scientist might use test tubes, flasks, and a Bunsen burner. But, whatxvii

Page 20

ABOUT THIS BOOKxviiimatters is what’s happening on the inside: what’s happening to the data, what results we get, and why. In the following pages, I introduce a wide range of software tools, but I keep my descriptions brief. More-comprehensive introductions can always be found else- where, and I’m more eager to delve into what those tools can do for you, and how they can aid you in your research and development. Focus always returns to the key concepts and challenges that are unique to each project in data science, and the process of organizing and harnessing available resources and information to achieve the project’s goals. To get the most out of this book, you should be reasonably comfortable with ele- mentary statistics—a college class or two is fine—and have some basic knowledge of a programming language. If you’re an expert in statistics, software development, or data science, you might find some parts of this book slow or trivial. That’s OK; skip or skim sections if you must. I don’t hope to replace anyone’s knowledge and experi- ence, but I do hope to supplement them by providing a conceptual framework for working through data science projects, and by sharing some of my own experiences in a constructive way. If you’re a beginner in data science, welcome to the field! I’ve tried to describe concepts and topics throughout the book so that they’ll make sense to just about any- one with some technical aptitude. Likewise, colleagues and managers of data scientists and developers might also read this book to get a better idea of how the data science process works from an inside perspective. For every reader, I hope this book paints a vivid picture of data science as a process with many nuances, caveats, and uncertainties. The power of data science lies not in figuring out what should happen next, but in realizing what might happen next and eventually finding out what does happen next. My sincere hope is that you enjoy the book and, more importantly, that you learn some things that increase your chances of success in the future. Roadmap The book is divided into three parts, representing the three major phases of the data science process. Part 1 covers the preparation phase: ■ Chapter 1 discusses my process-oriented perspective of data science projects and introduces some themes and concepts that are present throughout the book. ■ Chapter 2 covers the deliberate and important step of setting good goals for the project. Special focus is given to working with the project’s customer to gener- ate practical questions to address, and also to being pragmatic about the data’s ability to address those questions. ■ Chapter 3 delves into the exploration phase of a data science project, in which we try to discover helpful sources of data. I cover some helpful methods of data

The above is a preview of the first 20 pages. Register to read the complete e-book.

Support Author

0.00

Total Amount (¥)

Donation Count

Donation Amount (¥) Please enter an amount Minimum ¥1

Your name (optional)

Message (Optional)

Anonymous Donation

You will be redirected to Alipay to complete payment, then return here.

Recommended for You

Loading recommended books...

Failed to load, please try again later

← Back to List

Think Like a Data Scientist Tackle the Data Science Process Step-by-Step (Brian Godsey) (z-library.sk, 1lib.sk, z-lib.sk)