Statistics Done Wrong The Woefully Complete Guide (Alex Reinhart)（Z-Library）

M A K E S E N S E O F Y O U R D A T A , T H E R I G H T W A Y . • Choosing the right sample size and avoiding • Reporting your analysis and publishing your data and source code • Procedures to follow, precautions to take, and analytical software that can help Scientific progress depends on good research, and good research needs good statistics. But statistical analysis is tricky to get right, even for the best and brightest of us. You’d be surprised how many scientists are doing it wrong. Statistics Done Wrong is a pithy, essential guide to statistical blunders in modern science that will show you how to keep your research blunder-free. You’ll examine embarrassing errors and omissions in recent research, learn about the misconceptions and scientific politics that allow these mistakes to happen, and begin your quest to reform the way you and your peers do statistics. You’ll find advice on: false positives • Asking the right question, designing the right experiment, choosing the right statistical analysis, and sticking to the plan • How to think about p values, significance, insignificance, confidence intervals, and regression Scientists: Read this concise, powerful guide to help you produce statistically sound research. Statisticians: Give this book to everyone you know. SHELVE IN: M ATHEM ATICS/ PROBABILITY & STATISTICS $24.95 ($28.95 CDN) A B O U T T H E A U T H O R Alex Reinhart is a statistics instructor and PhD student at Carnegie Mellon University. He received his BS in physics at the University of Texas at Austin and does research on Statistics Done Wrong. The first step toward statistics done right is THE F INEST IN GEEK ENTERTA INMENT ™ www.nostarch.com locating radioactive devices using statistics and physics. FPO T H E W O E F U L L Y C O M P L E T E G U I D E A L E X R E I N H A R T S T A T I S T I C S D O N E W R O N G S T A T IS T IC S D O N E W R O N G S T A T IS T IC S D O N E W R O N G R E IN H A R T

StatiSticS Done Wrong

S t a t i S t i c S D o n e W R o n g t h e W o e f u l l y c o m p l e t e g u i d e by Alex Reinhart San Francisco

STATISTICS DONE WRONG. Copyright © 2015 by Alex Reinhart. All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any informa- tion storage or retrieval system, without the prior written permission of the copyright owner and the publisher. 19 18 17 16 15 1 2 3 4 5 6 7 8 9 ISBN-10: 1-59327-620-6 ISBN-13: 978-1-59327-620-1 Publisher: William Pollock Production Editor: Alison Law Cover Illustration: Josh Ellingson Developmental Editors: Greg Poulos and Leslie Shen Technical Reviewer: Howard Seltman Copyeditor: Kim Wimpsett Compositor: Alison Law Proofreader: Emelie Burnette For information on distribution, translations, or bulk sales, please contact No Starch Press, Inc. directly: No Starch Press, Inc. 245 8th Street, San Francisco, CA 94103 phone: 415.863.9900; info@nostarch.com www.nostarch.com Library of Congress Cataloging-in-Publication Data Reinhart, Alex, 1991- Statistics done wrong : the woefully complete guide / by Alex Reinhart. pages cm Includes index. Summary: "Discusses how to avoid the most common statistical errors in modern research, and perform more accurate statistical analyses" - Provided by publisher. ISBN 978-1-59327-620-1 - ISBN 1-59327-620-6 1. Statistics-Methodology. 2. Missing observations (Statistics) I. Title. QA276.R396 2015 519.5-dc23 2015002128 The xkcd cartoon by Randall Munroe is available under the Creative Commons Attribution- NonCommercial 2.5 License. No Starch Press and the No Starch Press logo are registered trademarks of No Starch Press, Inc. Other product and company names mentioned herein may be the trademarks of their respective owners. Rather than use a trademark symbol with every occurrence of a trademarked name, we are using the names only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The information in this book is distributed on an “As Is” basis, without warranty. While every precaution has been taken in the preparation of this work, neither the author nor No Starch Press, Inc. shall have any liability to any person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the information contained in it.

The first principle is that you must not fool yourself, and you are the easiest person to fool. —RICHARD P. FEYNMAN To consult the statistician after an experiment is finished is often merely to ask him to conduct a post mortem examination. He can perhaps say what the experiment died of. —R.A. FISHER

About the Author Alex Reinhart is a statistics instructor and PhD student at Carnegie Mellon University. He received his BS in physics at the University of Texas at Austin and does research on locating radioactive devices using physics and statistics.

BRIEF CONTENTS Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Chapter 1: An Introduction to Statistical Significance . . . . . . . . . . . 7 Chapter 2: Statistical Power and Underpowered Statistics . . . . . . 15 Chapter 3: Pseudoreplication: Choose Your Data Wisely . . . . . . . 31 Chapter 4: The p Value and the Base Rate Fallacy . . . . . . . . . . . . . 39 Chapter 5: Bad Judges of Significance . . . . . . . . . . . . . . . . . . . . . . . 55 Chapter 6: Double-Dipping in the Data . . . . . . . . . . . . . . . . . . . . . . . 63 Chapter 7: Continuity Errors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 Chapter 8: Model Abuse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 Chapter 9: Researcher Freedom: Good Vibrations? . . . . . . . . . . . . 89 Chapter 10: Everybody Makes Mistakes . . . . . . . . . . . . . . . . . . . . . . 97 Chapter 11: Hiding the Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .105 Chapter 12: What Can Be Done?. . . . . . . . . . . . . . . . . . . . . . . . . . . .119 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .131 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .147

CONTENTS IN DETA IL PREFACE xv Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii INTRODUCTION 1 1 AN INTRODUCTION TO STATISTICAL SIGNIFICANCE 7 The Power of p Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 Psychic Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Neyman-Pearson Testing . . . . . . . . . . . . . . . . . . . . . . . . 11 Have Confidence in Intervals . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 2 STATISTICAL POWER AND UNDERPOWERED STATISTICS 15 The Power Curve . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 The Perils of Being Underpowered . . . . . . . . . . . . . . . . . . . . . . . 18 Wherefore Poor Power? . . . . . . . . . . . . . . . . . . . . . . . . 20 Wrong Turns on Red . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Confidence Intervals and Empowerment . . . . . . . . . . . . . . . . . . 22 Truth Inflation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Little Extremes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 3 PSEUDOREPLICATION: CHOOSE YOUR DATA WISELY 31 Pseudoreplication in Action . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Accounting for Pseudoreplication . . . . . . . . . . . . . . . . . . . . . . . . 33 Batch Biology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Synchronized Pseudoreplication . . . . . . . . . . . . . . . . . . . . . . . . . 35

4 THE P VALUE AND THE BASE RATE FALLACY 39 The Base Rate Fallacy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 A Quick Quiz . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 The Base Rate Fallacy in Medical Testing . . . . . . . . . 42 How to Lie with Smoking Statistics . . . . . . . . . . . . . . . 43 Taking Up Arms Against the Base Rate Fallacy . . . . 45 If At First You Don’t Succeed, Try, Try Again . . . . . . . . . . . . . . 47 Red Herrings in Brain Imaging . . . . . . . . . . . . . . . . . . . . . . . . . . . 51 Controlling the False Discovery Rate . . . . . . . . . . . . . . . . . . . . . . 52 5 BAD JUDGES OF SIGNIFICANCE 55 Insignificant Differences in Significance . . . . . . . . . . . . . . . . . . . 55 Ogling for Significance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 6 DOUBLE-DIPPING IN THE DATA 63 Circular Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 Regression to the Mean . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 Stopping Rules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 7 CONTINUITY ERRORS 73 Needless Dichotomization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 Statistical Brownout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 Confounded Confounding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 8 MODEL ABUSE 79 Fitting Data to Watermelons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 Correlation and Causation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 Simpson’s Paradox . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 9 RESEARCHER FREEDOM: GOOD VIBRATIONS? 89 A Little Freedom Is a Dangerous Thing . . . . . . . . . . . . . . . . . . . . 91 Avoiding Bias . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 xii Contents in Detail

10 EVERYBODY MAKES MISTAKES 97 Irreproducible Genetics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 Making Reproducibility Easy . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 Experiment, Rinse, Repeat . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 11 HIDING THE DATA 105 Captive Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 Obstacles to Sharing . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 Data Decay . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 Just Leave Out the Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 Known Unknowns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 Outcome Reporting Bias . . . . . . . . . . . . . . . . . . . . . . . . 111 Science in a Filing Cabinet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 Unpublished Clinical Trials . . . . . . . . . . . . . . . . . . . . . . 114 Spotting Reporting Bias . . . . . . . . . . . . . . . . . . . . . . . . . 115 Forced Disclosure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 12 WHAT CAN BE DONE? 119 Statistical Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 Scientific Publishing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 Your Job . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126 NOTES 131 INDEX 147 Contents in Detail xiii

PREFACE A few years ago I was an undergraduate physics major at the University of Texas at Austin. I was in a seminar course, trying to choose a topic for the 25-minute presen- tation all students were required to give. “Something about conspiracy theories,” I told Dr. Brent Iverson, but he wasn’t satisfied with that answer. It was too broad, he said, and an engaging presentation needs to be focused and detailed. I studied the sheet of suggested topics in front of me. “How about scientific fraud and abuse?” he asked, and I agreed. In retrospect, I’m not sure how scientific fraud and abuse is a narrower subject than conspiracy theories, but it didn’t matter. After several slightly obsessive hours of research, I real- ized that scientific fraud isn’t terribly interesting—at least, not compared to all the errors scientists commit unintentionally. Woefully underqualified to discuss statistics, I nonetheless dug up several dozen research papers reporting on the numer- ous statistical errors routinely committed by scientists, read

and outlined them, and devised a presentation that satisfied Dr. Iverson. I decided that as a future scientist (and now a self- designated statistical pundit), I should take a course in statistics. Two years and two statistics courses later, I enrolled as a graduate student in statistics at Carnegie Mellon University. I still take obsessive pleasure in finding ways to do statistics wrong. Statistics Done Wrong is a guide to the more egregious sta- tistical fallacies regularly committed in the name of science. Because many scientists receive no formal statistical training— and because I do not want to limit my audience to the statisti- cally initiated—this book assumes no formal statistical training. Some readers may easily skip through the first chapter, but I suggest at least skimming it to become familiar with my expla- natory style. My goal is not just to teach you the names of common errors and provide examples to laugh at. As much as is pos- sible without detailed mathematics, I’ve explained why the statistical errors are errors, and I’ve included surveys showing how common most of these errors are. This makes for harder reading, but I think the depth is worth it. A firm understanding of basic statistics is essential for everyone in science. For those who perform statistical analyses for their day jobs, there are “Tips” at the end of most chapters to explain what statistical techniques you might use to avoid common pitfalls. But this is not a textbook, so I will not teach you how to use these techniques in any technical detail. I hope only to make you aware of the most common problems so you are able to pick the statistical technique best suited to your question. In case I pique your curiosity about a topic, a comprehen- sive bibliography is included, and every statistical misconcep- tion is accompanied by references. I omitted a great deal of mathematics in this guide in favor of conceptual understand- ing, but if you prefer a more rigorous treatment, I encourage you to read the original papers. I must caution you before you read this book. Whenever we understand something that few others do, it is tempting to find every opportunity to prove it. Should Statistics Done Wrong miraculously become a New York Times best seller, I expect to see what Paul Graham calls “middlebrow dismissals” in response to any science news in the popular press. Rather than taking the time to understand the interesting parts of scientific research, armchair statisticians snipe at news articles, using the vague xvi Preface

description of the study regurgitated from some overenthusi- astic university press release to criticize the statistical design of the research.* This already happens on most websites that discuss science news, and it would annoy me endlessly to see this book used to justify it. The first comments on a news article are always complaints about how “they didn’t control for this variable” and “the sample size is too small,” and 9 times out of 10, the commenter never read the scientific paper to notice that their complaint was addressed in the third paragraph. This is stupid. A little knowledge of statistics is not an excuse to reject all of modern science. A research paper’s statistical methods can be judged only in detail and in context with the rest of its methods: study design, measurement tech- niques, cost constraints, and goals. Use your statistical knowl- edge to better understand the strengths, limitations, and poten- tial biases of research, not to shoot down any paper that seems to misuse a p value or contradict your personal beliefs. Also, remember that a conclusion supported by poor statistics can still be correct—statistical and logical errors do not make a conclusion wrong, but merely unsupported. In short, please practice statistics responsibly. I hope you’ll join me in a quest to improve the science we all rely on. Acknowledgments Thanks to James Scott, whose statistics courses started my statis- tical career and gave me the background necessary to write this book; to Raye Allen, who made James’s homework assignments much more fun; to Matthew Watson and Moriel Schottlender, who gave invaluable feedback and suggestions on my drafts; to my parents, who gave suggestions and feedback; to Dr. Brent Iverson, whose seminar first motivated me to learn about statis- tical abuse; and to all the scientists and statisticians who have broken the rules and given me a reason to write. My friends at Carnegie Mellon contributed many ideas and answered many questions, always patiently listening as I tried to explain some new statistical error. My professors, particularly Jing Lei, Valérie Ventura, and Howard Seltman, prepared me with the necessary knowledge. As technical reviewer, Howard *Incidentally, I think this is why conspiracy theories are so popular. Once you believe you know something nobody else does (the government is out to get us!), you take every opportunity to show off that knowledge, and you end up reacting to all news with reasons why it was falsified by the government. Please don’t do the same with statistical errors. Preface xvii

caught several embarrassing errors; if any remain, they’re my responsibility, though I will claim they’re merely in keeping with the title of the book. My editors at No Starch dramatically improved the manu- script. Greg Poulos carefully read the early chapters and wasn’t satisfied until he understood every concept. Leslie Shen pol- ished my polemic in the final chapters, and the entire team made the process surprisingly easy. I also owe thanks to the many people who emailed me suggestions and comments when the guide became available online. In no particular order, I thank Axel Boldt, Eric Franzosa, Robert O’Shea, Uri Bram, Dean Rowan, Jesse Weinstein, Peter Hozák, Chris Thorp, David Lovell, Harvey Chapman, Nathaniel Graham, Shaun Gallagher, Sara Alspaugh, Jordan Marsh, Nathan Gouwens, Arjen Noordzij, Kevin Pinto, Elizabeth Page-Gould, and David Merfield. Without their com- ments, my explanations would no doubt be less complete. Perhaps you can join this list. I’ve tried my best, but this guide will inevitably contain errors and omissions. If you spot an error, have a question, or know a common fallacy I’ve missed, email me at alex@refsmmat.com. Any errata or updates will be published at http://www.statisticsdonewrong.com/ . xviii Preface

INTRODUCTION In the final chapter of his famous book How to Lie with Statistics, Darrell Huff tells us that “anything smacking of the medical profession” or backed by scientific laboratories and uni- versities is worthy of our trust—not uncondi- tional trust but certainly more trust than we’d afford the media or politicians.(After all, Huff’s book is filled with the misleading statistical trickery used in politics and the media.) But few people complain about statistics done by trained scientists. Scientists seek understanding, not ammuni- tion to use against political opponents. Statistical data analysis is fundamental to science. Open a random page in your favorite medical journal and you’ll be deluged with statistics: t tests, p values, proportional hazards models, propensity scores, logistic regressions, least-squares fits, and confidence intervals. Statisticians have provided scientists