Statistics for Data Scientists and Analysts Statistical approach to data-driven decision making using Python (Pant, DipendraMukhiya etc.) (z-library.sk, 1lib.sk, z-lib.sk)

Statistics for Data Scientists and Analysts Statistical approach to data- driven decision making using Python Dipendra Pant Suresh Kumar Mukhiya www.bpbonline.com

First Edition 2025 Copyright © BPB Publications, India ISBN: 978-93-65897-128 All Rights Reserved. No part of this publication may be reproduced, distributed or transmitted in any form or by any means or stored in a database or retrieval system, without the prior written permission of the publisher with the exception to the program listings which may be entered, stored and executed in a computer system, but they can not be reproduced by the means of publication, photocopy, recording, or by any electronic and mechanical means. LIMITS OF LIABILITY AND DISCLAIMER OF WARRANTY The information contained in this book is true to correct and the best of author’s and publisher’s knowledge. The author has made every effort to ensure the accuracy of these publications, but publisher cannot be held responsible for any loss or damage arising from any information in this book. All trademarks referred to in the book are acknowledged as properties of their respective owners but BPB Publications cannot guarantee the accuracy of this information. www.bpbonline.com

Dedicated to My dad Mahadev Pant and mom Nanda Pant My family members and my PhD Supervisor - Dipendra Pant My wife and children - Suresh Kumar Mukhiya

About the Authors Dipendra Pant is a Ph.D. candidate in Computer Science at the Norwegian University of Science and Technology (NTNU), Norway’s leading technical university. He holds Bachelor’s and Master’s degrees in Computer Engineering from Nepal, where he received the Chancellor’s Gold Medal from Kathmandu University for top Master’s grades. Before relocating to Norway, Dipendra gained experience in both academia and industry in Nepal and has published multiple high-quality research articles. Suresh Kumar Mukhiya is a Senior Software Engineer at Tryg Frorsikring Norge in Norway. He holds a Ph.D. in Computer Science from Høgskulen på Vestlandet HVL, Norway. He has extensive knowledge and experience in academia and the software industry, and have authored multiple books and high quality research articles.

About the Reviewer ❖ Dushyant Sengar is a senior consulting leader in the data science, AI, and financial services domain. His areas of expertise include credit risk modeling, customer and loyalty analytics, model risk management (MRM), ModelOps-driven product development, analytics strategies, and operations. He has managed analytics delivery and sales in Retail, Loyalty, and Banking domains at leading Analytics consulting firms globally where he was involved in practice development, delivery, training, and team building. Sengar has authored/co-authored 10+ books, peer- reviewed scientific publications, and media articles in industry publications and has presented as an invited speaker and participant at several national and international conferences. He has strong hands-on experience in data science (methods, strategies, and best practices) as well as in cross-functional team leadership, product strategy, people, program, and budget management. He is an active reader and passionate about helping organizations and individuals realize their full potential with AI.

Acknowledgements We would like to express our sincere gratitude to everyone who contributed to the completion of this book. First and foremost, we extend our heartfelt appreciation to our family for their unwavering support and encouragement throughout this journey. Their love has been a constant source of motivation. We are especially grateful to Laxmi Bhatta and Øystein Nytrø for their invaluable support and motivation during the writing process. We thank BPB Publications for arranging the reviewers, editors, and technical experts. Last but not least, we want to express our gratitude to the readers who have shown interest in our work. Your support and encouragement are deeply appreciated. Thank you to everyone who has played a part in making this book a reality.

Preface In an era where data is the new oil, the ability to extract meaningful insights from vast amounts of information has become an essential skill across various industries. Whether you are a seasoned data scientist, a statistician, a researcher, or someone beginning their journey in the world of data, understanding the principles of statistics and how to apply them using powerful tools like Python is crucial. This book was born out of our collective experience in academia and industry, where we recognized a significant gap between theoretical statistical concepts and their practical application using modern programming languages. We noticed that while there are numerous resources available on either statistics or Python programming, few integrate both in a hands-on, accessible manner tailored for data analysis and statistical modeling. "Statistics for Data Scientists and Analysts" is our attempt to bridge this gap. Our goal is to provide a comprehensive guide that not only explains statistical concepts but also demonstrates how to implement them using Python's rich ecosystem of libraries such as NumPy, Pandas, Matplotlib, Seaborn, SciPy, and scikit-learn. We believe that the best way to learn is by doing, so we've included numerous examples, code snippets, exercises, and real-world datasets to help you apply what you've learned immediately. Throughout this book, we cover a wide range of topics— from the fundamentals of descriptive and inferential statistics to advanced subjects like time series analysis,

survival analysis, and machine learning techniques. We've also dedicated a chapter to the emerging field of prompt engineering for data science, acknowledging the growing importance of AI and language models in data analysis. We wrote this book with a diverse audience in mind. Whether you have a background in Python programming or are new to the language, we've structured the content to be accessible without sacrificing depth. Basic knowledge of Python and statistics will be helpful but is not mandatory. Our aim is to equip you with the skills to explore, analyze, and visualize data effectively, ultimately empowering you to make informed decisions based on solid statistical reasoning. As you embark on this journey, we encourage you to engage actively with the material. Try out the code examples, tackle the exercises, and apply the concepts to your own datasets. Statistics is not just about numbers; it's a lens through which we can understand the world better. We are excited to share this knowledge with you and hope that this book becomes a valuable resource in your professional toolkit. Chapter 1: Foundations of Data Analysis and Python - In this chapter, you will learn the fundamentals of statistics and data, including their definitions, importance, and various types and applications. You will explore basic data collection and manipulation techniques. Additionally, you will learn how to work with data using Python, leveraging its powerful tools and libraries for data analysis. Chapter 2: Exploratory Data Analysis - This chapter introduces Exploratory Data Analysis (EDA), the process of examining and summarizing datasets using techniques like descriptive statistics, graphical displays, and clustering methods. EDA helps uncover key features, patterns, outliers,

and relationships in data, generating hypotheses for further analysis. You'll learn how to perform EDA in Python using libraries such as pandas, NumPy, SciPy, and scikit-learn. The chapter covers data transformation, normalization, standardization, binning, grouping, handling missing data and outliers, and various data visualization techniques. Chapter 3: Frequency Distribution, Central Tendency, Variability - Here, you will learn how to describe and summarize data using descriptive statistical techniques such as frequency distributions, measures of central tendency (mean, median, mode), and measures of variability (range, variance, standard deviation). You will use Python libraries like pandas, NumPy, SciPy, and Matplotlib to compute and visualize these statistics, gaining insights into how data values are distributed and how they vary. Chapter 4: Unraveling Statistical Relationships - This chapter focuses on measuring and examining relationships between variables using covariance and correlation. You will learn how these statistical measures assess how two variables vary together or independently. The chapter also covers identifying and handling outliers—data points that significantly differ from the rest, which can impact the validity of analyses. Finally, you will explore probability distributions, mathematical functions that model data distribution and the likelihood of various outcomes. Chapter 5: Estimation and Confidence Intervals - In this chapter, you will delve into estimation techniques, focusing on constructing confidence intervals for various parameters and data types. Confidence intervals provide a range within which the true population parameter is likely to lie with a certain level of confidence. You will learn how to calculate margin of error and determine sample sizes to assess the accuracy and precision of your estimates.

Chapter 6: Hypothesis and Significance Testing - This chapter introduces hypothesis testing and significance tests using Python. You will learn how to perform and interpret hypothesis tests for different parameters and data types, assessing the reliability and validity of results using p- values, significance levels, and statistical power. The chapter covers common tests such as t-tests, chi-square tests, and ANOVA, equipping you with the skills to make informed decisions based on statistical evidence. Chapter 7: Statistical Machine Learning - Here, you will learn how to implement various supervised learning techniques for regression and classification tasks, as well as unsupervised learning techniques for clustering and dimensionality reduction. Starting with the basics—training and testing data, loss functions, evaluation metrics, and cross-validation—you will implement models like linear regression, logistic regression, decision trees, random forests, and support vector machines. Using scikit-learn library you will build, train, and evaluate these models on real-world datasets. Chapter 8: Unsupervised Machine Learning - This chapter introduces unsupervised machine learning techniques that uncover hidden patterns in unlabeled data. We begin with clustering methods—including K-means, K- prototype, hierarchical clustering, and Gaussian mixture models—that group similar data points together. Next, we delve into dimensionality reduction techniques like Principal Component Analysis and Singular Value Decomposition, which simplify complex datasets while retaining essential information. Finally, we discuss model selection and evaluation strategies tailored for unsupervised learning, equipping you with the tools to assess and refine your models effectively.

Chapter 9: Linear Algebra, Nonparametric Statistics, and Time Series Analysis - In this chapter, you will explore advanced topics including linear algebra operations, nonparametric statistical methods that don't assume a specific data distribution, and time series analysis concepts for dealing with time-to-event data. Chapter 10: Generative AI and Prompt Engineering - This chapter introduces Generative AI and the concept of prompt engineering in the context of statistics and data science. You will learn how to write accurate and efficient prompts for AI models, understand the limitations and challenges associated with Generative AI, and explore tools like the GPT-4 API. This knowledge will help you effectively utilize Generative AI in data science tasks while avoiding common pitfalls. Chapter 11: Real World Statistical Applications - In the final chapter, you wil apply the concepts learned throughout the book to real-world data science projects. Covering the entire lifecycle from data cleaning and preprocessing to modeling and interpretation, you will work on projects involving statistical analysis of banking data and health data. This hands-on experience will help you implement data science solutions to practical problems, illustrating workflows and best practices in the field.

Code Bundle and Coloured Images Please follow the link to download the Code Bundle and the Coloured Images of the book: https://rebrand.ly/68f7c9 The code bundle for the book is also hosted on GitHub at https://github.com/bpbpublications/Statistics-for- Data-Scientists-and-Analysts. In case there’s an update to the code, it will be updated on the existing GitHub repository. We have code bundles from our rich catalogue of books and videos available at https://github.com/bpbpublications. Check them out! Errata We take immense pride in our work at BPB Publications and follow best practices to ensure the accuracy of our content to provide with an indulging reading experience to our subscribers. Our readers are our mirrors, and we use their inputs to reflect and improve upon human errors, if any, that may have occurred during the publishing processes involved. To let us maintain the quality and help us reach out to any readers who might be having difficulties due to any unforeseen errors, please write to us at : errata@bpbonline.com

Your support, suggestions and feedbacks are highly appreciated by the BPB Publications’ Family. Did you know that BPB offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.bpbonline.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at : business@bpbonline.com for more details. At www.bpbonline.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on BPB books and eBooks. Piracy If you come across any illegal copies of our works in any form on the internet, we would be grateful if you would provide us with the location address or website name. Please contact us at business@bpbonline.com with a link to the material. If you are interested in becoming an author If there is a topic that you have expertise in, and you are interested in either writing or contributing to a book, please visit www.bpbonline.com. We have worked with thousands of developers and tech professionals, just like you, to help them share their insights with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea. Reviews Please leave a review. Once you have read and used this book, why not leave a review on the site that you purchased it from? Potential readers can then see and use your unbiased opinion to make purchase decisions. We at BPB can understand what you think about our products, and our authors can see your feedback on their book. Thank you! For more information about BPB, please visit www.bpbonline.com. Join our book’s Discord space Join the book’s Discord Workspace for Latest updates, Offers, Tech happenings around the world, New Release and Sessions with the Authors:

https://discord.bpbonline.com

Table of Contents 1. Foundations of Data Analysis and Python Introduction Structure Objectives Environment setup Software installation Launch application Basic overview of technology Python pandas NumPy Sklearn Matplotlib Statistics, data and its importance Types of data Qualitative data Quantitative data Level of measurement Nominal data Ordinal data Discrete data Continuous data Interval data

Ratio data Distinguishing qualitative and quantitative data Univariate, bivariate, and multivariate data Univariate data and univariate analysis Bivariate data Multivariate data Data sources, methods, populations, and samples Data source Collection methods Population and sample Data preparation tasks Data quality Cleaning Missing values Imputation Duplicates Outliers Wrangling and manipulation Conclusion 2. Exploratory Data Analysis Introduction Structure Objectives Exploratory data analysis and its importance Data aggregation Mean Median Mode Variance

Standard deviation Quantiles Data normalization, standardization, and transformation Data normalization Normalization of NumPy array Normalization of pandas data frame Data standardization Standardization of NumPy array Standardization of data frame Data transformation Data binning, grouping, encoding Data binning Data grouping Data encoding Missing data, detecting and treating outliers Visualization and plotting of data Line plot Pie chart Bar chart Histogram Scatter plot Stacked area plot Dendrograms Violin plot Word cloud Graph Conclusion 3. Frequency Distribution, Central Tendency, Variability

Introduction Structure Objectives Measure of frequency Frequency tables and distribution Relative and cumulative frequency Measure of central tendency Measures of variability or dispersion Measure of association Covariance and correlation Chi-square Cramer’s V Contingency coefficient Measures of shape Skewness Kurtosis Conclusion 4. Unravelling Statistical Relationships Introduction Structure Objectives Covariance Correlation Outliers and anomalies Probability Probability distribution Uniform distribution Normal distribution