Probabilistic Machine Learning for Finance and Investing A Primer to Generative AI with Python (Deepak K. Kanungo)（Z-Library）

Prob a b ilistic M a chine Lea rning Prob a b ilistic M a chine Lea rning for Fina nce a nd Investing for Fina nce a nd Investing Deepak K. Kanungo Probabilistic Machine Learning for Finance and Investing A Primer to Generative AI with Python

MACHINE LE ARNING “In his no-nonsense defiant style, Kanungo dismisses modern orthodoxies to deliver a superb analysis of probabilistic machine learning; not as a solution, but the most sensible way forward for FinTech.” —Ian Angell Professor Emeritus, London School of Economics “Explaining the flaws of conventional models, and the realistic predictions of probabilistic ML models for finance and investing, this book is a significant leap forward in minimizing the reliance on intuition.” —Bruno Rignel Chief Investment Officer, Alpha Key Capital Management Probabilistic Machine Learning for Finance and Investing Twitter: @oreillymedia linkedin.com/company/oreilly-media youtube.com/oreillymedia There are several reasons why probabilistic machine learning represents the next-generation ML framework and technology for finance and investing. This generative ensemble learns continually from small and noisy financial datasets while seamlessly enabling probabilistic inference, retrodiction, prediction, and counterfactual reasoning. Probabilistic ML also lets you systematically encode personal, empirical, and institutional knowledge into ML models. Whether they’re based on academic theories or ML strategies, all financial models are subject to modeling errors that can be mitigated but not eliminated. Probabilistic ML systems treat uncertainties and errors of financial and investing systems as features, not bugs. And they quantify uncertainty generated from inexact inputs and outputs as probability distributions, not point estimates. This makes for realistic financial inferences and predictions that are useful for decision-making and risk management. Unlike conventional AI, these systems are capable of warning us when their inferences and predictions are no longer useful in the current market environment. By moving away from flawed statistical methodologies and a restrictive conventional view of probability as a limiting frequency, you’ll move toward an intuitive view of probability as logic within an axiomatic statistical framework that comprehensively and successfully quantifies uncertainty. This book shows you how. Deepak K. Kanungo is an algorithmic derivatives trader, instructor, and CEO of Hedged Capital LLC, an AI-powered proprietary trading company he founded in 2009. Since 2019, Deepak has taught tens of thousands of O’Reilly Media subscribers worldwide the concepts, processes, and machine learning technologies for algorithmic trading, investing, and finance with Python. He also served as a financial advisor at Morgan Stanley during the Great Financial Crisis. US $79.99 CAN $99.99 ISBN: 978-1-492-09767-9

Deepak K. Kanungo Probabilistic Machine Learning for Finance and Investing A Primer to Generative AI with Python Boston Farnham Sebastopol TokyoBeijing

978-1-492-09767-9 [LSI] Probabilistic Machine Learning for Finance and Investing by Deepak K. Kanungo Copyright © 2023 Hedged Capital L.L.C. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Michelle Smith Development Editor: Jeff Bleiel Production Editor: Aleeya Rahman Copyeditor: nSight, Inc. Proofreader: Liz Wheeler Indexer: Sue Klefstad Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea August 2023: First Edition Revision History for the First Edition 2023-08-14: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781492097679 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Probabilistic Machine Learning for Finance and Investing, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the author and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

To my parents, from whom I learned common sense and what it means to be human.

Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix 1. The Need for Probabilistic Machine Learning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Finance Is Not Physics 2 All Financial Models Are Wrong, Most Are Useless 4 The Trifecta of Modeling Errors 6 Errors in Model Specification 6 Errors in Model Parameter Estimates 7 Errors from the Failure of a Model to Adapt to Structural Changes 10 Probabilistic Financial Models 10 Financial AI and ML 12 Probabilistic ML 16 Probability Distributions 17 Knowledge Integration 17 Parameter Inference 19 Generative Ensembles 19 Uncertainty Awareness 20 Summary 20 References 21 Further Reading 22 2. Analyzing and Quantifying Uncertainty. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 The Monty Hall Problem 24 Axioms of Probability 26 Inverting Probabilities 30 v

Simulating the Solution 33 Meaning of Probability 35 Frequentist Probability 36 Epistemic Probability 37 Relative Probability 40 Risk Versus Uncertainty: A Useless Distinction 41 The Trinity of Uncertainty 44 Aleatory Uncertainty 44 Epistemic Uncertainty 46 Ontological Uncertainty 49 The No Free Lunch Theorems 51 Investing and the Problem of Induction 54 The Problem of Induction, NFL Theorems, and Probabilistic Machine Learning 59 Summary 60 References 61 3. Quantifying Output Uncertainty with Monte Carlo Simulation. . . . . . . . . . . . . . . . . . . . 63 Monte Carlo Simulation: Proof of Concept 64 Key Statistical Concepts 66 Mean and Variance 66 Expected Value: Probability-Weighted Arithmetic Mean 67 Why Volatility Is a Nonsensical Measure of Risk 68 Skewness and Kurtosis 69 The Gaussian or Normal Distribution 70 Why Volatility Underestimates Financial Risk 71 The Law of Large Numbers 75 The Central Limit Theorem 76 Theoretical Underpinnings of MCS 77 Valuing a Software Project 78 Building a Sound MCS 83 Summary 84 References 85 4. The Dangers of Conventional Statistical Methodologies. . . . . . . . . . . . . . . . . . . . . . . . . . 87 The Inverse Fallacy 88 NHST Is Guilty of the Prosecutor’s Fallacy 94 The Confidence Game 98 Single-Factor Market Model for Equities 100 vi | Table of Contents

Simple Linear Regression with Statsmodels 101 Confidence Intervals for Alpha and Beta 104 Unveiling the Confidence Game 105 Errors in Making Probabilistic Claims About Population Parameters 105 Errors in Making Probabilistic Claims About a Specific Confidence Interval 106 Errors in Making Probabilistic Claims About Sampling Distributions 106 Summary 109 References 111 Further Reading 112 5. The Probabilistic Machine Learning Framework. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 Investigating the Inverse Probability Rule 114 Estimating the Probability of Debt Default 118 Generating Data with Predictive Probability Distributions 123 Summary 127 Further Reading 128 6. The Dangers of Conventional AI Systems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 AI Systems: A Dangerous Lack of Common Sense 130 Why MLE Models Fail in Finance 132 An MLE Model for Earnings Expectations 133 A Probabilistic Model for Earnings Expectations 136 Markov Chain Monte Carlo Simulations 142 Markov Chains 143 Metropolis Sampling 145 Summary 149 References 150 7. Probabilistic Machine Learning with Generative Ensembles. . . . . . . . . . . . . . . . . . . . . 151 MLE Regression Models 153 Market Model 154 Model Assumptions 155 Learning Parameters Using MLE 155 Quantifying Parameter Uncertainty with Confidence Intervals 156 Predicting and Simulating Model Outputs 156 Probabilistic Linear Ensembles 156 Prior Probability Distributions P(a, b, e) 158 Likelihood Function P(Y| a, b, e, X) 159 Marginal Likelihood Function P(Y|X) 159 Table of Contents | vii

Posterior Probability Distributions P(a, b, e| X, Y) 160 Assembling PLEs with PyMC and ArviZ 161 Define Ensemble Performance Metrics 162 Analyze Data and Engineer Features 164 Develop and Retrodict Prior Ensemble 168 Train and Retrodict Posterior Model 176 Test and Evaluate Ensemble Predictions 185 Summary 188 References 189 Further Reading 189 8. Making Probabilistic Decisions with Generative Ensembles. . . . . . . . . . . . . . . . . . . . . . 191 Probabilistic Inference and Prediction Framework 193 Probabilistic Decision-Making Framework 195 Integrating Subjectivity 195 Estimating Losses 197 Minimizing Losses 200 Risk Management 201 Capital Preservation 202 Ergodicity 202 Generative Value at Risk 206 Generative Expected Shortfall 209 Generative Tail Risk 210 Capital Allocation 211 Gambler’s Ruin 212 Expected Valuer’s Ruin 214 Modern Portfolio Theory 218 Markowitz Investor’s Ruin 221 Kelly Criterion 225 Kelly Investor’s Ruin 229 Summary 230 References 231 Further Reading 232 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233 viii | Table of Contents

Preface Generative AI, and Chat GPT-4 in particular, is all the rage these days. Probabilistic machine learning (ML) is a type of generative AI that is ideally suited for finance and investing. Unlike deep neural networks, on which ChatGPT is based, probabilistic ML models are not black boxes. These models also enable you to infer causes from effects in a fairly transparent manner. This is important in heavily regulated indus‐ tries, such as finance and healthcare, where you have to explain the basis of your deci‐ sions to many stakeholders. Probabilistic ML also enables you to explicitly and systematically encode personal, empirical, and institutional knowledge into ML models to sustain your organization’s competitive advantages. What truly distinguishes probabilistic ML from its conven‐ tional counterparts is its capability of seamlessly simulating new data and counterfac‐ tual knowledge conditioned on the observed data and model assumptions on which it was trained and tested, regardless of the size of the dataset or the ordering of the data. Probabilistic models are generative models that know their limitations and honestly express their ignorance by widening the ranges of their inferences and predictions. You won’t get such quantified doubts from ChatGPT’s confident hallucinations, more commonly known as fibs and lies. All ML models are built on the assumption that patterns discovered in training or in- sample data will persist in testing or out-of-sample data. However, when nonproba‐ bilistic ML models encounter patterns in data that they have never been trained or tested on, they make egregious inferences and predictions because of the inherent foundational flaws of their statistical models. Furthermore, these ML models do it with complete confidence and without warning decision makers of their uncertainties. The increasing adoption of nonprobabilistic ML models for decision making in finance and investments can lead to catastrophic consequences for individuals and society at large, including bankruptcies and economic recessions. It is imperative that all ML models quantify the uncertainty of their inferences and predictions on unseen ix

data to support sound decision making in a complex world with three-dimensional uncertainties. Leading companies clearly understand the limitations of standard AI technologies and are developing their probabilistic versions to extend their applica‐ bility to more complex problems. Google recently introduced TensorFlow Probability to extend its established TensorFlow platform. Similarly, Facebook and Uber have introduced Pyro to extend their PyTorch platforms. Currently, the most popular open source probabilistic ML technologies are PyMC and Stan. PyMC is written in Python, and Stan is written in C++. This book uses the extensive ecosystem of user-friendly Python libraries. Who Should Read This Book? The primary audience of this book is the thinking practitioner in the finance and investing discipline. A thinking practitioner is someone who doesn’t merely want to follow instructions from a manual or cookbook. They want to understand the under‐ lying concepts for why they must adopt a process, model, or technology. Generally, they are intellectually curious and enjoy learning for its own sake. At the same time, they are not looking for onerous mathematical proofs or tedious academic tomes. I have provided many scholarly references in each chapter for readers who are looking for the mathematical and technical details underlying the concepts and reasoning presented in this book. A thinking practitioner could be an individual investor, analyst, developer, manager, project manager, data scientist, researcher, portfolio manager, or quantitative trader. These thinking practitioners understand that they need to learn new concepts and technologies continually to advance their careers and businesses. A practical depth of understanding gives them the confidence to apply what they learn to develop creative solutions for their unique challenges. It also gives them a framework to explore and learn related technologies and concepts more easily. In this book, I am assuming that readers have a basic familiarity with finance, statis‐ tics, machine learning, and Python. I am not assuming that they have read any partic‐ ular book or mastered any particular skill. I am only assuming that they have a willingness to learn, especially when ChatGPT, Bard, and Bing AI can easily explain any code or formula in this book. Why I Wrote This Book There is a paucity of general probabilistic ML books, and none that is dedicated entirely to finance and investing problems. Because of the idiosyncratic complexities of these domains, any naive application of ML in general and probabilistic ML in par‐ ticular is doomed to failure. A depth of understanding of the foundations of these domains is pivotal to having any chance of succeeding. This book is a primer that x | Preface

endeavors to give the thinking practitioner a solid grounding in the foundational concepts of probabilistic ML and how to apply it to finance and investing problems, using simple math and Python code. There is another reason why I wrote this book. To this day, books are still a medium for serious discourse. I wanted to remind the readers about the continued grave flaws of modern financial theory and conventional statistical inference methodology. It is outrageous that these pseudoscientific methods are still taught in academia and prac‐ ticed in industry despite their deep flaws and pathetic performance. They continue to waste billions of research dollars producing junk studies, tarnish the reputation of the scientific enterprise, and contribute significantly to economic disasters and human misery. We are at a crossroads in the evolution of AI technologies, with most experts predict‐ ing exponential growth in its use, fundamentally transforming the way we live, work, and interact with one another. The danger that AI systems will take over humanity imminently is silly science fiction, because even the most advanced AI system lacks the common sense of a toddler. The real clear and present danger is that fools might end up developing and managing these powerful savants based on the spurious mod‐ els of conventional finance and statistics. This will most likely lead to catastrophes faster and bigger than we have ever experienced before. My criticisms are supported by simple math, common sense, data, and scholarly works that have been published over the past century. Perhaps one added value of this book is in retrieving many of those forgotten academic publications from the dusty archives of history and making readers aware of their insights in plain, unequivocal language using logic, simple math, or code that anyone with a high school degree can understand. Clearly, the conventional mode of expressing these criticisms hasn’t worked at all. The stakes for individuals, society, and the scientific enterprise are too high for us to care if plainly spoken mathematical and scientific truths might offend someone or tarnish a reputation built on authoring or supporting bogus theories. Navigating This Book The contents of this book may be divided into two logical parts interwoven unevenly throughout each chapter. One part examines the appalling uselessness of the prevail‐ ing economics, statistical, and machine learning models for finance and investing domains. The other part examines why probabilistic machine learning is a less wrong, more useful model for these problem domains. The singular focus of this pri‐ mer is on understanding the foundations of this complex, multidisciplinary field. Only pivotal concepts and applications are covered. Sometimes less is indeed more. The book is organized as follows, with each chapter having at least one of the main concepts in finance and investing applied in a hands-on Python code exercise: Preface | xi

• Chapter 1, “The Need for Probabilistic Machine Learning” examines some of the woeful inadequacies of theoretical finance, how all financial models are afflicted with a trifecta of errors, and why we need a systematic way of quantifying the uncertainty of our inferences and predictions. The chapter explains why proba‐ bilistic ML provides a useful framework for finance and investing. • Chapter 2, “Analyzing and Quantifying Uncertainty” uses the Monty Hall prob‐ lem to review the basic rules of probability theory, examine the meanings of probability, and explore the trinity of uncertainties that pervade our world. The chapter also explores the problem of induction and its algorithmic restatement, the no free lunch (NFL) theorems, and how they underpin finance, investing, and probabilistic ML. • Chapter 3, “Quantifying Output Uncertainty with Monte Carlo Simulation” reviews important statistical concepts to explain why Monte Carlo simulation (MCS), one of the most important numerical techniques, works by generating approximate probabilistic solutions to analytically intractable problems. • Chapter 4, “The Dangers of Conventional Statistical Methodologies” exposes the skullduggery of conventional statistical inference methodologies commonly used in research and industry, and explains why they are the main cause of false research findings that plague the social and economic sciences. • Chapter 5, “The Probabilistic Machine Learning Framework” explores the proba‐ bilistic machine framework and demonstrates how inference from data and sim‐ ulation of new data are logically and seamlessly integrated in this type of generative model. • Chapter 6, “The Dangers of Conventional AI Systems” exposes the dangers of conventional AI systems, especially their lack of basic common sense and how they are unaware of their own limitations, which pose massive risks to all their stakeholders and society at large. Markov chain Monte Carlo simulations are introduced as a dependent sampling method for solving complex problems in finance and investing. • Chapter 7, “Probabilistic Machine Learning with Generative Ensembles” explains how probabilistic machine learning is essentially a form of ensemble machine learning. It shows readers how to develop a prototype of a generative linear ensemble for regression problems in finance and investing using PyMC, Xarray, and ArviZ Python libraries. • Chapter 8, “Making Probabilistic Decisions with Generative Ensembles” shows how to apply generative ensembles to risk management and capital allocation decisions in finance and investing. The implications of ergodicity and the pitfalls of using ensemble averages for financial decision making are explored. The strengths and weaknesses of capital allocation algorithms, including the Kelly cri‐ terion, are examined. xii | Preface

Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program ele‐ ments such as variable or function names, databases, data types, environment variables, statements, and keywords. Constant width bold Shows commands or other text that should be typed literally by the user. Constant width italic Shows text that should be replaced with user-supplied values or by values deter‐ mined by context. This element signifies a tip or suggestion. This element signifies a general note. This element indicates a warning or caution. Using Code Examples Supplemental material (code examples) is available for download at https://oreil.ly/ supp-probabilistic-ML. If you have a technical question or a problem using the code examples, please send email to support@oreilly.com. Preface | xiii

This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require per‐ mission. We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Probabilistic Machine Learning for Finance and Investing by Deepak K. Kanungo (O’Reilly). Copyright 2023 Hedged Capital L.L.C., 978-1-492-09767-9.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit https://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-889-8969 (in the United States or Canada) 707-829-7019 (international or local) 707-829-0104 (fax) support@oreilly.com https://www.oreilly.com/about/contact.html xiv | Preface

We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/Probabilistic_ML. For news and information about our books and courses, visit https://oreilly.com. Find us on LinkedIn: https://linkedin.com/company/oreilly-media Follow us on Twitter: https://twitter.com/oreillymedia Watch us on YouTube: https://youtube.com/oreillymedia Acknowledgments I would like to thank Michelle Smith, Jeff Bleiel, and the entire O’Reilly Media team for making this book possible. It was a pleasure working with everyone, especially Jeff, whose honest and insightful feedback helped me improve the contents of this book. I would also like to thank the expert reviewers of my book, Abdullah Karasan, Juan Manuel Contreras, and Isaac Rhea, for their valuable comments. Furthermore, I would like to thank the following readers of the early releases of the book for their equally valuable feedback: Ian Angell, Bruno Rignel, Jonathan Hugenschmidt, Autumn Peters, and Mike Shwe. Preface | xv

CHAPTER 1 The Need for Probabilistic Machine Learning Essentially, all models are wrong, but some are useful. However, the approximate nature of the model must always be borne in mind. —George Box, eminent statistician A map will enable you to go from one geographic location to another. It is a very use‐ ful mathematical model for navigating the physical world. It becomes even more use‐ ful if you automate it into a GPS system using artificial intelligence (AI) technologies. However, neither the mathematical model nor the AI-powered GPS system will ever be able to capture the human experience and richness of the terrain it represents. That’s because all models have to simplify the complexities of the real world, thus enabling us to focus on some of the features of a phenomenon that interest us. George Box, an eminent statistician, famously said, “all models are wrong, but some are useful.” This deeply insightful quip is our mantra. We accept that all models are wrong because they are inadequate and incomplete representations of reality. Our goal is to build financial systems based on models and supporting technologies that enable useful inferences and predictions for decision making and risk management in the face of endemic uncertainty, incomplete information, and inexact measurements. All financial models, whether derived theoretically or discovered empirically by humans and machines, are not only wrong but are also at the mercy of three types of errors. In this chapter, we explain this trifecta of errors with an example from con‐ sumer credit and explore it using Python code. This exemplifies our claim that inac‐ curacies of financial models are features, not bugs. After all, we are dealing with people, not particles or pendulums. 1

1 David Orrell and Paul Wilmott, “Going Random,” in The Money Formula: Dodgy Finance, Pseudo Science, and How Mathematicians Took Over the Markets (West Sussex, UK: Wiley, 2017). Finance is not an accurate physical science like physics, dealing with precise estimates and predictions, as academia will have us believe. It is an inexact social study grap‐ pling with a range of values with varying plausibilities that change continually, often abruptly. We conclude the chapter by explaining why AI in general and probabilistic machine learning (ML) in particular offers the most useful and promising theoretical frame‐ work and technologies for developing the next generation of systems for finance and investing. What Is a Model? AI systems are based on models. A model maps functional relationships among its inputs and outputs variables based on assumptions and constraints. In general, input variables are called independent variables and output variables are called dependent variables. In high school, you learned that the equation of any line in the XY plane can be expressed as y = mx + b, where m is the slope and b is the y-intercept of the line. For example, if you assume that consumer spending—the output/dependent variable y— has a linear relationship with personal income—the input/independent variable x— the equation for the line is called a model for consumer spending. Moreover, the slope m and the intercept b are referred to as the model’s parameters. They are treated as constants, and their specific values define unique functional relationships or models. Depending on the type of functional relationships, the parameters, and the nature of inputs and outputs variables, models may be classified as deterministic or probabilis‐ tic. In a deterministic model, there are no uncertainties about the type of functional relationships, the parameters, or the inputs or outputs of the model. The exact oppo‐ site is true for probabilistic models discussed in this book. Finance Is Not Physics Adam Smith, generally recognized as the founder of modern economics, was in awe of Newton’s laws of mechanics and gravitation.1 Since then, economists have endea‐ vored to make their discipline into a mathematical science like physics. They aspire to formulate theories that accurately explain and predict the economic activities of human beings at the micro and macro levels. This desire gathered momentum in the early 20th century with economists like Irving Fisher and culminated in the econo‐ physics movement of the late 20th century. 2 | Chapter 1: The Need for Probabilistic Machine Learning