Troubleshooting Java Read, debug, and optimize JVM applications (Final Release) (Laurentiu Spilca [Laurențiu Spilcă]) (Z-Library)

M A N N I N G Laurenţiu Spilcă Read, debug, and optimize JVM applications Java TROUBLESHOOTING

Tips for applying investigation techniques  Always look for the root cause of a problem before deciding how to solve it. Focusing on the apparent problem might only be sweeping it under the rug.  An exception thrown at run time is not always in itself the problem. It could be a conse- quence of the real problem. Always look for the root cause.  No one investigation technique applies to every troubleshooting situation. Remember that for most scenarios, you'll have to apply a combination of investigation techniques.  The more you are aware of all the investigation techniques, the easier it will be for you to find the proper combination to quickly solve a scenario.  In most cases, complex investigation techniques can help, but always remember that some- times a simple log line in the right place can do miracles.  Sometimes a good night’s sleep is better than any troubleshooting technique. In an app that behaves normally, you will see this type of a pattern. The memory fills, and at a certain point the GC cleans the unneeded data, freeing up the memory. These are moments in which the GC cleaned the unneeded data, making space for new data to be added in memory. When an app has a memory leak, you will see the used memory continuously grows. The GC makes efforts to free the memory but can’t deallocate enough objects since the app holds the references for most of them. Normal behavior Abnormal behavior

Troubleshooting Java READ, DEBUG, AND OPTIMIZE JVM APPLICATIONS LAURENŢIU SPILCĂ M A N N I N G SHELTER ISLAND

For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact Special Sales Department Manning Publications Co. 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 Email: orders@manning.com ©2023 by Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps. Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine. The author and publisher have made every effort to ensure that the information in this book was correct at press time. The author and publisher do not assume and hereby disclaim any liability to any party for any loss, damage, or disruption caused by errors or omissions, whether such errors or omissions result from negligence, accident, or any other cause, or from any usage of the information herein. Manning Publications Co. Development editor: Marina Michaels 20 Baldwin Road Technical development editor: Nick Watts PO Box 761 Review editor: Marina Michaels Shelter Island, NY 11964 Production editor: Deirdre S. Hiam Copy editor: Michele Mitchell Proofreader: Katie Tennant Technical proofreader: Jean-François Morin Typesetter: Gordan Salinovic Cover designer: Marija Tudor ISBN 9781617299773 Printed in the United States of America

iii contents preface vii acknowledgments ix about this book xi about the author xiv about the cover illustration xv PART 1 THE BASICS OF INVESTIGATING A CODEBASE .................1 1 Revealing an app’s obscurities 3 1.1 How to more easily understand your app 4 1.2 Typical scenarios for using investigation techniques 7 Demystifying the unexpected output 8 ■ Learning certain technologies 13 ■ Clarifying slowness 13 ■ Understanding app crashes 15 ■ What you will learn in this book 17 2 Understanding your app’s logic through debugging techniques 19 2.1 When analyzing code is not enough 21 2.2 Investigating code with a debugger 24 What is the execution stack trace, and how do I use it? 29 Navigating code with the debugger 34 2.3 When using the debugger might not be enough 40

CONTENTSiv 3 Finding problem root causes using advanced debugging techniques 43 3.1 Minimizing investigation time with conditional breakpoints 44 3.2 Using breakpoints that don’t pause the execution 48 3.3 Dynamically altering the investigation scenario 51 3.4 Rewinding the investigation case 54 4 Debugging apps remotely 61 4.1 What is remote debugging? 63 4.2 Investigating in remote environments 65 The scenario 66 ■ Finding issues in remote environments 67 5 Making the most of logs: Auditing an app’s behavior 79 5.1 Investigating issues with logs 83 Using logs to identify exceptions 84 ■ Using exception stack traces to identify what calls a method 85 ■ Measuring time spent to execute a given instruction 86 ■ Investigating issues in multithreaded architectures 87 5.2 Implementing logging 89 Persisting logs 89 ■ Defining logging levels and using logging frameworks 90 ■ Problems caused by logging and how to avoid them 97 5.3 Logs vs. remote debugging 101 PART 2 DEEP ANALYSIS OF AN APP’S EXECUTION ..................105 6 Identifying resource consumption problems using profiling techniques 107 6.1 Where would a profiler be useful? 108 Identifying abnormal usage of resources 108 ■ Finding out what code executes 109 ■ Identifying slowness in an app’s execution 110 6.2 Using a profiler 110 Installing and configuring VisualVM 110 ■ Observing the CPU and memory usage 113 ■ Identifying memory leaks 123 7 Finding hidden issues using profiling techniques 129 7.1 Sampling to observe executing code 130 7.2 Profiling to learn how many times a method executed 139

CONTENTS v 7.3 Using a profiler to identify SQL queries an app executes 141 Using a profiler to retrieve SQL queries not generated by a framework 141 ■ Using the profiler to get the SQL queries generated by a framework 146 ■ Using the profiler to get programmatically generated SQL queries 149 8 Using advanced visualization tools for profiled data 154 8.1 Detecting problems with JDBC connections 155 8.2 Understanding the app’s code design using call graphs 169 8.3 Using flame graphs to spot performance problems 171 8.4 Analyzing queries on NoSQL databases 175 9 Investigating locks in multithreaded architectures 178 9.1 Monitoring threads for locks 179 9.2 Analyzing thread locks 184 9.3 Analyzing waiting threads 193 10 Investigating deadlocks with thread dumps 202 10.1 Getting a thread dump 203 Getting a thread dump using a profiler 205 ■ Generating a thread dump from the command line 207 10.2 Reading thread dumps 210 Reading plain-text thread dumps 210 ■ Using tools to better grasp thread dumps 216 11 Finding memory-related issues in an app’s execution 221 11.1 Sampling and profiling for memory issues 222 11.2 Using heap dumps to find memory leaks 229 Obtaining a heap dump 230 ■ Reading a heap dump 234 Using the OQL console to query a heap dump 239 PART 3 FINDING PROBLEMS IN LARGE SYSTEMS ....................247 12 Investigating apps’ behaviors in large systems 249 12.1 Investigating communication between services 250 Using HTTP server probes to observe HTTP requests 251 ■ Using HTTP client probes to observe HTTP requests the app sends 254 Investigating low-level events on sockets 255 12.2 The relevance of integrated log monitoring 258

CONTENTSvi 12.3 Using deployment tools in investigations 264 Using fault injection to mimic hard-to-replicate issues 266 ■ Using mirroring to facilitate testing and error detection 267 appendix A Tools you’ll need 271 appendix B Opening a project 272 appendix C Recommended further reading 275 appendix D Understanding Java threads 277 appendix E Memory management in Java apps 293 index 305

vii preface What does a software developer actually do for a living? “Implement software” is the most common answer to this question. But what does that mean? Is it only writing code? Well, no. While it is true that code is the result of everything a software devel- oper does, the activity of writing code takes only a small part of a software developer’s working time. Most of a software developer’s time is actually used designing solutions, reading existing code, understanding how it executes, and learning new things. Writ- ing code is the result of a software developer successfully accomplishing all of these tasks. Therefore, a programmer spends most of their time reading existing solutions rather than effectively writing new capabilities. Clean coding as a subject has, in the end, the same purpose: teaching developers how to write easier-to-read solutions. Developers realize that it’s more efficient to write an easier-to-read solution from the beginning than spend time trying to understand it later. But we need to be honest and admit that not all solutions are clean enough to quickly comprehend. We’ll always face scenarios in which we will need to understand the execution of some foreign capability. The reality is that software developers spend a lot of time investigating how apps work. They read and examine code in their app’s codebases and associated dependencies to figure out why something doesn’t work the way they expect. Developers sometimes read code only to learn about or better understand a given dependency. In many cases, reading code isn’t enough, and you have to find alternative (sometimes more complicated) ways to figure out what your app does. To understand how the environment affects your app or the JVM instance your Java app runs on, you

PREFACEviii may use a combination of profiling, debugging, and log investigations. If you know your options well and how to choose from among them, you will save valuable time. Remember, this is what developers spend most of their time doing. This development activity can be very beneficial. I designed this book to help people optimize the way they investigate software development challenges. In it, you’ll find the most relevant investigation techniques, which are applied with examples. We’ll discuss debugging, profiling, using logs, and efficiently combining these techniques. Throughout the book, I’ll give you valuable tips and tricks that will help you to become more efficient and solve problems (even the most difficult of them) faster. In other words, this book’s purpose, overall, is to make you more efficient as a developer. I hope this book brings significant value to you and helps you to become more effi- cient in quickly finding the root causes of issues you investigate.

ix acknowledgments This book wouldn’t be possible without the many smart, professional, and friendly people who helped me out throughout its development process. I want to say a big thanks to my wife Daniela, who was there for me, helped with valuable opinions, and continuously supported and encouraged me. I’d also like to send special thanks to all the colleagues and friends whose valuable advice helped me with the very first table of contents and proposal. I’d like to thank the entire Manning team for their huge help in making this a valuable resource. I’d especially want to call out Marina Michaels, Nick Watts, and Jean-François Morin for being incredibly supportive and professional. Their advice brought great value to this book. Thans go as well to Deirdre Hiam, my project man- ager; Michele Mitchell, my copyeditor; and Katie Tennant, my proofreader. I’d like to thank my friend Ioana Göz for the drawings she created for the book. She turned my thoughts into the cartoons you’ll see throughout the book. I’d also like to thank everyone who reviewed the manuscript and provided useful feedback that helped me improve the content of this book. I’d like to specifically call out the reviewers from Manning—Alex Gout, Alex Zuroff, Amrah Umudlu, Anand Natarajan, Andres Damian Sacco, Andriy Stosyk, Anindya Bandopadhyay, Atul Shrini- was Khot, Becky Huett, Bonnie Malec, Brent Honadel, Carl Hope, Cătălin Matei, Christopher Kardell, Cicero Zandona, Cosimo Damiano Prete, Daniel R. Carl, Desh- uang Tang, Fernando Bernardino, Gabor Hajba, Gaurav Tuli, Giampiero Granatella, Giorgi Tsiklauri, Govinda Sambamurthy, Halil Karaköse, Hugo Figueiredo, Jacopo Bis- cella, James R. Woodruff, Jason Lee, Javid Asgarov, Jean-Baptiste Bang Nteme, Jeroen

ACKNOWLEDGMENTSx van Wilgenburg, Joel Caplin, Jürg Marti, Krzysztof Kamyczek, Latif Benzzine, Leon- ardo Gomes da Silva, Manoj Reddy, Marcus Geselle, Matt Deimel, Matt Welke, Michael Kolesidis, Michael Wall, Michal Owsiak, Oliver Korten, Olubunmi Ogunsan, Paolo Brunasti, Peter Szabós, Prabhuti Prakash, Rajesh Balamohan, Rajesh Mohanan, Raveesh Sharma, Ruben Gonzalez-Rubio, Aboudou SamadouSare, Simeon Leyzerzon, Simone Cafiero, SravanthiReddy, Sveta Natu, Tan Wee, Tanuj Shroff, Travis Nelson, Yakov Boglev, and Yuri Klayman—as well friends who advised me: Maria Chițu, Adrian Buturugă, Mircea Vacariuc, Cătălin Matei.

xi about this book Who should read this book Since you opened this book, I assume you are a developer using a JVM language. You might use Java, but you could also use Kotlin or Scala. Regardless of the JVM language you’re using, you’ll find this book’s content valuable. It teaches you relevant investiga- tion techniques you can use to identify the root causes of problems (i.e., bugs) and how to easily learn new technologies. As a software developer, you may have already noticed how much time you spend understanding what an app does. Like other developers, you probably spend more time reading code, debugging, or using logs than writing code. So why not become more efficient in what you do most during your working day? In this book, we’ll discuss, and apply examples to, the following topics: ■ Simple and advanced debugging techniques ■ Efficiently using logs to understand app behaviors ■ Profiling CPU and memory resource consumption ■ Profiling to find executing code ■ Profiling to understand how an app works with persisted data ■ Analyzing how apps communicate with one another ■ Monitoring system events Regardless of your experience, you will find this book helpful in learning new investi- gation techniques, or, if you’re already an experienced developer, you will find this is a good refresher.

ABOUT THIS BOOKxii The prerequisite for reading this book is understanding the basics of the Java lan- guage. I intentionally designed all the examples with Java (even if they apply to any JVM language) for consistency. If you understand Java at a basic level (classes, meth- ods, basic instructions such as decisional or repetitive instructions and declaring vari- ables), you should be able to understand the discussions in the book. How this book is organized: A roadmap The book is divided into three parts that cover 12 chapters. We’ll start our discussion (in the first part of the book) with debugging techniques. We’ll discuss and apply both simple and more advanced debugging techniques and where you can use them to save time when investigating various scenarios. I chose to start our discussion with debug- ging because this is usually the first step in investigating how some capability of an app behaves during its development phase. Some people asked me why I didn’t start with logs first, since they are the first investigation technique for production issues. While this is true, a developer has to deal with a debugger when they start implementing fea- tures, so I figured a better arrangement of the chapters would be to begin with debug- ging techniques. In the first chapter, we discuss the relevance of the investigation techniques the book discusses and figure out a plan for learning them. Chapters 2, 3, and 4 focus on debugging and teach you relevant skills, from adding a simple breakpoint to debug- ging apps in remote environments. Chapter 5, which is the last chapter in part 1, dis- cusses logging. Debugging and using logs are the simplest (and most frequently used) investigation techniques for building an application. The second part of the book discusses profiling techniques. The popular opinion is that profiling is more advanced and less used with modern apps than debugging and researching logs. While I agree that profiling is more advanced, I demonstrate that you can use many profiling techniques to be more efficient when investigating issues in modern JVM apps or studying frameworks considered essential. Chapter 6, which begins the book’s second part, discusses identifying whether your app has faults in its management of CPU and memory resources. Chapter 7 goes into detail on this topic and shows you how to get to the part of the app that causes specific latencies and how to observe what your app executes at a given time. In chapters 6 and 7, we use VisualVM, a free tool. Chapter 8 continues the discussion from chapter 7 with more advanced visualization tools that you typically only get with a licensed profiling tool. For the details discussed in this chapter, we use JProfiler, which is not free to use. Chapters 9 and 10 focus on more subtle profiling techniques. You’ll learn skills that can save you time when dealing with issues deeply hidden in the multithreaded architecture behind an app’s execution. Chapter 11 ends part 2 by addresssing how to investigate an app’s memory management. The book ends with part 3, which has just one chapter: chapter 12. In it, we go beyond an app’s borders to discuss investigating issues in an extensive system com- posed of multiple apps.

ABOUT THIS BOOK xiii The chapters are in the order in which I recommend you read them, but each focuses on a different topic. So, if you are interested in a specific topic, you can jump directly to that chapter. For example, if you’re interested in investigating issues with memory management, you can go straight to chapter 11. About the code This book contains many examples of source code, both in numbered listings and in line with normal text. In both cases, source code is formatted in a fixed-width font like this to separate it from ordinary text. Sometimes code is also in bold to highlight code that has changed from previous steps in the chapter, such as when a new feature adds to an existing line of code. In many cases, the original source code has been reformatted; we’ve added line breaks and reworked indentation to accommodate the available page space in the book. In rare cases, even this was not enough, and listings include line-continuation markers (➥). Additionally, comments in the source code have often been removed from the listings when the code is described in the text. Code annotations accompany many of the listings and highlight important concepts. You can get executable snippets of code from the liveBook (online) version of this book at https://livebook.manning.com/book/troubleshooting-java. The complete code for the examples in the book is available for download from the Manning website at www.manning.com. liveBook discussion forum Purchase of Troubleshooting Java includes free access to liveBook, Manning’s online reading platform. Using liveBook’s exclusive discussion features, you can attach comments to the book globally or to specific sections or paragraphs. It’s easy to make notes for yourself, ask and answer technical questions, and receive help from the author and other users. To access the forum, go to https://livebook.manning.com/book/ troubleshooting-java/ discussion. You can also learn more about Manning’s forums and the rules of conduct at https://livebook.manning.com/discussion. Manning’s commitment to our readers is to provide a venue where a meaningful dialogue between individual readers and between readers and the author can take place. It is not a commitment to any specific amount of participation on the part of the author, whose contribution to the forum remains voluntary (and unpaid). We suggest you try asking him some challenging questions lest his interest stray! The forum and the archives of previous discussions will be accessible from the publisher’s website as long as the book is in print. Author online I recommend you keep in touch with me online. You’ll definitely find plenty of good learning material related to troubleshooting Java apps on my YouTube channel: youtube.com/c/laurentiuspilca, and you can follow me on Twitter @laurspilca.

xiv about the author LAURENŢIU SPILCĂ is a dedicated development lead and trainer at Endava, where he is responsible for leading and consulting on mul- tiple projects from various locations in Europe, the United States, and Asia. He has been working in software development since 2007. Laurenţiu believes it’s essential to not only deliver high-quality soft- ware but to also share knowledge and help others upskill. This belief has driven him to design and teach courses related to Java technologies and deliver pre- sentations and workshops. Laurenţiu is also the author of Spring Security in Action (Man- ning, 2020), and he recently finished Spring Start Here (Manning, 2021).

xv about the cover illustration The figure on the cover of Troubleshooting Java is “Homme de l’Istrie,” or “Man from Istria,” taken from a collection by Jacques Grasset de Saint-Sauveur, published in 1797. Each illustration is finely drawn and colored by hand. In those days, it was easy to identify where people lived and what their trade or sta- tion in life was just by their dress. Manning celebrates the inventiveness and initiative of the computer business with book covers based on the rich diversity of regional cul- ture centuries ago, brought back to life by pictures from collections such as this one.

Part 1 The basics of investigating a codebase As a software developer, working on real-world apps often involves investi- gating how your code works. You have to understand the app’s behavior when fixing problems as well as when implementing new features. You use several techniques for this purpose, such as debugging, logging, profiling, and so on, which we will analyze deeply in this book. In part 1, we start with the first techniques a developer is exposed to: debug- ging and logging. When working on an app, a developer must often engage in debugging. For example, say you have a small piece of code, and you need to understand how it works. You use the debugger to pause the application’s execu- tion and dive deep into how the app processes the data. Then, when your app runs in an environment, you can rely a lot on logs, which give you needed clues about where something could go wrong. In chapter 1, we’ll discuss the need for knowing investigation techniques and obtain a big-picture view of them, which we’ll detail throughout the rest of the book. We’ll then take these techniques in the order a developer is exposed to them. In chapters 2 through 4, we discuss debugging. In chapter 5, we go through essential details about implementing and using logs in investigations.

2 PART 1 The basics of investigating a codebase Reading code Starting to build an app Debugging Log analysis Profiling profile.ifPresentOrElse( p -> { healthMetric.setProfile(p); healthMetricRepository.save(healthMetric); }, () -> { throw new NonExistentHealthProfileException(); });