Effective Software Testing A developers guide (Final Release) (Mauricio Aniche) (Z-Library)

📄 File Format: PDF

💾 File Size: 7.2 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

M A N N I N G Maurício Aniche Forewords by Arie van Deursen and Steve Freeman

📄 Page 2

Developer Builds a feature T o guide eesting t d velopment Requirement analysis Test-driven development (chapter 8) Design for testability (chapter 7) Design by contracts (chapter 4) Effective and systematic testing Specification testing (chapter 2) Bound yar testing (chapter 2) Structural testing (chapter 3) Intelligent testing Mutation testing (chapter 3) Larger tests Integration testing (chapter 9) System testing (chapter 9)Unit w h ds it ifferent roles and ilitiresponsib es Unit testing Property- based testing (chapter 5) Mocks, stubs, and fakes (chapter 6) Automated test suite T odeest c quality (chapter 10) The different techniques a developer should use to effectively and systematically test a software system

📄 Page 3

Effective Software Testing

📄 Page 4

(This page has no text content)

📄 Page 5

Effective Software Testing A DEVELOPER’S GUIDE MAURÍCIO ANICHE Forewords by ARIE VAN DEURSEN and STEVE FREEMAN MANN I NG SHELTER ISLAND

📄 Page 6

For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact Special Sales Department Manning Publications Co. 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 Email: orders@manning.com ©2022 by Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps. Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine. The author and publisher have made every effort to ensure that the information in this book was correct at press time. The author and publisher do not assume and hereby disclaim any liability to any party for any loss, damage, or disruption caused by errors or omissions, whether such errors or omissions result from negligence, accident, or any other cause, or from any usage of the information herein. Manning Publications Co. Development editors: Kristen Watterson and 20 Baldwin Road Toni Arritola PO Box 761 Technical development editor: Frances Buontempo Shelter Island, NY 11964 Review editor: Mihaela Batinić Production editor: Andy Marinkovich Copy editor: Tiffany Taylor Proofreader: Jason Everett Technical proofreader: Kevin Orr Typesetter: Dennis Dalinnik Cover designer: Marija Tudor ISBN: 9781633439931 Printed in the United States of America

📄 Page 7

brief contents 1 ■ Effective and systematic software testing 1 2 ■ Specification-based testing 30 3 ■ Structural testing and code coverage 63 4 ■ Designing contracts 97 5 ■ Property-based testing 117 6 ■ Test doubles and mocks 141 7 ■ Designing for testability 172 8 ■ Test-driven development 198 9 ■ Writing larger tests 215 10 ■ Test code quality 258 11 ■ Wrapping up the book 276v

📄 Page 8

(This page has no text content)

📄 Page 9

contents forewords xiii preface xvi acknowledgments xviii about this book xxi about the author xxv about the cover illustration xxvi 1 Effective and systematic software testing 1 1.1 Developers who test vs. developers who do not 2 1.2 Effective software testing for developers 11 Effective testing in the development process 12 ■ Effective testing as an iterative process 14 ■ Focusing on development and then on testing 14 ■ The myth of “correctness by design” 15 ■ The cost of testing 15 ■ The meaning of effective and systematic 15 The role of test automation 16 1.3 Principles of software testing (or, why testing is so difficult) 16 Exhaustive testing is impossible 16 ■ Knowing when to stop testing 17 ■ Variability is important (the pesticide paradox) 17 Bugs happen in some places more than others 17 ■ No matter what testing you do, it will never be perfect or enough 18 ■ Context is king 18 ■ Verification is not validation 18vii

📄 Page 10

CONTENTSviii1.4 The testing pyramid, and where we should focus 19 Unit testing 19 ■ Integration testing 20 ■ System testing 21 When to use each test level 23 ■ Why do I favor unit tests? 23 What do I test at the different levels? 24 ■ What if you disagree with the testing pyramid? 25 ■ Will this book help you find all the bugs? 27 2 Specification-based testing 30 2.1 The requirements say it all 31 Step 1: Understanding the requirements, inputs, and outputs 33 Step 2: Explore what the program does for various inputs 34 Step 3: Explore possible inputs and outputs, and identify partitions 35 ■ Step 4: Analyze the boundaries 37 ■ Step 5: Devise test cases 39 ■ Step 6: Automate the test cases 41 Step 7: Augment the test suite with creativity and experience 43 2.2 Specification-based testing in a nutshell 45 2.3 Finding bugs with specification testing 46 2.4 Specification-based testing in the real world 54 The process should be iterative, not sequential 54 ■ How far should specification testing go? 55 ■ Partition or boundary? It does not matter! 55 ■ On and off points are enough, but feel free to add in and out points 55 ■ Use variations of the same input to facilitate understanding 55 ■ When the number of combinations explodes, be pragmatic 56 ■ When in doubt, go for the simplest input 56 ■ Pick reasonable values for inputs you do not care about 56 ■ Test for nulls and exceptional cases, but only when it makes sense 56 ■ Go for parameterized tests when tests have the same skeleton 57 ■ Requirements can be of any granularity 57 How does this work with classes and state? 57 ■ The role of experience and creativity 59 3 Structural testing and code coverage 63 3.1 Code coverage, the right way 64 3.2 Structural testing in a nutshell 68 3.3 Code coverage criteria 69 Line coverage 69 ■ Branch coverage 69 ■ Condition + branch coverage 70 ■ Path coverage 71 3.4 Complex conditions and the MC/DC coverage criterion 72 An abstract example 72 ■ Creating a test suite that achieves MC/DC 73 3.5 Handling loops and similar constructs 75

📄 Page 11

CONTENTS ix3.6 Criteria subsumption, and choosing a criterion 75 3.7 Specification-based and structural testing: A running example 77 3.8 Boundary testing and structural testing 82 3.9 Structural testing alone often is not enough 82 3.10 Structural testing in the real world 84 Why do some people hate code coverage? 84 ■ What does it mean to achieve 100% coverage? 86 ■ What coverage criterion to use 88 MC/DC when expressions are too complex and cannot be simplified 88 ■ Other coverage criteria 89 ■ What should not be covered? 90 3.11 Mutation testing 90 4 Designing contracts 97 4.1 Pre-conditions and post-conditions 98 The assert keyword 99 ■ Strong and weak pre- and post- conditions 100 4.2 Invariants 102 4.3 Changing contracts, and the Liskov substitution principle 105 Inheritance and contracts 107 4.4 How is design-by-contract related to testing? 109 4.5 Design-by-contract in the real world 110 Weak or strong pre-conditions? 110 ■ Input validation, contracts, or both? 110 ■ Asserts and exceptions: When to use one or the other 112 ■ Exception or soft return values? 113 ■ When not to use design-by-contract 113 ■ Should we write tests for pre-conditions, post-conditions, and invariants? 114 ■ Tooling support 114 5 Property-based testing 117 5.1 Example 1: The passing grade program 118 5.2 Example 2: Testing the unique method 122 5.3 Example 3: Testing the indexOf method 124 5.4 Example 4: Testing the Basket class 129 5.5 Example 5: Creating complex domain objects 136 5.6 Property-based testing in the real world 137 Example-based testing vs. property-based testing 137 ■ Common issues in property-based tests 138 ■ Creativity is key 139

📄 Page 12

CONTENTSx6 Test doubles and mocks 141 6.1 Dummies, fakes, stubs, spies, and mocks 143 Dummy objects 143 ■ Fake objects 144 ■ Stubs 144 Mocks 144 ■ Spies 144 6.2 An introduction to mocking frameworks 145 Stubbing dependencies 145 ■ Mocks and expectations 150 Capturing arguments 153 ■ Simulating exceptions 157 6.3 Mocks in the real world 158 The disadvantages of mocking 159 ■ What to mock and what not to mock 160 ■ Date and time wrappers 164 ■ Mocking types you do not own 166 ■ What do others say about mocking? 168 7 Designing for testability 172 7.1 Separating infrastructure code from domain code 173 7.2 Dependency injection and controllability 181 7.3 Making your classes and methods observable 184 Example 1: Introducing methods to facilitate assertions 184 Example 2: Observing the behavior of void methods 186 7.4 Dependency via class constructor or value via method parameter? 189 7.5 Designing for testability in the real world 191 The cohesion of the class under test 192 ■ The coupling of the class under test 193 ■ Complex conditions and testability 193 Private methods and testability 193 ■ Static methods, singletons, and testability 194 ■ The Hexagonal Architecture and mocks as a design technique 194 ■ Further reading about designing for testability 195 8 Test-driven development 198 8.1 Our first TDD session 199 8.2 Reflecting on our first TDD experience 206 8.3 TDD in the real world 208 To TDD or not to TDD? 208 ■ TDD 100% of the time? 209 Does TDD work for all types of applications and domains? 209 What does the research say about TDD? 209 ■ Other schools of TDD 211 ■ TDD and proper testing 212

📄 Page 13

CONTENTS xi9 Writing larger tests 215 9.1 When to use larger tests 216 Testing larger components 216 ■ Testing larger components that go beyond our code base 224 9.2 Database and SQL testing 229 What to test in a SQL query 229 ■ Writing automated tests for SQL queries 231 ■ Setting up infrastructure for SQL tests 236 Best practices 238 9.3 System tests 239 An introduction to Selenium 239 ■ Designing page objects 242 Patterns and best practices 251 9.4 Final notes on larger tests 254 How do all the testing techniques fit? 254 ■ Perform cost/benefit analysis 255 ■ Be careful with methods that are covered but not tested 255 ■ Proper code infrastructure is key 255 ■ DSLs and tools for stakeholders to write tests 256 ■ Testing other types of web systems 256 10 Test code quality 258 10.1 Principles of maintainable test code 259 Tests should be fast 259 ■ Tests should be cohesive, independent, and isolated 259 ■ Tests should have a reason to exist 260 Tests should be repeatable and not flaky 260 ■ Tests should have strong assertions 261 ■ Tests should break if the behavior changes 261 ■ Tests should have a single and clear reason to fail 262 ■ Tests should be easy to write 262 ■ Tests should be easy to read 262 ■ Tests should be easy to change and evolve 266 10.2 Test smells 267 Excessive duplication 267 ■ Unclear assertions 268 ■ Bad handling of complex or external resources 268 ■ Fixtures that are too general 269 ■ Sensitive assertions 270 11 Wrapping up the book 276 11.1 Although the model looks linear, iterations are fundamental 276 11.2 Bug-free software development: Reality or myth? 277 11.3 Involve your final user 278 11.4 Unit testing is hard in practice 278

📄 Page 14

CONTENTSxii11.5 Invest in monitoring 279 11.6 What’s next? 280 appendix Answers to exercises 281 References 289 index 295

📄 Page 15

forewords In modern software development, software testing steers the design, implementation, evolution, quality assurance, and deployment of software systems. To be an effective developer, you must become an effective software tester. This book helps you to achieve that goal. Put simply, testing is nothing but executing a piece of software to see if it behaves as expected. But testing is also hard. Its difficulty surfaces when thinking about the full set of test cases to be designed and executed. Out of the infinitely many possible test cases, which one should you write? Did you do enough testing to move the system to production? What extra tests do you need? Why these tests? And, if you need to change the system, how should you set up the test suite so that it supports rather than impedes future change? This book doesn’t shy away from such complex questions. It covers key testing techniques like design by contract, property-based testing, boundary testing, test ade- quacy criteria, mutation testing, and the proper use of mock objects. Where relevant, it gives pointers to additional research papers on the topic. At the same time, this book succeeds in making sure the test cases themselves and the testing process remain as simple as can be justified. It does so by always taking the perspective of the developer who is actually designing and running the tests. The book is full of examples, ensuring that the reader can get started with applying the techniques in their own projects straight away. This book emerged out of a course taught at Delft University of Technology for many years. In 2003 I introduced a course on software testing in the undergraduatexiii

📄 Page 16

FOREWORDSxivcurriculum. In 2016, Maurício Aniche joined me in teaching the course, and in 2019 he took over the course entirely. Maurício is a superb lecturer, and in 2021 the stu- dents elected him as Teacher of the Year of the faculty of Electrical Engineering, Mathe- matics, and Computer Science. At TU Delft, we teach testing in the very first year of our Computer Science and Engineering bachelor program. It has been difficult finding a book that aligns with our vision that an effective software engineer must be an effective software tester. Many academic textbooks focus on research results. Many developer-oriented texts focus on specific tools or processes. Maurício Aniche’s Effective Software Testing fills that gap by finding the sweet spot between theory and practice. It is written with the working developer in mind, offer- ing you state-of-the-art software testing techniques. At the same time, it is perfect for undergraduate university courses, training the next generations of computer scientists to become effective software testers. —DR. ARIE VAN DEURSEN, Professor in Software Engineering, Delft University of Technology, The Netherlands Effective Software Testing by Maurício Aniche is a practical introductory book that helps developers test their code. It’s a compact tour through the essentials of software test- ing that covers major topics every developer should know about. The book’s combina- tion of theory and practice shows the depth of Maurício’s experience as an academic and as a working programmer. My own path into software was rather haphazard: some programming courses at university, ad-hoc training on the job, and eventually a conversion course leading to a PhD. This left me envious of programmers who had taken the right courses at the right time and had the theoretical depth that I lacked. I periodically discovered that one of my ideas, usually with a half-baked implementation, turned out to be an estab- lished concept that I hadn’t heard of. That’s why I think it’s important to read intro- ductory material, such as this book. Throughout much of my software life, I saw testing as a necessary evil that mostly involved the tedium of following text instructions by hand. Nowadays it’s obvious to most that test automation is best done by computers, but it’s taken decades for that to become so widely accepted. That’s why, to me, test-driven development, when I first came across it, initially seemed crazy—and then essential. That said, I see a lot of test code in the wild that really isn’t clear. Obviously, this is easier to see in hindsight, without the immediate pressure of deadlines or after the domain model has settled. But I believe that this test code would be improved if more programmers used the techniques described in this book to structure and reason about the problems they’re working on. This doesn’t mean that we all must turn into academics, but the light application of a few concepts can make a big difference. For example, I find design-by-contract helpful when working with components that

📄 Page 17

FOREWORDS xvmaintain state. I might not always add explicit pre- and post-conditions to my code, but the concepts help me to think about, or discuss, what the code should do. Obviously, software testing is a huge subject for developers, but this book is a good way to get started. And, for those of us who’ve been around a bit longer, it’s a good reminder of techniques that we’ve neglected or maybe missed the first time around. It’s also good to see sections on software testing as a practice, in particular the brief introduction to larger-scale testing and, my favorite, sustaining test code quality. So many real-life test suites turn into a source of frustration because they haven’t been maintained. Maurício’s experience shows in the practical guidance and heuristics that he includes in the explanation of each technique. He is careful to provide the tools, but lets the reader find their own path (although it’s probably a good idea to take his advice). And, of course, the contents of the book itself have been thoroughly tested as it was originally developed in the open for his course at TU Delft. On a personal note, I used to meet Maurício when I guest lectured for his course, after which we would stop for pickled herrings (a taste that is uniquely appealing to Northern European palates) at a historic market stall in the town center. We would discuss programming and testing techniques, and life in the Netherlands. I was impressed with his care to do his best for his students, and with his ideas for his research. I look forward to the day when I can get on the train to Delft again. —DR. STEVE FREEMAN, author of Growing Object-Oriented Software, Guided by Tests (Addison-Wesley Professional)

📄 Page 18

preface Every software developer remembers a specific bug that affected their career. Let me tell you about mine. In 2006, I was the technical lead for a small development team that was building an application to control payments at gas stations. At the time, I was finishing my computer science undergraduate studies and beginning my career as a software developer. I had only worked on two serious web applications previously. And as the lead developer, I took my responsibility very seriously. The system needed to communicate directly with gas pumps. As soon as a cus- tomer finished refueling, the gas pump notified our system, and the application started its process: gathering information about the purchase (type of fuel, quantity in liters), calculating the final price, taking the user through the payment process, and storing the information for future reporting. The software system had to run on a dedicated device with a 200 MHz processor, 2 MB of RAM, and a few megabytes of permanent storage. This was the first time any- one had tried to use the device for a business application. So, there was no previous project from which we could learn or borrow code. We also could not reuse any exter- nal libraries, and we even had to implement our own simplistic database. The system required refuelings, and simulating them became a vital part of our development flow. We would implement a new feature, run the system, run the sim- ulator, simulate a few gas purchases, and manually check that the system responded correctly. After a few months, we had implemented the important features. Our (manual) tests, including tests performed by the company, succeeded. We had a version thatxvi

📄 Page 19

PREFACE xviicould be tested in the wild! But real-world testing was not simple: an engineering team had to make physical changes at a gas station so the pumps could talk to our software. To my surprise, the company decided to schedule the first pilot in the Dominican Republic. I was excited not only to see my project go live but also to visit such a beauti- ful country. I was the only developer who traveled to the Dominican Republic for the pilot, so I was responsible for fixing any last-minute bugs. I watched the installation and fol- lowed along when the software ran for the first time. I spent the entire day monitoring the system, and everything seemed fine. That night we went out to celebrate. The beer was cold, and I was proud of myself. I went to bed early so I would be ready to meet the stakeholders the next morning and discuss the project’s next steps. But at 6:00 a.m., my hotel telephone rang. It was the owner of the pilot gas station: “The software apparently crashed during the night. The night workers did not know what to do, and the gas pumps were not delivering a sin- gle drop of fuel, so the station could not sell anything the entire night!” I was shaken. How could that have happened? I went straight to the site and started debugging the system. The bug was caused by a situation we had not tested: more refuelings than the system could handle. We knew we were using an embedded device with limited memory, so we had taken precau- tions. But we never tested what would happen if the limit was reached—and there was a bug! Our tests were all done manually: to simulate refueling, we went to the simulator, clicked a button on a pump, started pumping gas, waited some number of seconds (on the simulator, the longer we waited, the more liters of fuel we purchased), and then stopped the refueling process. If we wanted to simulate 100 gas purchases, we had to click 100 times in the simulator. Doing so was slow and painful. So, at develop- ment time, we tried only two or three refuelings. We probably tested the exception- handling mechanism once, but that was not enough. The first software system for which I was the lead developer did not even work a full day! What could I have done to prevent the bug? It was time for me to change how I was building software—and this led me to learn more about software testing. Sure, in college I had learned about many testing techniques and the importance of software testing, but you only recognize the value of some things when you need them. Today, I cannot imagine building a system without building an automated test suite along with it. The automated test suite can tell me in seconds whether the code I wrote is right or wrong, so I am much more productive. This book is my attempt to help developers avoid the mistakes I made.

📄 Page 20

acknowledgments This is not my first technical book, but it is the first one I have put my heart into. And it was only possible due to the help and inspiration of many people. First, by far the most important person who led me to write this book is Prof. Dr. Arie van Deursen. Arie was my post-doc supervisor and later my colleague in the Software Engineering Research Group (SERG) at Delft University of Technology. In 2017, he invited me to co-teach his software testing course for first-year computer science stu- dents (yes, Delft teaches software testing from the start!). While co-teaching with him, I learned a great deal about his views on theoretical and practical software testing. Arie’s passion for educating people on this topic inspired me, and I keep working to improve TU Delft’s software testing course (which is now my full responsibility). This book is a natural result of the interest he triggered in me years ago. Other colleagues at TU Delft have also influenced me significantly. Frank Mulder, who now co-teaches software testing with me, is a very experienced software developer and not afraid to challenge the software development status quo. I have lost count of how many discussions we have had about different practices over the years. We also take these discussions into the lecture hall, and our students have almost as much fun as we do as we present our views. Many of the pragmatic discussions in this book began as conversations with Frank. My thanks go to Wouter Polet. Wouter has been my teaching assistant for many years. When the Covid pandemic began, I told Wouter that we should make the lecture notes available for students who couldn’t attend class. He took that as a mission and quickly built a website containing transcripts of videos I had made a few years earlier.xviii