Practical Process Automation Orchestration and Integration in Microservices and Cloud Native Architectures (Bernd Ruecker) (Z-Library)

📄 File Format: PDF

💾 File Size: 19.4 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

Bernd Ruecker Practical Process Automation Orchestration and Integration in Microservices and Cloud Native Architectures

📄 Page 2

(This page has no text content)

📄 Page 3

Bernd Ruecker Practical Process Automation Orchestration and Integration in Microservices and Cloud Native Architectures Boston Farnham Sebastopol TokyoBeijing

📄 Page 4

Practical Process Automation by Bernd Ruecker Copyright © 2021 Bernd Ruecker. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Melissa Duffield Development Editor: Michele Cronin Production Editor: Deborah Baker Copyeditor: Rachel Head Proofreader: Kim Wimpsett Indexer: Potomac Indexing, LLC Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea March 2021: First Edition Revision History for the First Edition 2021-03-16: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781492061458 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Practical Process Automation, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the author, and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. This work is part of a collaboration between O’Reilly and Camunda. See our statement of editorial inde‐ pendence. 978-1-098-10645-8 [LSI]

📄 Page 5

Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix 1. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Process Automation 1 Wild West Integrations 4 Workflow Engines and Executable Process Models 7 A Business Scenario 9 Long-Running Processes 10 Business Processes, Integration Processes, and Workflows 11 Business–IT Collaboration 12 Business Drivers and the Value of Process Automation 13 Not Your Parents’ Process Automation Tools 14 A Brief History of Process Automation 14 The Story of Camunda 18 Conclusion 19 Part I. Fundamentals 2. Workflow Engines and Process Solutions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 The Workflow Engine 23 Core Capabilities 23 Additional Features of Workflow Platforms 25 Architecture 26 A Process Solution 28 An Executable Example 29 Applications, Processes, and Workflow Engines 37 Typical Workflow Tools in a Project’s Life Cycle 38 iii

📄 Page 6

Graphical Process Modeler 38 Collaboration Tools 40 Operations Tooling 41 Tasklist Applications 42 Business Monitoring and Reporting 42 Conclusion 44 3. Developing Process Solutions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Business Process Model and Notation (BPMN) 45 Start and End Events 48 The Token Concept: Implementing Control Flow 48 Sequence Flows: Controlling the Flow of Execution 49 Tasks: Units of Work 49 Gateways: Steering Flow 51 Events: Waiting for Something to Happen 52 Message Events: Waiting for a Trigger from the Outside 52 Combining Process Models and Programming Code 54 Publish/Subscribe to a Process 54 Referencing Code in Process Models 57 Using Prebuilt Connectors 58 Model or Code? 59 Testing Processes 62 Versioning of Process Solutions 63 Running Versions in Parallel 63 Conclusion 65 4. Orchestrate Anything. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 Orchestrate Software 68 Service-Oriented Architecture (SOA) Services 69 Microservices 70 Serverless Functions 71 Modular Monoliths 74 Deconstructing the Monolith 75 Orchestrate Decisions 76 Decision Model and Notation (DMN) 78 Decisions in a Process Model 80 Orchestrate Humans 81 Task Assignment 82 Additional Tool Support 84 The User Interface of User Tasks 85 Orchestrate RPA Bots 88 Orchestrate Physical Devices and Things 91 iv | Table of Contents

📄 Page 7

Conclusion 92 5. Championing Workflow Engines and BPMN. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 Limitations of Other Implementation Options 93 Hardcoded Processes 94 Batch Processing 94 Data Pipelines and Streaming 96 The Actor Model 98 Stateful Functions 99 Process Modeling Languages 100 Workflow Patterns 101 Benefits of Graphical Process Visualizations 102 Textual Process Modeling Approaches 104 Typical Concerns About Graphical Modeling 106 Graphical Versus Textual Approaches 108 Process Automation with Blockchain? 108 Conclusion 111 Part II. Process Automation in the Enterprise 6. Solution Architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 When to Use a Workflow Engine 115 Architecture Trade-Offs 116 Running the Workflow Engine 117 Decentralized Engines 117 Sharing Engines 118 Ownership of Process Models 119 Using the Workflow Engine as a Communication Channel 119 In-House Workflow Platforms 120 Performance and Scalability 121 Developer Experience and Continuous Delivery 122 Evaluating Workflow Engines 123 Conclusion 126 7. Autonomy, Boundaries, and Isolation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 Strong Cohesion and Low Coupling 127 Domain-Driven Design, Bounded Contexts, and Services 129 Boundaries and Business Processes 130 Respect Boundaries and Avoid Process Monoliths 131 Foster Your Understanding of Responsibilities 136 Long-Running Behavior Helps You Defend Boundaries 138 Table of Contents | v

📄 Page 8

How Processes Communicate Across Boundaries 139 Call Activities: Handy Shortcuts Only Within the Boundary 140 Crossing Boundaries Is an API Call 141 Decentralized Workflow Tooling 144 Conclusion 145 8. Balancing Orchestration and Choreography. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147 Event-Driven Systems 147 Emergent Behavior 150 Event Chains 150 The Risk of Distributed Monoliths 154 Contrasting Orchestration and Choreography 155 Introducing Commands 155 Messages, Events, and Commands 157 Terminology and Definitions 158 Avoiding Event Chains by Using Commands 158 The Direction of Dependency 161 Finding the Right Balance 162 Deciding Whether to Use Commands or Events 162 Mixing Commands and Events 162 Designing Responsibilities 165 Evaluating Change Scenarios to Validate Decisions 166 Debunking Common Myths 168 Commands Do Not Require Synchronous Communication 168 Orchestration Does Not Need to Be Central 170 Choreography Does Not Automatically Lead to More Decoupling 170 The Role of Workflow Engines 170 Conclusion 172 9. Workflow Engines and Integration Challenges. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 Communication Patterns for Service Invocation 173 Synchronous Request/Response 174 Asynchronous Request/Response 176 BPMN and Being Ready to Receive 178 Aggregating Messages 180 Poisoned and Dead Messages 181 Synchronous Facades Hiding Asynchronous Communication 181 Transactions and Consistency 183 Eventual Consistency 185 Business Strategies to Handle Inconsistency 186 The Saga Pattern and Compensation 187 Chaining Resources by Using the Outbox Pattern 189 vi | Table of Contents

📄 Page 9

Eventual Consistency Applies to Every Form of Remote Communication 191 The Importance of Idempotency 192 Conclusion 193 10. Business–IT Collaboration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195 A Typical Project 195 The Moral of the Story 199 Including All the People: BizDevOps 200 Development 200 Business 201 Operations 202 The Power of One Joined Model 205 From a Process Pyramid to a House 206 Who Does the Modeling? 209 Creating Better Process Models 211 Extracting (Integration) Logic into Subprocesses 211 Distinguishing Between Results, Exceptions, and Errors 213 Increasing Readability 215 Conclusion 217 11. Process Visibility. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 The Value of Process Visibility 219 Getting the Data 221 Leverage Audit Data from Your Workflow Engine 221 Model Events to Measure Key Performance Indicators 222 Status Inquiries 223 Understanding Processes That Span Multiple Systems 224 Observability and Distributed Tracing Tools 225 Custom Centralized Monitoring 226 Data Warehouses, Data Lakes, and Business Intelligence Tools 228 Process Mining 229 Process Event Monitoring 230 Current Market Dynamics 231 Setting Up Process Reporting and Monitoring 232 Typical Metrics and Reports 232 Allowing for a Deeper Understanding 233 Conclusion 234 Table of Contents | vii

📄 Page 10

Part III. Get Going! 12. The Journey to Introduce Process Automation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237 Understanding the Adoption Journey 238 Failures You Want to Avoid 238 A Success Story 240 The Pattern of Successful Adoption Journeys 242 Different Journeys for Different Scenarios 246 Starting Your Journey 248 Bottom-up Versus Top-down Adoption 249 Proofs of Concepts 250 Presenting the Business Case 251 Don’t Build Your Own Platform 253 Dos and Don’ts Around Reuse 254 From Project to Program: Scaling Adoption 254 Perception Management: What Is Process Automation? 255 Establishing a Center of Excellence 255 Managing Architecture Decisions 256 Decentralized Workflow Tooling 257 Roles and Skill Development 258 Conclusion 259 13. Parting Words. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261 Current Architecture Trends Influence Process Automation 261 Rethinking Business Processes and the User Experience 262 Where to Go from Here 264 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265 viii | Table of Contents

📄 Page 11

Preface I remember very clearly when I first decided to use a small open source workflow engine, implemented in Java, to write a piece of business software for a friend 20 years ago. This decision changed my life. I got very enthusiastic about process auto‐ mation and engaged in the community of that open source project. Ultimately this experience pushed me toward cofounding my own company, which went on to become the leading vendor of source-available process automation tooling (I could never, ever have dreamed of the big names now using our software!). My aim with this book is not only to share my excitement about process automation, but also to explain how to apply process automation technology in real life, in a pragmatic and developer-friendly way. But first, an anecdote. During high school a good friend of mine started their own business; a specialized retail store for graphics cards. You may remember these cards if you’ve assembled a computer—they could be “modded” to get more power out of the chip, which allowed gamers to buy cheaper cards and achieve better performance. The business model required handling each physical graphic card as an individual item and establishing very specific procedures around sales and distribution. My friend was successful with this business model. Actually, very successful. So suc‐ cessful that the process, which was based on manual handling and emails, broke down. Orders were delayed, and piles of graphics cards, as well as unprocessed returned parcels, started filling the rooms. We discussed remedies to this situation and finally ended up developing a piece of custom software that automated some of their processes while supporting the specifics of their business model. It had a pretty narrow focus, but helped them to remove all the piles of stuff. They reduced the cycle time so that orders were shipped within a day. Manual work in the redesigned process was reduced to steps that involved the physical goods (e.g., packing the parcel), while other tasks were automa‐ ted (generating and printing the invoice and the shipping label, sending customer confirmations, etc.). Customers got transparency into the status of their orders, and ix

📄 Page 12

we even provided a very simple self-service tracking portal. The software escalated issues if some process got stuck for too long, so it was no longer necessary to wait for customers to complain to take corrective action. Overall, as hands-on as the software was, it was a huge success. Back then I would never have phrased it like this, but I experienced the advantages of process automation firsthand: improved process quality, reduced cycle times, auto‐ mation of boring tasks, ability to scale, and reducing operational spend. Over the next 20 years, I saw core processes and support processes being automated in all industries. I saw NASA processing data from the Mars robot using an automa‐ ted process on Earth in order to send back control signals to space. I saw insurance companies automating onboarding and claim handling processes, including the reporting of accidents via apps and fully automated handling of these reports. I saw process automation technology being applied to trading and money transfer use cases, and to many different processes in telecommunication. I even saw actual lab robots being controlled by a workflow engine. Process automation is everywhere, and it is super exciting. The need for automation is growing almost on a daily basis. Digital transformation is happening, allowing completely new business models and requiring companies to change business pro‐ cesses at a fundamental level. Recently, the COVID-19 pandemic brought this into focus: businesses needed to switch from paperwork being signed on-site to electronic processes basically overnight; companies needed to scale complete processes that had been relatively uncommon before, like airlines canceling tickets and compensating for flights; and organizations rapidly pivoted to completely new business models, like the distribution of face masks. These are only a few examples of the bigger trend Gartner calls “hyperautomation.” Companies embark upon this journey for many reasons: existing processes might be too inefficient, too slow, too expensive to operate, impossible to scale, or simply not flexible enough to support new business models (or all of those things at the same time!). And manually executed or poorly automated processes don’t provide enough data to gain actionable insight into what is going on, making it hard to learn and adapt. This makes the business vulnerable to competitors that have already embraced digital transformation and process automation. Process automation typically addresses processes that need to be tailor-made to an organization’s needs. Therefore, they cannot be bought as off-the-shelf application software. Even if these processes are often the same across different organizations (e.g., customer onboarding, order management, claim settlement), the way each orga‐ nization designs and implements them is unique and can be a differentiator for them in their market. Process automation enables organizations to be more competitive, x | Preface

📄 Page 13

conduct their business more efficiently, save cost, increase revenue, and progress in their digital transformation. Chances are high that you work in such a company, maybe as a software architect, enterprise architect, business analyst, or developer. Process automation will be one of the key tools in your toolbox. My mission with this book is to help you on your journey by sharing what I’ve learned through 20 years of firsthand experience with process automation. Process Automation Tools and Techniques There are many ways to automate processes, from plain software development to batch processing, event-driven microservices, and any other development practice you can think of. But automating processes has specific characteristics and requirements, and there is dedicated software built for addressing these. Analysts define different software mar‐ ket categories that are related to process automation: for example, digital process automation (DPA), intelligent business process management suites (iBPMSs), low- code platforms, robotic process automation (RPA), microservice orchestration, pro‐ cess orchestration, process monitoring, process mining, decision support, and automation. All the different software categories provide tools and technologies that allow organi‐ zations to coordinate, automate, and improve business processes. These processes can include people, software, decisions, bots, and things. That’s a broad scope. So what will we focus on in this book? The Scope of This Book This book looks at how process automation can be applied in modern system archi‐ tectures and software development practices. It examines how tool support needs to look like to become a vital part of every developer’s toolbox. It demonstrates that the core component to make this happen is a lightweight and developer-friendly work‐ flow engine, which will be explored in great detail throughout the book. Along the way, we’ll discuss some typical misconceptions. Workflow engines are not alien in software development, like some people may expect. And even if neither ana‐ lyst reports nor tools from big vendors are particularly developer-focused or developer-friendly, there are alternative tools available today, as you will see through‐ out this book. Some of these might not fit into the categories mentioned earlier, but others do. Preface | xi

📄 Page 14

That said, I will not dedicate a lot of time to what analysts say about process automa‐ tion software, but focus on giving practical advice about workflow engines in the con‐ text of software development in modern architectures. In this context, I will weave together ideas from microservices, event-driven systems, and domain-driven design. This might give you a new perspective on process automation. Who This Book Is For This book targets software developers and software or system architects who want to learn about process automation. You might prefer to be called a software engineer instead of a developer, and that is perfectly OK. In this book I use the term soft‐ ware developer, simply because I had to decide on one. If you are a software developer, you might want to use a workflow engine in your application, service, or microservice to solve hands-on problems. This book will help you understand which problems a workflow engine can solve for you, and how to get started. If you are a system architect, this book will help you understand opportunities and pitfalls around process automation. It will guide you through some tough architec‐ tural decisions and trade-offs, including how using a workflow engine compares to alternative approaches or whether a workflow engine should be operated centrally. But you can also benefit if you work in other roles. For example: • If you are an IT manager, this book can help you make better-informed decisions and ask the right questions internally. • If you are a business analyst, this book can help you if you are motivated to think outside the box and understand the technical side of things. Overall, you will need some general experience in the field of software engineering, but no other specific knowledge. xii | Preface

📄 Page 15

The Architect Always Implements Discussing concepts is only half the fun if you cannot point to concrete code exam‐ ples. Runnable code forces you to be precise, to think about details you can leave out on the conceptual level—and, most importantly, it often explains things best. I am personally a big fan of the motto “the architect always implements.” The downside is that I have to decide on a concrete technology (which might not be the technology of your choice) and on concrete products (which might be outdated by the time the book is printed). I’ve attempted to be as vendor-neutral as possible, but as cofounder of a process automation vendor, Camunda, I am of course opinionated and tend to use the tooling I know best, which is that my company provides. My opinions of course also influence our product, which means some alignment is unavoidable. But as a process automation addict with 20 years of real-world experi‐ ence, this book is rooted in the frontline customer engagements that have formed those opinions. In some places I do use executable source code, as anything else would make it harder to understand certain concepts. In these cases I use the process automation platform from Camunda. Accompanying Website and Code Examples In addition to this book, you can find supplemental material (code examples, etc.) for download at https://ProcessAutomationBook.com. This website also links to source code available on GitHub. These examples will not only help you better understand the concepts described in the book, but also give you a great opportunity to play with technology whenever you are bored with reading. If you have a technical question or a problem using the code examples, please send an email to bookquestions@oreilly.com. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a significant amount of example code from this book into your product’s documentation does require permission. Preface | xiii

📄 Page 16

We appreciate, but generally do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Practical Process Auto‐ mation by Bernd Ruecker (O’Reilly). Copyright 2021 Bernd Ruecker, 978-1-492-06145-8.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. Feedback I am always happy to take any feedback via feedback@ProcessAutomationBook.com. How to Read This Book In general, I recommend that you read Chapter 1 and Chapter 2 first and sequentially. This gives you the basic knowledge to understand what the book covers and how it applies to your scenario. From there on, you might simply continue reading or fast-forward to the chapters that look most interesting to you. While there is of course some logical plot through‐ out the book, I tried to cross-reference in case you jump over certain parts. However, there are a few deviations I can recommend: • If you’ve had bad experiences with business process management (BPM) in the past, you might want to read “Not Your Parents’ Process Automation Tools” on page 14 first, as this should assure you that you have the right book in your hands. • If you have experience with event-driven systems and believe you don’t need orchestration, you might want to sneak a peek at Chapter 8 to get a better feeling for why this book is relevant to you. Also look at Chapter 2 to get a better under‐ standing of what I mean by process automation. • If you are a fan of microservices or domain-driven design (DDD), you might be skeptical about how process automation can fit into this world. I recommend that you read Chapter 7 early on, as this best demonstrates how the thinking about process automation in this book is different from many traditional approaches in the field. • If you are an IT manager who has been pulled into a business or process automa‐ tion project in an off-guard moment, you might want to start with Chapter 12, as this will give you some guidance on how to shape your journey. • If you are happy to follow my recommendation to use a BPMN-based workflow engine, you can skip Chapter 5. xiv | Preface

📄 Page 17

Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program ele‐ ments such as variable or function names, databases, data types, environment variables, statements, and keywords. This element signifies a tip or suggestion. This element signifies a general note. This element indicates a warning or caution. O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit http://oreilly.com. Preface | xv

📄 Page 18

How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) You can access the web page for this book, where we list errata, examples, and any additional information, at https://oreil.ly/Practical_Process_Automation. Email bookquestions@oreilly.com to comment or ask technical questions about this book. For news and more information about our books and courses, see our website at http://www.oreilly.com. Find us on Facebook: http://facebook.com/oreilly Follow us on Twitter: http://twitter.com/oreillymedia Watch us on YouTube: http://youtube.com/oreillymedia Acknowledgments I want to thank all the people who helped me to write this book. First and foremost that includes all the people I’ve met over the last decade, for example in the Camunda community, within customer projects, or at conferences. Countless discussions hel‐ ped me understand the world of process automation, and constant feedback shaped not only the Camunda platform, but also my teaching material around it. I want to thank each and every person at Camunda. Camunda is not only a great place to work, especially because of all the great colleagues, but it is also changing the world of process automation. What we’ve achieved with the company is far more than I could have ever dreamed of when I cofounded it. And every day is still a lot of fun, so let’s keep rolling. :-) Furthermore, I want to thank my good friend Martin Schimak, who helped me shape the initial thoughts captured in this book. Martin was also a great sparring partner for structuring the book. I am also very grateful to all the great tech reviewers who pro‐ vided super-helpful feedback. These folks invested a bunch of free time helping to improve this book, so thank you to (listed alphabetically) Tiese Barrell, Adam xvi | Preface

📄 Page 19

Bellamare, Rutger van Bergen, Colin Breck, Joe Bowbeer, Norbert Kuchenmeister, Kamil Litman, Chris McKinty, Surush Samani, Volker Stiehl, and all the others. Of course, I also thank my family for having endured not only a pandemic, but also me working on this book. And last but not least, I want to thank the whole team at O’Reilly for making the book-writing process not only painless, but pretty enjoyable. Preface | xvii

📄 Page 20

(This page has no text content)