Flow Architectures The Future of Streaming and Event-Driven Integration (James Urquhart) (z-library.sk, 1lib.sk, z-lib.sk)

Page 1

James Urquhart Flow Architectures The Future of Streaming and Event-Driven Integration

Page 2

(This page has no text content)

Page 3

James Urquhart Flow Architectures The Future of Streaming and Event-Driven Integration Boston Farnham Sebastopol TokyoBeijing

Page 4

978-1-492-07589-9 [LSI] Flow Architectures by James Urquhart Copyright © 2021 James Urquhart. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Melissa Duffield Development Editor: Amelia Blevins Production Editor: Kate Galloway Copyeditor: Piper Editorial, LLC Proofreader: Justin Billing Indexer: WordCo Indexing Services, Inc. Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea January 2021: First Edition Revision History for the First Edition 2021-01-06: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781492075899 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Flow Architectures, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the author, and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

Page 5

To my children, Owen and Emery: You each inspire me every day. I am so very proud of you. And to my wife, Mia: You are my favorite.

Page 6

(This page has no text content)

Page 7

Table of Contents Foreword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi The 10-Year Impact of the World Wide Flow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii 1. Introduction to Flow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 What Is Flow? 3 Flow and Integration 6 Flow and Event-Driven Architectures 9 The Ancestors of Flow 12 Code and Flow 13 The Chapters Ahead 15 2. The Business Case for Flow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Drivers for Flow Adoption 18 Improving Customer Experience 18 Improved Organizational Efficiency 20 Innovation and Experimentation 24 Enablers of Flow Adoption 26 Lowering the Cost of Stream Processing 26 Increasing the Flexibility of Data Flow Design 29 Creating the Great Flow Ecosystem 33 What Businesses Will Require from Flow 36 The Effects of Flow Adoption 37 Expanding the Use of Timely Data 37 The Importance (and Peril) of Flow Networks 38 Flow’s Impact on Jobs and Expertise 40 v

Page 8

Flow and New Business and Institutional Models 41 Flow and Scale 43 Next Steps 44 3. Understanding the Flow Value Chain. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Recap: The High-Level Properties for Flow 45 Wardley Mapping and Promise Theory 46 Wardley Mapping 47 Promise Theory 48 Building a Flow Integration Value Chain 51 Establishing a Scope for the Map 51 Establishing Our Users and User Need 52 Flow Integration Components 53 Interaction Components 56 The Final Piece 60 Mapping Our Value Chain 61 Determining a Measure of Technology Evolution 61 Turning Our Value Chain into a Map 65 Our Final Model and Next Steps 70 4. Evaluating the Current Streaming Market. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 Service Buses and Message Queues 74 Message Queues 75 Service Buses 76 Mapping Service Buses and Message Queues 77 Internet of Things 78 MQTT 79 HTTP and WebSocket 81 Mapping Internet of Things Architectures 82 Event Processing 83 Functions, Low-Code, and No-Code Processors 84 Log-Based Stream Processing Platforms 85 Stateful Stream Processing 87 Mapping Event Processing Platforms 88 Streaming Architectures and Integration Today 89 Next Steps 91 5. Evaluating the Emergence of Flow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 Mapping the Evolution to Flow 94 Gameplay 97 Market: Standards Game 98 Accelerators: Exploiting Network Effects 99 vi | Table of Contents

Page 9

Ecosystem: Cocreation 100 The Others 101 Inertia 102 Vendor Inertia 102 Enterprise Inertia 104 Here Be Dragons 105 Flow Requirements, Challenges, and Opportunities 108 Security 108 Agility 113 Timeliness 116 Manageability 120 Memory 125 Control of Intellectual Property 126 Flow Pattern Challenges and Opportunities 128 The Collector Pattern 128 The Distributor Pattern 129 The Signal Pattern 130 The Facilitator Pattern 132 The Unexpected 133 6. Building for a Flow Future. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 Identifying Flow in Your Business 136 Flow Use Cases 137 Modeling Flow 148 “Event-First” Use Cases for Flow 151 Messaging Versus Eventing 152 Discrete Events Versus Event Series 155 Single Actions Versus Workflows 159 Driving Flow Forward 162 Driving Technology Development 162 Driving Flow Networks 166 We Will Make Flow Happen 170 Appendix. Evaluating the Current Flow Market. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 Table of Contents | vii

Page 10

(This page has no text content)

Page 11

Foreword As the human race continues its complicated relationship with technology, one thing is abundantly clear, the need for secure hyperconnectivity. Technology is being pushed to its limits to provide more convenience, personalization, or to save us time and money. It is being pushed to do more, and to do it faster and cheaper. How will technology keep up with our demands? Any technology that we interface with today is a distributed system that involves many components that all need to collaborate and communicate. Every information system, every modern technology, is a distributed system. The way these systems communicate can be viewed in a straightforward manner: they are either making statements or asking questions. Every communication is either trying to let someone else know you did something, or you are asking a question for which you need an answer. Modern terms for these statements are streams and services. All modern dis‐ tributed systems have service-oriented architecture components, and event stream processing has been rising in popularity. So, as we continue to push technology forward, how does it need to respond? For sys‐ tems to do more, they simply need more services and more events. These events and streams form flows of information that enter into the system to be analyzed for pat‐ terns, trends, and casualties. These new and expanded services give rise to new ways to interact with the technology, whether as a human or a digital system. For the sys‐ tem to act faster there are certainly areas of research to speed up the core backend systems, but in our modern world the majority of “faster” is just latency. How far away are you from the technology and can that technology move closer to you? This is what I term edge computing. There may be better definitions out there, but for me, edge computing is how technology continues to evolve to do things faster, and to do more. That is, getting closer to who asks the questions and being able to process more events and data without having to go to a central cloud resource. ix

Page 12

To put this into perspective, I’ll draw an analogy to banking. Not too long ago you had to physically drive to a bank to interact with any of the bank services, like depos‐ iting a check or withdrawing cash. We did have an interim wave where ATMs were popping up all over the place to allow more access points. Now we simply pull out our phones, or tablets, or in the worse case scenario possibly our laptops, to interact with every banking service one could imagine. Waiting in a restaurant for 5–10 minutes to have the waiter bring the check and return it to us for signature may be ok, but imagine being on your mobile phone and waiting 10 minutes to check out from Amazon. Speed matters, access matters, and secure access matters even more. As systems continue to need access to more information—to be able to process the information where it is relevant, and to move closer to where the system’s services are needed—it becomes clear that we need a new approach to connectivity. Not only do we need more things connected, we also need requesters, responders, processors, and analytics to be location transparent, mobile and secure. New technologies will emerge and give rise to a secure and hyperconnected world, not only for humans, but for all of the digital systems, services, and devices. All of these factors create an environment ripe for massive, exponential innovation. Think for a moment about what we did with the internet and the global cellular net‐ works, and the approximately five billion connected people. Now, imagine the possi‐ bilities when the world’s digital systems, services, and devices are always hyper- connected, location independent, secured, and can interact securely at any point in time. The experimentation phase goes into hyperdrive when secure access to diverse technologies, information and services has been democratized. When mixed with machine learning and artificial intelligence analyzing all of the events and data flows, the possibilities are endless. Once this secure and hyper-connected future becomes reality, what is next? How do these systems make sense of additional information, and how do they evolve to inter‐ act? In this book James walks us through the basic concepts of FLOW, and what sys‐ tems will need to do in order to take advantage of these new capabilities. Along the way you will begin to understand how FLOW affects and shapes modern distributed systems and ultimately our expectations of those systems and the technology they offer. We need a new approach to connecting, securing, and communicating within these modern systems, and this book is a great place to start understanding how these systems of the future will transform our world. — Derek Collison, creator of NATS.io x | Foreword

Page 13

Preface This is a book about the future of software integration and its effect on the evolution of economic processes. It is a book meant to bring technologists and decision makers on a journey to a future vision of fundamental change in the way we and our institu‐ tions work together. It is also, therefore, a book about opportunity. On the surface, it may seem like this is a book about technology. I will certainly sur‐ vey technologies available today or in the near future that contribute to the way activ‐ ity is communicated between organizations. However, the most important concept you should take from reading this book is the effect of new technologies on the evolu‐ tion of institutional cooperation and collaboration. The fluid, real-time communication of system state is much like the flow of water in a river, or the flow of traffic in a highway system; it is a flow of activity. The path this flow follows is defined by the available activity types, and the needs and desires of the entities that wish to consume them. The easier it is for data to find the right path to the right entities, the easier it is for the system to adapt and discover desired behaviors. Integration is key to the way our digital economy will evolve, as data is what drives economic activity. Today, integration across organizational boundaries is expensive and slow. We rely a lot on human action, bulk data processing, and historical data to make our markets run and our institutions operate. So, the question becomes, what if we could make the exchange of real-time data relatively cheap and nearly instantaneous? With the evolution of technologies that I describe in this book, it will soon become much less expensive to connect cause and effect across the internet. This, in turn, will lead to both dramatic growth in the number of integrations that organizations exe‐ cute, and an acceleration of economic experimentation. These are the conditions for a Cambrian explosion, of sorts—a hearty soil for dramatic new solutions that change the way the world works. And I will demonstrate why this explosion is nearly xi

Page 14

inevitable. Like HTTP created the World Wide Web and linked the world’s informa‐ tion, what I call “flow” will create the World Wide Flow and link the world’s activity. This was a challenging book to write in many ways, not the least of which is that large parts of it are speculative. It may be a decade before any World Wide Flow technolo‐ gies are considered “mainstream.” It will probably be three to five years before the reasonably strong contenders for the necessary programming interfaces and data protocols are available for experimentation. So why write the book now? The answer lies in the lessons learned from decades of distributed systems evolution prior to today. We, as a global technology community, tend to dismiss the possibilities that lie before us; to focus on how to make incremental improvement to the world we know today rather than prepare for tomorrow’s disruptions. There is a good reason for that: none of us have a crystal ball. However, we do have tools that can give us insight into general trends that are likely to happen. We can intelligently analyze the landscape of technologies and user needs to find places where evolution—or revolution—is likely to happen. I will use two of these—Wardley Mapping and Promise Theory—to demonstrate both what compo‐ nents are required for flow, and why those components will evolve from their form today to a form that encourages ubiquitous event-driven integration. Cloud computing caught many organizations flat-footed, which resulted in advan‐ tages for their competitors or missed opportunities to serve their missions. My goal with this book is to help you see not only what flow means, but why it is almost cer‐ tain to be. I’ll start with an unusual introduction: a hypothetical technology journal article from ten years in our future. The stories described there will give you some context for flow and how it might drive change in the coming decade. Chapter 1 will then introduce you to the concept of flow, from its basic definition to the key concepts that fall from that definition. Chapter 2 will make the case for why businesses, governments, non-profits, and a number of other institutions will embrace flow as it emerges. In Chapter 3, I’ll use Wardley Mapping and Promise Theory to make the case for why flow is almost certain to happen, and what the key components of flow systems are likely to be. Chapter 4 surveys the messaging and event-driven architectures available today that will either guide or form the basis of flow systems in the future. Building on that and the Wardley Map we defined in Chapter 3, Chapter 5 will discuss where key innova‐ tions are required to support true flow systems in the future. Chapter 6 then con‐ cludes the discussion with a survey of the things you can do today to both prepare for a flow future, and to help make that future happen. xii | Preface

Page 15

Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program ele‐ ments such as variable or function names, databases, data types, environment variables, statements, and keywords. This element signifies a general note or suggestion. O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit http://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/flow-arch. Preface | xiii

Page 16

1 Actually, Simon and other practitioners of Wardley Mapping have used the techniques in domains ranging from politics and culture to investment and revitalization. Email bookquestions@oreilly.com to comment or ask technical questions about this book. For news and information about our books and courses, visit http://oreilly.com. Find us on Facebook: http://facebook.com/oreilly Follow us on Twitter: http://twitter.com/oreillymedia Watch us on YouTube: http://www.youtube.com/oreillymedia Acknowledgments I have many people to thank for the insights and analysis you are about to read. I’ll start with two great thinkers who I have had the immense honor to know and con‐ verse with often throughout the last 15 years or so. Simon Wardley, the creator of Wardley Mapping, has created one of the most amazing tools for situational aware‐ ness in business or technical strategy.1 Mark Burgess, the creator of Promise Theory, provides an algebra for studying intent that perfectly compliments Wardley Mapping, but also stands on its own as a powerful analysis tool. To both of them, I owe a debt of gratitude for not only their part in this book, but for educating me about strategic analysis. This book was reviewed by an all-star collection of forward-thinking technologists and entrepreneurs. Derek Collison created NATS.io, and is the founder of Synadia, a company working hard to change the way event-driven integration is managed. Simon Crosby created the Xen hypervisor, and went on to found swim.ai, which is delivering a potentially game changing stateful event processing platform. Paul But‐ terworth founded Forte Software, the technology vendor at which I cut my meta‐ phorical distributed systems teeth, and now sits as co-founder of event-driven development platform Vantiq. Adrian Cockcroft built his reputation as the chief cloud architect at Netflix, and then as VP of Strategy for AWS. I owe all four an immense gratitude for their tireless work and advice over the last year. Clemens Vassars, Chief Messaging Architect at Microsoft, deserves special mention for walking me through how one selects event processing architectures, and for giv‐ ing me the language needed to describe that to you. Mark Anderson, CEO of the Strategic News Service, shared his insights on the rela‐ tionship between flow and interaction. My conversations with Mark over the years have strongly influenced my vision of flow. xiv | Preface

Page 17

I also owe thanks to a number of other technologists that contributed to various aspects of the book, including Jesse Bean, Mark Heckler, Stacey Higgenbotham, Sutha Kamal, Mark Kropf, Richard Seroter, Jason Shephard, Sina Sojoodi, Henri van den Bulk, and James Watters. Thank you to my current and former colleagues at Pivotal/VMware, especially my colleagues on the Field CxO team for your support and encouragement. You guys are amazing, and made it very easy to focus on this work when I needed to. I have learned important lessons about technology and management from each and every one of you. To my mentor, the great Tom Lounibos, I again offer my deepest gratitude for your encouragement, advice, and occasional kick in the butt. This book is not possible without my editor at O’Reilly, Amelia Blevins. Amelia, your patience is incredible, and your advice was always exactly what I needed when I needed it. Thank you. Thanks also go to Ryan Shaw and Mike Loukides for encourag‐ ing me to write. Special thanks go to Tim O’Reilly for the wonderful tea and scones, and for asking the right questions at the right time. Finally, and most importantly, to my wife, Mia, and my children, Owen and Emery. You were there every step of the way, and did nothing but encourage me and push me when I needed a push. Not a day goes by that I am not grateful for what life has given me, and for the love and laughter that we share. I love you all so deeply. And Mia, you really are my favorite. Preface | xv

Page 18

(This page has no text content)

Page 19

The 10-Year Impact of the World Wide Flow This article assumes that fictional flow standards, the Event Meta‐ data Protocol and the Event Subscription Interface—collectively known as the Internet Event Flow Suite—was introduced in 2021 and adopted by a number of tech companies. This created the plat‐ form by which Event Flow Protocol became standard tooling that sent and received data streams (including events). The article purposely avoids talking about the specifics of how the Internet Event Flow Suite works. It notes that the Event Metadata Protocol (which is modeled on the CNCF Cloud Events Protocol) is an event data protocol, and that the Event Subscription Interface enables consumers to subscribe to streams (which is loosely based on today’s MQTT and NATS.io subscription application program interfaces [APIs]). Again, the article avoids discussing specifics, as they are not critical to the outcomes described here. San Francisco (2034)—Ten years ago this month, the Internet Event Flow Suite, or IEFS, standard was introduced by the Cloud Native Computing Foundation (CNCF) and the World Wide Web Consortium (W3C). The world economy has undergone significant changes in the decade since. From powerful new online services to funda‐ mental changes in the way business has been automated, the World Wide Flow (WWF) has had great impact—both good and bad—on the way our world works. We thought it would be fun to look at some of the incredible technologies and services enabled by the WWF, and pay tribute to the incredible rise of event-driven integra‐ tion and shared data flows. The WWF in Finance Our financial system is based on the exchange of data (money) between parties engaged in various value-building activities. It is no surprise, then, that financial service companies were among the first to embrace IEFS, including the Event xvii

Page 20

Metadata Protocol (EMP) and Event Subscription Interface (ESI). The credit card industry was quick to leverage IEFS to offer their customers real-time budget man‐ agement, parental controls, and other time-sensitive services. However, integrating real-time credit and debit card transaction data with new cen‐ tral banking reporting services was the beginning of what was to become the US gov‐ ernment’s “Real-Time Economy” (RTE) initiative. The RTE program reports key economic data, including inflation, GDP, and purchasing trends every business day, and offers IEFS interfaces to enable subscribers to receive that data as soon as it is published. For example, the IEFS interfaces are used by banks to rapidly adjust their risk analysis for loans and investments across their portfolios. Interestingly, the Fed‐ eral Reserve has reported that it intends to boost this reporting to hourly (for data four hours in arrears) as early as next year, and banks are eagerly anticipating the change. The equities markets were also eager to take advantage of real-time flow standards. Though the stock and commodity markets were already using data flows to enable high-frequency trading and other automated trading programs, the introduction of standards for event data flows greatly reduced the cost of accessing and processing those streams. The rapidly decreasing cost of trading through software APIs com‐ bined with these flows resulted in the incredible rise of hobbyist trading, as well as the birth of a cadre of new financial services firms aimed at using artificial intelligence algorithms and other computational approaches to attempt to out-think the markets. Most of these algorithms failed to maintain any kind of advantage over the markets, and a few were outright disastrous. For example, Komputrade, a two-employee startup based in Milwaukee, Wisconsin, was blamed for the Komputrade Crash of March 8, 2023, in which every US equity market was halted for an hour and a half due to the unusual activity the startup’s algorithms generated. Most other hobbyist algorithms failed to exceed market averages, but a few made their inventors extremely wealthy. Neuroquity founder Imani Abioye famously became Africa’s richest woman when her commodity trading company was acquired by JPMorgan Chase in 2028 for $34 billion. The WWF in Retail The retail industry benefited from the WWF through a combination of reduced costs and increased visibility into the real-time state of the business. Initially, IEFS and EMP were used to standardize the exchange of real-time inventory data across retail supply chains. Within a few short years, any retailer could count on being able to receive inventory event flows from any supplier using the inventory processing sys‐ tem of their choice. xviii | The 10-Year Impact of the World Wide Flow