Building Event-Driven Microservices (Adam Bellemare) (Z-Library)

📄 File Format: PDF

💾 File Size: 5.1 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

(This page has no text content)

📄 Page 2

  1. Preface   1. Conventions Used in This Book 2. O’Reilly Online Learning 3. How to Contact Us 4. Acknowledgments 2. 1. Why Event-Driven Microservices   1. What Are Event-Driven Microservices? 2. Introduction to Domain-Driven Design and Bounded Contexts   1. Leveraging Domain Models and Bounded Contexts 2. Aligning Bounded Contexts with Business Requirements 3. Communication Structures   1. Business Communication Structures 2. Implementation Communication Structures 3. Data Communication Structures 4. Conway’s Law and Communication Structures 4. Communication Structures in Traditional Computing   1. Option 1: Make a New Service 2. Option 2: Add It to the Existing Service 3. Pros and Cons of Each Option 4. The Team Scenario, Continued 5. Conflicting Pressures 5. Event-Driven Communication Structures   1. Events Are the Basis of Communication

📄 Page 3

2. Event Streams Provide the Single Source of Truth 3. Consumers Perform Their Own Modeling and Querying 4. Data Communication Is Improved Across the Organization 5. Accessible Data Supports Business Communication Changes 6. Asynchronous Event-Driven Microservices   1. Example Team Using Event-Driven Microservices 7. Synchronous Microservices   1. Drawbacks of Synchronous Microservices 2. Benefits of Synchronous Microservices 8. Summary 3. 2. Event-Driven Microservice Fundamentals   1. Building Topologies   1. Microservice Topology 2. Business Topology 2. The Contents of an Event 3. The Structure of an Event   1. Unkeyed Event 2. Entity Event 3. Keyed Event 4. Materializing State from Entity Events 5. Event Data Definitions and Schemas 6. Microservice Single Writer Principle 7. Powering Microservices with the Event Broker   1. Event Storage and Serving 2. Additional Factors to Consider

📄 Page 4

8. Event Brokers Versus Message Brokers   1. Consuming from the Immutable Log 2. Providing a Single Source of Truth 9. Managing Microservices at Scale   1. Putting Microservices into Containers 2. Putting Microservices into Virtual Machines 3. Managing Containers and Virtual Machines 10. Paying the Microservice Tax 11. Summary 4. 3. Communication and Data Contracts   1. Event-Driven Data Contracts   1. Using Explicit Schemas as Contracts 2. Schema Definition Comments 3. Full-Featured Schema Evolution 4. Code Generator Support 5. Breaking Schema Changes 2. Selecting an Event Format 3. Designing Events   1. Tell the Truth, the Whole Truth, and Nothing but the Truth 2. Use a Singular Event Definition per Stream 3. Use the Narrowest Data Types 4. Keep Events Single-Purpose 5. Minimize the Size of Events 6. Involve Prospective Consumers in the Event Design 7. Avoid Events as Semaphores or Signals

📄 Page 5

4. Summary 5. 4. Integrating Event-Driven Architectures with Existing Systems   1. What Is Data Liberation?   1. Compromises for Data Liberation 2. Converting Liberated Data to Events 2. Data Liberation Patterns 3. Data Liberation Frameworks 4. Liberating Data by Query   1. Bulk Loading 2. Incremental Timestamp Loading 3. Autoincrementing ID Loading 4. Custom Querying 5. Incremental Updating 6. Benefits of Query-Based Updating 7. Drawbacks of Query-Based Updating 5. Liberating Data Using Change-Data Capture Logs   1. Benefits of Using Data Store Logs 2. Drawbacks of Using Data Base Logs 6. Liberating Data Using Outbox Tables   1. Performance Considerations 2. Isolating Internal Data Models 3. Ensuring Schema Compatibility 4. Capturing Change-Data Using Triggers 7. Making Data Definition Changes to Data Sets Under Capture  

📄 Page 6

1. Handling After-the-Fact Data Definition Changes for the Query and CDC Log Patterns 2. Handling Data Definition Changes for Change-Data Table Capture Patterns 8. Sinking Event Data to Data Stores 9. The Impacts of Sinking and Sourcing on a Business 10. Summary 6. 5. Event-Driven Processing Basics   1. Composing Stateless Topologies   1. Transformations 2. Branching and Merging Streams 2. Repartitioning Event Streams   1. Example: Repartitioning an Event Stream 3. Copartitioning Event Streams   1. Example: Copartitioning an Event Stream 4. Assigning Partitions to a Consumer Instance   1. Assigning Partitions with the Partition Assignor 2. Assigning Copartitioned Partitions 3. Partition Assignment Strategies 5. Recovering from Stateless Processing Instance Failures 6. Summary 7. 6. Deterministic Stream Processing   1. Determinism with Event-Driven Workflows 2. Timestamps   1. Synchronizing Distributed Timestamps

📄 Page 7

2. Processing with Timestamped Events 3. Event Scheduling and Deterministic Processing   1. Custom Event Schedulers 2. Processing Based on Event Time, Processing Time, and Ingestion Time 3. Timestamp Extraction by the Consumer 4. Request-Response Calls to External Systems 4. Watermarks   1. Watermarks in Parallel Processing 5. Stream Time   1. Stream Time in Parallel Processing 6. Out-of-Order and Late-Arriving Events   1. Late Events with Watermarks and Stream Time 2. Causes and Impacts of Out-of-Order Events 3. Time-Sensitive Functions and Windowing 7. Handling Late Events 8. Reprocessing Versus Processing in Near-Real Time 9. Intermittent Failures and Late Events 10. Producer/Event Broker Connectivity Issues 11. Summary and Further Reading 8. 7. Stateful Streaming   1. State Stores and Materializing State from an Event Stream 2. Recording State to a Changelog Event Stream 3. Materializing State to an Internal State Store   1. Materializing Global State

📄 Page 8

2. Advantages of Using Internal State 3. Disadvantages of Using Internal State 4. Scaling and Recovery of Internal State 4. Materializing State to an External State Store   1. Advantages of External State 2. Drawbacks of External State 3. Scaling and Recovery with External State Stores 5. Rebuilding Versus Migrating State Stores   1. Rebuilding 2. Migrating 6. Transactions and Effectively Once Processing   1. Example: Stock Accounting Service 2. Effectively Once Processing with Client-Broker Transactions 3. Effectively Once Processing Without Client-Broker Transactions 7. Summary 9. 8. Building Workflows with Microservices   1. The Choreography Pattern   1. A Simple Event-Driven Choreography Example 2. Creating and Modifying a Choreographed Workflow 3. Monitoring a Choreographed Workflow 2. The Orchestration Pattern   1. A Simple Event-Driven Orchestration Example 2. A Simple Direct-Call Orchestration Example 3. Comparing Direct-Call and Event-Driven Orchestration

📄 Page 9

4. Creating and Modifying an Orchestration Workflow 5. Monitoring the Orchestration Workflow 3. Distributed Transactions   1. Choreographed Transactions: The Saga Pattern 2. Orchestrated Transactions 4. Compensation Workflows 5. Summary 10. 9. Microservices Using Function-as-a-Service   1. Designing Function-Based Solutions as Microservices   1. Ensure Strict Membership to a Bounded Context 2. Commit Offsets Only After Processing Has Completed 3. Less Is More 2. Choosing a FaaS Provider 3. Building Microservices Out of Functions 4. Cold Start and Warm Starts 5. Starting Functions with Triggers   1. Triggering Based on New Events: The Event-Stream Listener 2. Triggering Based on Consumer Group Lag 3. Triggering on a Schedule 4. Triggering Using Webhooks 5. Triggering on Resource Events 6. Performing Business Work with Functions 7. Maintaining State 8. Functions Calling Other Functions

📄 Page 10

  1. Event-Driven Communication Pattern 2. Direct-Call Pattern 9. Termination and Shutdown 10. Tuning Your Functions   1. Allocating Sufficient Resources 2. Batch Event-Processing Parameters 11. Scaling Your FaaS Solutions 12. Summary 11. 10. Basic Producer and Consumer Microservices   1. Where Do BPCs Work Well?   1. Integration with Existing and Legacy Systems 2. Stateful Business Logic That Isn’t Reliant Upon Event Order 3. When the Data Layer Does Much of the Work 4. Independent Scaling of the Processing and Data Layer 2. Hybrid BPC Applications with External Stream Processing   1. Example: Using an External Stream-Processing Framework to Join Event Streams 3. Summary 12. 11. Heavyweight Framework Microservices   1. A Brief History of Heavyweight Frameworks 2. The Inner Workings of Heavyweight Frameworks 3. Benefits and Limitations 4. Cluster Setup Options and Execution Modes   1. Use a Hosted Service

📄 Page 11

2. Build Your Own Full Cluster 3. Create Clusters with CMS Integration 5. Application Submission Modes   1. Driver Mode 2. Cluster Mode 6. Handling State and Using Checkpoints 7. Scaling Applications and Handling Event Stream Partitions   1. Scaling an Application While It Is Running 2. Scaling an Application by Restarting It 3. Autoscaling Applications 8. Recovering from Failures 9. Multitenancy Considerations 10. Languages and Syntax 11. Choosing a Framework 12. Example: Session Windowing of Clicks and Views 13. Summary 13. 12. Lightweight Framework Microservices   1. Benefits and Limitations 2. Lightweight Processing 3. Handling State and Using Changelogs 4. Scaling Applications and Recovering from Failures   1. Event Shuffling 2. State Assignment 3. State Replication and Hot Replicas 5. Choosing a Lightweight Framework

📄 Page 12

  1. Apache Kafka Streams 2. Apache Samza: Embedded Mode 6. Languages and Syntax 7. Stream-Table-Table Join: Enrichment Pattern 8. Summary 14. 13. Integrating Event-Driven and Request-Response Microservices   1. Handling External Events   1. Autonomously Generated Events 2. Reactively Generated Events 2. Handling Autonomously Generated Analytical Events 3. Integrating with Third-Party Request-Response APIs 4. Processing and Serving Stateful Data   1. Serving Real-Time Requests with Internal State Stores 2. Serving Real-Time Requests with External State Stores 5. Handling Requests Within an Event-Driven Workflow   1. Processing Events for User Interfaces 6. Micro-Frontends in Request-Response Applications 7. The Benefits of Microfrontends   1. Composition-Based Microservices 2. Easy Alignment to Business Requirements 8. Drawbacks of Microfrontends   1. Potentially Inconsistent UI Elements and Styling 2. Varying Microfrontend Performance 3. Example: Experience Search and Review Application

📄 Page 13

9. Summary 15. 14. Supportive Tooling   1. Microservice-to-Team Assignment System 2. Event Stream Creation and Modification 3. Event Stream Metadata Tagging 4. Quotas 5. Schema Registry 6. Schema Creation and Modification Notifications 7. Offset Management 8. Permissions and Access Control Lists for Event Streams 9. State Management and Application Reset 10. Consumer Offset Lag Monitoring 11. Streamlined Microservice Creation Process 12. Container Management Controls 13. Cluster Creation and Management   1. Programmatic Bringup of Event Brokers 2. Programmatic Bringup of Compute Resources 3. Cross-Cluster Event Data Replication 4. Programmatic Bringup of Tooling 14. Dependency Tracking and Topology Visualization   1. Topology Example 15. Summary 16. 15. Testing Event-Driven Microservices   1. General Testing Principles 2. Unit-Testing Topology Functions

📄 Page 14

  1. Stateless Functions 2. Stateful Functions 3. Testing the Topology 4. Testing Schema Evolution and Compatibility 5. Integration Testing of Event-Driven Microservices 6. Local Integration Testing   1. Create a Temporary Environment Within the Runtime of Your Test Code 2. Create a Temporary Environment External to Your Test Code 3. Integrate Hosted Services Using Mocking and Simulator Options 4. Integrate Remote Services That Have No Local Options 7. Full Remote Integration Testing   1. Programmatically Create a Temporary Integration Testing Environment 2. Testing Using a Shared Environment 3. Testing Using the Production Environment 8. Choosing Your Full-Remote Integration Testing Strategy 9. Summary 17. 16. Deploying Event-Driven Microservices   1. Principles of Microservice Deployment 2. Architectural Components of Microservice Deployment   1. Continuous Integration, Delivery, and Deployment Systems 2. Container Management Systems and Commodity Hardware 3. The Basic Full-Stop Deployment Pattern

📄 Page 15

4. The Rolling Update Pattern 5. The Breaking Schema Change Pattern   1. Eventual Migration via Two Event Streams 2. Synchronized Migration to the New Event Stream 6. The Blue-Green Deployment Pattern 7. Summary 18. 17. Conclusion   1. Communication Layers 2. Business Domains and Bounded Contexts 3. Shareable Tools and Infrastructure 4. Schematized Events 5. Data Liberation and the Single Source of Truth 6. Microservices 7. Microservice Implementation Options 8. Testing 9. Deploying 10. Final Words 19. Index

📄 Page 16

Building Event-Driven Microservices Leveraging Organizational Data at Scale Adam Bellemare

📄 Page 17

Building Event-Driven Microservices by Adam Bellemare Copyright © 2020 Adam Bellemare. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com.   Acquisitions Editor: Melissa Duffield Development Editor: Corbin Collins Production Editor: Christopher Faucher Copyeditor: Rachel Monaghan Proofreader: Kim Wimpsett Indexer: Potomac Indexing, LLC Interior Designer: David Futato

📄 Page 18

Cover Designer: Karen Montgomery Illustrator: O’Reilly Media, Inc.   August 2020: First Edition Revision History for the First Edition   2020-07-02: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781492057895 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Building Event-Driven Microservices, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the author, and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights. 978-1-492-05789-5 [LSI]

📄 Page 19

Preface I wrote this book to be the book that I wish I’d had when I started out on my journey into the world of event-driven microservices. This book is a culmination of my own personal experiences, discussions with others, and the countless blogs, books, posts, talks, conferences, and documentation related to one part or another of the event-driven microservice world. I found that many of the works I read mentioned event-driven architectures either only in passing or with insufficient depth. Some covered only a specific aspect of the architecture and, while helpful, provided only a small piece of the puzzle. Other works proved to be reductive and dismissive, asserting that event-driven systems are really only useful for one system to send an asynchronous message directly to another as a replacement for synchronous request- response systems. As this book details, there is far more to event- driven architectures than this. The tools that we use shape and influence our inventions significantly. Event-driven microservice architectures are made possible by a whole host of technologies that have only recently become readily accessible. Distributed, fault-tolerant, high-capacity, and high-speed event brokers underpin the architectures and design patterns in this book. These technological solutions are based on the convergence of big data with the need for near-real-time event processing. Microservices are facilitated by the ease of containerization and the requisitioning of compute resources, allowing for simplified hosting, scaling, and management of hundreds of thousands of microservices. The technologies that support event-driven microservices have a significant impact on how we think about and solve problems, as well as on how our businesses and organizations are structured. Event- driven microservices change how a business works, how problems can be solved, and how teams, people, and business units communicate.

📄 Page 20

These tools give you a truly new way of doing things that has not been possible until only recently. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables, statements, and keywords. Constant width bold Shows commands or other text that should be typed literally by the user. Constant width italic Shows text that should be replaced with user-supplied values or by values determined by context. TIP This element signifies a tip or suggestion. NOTE This element signifies a general note. WARNING