JSON at Work (Tom Marrs) (Z-Library)

Tom Marrs JSON at Work PRACTICAL DATA INTEGRATION FOR THE WEB

Tom Marrs JSON at Work Practical Data Integration for the Web Boston Farnham Sebastopol TokyoBeijing

978-1-449-35832-7 [LSI] JSON at Work by Tom Marrs Copyright © 2017 Vertical Slice, Inc. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com/safari). For more information, contact our corporate/insti‐ tutional sales department: 800-998-9938 or corporate@oreilly.com. Editor: Meg Foley Production Editor: Nicholas Adams Copyeditor: Sharon Wilkey Proofreader: Charles Roumeliotis Indexer: Ellen Troutman-Zaig Interior Designer: David Futato Cover Designer: Randy Comer Illustrator: Rebecca Demarest July 2017: First Edition Revision History for the First Edition 2017-06-16: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781449358327 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. JSON at Work, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

To everyone who produces or consumes JSON data with web/mobile applications, REST APIs, and messaging systems—I hope this makes your job easier. To the unsung JSON community that produces JSON-based tools and libraries for the rest of us—thank you for all your hard work to make JSON useful and meaningful.

Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii Part I. JSON Overview and Platforms 1. JSON Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 JSON Is a Standard 3 A Brief Sample 4 Why JSON? 6 Core JSON 8 JSON Data Types 8 JSON Value Types 11 JSON Versions 14 JSON Comments 14 JSON File and MIME Type 14 JSON Style Guidelines 15 Our Example—MyConference 17 Our Technical Stack 17 Our Architectural Style—noBackEnd 17 Model JSON Data with JSON Editor Online 18 Generate Sample JSON Data with JSON Generator 20 Create and Deploy a Stub API 20 What We Covered? 24 What’s Next? 24 2. JSON in JavaScript. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 Node.js Setup 25 JSON Serialization/Deserialization with JSON.stringify() and JSON.parse() 26 v

The JSON Stringifier/Parser Object 26 JSON Serialization with Simple JavaScript Data Types 26 JSON Serialization with an Object and toJSON() 29 JSON Deserialization Using eval() 30 JSON Deserialization with an Object and JSON.parse() 31 JavaScript Objects and JSON 32 Node REPL 33 Where to Learn More About JavaScript Objects 35 Unit Testing with a Stub API 35 Unit Test Style—TDD and BDD 35 Just Enough Unit Testing with Mocha and Chai 36 Setting Up the Unit Test 36 Unirest 36 Test Data 37 Speakers Unit Test 37 Building a Small Web Application 39 Yeoman 39 Iteration 1—Generate a Web Application with Yeoman 41 Iteration 2—Make an HTTP Call with jQuery 45 Iteration 3—Consume Speaker Data from a Stub API and Use a Template 49 How to Go Deeper with JavaScript 54 What We Covered 55 What’s Next? 55 3. JSON in Ruby on Rails. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 Ruby on Rails Setup 57 Ruby JSON Gems 58 JSON Serialization/Deserialization with MultiJson 58 The MultiJson Object 58 JSON Serialization/Deserialization with Simple Ruby Data Types 59 JSON Deserialization with Objects and MultiJson 61 A Word on Camel Casing and JSON 63 JSON Serialization with Objects and ActiveSupport 64 JSON Deserialization with Objects and ActiveSupport 65 Unit Testing with a Stub API 66 Just Enough Unit Testing with Minitest 66 Setting Up the Unit Test 67 Test Data 68 JSON and Minitest Testing with APIs 68 Speakers Unit Test 68 Further Reading on Ruby and Minitest 72 What Is Missing in the Unit Tests? 72 vi | Table of Contents

Build a Small Web API with Ruby on Rails 73 Choose a JSON Serializer 73 speakers-api-1—Create an API with Camel-Cased JSON 75 speakers-api-2—Create an API that Customizes the JSON Representation 82 Further Reading on Rails and Rails-based APIs 84 What We Covered 84 What’s Next? 84 4. JSON in Java. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 Java and Gradle Setup 85 Gradle Overview 85 Just Enough Unit Testing with JUnit 87 Java-Based JSON Libraries 87 JSON Serialization/Deserialization with Jackson 88 Serialization/Deserialization with Simple Java Data Types 88 Serialization/Deserialization with Java Objects 91 Unit Testing with a Stub API 96 Test Data 96 JSON and JUnit Testing with APIs 96 Build a Small Web API with Spring Boot 100 Create the Model 101 Create the Controller 103 Register the Application 104 Write the Build Script 105 Deploy the API 107 Test the API with Postman 107 What We Covered 108 What’s Next? 109 Part II. The JSON Ecosystem 5. JSON Schema. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 JSON Schema Overview 113 What Is JSON Schema? 113 Syntactic Versus Semantic Validation 114 A Simple Example 114 JSON Schema on the Web 115 Why JSON Schema? 116 My Journey with JSON Schema 117 The Current State of the JSON Schema Standard 117 JSON Schema and XML Schema 117 Table of Contents | vii

Core JSON Schema—Basics and Tooling 118 JSON Schema Workflow and Tooling 118 Core Keywords 120 Basic Types 121 Numbers 125 Arrays 126 Enumerated Values 128 Objects 129 Pattern Properties 131 Regular Expressions 133 Dependent Properties 135 Internal References 136 External References 138 Choosing Validation Rules 141 How to Design and Test an API with JSON Schema 146 Our Scenario 146 Model a JSON Document 146 Generate a JSON Schema 148 Validate the JSON Document 151 Generate Sample Data 152 Deploy a Stub API with json-server 155 Final Thoughts on API Design and Testing with JSON Schema 157 Validation Using a JSON Schema Library 157 Where to Go Deeper with JSON Schema 158 What We Covered 158 What’s Next? 158 6. JSON Search. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 Why JSON Search? 159 JSON Search Libraries and Tools 160 Honorable Mention 160 What to Look For 160 Test Data 161 Setting Up Unit Tests 162 Comparing JSON Search Libraries and Tools 163 JSONPath 163 JSON Pointer 170 jq 173 JSON Search Library and Tool Evaluations—The Bottom Line 184 What We Covered 185 What’s Next? 185 viii | Table of Contents

7. JSON Transform. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187 Types of JSON Transformation 187 What to Look For in a JSON Transform Library 188 Test Input Data 189 JSON-to-HTML Transformation 191 Target HTML Document 191 Mustache 192 Handlebars 198 JSON-to-HTML Transformation Evaluations—The Bottom Line 204 JSON-to-JSON Transform 204 The Issues 205 JSON-to-JSON Transform Libraries 205 Honorable Mention 205 Target JSON Output 206 JSON Patch 207 JSON-T 213 Mustache 217 Handlebars 219 JSON-to-JSON Transformation Evaluations—The Bottom Line 221 JSON-XML Transformation 222 JSON-XML Transformation Conventions 222 The Issues with JSON-XML Transformation Conventions 231 XML-JSON Transform—The Bottom Line 231 JSON-XML Transformation Unit Test 233 What We Covered 235 What’s Next? 235 Part III. JSON in the Enterprise 8. JSON and Hypermedia. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239 Comparing Hypermedia Formats 240 Defining Key Terms 241 My Opinion on Hypermedia 241 Siren 242 JSON-LD 244 Collection+JSON 249 json:api 250 HAL 254 Conclusions on Hypermedia 259 Recommendations for Working with Hypermedia 260 Practical Issues with Hypermedia 260 Table of Contents | ix

Testing with HAL in the Speakers API 261 Test Data 261 HAL Unit Test 263 Server-Side HAL 267 Going Deeper with Hypermedia 268 What We Covered 268 What’s Next? 268 9. JSON and MongoDB. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269 What About BSON? 269 MongoDB Setup 270 MongoDB Server and Tools 270 MongoDB Server 270 Importing JSON into MongoDB 271 MongoDB Command Shell 273 Basic CRUD with mongo 274 Exporting from MongoDB to a JSON Document 277 What About Schema? 280 RESTful API Testing with MongoDB 281 Test Input Data 282 Providing a RESTful Wrapper for MongoDB 282 What We Covered 285 What’s Next? 285 10. JSON Messaging with Kafka. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287 Kafka Use Cases 288 Kafka Concepts and Terminology 288 The Kafka Ecosystem—Related Projects 289 Kafka Environment Setup 290 Why Do I Need ZooKeeper? 290 Kafka Command-Line Interface (CLI) 291 How to Publish a JSON Message with the CLI 291 Start ZooKeeper 291 Start Kafka 292 Create a Topic 292 List Topics 293 Start a Consumer 293 Publish a JSON Message 294 Consume a JSON Message 295 Clean Up and Shut Down Kafka 295 Kafka Libraries 297 End-to-End Example—Speaker Proposals at MyConference 297 x | Table of Contents

Test Data 297 Architecture Components 299 Set Up the Kafka Environment 300 Set Up Fake Email Server and Client—MailCatcher 301 Set Up Node.js Project Environment 302 Speaker Proposal Producer (Send Speaker Proposals) 302 Proposal Reviewer (Consumer/Producer) 302 Speaker Notifier (Consumer) 308 Review Notification Email Messages with MailCatcher 313 What We Covered 315 A. Installation Guides. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317 B. JSON Community. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337 Table of Contents | xi

Preface JavaScript Object Notation (JSON) has become the de facto standard for RESTful interfaces, but an ecosystem of little-known standards, tools, and technologies is available that architects and developers can start using today to build well-designed applications. JSON is more than just a simple replacement for XML when you make an AJAX call. It is becoming the backbone of any serious data interchange over the internet. Solid standards and best practices can be used to harness the energy and enthusiasm around JSON to build truly elegant, useful, and efficient applications. The only thing missing is a book to pull everything together. This book aims to help developers leverage JSON so that they can build enterprise-class applications and services. My goals are to promote the use of JSON tooling and the concept of mes‐ sage/document design as a first-class citizen in the fast-growing API community. My journey into JSON began in 2007 when I was leading a large web portal project, and we had to populate a drop-down list with several thousand entries. At that time, I was reading Head First AJAX by Rebecca Riordan (O’Reilly), so I had a decent archi‐ tectural approach. AJAX would solve overall latency and page load issues, but what about the data? I had been using XML successfully for several years, but it seemed like overkill for the task at hand—moving data from the backend of a web application to the View. Head First AJAX mentioned a new data format called JSON, and it looked like the way to go. My team began looking into APIs that would convert our Java objects into JSON, and chose the one that had the simplest and shortest JUnit tests—the goal was to do the simplest thing that could possibly work. We put the application under rigorous load testing, and the Java-to-JSON conversion was never a performance issue. The application scaled up in production, and the users saw their drop-down list in a timely manner. Along my journey, I considered the use of JSON with web applications, RESTful APIs, and messaging. As of 2009, I was still working with XML because XML Schema pro‐ vided the semantic validation needed for meaningful data interchange. So, my posi‐ tion at that time was to use JSON for web user interfaces, or UIs (for speed), and xiii

XML for Web Services and Messaging (for integration). But then I heard about JSON Schema in 2010, and found that I had no further need for XML. The JSON Schema specification is still under development, but it’s sufficiently mature enough now to use for enterprise-class integration. At this point, I was hooked on or, more accurately, obsessed with JSON. I began look‐ ing around the internet to see what else JSON could do, and I found copious APIs, online tools, search capabilities, and more. In short, anything that has been done with XML can (and should) now be done with JSON. I then began to look for JSON in books, and was disappointed when I could find only a chapter or two on the topic in a JavaScript or RESTful Web Services book. I saw a growing JSON community along with lots of tool support and articles and blogs, but there was no single place—other than Douglas Crockford’s JSON site—that pulled everything together. Audience, Assumptions, and Approach This book is for architects and developers who design/implement web and mobile applications, RESTful APIs, and messaging applications. Code examples are in Java‐ Script, Node.js, Ruby on Rails, and Java. If you’re a Groovy, Go, Scala, Perl, Python, Clojure, or C# developer, you’ll need to follow along with the code examples pro‐ vided. But rest assured that most major/modern languages provide excellent JSON support. For the architect, I’ve provided guidelines, best practices, and architecture and design diagrams where appropriate. But in addition to providing visionary lead‐ ership, real architects prove their ideas with working code. While I love working with JSON and writing code, it’s entirely meaningless without use cases, and a business and technical context. For developers, this book is packed with code examples, tool‐ ing, and Unit Tests, along with a GitHub repository (see “Code Examples” on page xvii). Chapters 5–10 only have code examples only in Node.js to keep things simple and focused. But it’s not hard to translate these examples into your platform of choice. What Does “At Work” Mean? When I wrote JBoss at Work with Scott Davis back in the mid-2000s, our vision was to write a book that developers could use at work on their daily jobs. In the same manner, the purpose of JSON at Work is to provide practical examples to developers based on my real-world integration experience with JSON. To that end, I’ve baked Unit Testing (wherever feasible) into every chapter. It’s simple: if there’s no test for a piece of code, then that code doesn’t exist. Period. xiv | Preface

Expect to roll up your sleeves and look at code. Whether you’re an architect or devel‐ oper, you’ll find something here to help you on your job. What You’ll Learn By reading and following this book’s examples, you’ll learn how to do the following: • JSON basics and how to model JSON data • Use JSON with Node.js, Ruby on Rails, and Java • Structure JSON documents with JSON Schema to design and test APIs • Search the contents of JSON documents with JSON Search tools • Convert JSON documents to other data formats with JSON Transform tools • Use JSON as part of an enterprise architecture • Compare JSON-based Hypermedia formats, including HAL and json:api • Leverage MongoDB to store and access JSON documents • Use Apache Kafka to exchange JSON-based messages between services • Use freely available JSON tools and utilities to simplify testing • Invoke APIs in your favorite programming language with simple utilities and libraries What You’ll Work With Here’s a sample of the JSON tooling you’ll use in this book: • JSON editors/modelers • Unit-Testing tools (e.g., Mocha/Chai, Minitest, JUnit) • JSON Validators • A JSON Schema Generator • JSON Search tools • JSON Transform (templating) tools Who This Book Is Not For This book is not for you if your only interest in JSON is to make AJAX calls from JavaScript. Although I cover this topic, it’s just the tip of the iceberg. Plenty of Java‐ Script books have the chapter you’re looking for. Preface | xv

Developers looking for a deep reference on REST, Ruby on Rails, Java, JavaScript, etc. won’t find it here. This book relies on these technologies, but focuses on how to use JSON with these languages and technologies. Organization This book consists of the following parts: • Part I, JSON Overview and Platforms • Part II, The JSON Ecosystem • Part III, JSON in the Enterprise • Appendices Part I, JSON Overview and Platforms • Chapter 1, JSON Overview, starts with an overview of the JSON data format, describes best practices in JSON usage, and introduces the tools used throughout the book. • Chapter 2, JSON in JavaScript, shows how to use JSON with JavaScript, Node.js, and Mocha/Chai Unit Tests. • Chapter 3, JSON in Ruby on Rails, describes how to convert between Ruby objects and JSON, and integrate with Rails. • Chapter 4, JSON in Java, tells you how to use JSON with Java and Sprint Boot. Part II, The JSON Ecosystem • Chapter 5, JSON Schema, helps you structure JSON documents with JSON Schema. Along the way, you’ll generate a JSON Schema and design an API with it. • Chapter 6, JSON Search, shows how to search JSON documents with jq and JSONPath. • Chapter 7, JSON Transform, provides the tools you’ll need transform a poorly designed JSON document to a better designed/more useful JSON document. Plus, it shows how to convert between JSON and other formats such as XML and HTML. xvi | Preface

Part III, JSON in the Enterprise • Chapter 8, JSON and Hypermedia, looks at how to use JSON with several well- known Hypermedia formats (e.g., HAL and jsonapi). • Chapter 9, JSON and MongoDB, shows how to leverage MongoDB to store and access JSON documents. • Chapter 10, JSON Messaging with Kafka, describes how to use Apache Kafka to exchange JSON-based messages between services. Appendices • Appendix A, Installation Guides, shows how to install the applications you’ll need to run the code examples in this book. • Appendix B, JSON Community, provides further information and links to con‐ nect you to the JSON community (e.g., standards and tutorials) and to help you go further with JSON. Code Examples All code examples for this book are freely available from the JSON at Work examples GitHub repository. This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a signifi‐ cant amount of example code from this book into your product’s documentation does require permission. We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “JSON at Work by Tom Marrs (O’Reilly). Copyright 2017 Vertical Slice, Inc., 978-1-449-35832-7.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. Preface | xvii

O’Reilly Safari Safari (formerly Safari Books Online) is a membership-based training and reference platform for enterprise, government, educators, and individuals. Members have access to thousands of books, training videos, Learning Paths, interac‐ tive tutorials, and curated playlists from over 250 publishers, including O’Reilly Media, Harvard Business Review, Prentice Hall Professional, Addison-Wesley Profes‐ sional, Microsoft Press, Sams, Que, Peachpit Press, Adobe, Focal Press, Cisco Press, John Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks, Packt, Adobe Press, FT Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett, and Course Technology, among others. For more information, please visit http://oreilly.com/safari. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at http://bit.ly/json-at-work. To comment or ask technical questions about this book, send email to bookques‐ tions@oreilly.com. For more information about our books, courses, conferences, and news, see our web‐ site at http://www.oreilly.com. Find us on Facebook: http://facebook.com/oreilly Follow us on Twitter: http://twitter.com/oreillymedia Watch us on YouTube: http://www.youtube.com/oreillymedia xviii | Preface