Unifying Business, Data, and Code Designing Data Products With JSON Schema (Ron Itelman, Juan Cruz Viotti) (Z-Library)

📄 File Format: PDF

💾 File Size: 14.6 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

Ron Itelman & Juan Cruz Viotti Unifying Business, Data, and Code Designing Data Products with JSON Schema

📄 Page 2

DATA “There simply isn’t any other resource that provides the same explanation that can take you from zero to hero.” —Ben Hutton JSON Schema Specification Lead, Postman “I keep thinking, why is this simplicity missing from an entire field?” —Hala Nelson Author of Essential Math for AI (O’Reilly) Unifying Business, Data, and Code Twitter: @oreillymedia linkedin.com/company/oreilly-media youtube.com/oreillymedia In the modern symphony of business, each section—from the technical to the managerial—must play in harmony. Authors Ron Itelman and Juan Cruz Viotti introduce a bold methodology to synchronize your business and technical teams, transforming them into a single, high-performing unit. Misalignment between business and technical teams halts innovation. You’ll learn how to transcend the root causes of project failure—the ambiguity, knowledge gaps, and blind spots that lead to wasted efforts. The unifying methodology in this book will teach you these alignment tools and more: • The four facets of data products: A simple blueprint that encapsulates data and business logic helps eliminate the most common causes of wasted time and misunderstanding • The concept compass: An easy way to identify the biggest sources of misalignment • Success spectrums: Define the required knowledge and road map your team needs to achieve success • JSON Schema: Leverage JSON and JSON Schema to technically implement the strategy at scale, including extending JSON Schema with custom keywords, understanding JSON Schema annotations, and hosting your own schema registry • Data hygiene: Learn how to design high-quality datasets aligned with creating real business value, and protect your organization from the most common sources of pain Ron Itelman is cofounder of Intelligence.AI. His expertise is in collaborative intelligence, at the intersection of AI and psychology, to model behavior and knowledge in order to generate customized learning experiences. Juan Cruz Viotti is cofounder of Intelligence.AI. In addition to leading his own open source lab, Sourcemeta, based on award-winning research at the University of Oxford, he leads the desktop engineering team at Postman and collaborates with the JSON Schema organization. 9 7 8 1 0 9 8 1 4 5 0 0 2 5 6 5 9 9 US $65.99 CAN $82.99 ISBN: 978-1-098-14500-2

📄 Page 3

Ron Itelman and Juan Cruz Viotti Unifying Business, Data, and Code Designing Data Products with JSON Schema Boston Farnham Sebastopol TokyoBeijing

📄 Page 4

978-1-098-14500-2 [LSI] Unifying Business, Data, and Code by Ron Itelman and Juan Cruz Viotti Copyright © 2024 Intelligence.AI LLC. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Aaron Black Development Editor: Corbin Collins Production Editor: Ashley Stussy Copyeditor: nSight, Inc. Proofreader: Tove Innis Indexer: WordCo Indexing Services, Inc. Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea February 2024: First Edition Revision History for the First Edition 2024-01-24: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781098145002 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Unifying Business, Data, and Code, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the authors and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

📄 Page 5

Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi 1. The Need for a Unifying Data Strategy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Your Quest for Data-Driven Breakthroughs Begins 1 There Are Usually Multiple, Conflicting North Stars 2 The Good, the Bad, and the Ugly of Data Problems 3 The Problem with Problems 7 Unifying Concepts: The Key to Innovation 9 What a Unifying Data Strategy Will Do for Agile 11 Defining Being Agile 12 Agile Theater 13 Agile, Waterfall, and Unifying 13 Defining a Unifying Data Strategy Approach 14 Understanding the Phrase Being Data Driven 15 To Be Data Driven, Be Data Centric 16 Bottlenecks Preventing Teams from Being Data Driven 17 This Book’s Project: Intelligence.AI Coffee Beans 18 Summary 19 2. The Lingua Franca of Data: JSON. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Introducing JSON 21 A Simple JSON Example 23 JSON Viewing and Authoring Tools 24 Overview of JSON Grammar 26 Booleans 27 Numbers 27 Strings 27 Arrays 28 iii

📄 Page 6

Objects 29 Null 31 Learning More 31 Minification 32 Alternative Representations 34 Creating a JSON Document 36 A Product Entry 36 A Store Order 37 Summary 38 3. Data-Centric Innovation: A Guide for Data Champions. . . . . . . . . . . . . . . . . . . . . . . . . . . 39 Data Transformations Require Data Champions 40 The Rise of the Data Product Manager 42 Alignment Is a Journey, Not a Destination 43 Evaluating Alignment from a Holistic Perspective 43 The Goal Isn’t Alignment, It’s Effective Alignment 45 Strategies for Setting Up Teams for Success 46 Incorporating a Product Management Mindset 48 Defining Data Users’ Needs 49 Defining Product Features 50 Defining and Measuring Success 52 Unifying Versus Aligning 52 Summary 54 4. Concept-First Design for Data Products. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 Packaging and Products: An Example Using Coffee 59 The Four Facets of a Data Product 60 Getting Started with Concept-First Design 63 A Blueprint for Unifying 64 Mapping the Conceptual Terrain: Assessing Concepts 65 Facilitating Assessments of Conceptual Alignment Across Technical and Nontechnical Teams 67 Smooth Is Slow, Slow Is Fast 69 Summary 70 5. A Universal Language for Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 What Is JSON Schema? 74 What Is a Schema? 74 The Building Blocks of JSON Schema 75 Vocabularies and Dialects 75 Meta-Schemas: Schemas That Describe Other Schemas 76 Understanding JSON Schemas 76 iv | Table of Contents

📄 Page 7

Step 1: Determining the Schema Dialect: The $schema Keyword 78 Step 2: Determining the Schema Vocabularies 79 Step 3: Understanding Schema Vocabularies 81 Step 4: Understanding Schema Keywords 82 JSON Schema as a Recursive Data Structure 86 Referencing Schemas 87 What does duplication look like? 87 Local referencing 88 Remote referencing 90 Your First JSON Schema Project 91 Writing a Schema: Step by Step 91 Generating a Web Form 95 Summary 97 6. The Art of Alignment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 Enemies of Alignment: Ambiguity and Assumptions 100 Ambiguity: The Culprit in the Illusion of Communication 101 Assumptions: Ambiguity’s Best Friend 102 Defining Success: Symmetry Between Concepts and JSON Schema Equals Minimal Ambiguity 102 Illuminating Misalignment with a Concept Compass 104 Step 1: Harmonizing the What 105 Step 2: Harmonizing the Way 106 Step 3: Harmonizing the How 108 Harmonized Concepts 109 Validating Concepts: Belief Scoring and Hypotheticals 111 Counterfactuals 111 Belief Scoring 112 Summary 113 7. The Science of Synchronization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 An Introduction to Thinking in Networks 116 Example of Thinking in Networks: Athletes Versus Artists 116 Graphs: The Visual Language of Networks 117 Networks of Entities: Knowledge Graphs 118 A Simple Knowledge Graph 119 Challenges with Knowledge Graphs 119 Aligning Knowledge for the 99% 120 Fundamentals of CLEAN Data Governance 120 Collaboration 122 Knowledge 123 Business Logic 124 Table of Contents | v

📄 Page 8

Activity 124 CLEAN Data Governance in Practice 125 The Four Facets of Data Products and CLEAN 126 The Four Horsemen of Data Death 127 Ignorance 128 Siloed Incentives 128 Shortsightedness 128 Indecisiveness 128 The Power of Design in Collaborative Networks 129 Summary 130 8. The Two Fundamental Operations of Schemas. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 Validating the Structure of Data 134 Using an Online Validator 135 Validation Example 136 JSON Schema as a Constraints Language 137 Boolean Schemas 139 Heterogeneous Data Structures 140 The format Keyword 142 Using Annotations to Define Meaning 144 Annotation Extraction Example 144 A Simple Use Case: Deprecations 145 Runtime Extraction 146 Standard Output Formats 148 Revisiting the format Keyword 150 Using an Online Validator 151 Thinking in Schemas 151 Summary 152 9. Illuminating Pathways of Acceleration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 How Ambiguity, Knowledge Gaps, and Blind Spots Influence Decisions and Progress Toward Goals 155 Which Is Bigger: Greenland or the US? 156 Mapping Pathways of Processes and Progress 157 Measuring Progress Toward Goals 157 Defining Decisions and Steps with Process Maps 158 How Process Maps Reveal Ambiguity 159 Visualizing and Removing Ambiguity in Processes 160 Enriching Process Maps with Annotations 162 Process Maps Reveal Innovation Opportunities 163 Summary 164 vi | Table of Contents

📄 Page 9

10. Spectrums of Success. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165 An Introduction to Knowledge Frameworks 166 Knowledge Experiences and Pathways 167 A Tool for Designing Knowledge Experiences 169 From Structured Knowledge to Computational Knowledge 171 Success Spectrums 172 Mapping Progress and Value 172 Visualizing and Adding “Next Best States” 173 Removing Blind Spots 174 Embracing Multiperspective Design and Road Maps 176 Defining KPIs for Success Measures and Metrics (Assessments) 178 Using Demons and Magical Thinking for Innovation 179 Faster Horses 180 Imagining Magical Possibilities 181 Problem Landscapes: Quantifying Pain Points Threatening Value 182 Nudges: The Right Information at the Right Time 183 A Real-Life Problem Landscape and Demon Example That Led to a Unified Data Product Model 184 Understanding the Problem Landscape 184 The Staggering Impact 185 A Meeting of Minds and the Birth of a Solution 185 Beyond Data Products: Data Product Management 187 The Circular Nature of Unifying 188 Summary 189 11. Deploying a JSON Schema Registry. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 Schemas Over HTTP 191 Step 1: Setting Up a GitHub Repository 192 Creating a GitHub Repository 192 Uploading Your First Schema 193 Step 2: Deploying to Cloudflare Pages 195 Creating a New Cloudflare Pages Website Project 195 Step 3: Configuring HTTP Headers 200 Inspecting the Current HTTP Headers 201 Declaring Custom HTTP Headers on Cloudflare Pages 201 Checking the Results 202 Step 4: Creating a Landing Page 204 Adding an HTML Entry Point 204 Step 5: Adding a Custom Domain 205 Configuring a Custom Domain in Cloudflare Pages 206 Setting Up a CNAME DNS Record 208 Checking the Results 209 Table of Contents | vii

📄 Page 10

Best Practices 210 Schemas Are Immutable 210 Adopt a Versioning Strategy 210 Summary 211 12. Designing Data Products Using JSON Schema. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213 First Facet: Data 214 An Example CSV Dataset 214 A JSON Row Representation 215 Second Facet: Structure 215 General-Purpose Concepts 215 Application-Specific Concepts 220 Dataset Entries 220 The Dataset Schema 221 Third Facet: Meaning 222 Timestamp 223 IP Address 223 Email 224 US State 224 Currency 225 Price 226 Milestone 227 Analytics Entry 227 Fourth Facet: Context 228 The Signup Analytics Schema 229 Summary 229 Automated Schema Extraction 229 Next Steps 231 13. Extending JSON Schema. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233 Simple Case: Unknown Keywords 234 Extracting Unknown Keywords as Annotations 234 Pros and Cons of This Approach 235 Complex Case: Authoring Vocabularies 236 The JSON Schema Vocabulary System 236 Step 1: Writing a Specification 237 Step 2: Writing a Vocabulary Meta-Schema 241 Step 3: Extending an Implementation 244 Consuming Vocabularies 247 Defining a Dialect 247 Making Use of the Dialect 249 Example: Extracting Annotations with Hyperjump 249 viii | Table of Contents

📄 Page 11

Summary 251 14. Introducing JSON Unify. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253 Introducing the Dataset Vocabulary 253 Revisiting the Signup Analytics Example 254 JSON Schema Bundling 255 The Bundling Process 258 Bundling Our Example Data Product 259 Referencing Remote Data 261 The Problem of Streaming JSON 262 Introducing JSON Lines 262 Extracting Meaning 263 A Simple Example 263 Using Logic Operators 264 The Signup Analytics Example 265 Dataset Lineage 266 Filtering 267 Transforming 268 Aggregation 269 Summary 271 15. Principles of Designing Intelligence. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273 Your Unifying Journey So Far 273 A Constellation of Deeper Principles Guides Unifying 274 1. The Principle of Alignment 275 Transforming the Abstract to Concrete 275 What You See Can Kill You, and the Same Is True in Data 276 2. The Principle of Information 278 Understanding Uncertainty 278 3. The Principle of Learning 280 Defining Learning 280 Defining Errors 282 4. The Principle of Integrated Simplicity 282 Complexity Reduction 282 Decomposition 283 Compression 283 Memoization 283 Integrating in Communication Networks 283 5. The Principle of Continuums 284 Making Things Measurable 284 The Dangers of Misusing Measurements 284 A Continuum Example for a Control Strategy Problem 285 Table of Contents | ix

📄 Page 12

6. The Principle of State Transitions 286 A Simple State Machine 287 Simplifying State Transitions 287 7. The Principle of Decidability 288 What Is Decidability? 288 Two Key Approaches to Problem Solving 289 Making Informed Decisions 289 Real-World Decidability to Reduce Misalignment in Teams 290 8. The Principle of Heuristics 290 Awareness and Ethical Considerations 291 Connection to Decision Making in Business 292 9. The Principle of Mastery 292 Levels of Mastery in Knowledge 293 Strategies for Mastery 294 10. The Principle of Wisdom 295 Summary 296 16. Toward Unified Intelligence. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297 Functional Artificial Intelligence 298 Your AI Is Only as Good as Your Data 298 Beware Illusions Within Vetting Processes 299 Question Assumptions 299 Collective Intelligence 299 Collaborative Intelligence 301 Unified Intelligence 302 Collaborative Learning Networks 302 Personalized Knowledge 303 Anticipatory Design: Personalization and Digital Twins 305 Codifying Principles of Intelligence 306 Continuous Human–Machine Learning Loops 308 Applying Wisdom in Practice 308 Conceptual Zoomability 309 Wisdom Graphs: Connecting Concepts, Actions, and Outcomes 311 Cognitive Primitives: Standardizing Cognitive Experience Design 312 The Value of Unifying 314 Prioritize Knowledge Before AI 314 A Tale of Simple Knowledge Versus Complex Intelligence 315 Follow the Principle of Integrated Simplicity 315 Summary 315 Going Beyond This Book 316 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317 x | Table of Contents

📄 Page 13

Preface In the 1840s, a Hungarian physician named Ignaz Semmelweis encountered a per‐ plexing challenge while working in the maternity clinic at the General Hospital in Vienna. A significant number of women were succumbing to a mysterious ailment known as “childbed fever,” which plagued many European hospitals. Semmelweis made a striking observation: the maternity ward overseen by male doctors had a significantly higher mortality rate than the one managed by midwives. Furthermore, he noticed that doctors often proceeded directly from performing autopsies to examining expectant mothers. After a colleague pricked his own finger while doing an autopsy, resulting in the colleague falling ill and eventually dying, Semmelweis had a revelatory moment: perhaps what killed his colleague might be also killing the women in childbirth. Semmelweis theorized that contaminants from the cadavers that doctors were oper‐ ating on and using to teach medical students might be transferring to the women, leading to the fever. To test this hypothesis, he implemented a policy in 1847 that required doctors to wash their hands with a chlorine solution to eliminate what he called “cadaverous particles,” before examining pregnant women. Following the implementation of this handwashing policy, the maternal mortality rate in the doctors’ ward plummeted from 18% to a mere 2%. However, Semmelweis’s ideas were met with skepticism from the medical community because they challenged the scientific beliefs at the time, and germ theory had not yet been developed. Semmelweis could offer no theoretical explanation for his findings, and he was mocked and ridiculed. In 1865, Semmelweis suffered from a nervous breakdown, resulting in his being committed to an asylum in Vienna by his colleagues, where he was beaten by guards and tragically died from a gangrenous wound on his right hand just 14 days later, at the age of 47. xi

📄 Page 14

The story of Ignaz Semmelweis offers a few valuable insights: Human behavior is constrained by bias Embracing new perspectives often challenges our deeply held beliefs. Such changes are frequently met with resistance—even from those equipped with knowledge and influence. It underscores the profound impact of cognitive bias and societal norms on human judgment. Interconnected systems are impacted greatly by hygiene The vast and intricate systems we see, such as hospital protocols or childbirth procedures, can be dramatically influenced by elements so minuscule they’re often invisible, such as germs. This highlights the delicate balance and intercon‐ nectedness of our world, from the microscopic to the grand scale. Simple actions can have massive ripple effects At times, the most straightforward measures, like handwashing, become our most potent solutions. Understanding the methods to mitigate tiny threats can prove pivotal, with ramifications felt on a monumental scale. What You Can’t See Can Kill You, and the Same Is True for Data The transformative shift in our understanding of disease causation can be dated back to the 1860s. Louis Pasteur’s revolutionary experiments demonstrated that microor‐ ganisms were responsible for fermentation and spoilage, laying the foundation for germ theory and paving the way for monumental advancements such as vaccines, antiseptics, and sterilization techniques. In marked contrast, Ignaz Semmelweis made essential observations decades earlier but remained largely overlooked due to his lack of a robust scientific theory. The divergence in their legacies—Pasteur’s transformative influence versus Semmelweis’s limited recognition—emphasizes the critical need for both theoretical and practical foundations in tackling complex problems. Unifying Business, Data, and Code seeks to bridge this very gap in the field of organi‐ zational data management and the design of intelligent systems. We aim to furnish you with both a robust theoretical framework and actionable practical tools, applica‐ ble whether you’re brainstorming strategies on a whiteboard or coding sophisticated algorithms. Diverging from books that concentrate on either technical or managerial facets of data and intelligent system design, Unifying Business, Data, and Code takes a holistic stance that merges both strategic perspectives. We’ve discovered that a technically sound strategy lacking managerial integration is doomed to fail—and the reverse is xii | Preface

📄 Page 15

equally true. This synthesis enables you to make better-informed decisions, effectively bridging the divide between IT and business strategy. Just as neglecting basic hand hygiene had devastating repercussions in Semmelweis’s time, modern organizations face concealed yet significant risks from poor data man‐ agement. In essence, the primary challenges compromising your organizational data hygiene can be distilled into three categories: Ambiguity There are multiple possible interpretations. Knowledge gaps Missing information obstructs problem solving. Blind spots There is a lack of awareness of ambiguity and knowledge gaps and their effects on organizational outcomes. This book will guide you through the process of identifying poor data hygiene and the root causes of misalignment that it leads to within your organization. Armed with this understanding, you’ll be equipped to drive innovation and transformation through a strategic data management approach, unlocking the benefits of intelligent system design for superior results. Hidden Threats to Organizations: A Modern Parallel In the expansive world of organizational dynamics, hidden levels of granularity shape our actions and decisions, yet remain unseen in our daily routines. This book journeys into these enigmatic depths. True organizational coherence demands the dexterity to zoom out, transcending individual roles and looking at the vast networks that knit an organization together. At the same time, mastering the finesse to “zoom in” becomes crucial to tackle nuanced data challenges. Like the invisible germs Sem‐ melweis grappled with, these subtle issues can reverberate and escalate unpredictably, leading to profound consequences. As an example, imagine a retail company using data analytics to forecast demand. A column in the database is ambiguously labeled as Total Sales Revenue. An analyst assumes this means Net Sales, the revenue after returns and discounts, but it actually represents Gross Sales, the revenue generated before any expenses. This simple misun‐ derstanding skews the demand forecast, as the report doesn’t take into account the item’s terrible quality and high return rates. The company ends up overstocking the flawed items and understocking good ones. Inventory costs balloon, customers are left dissatisfied and lose trust in the brand. The flawed decision making results in a hunt for who to blame for the error, and as the culture becomes toxic, top performers who care about the company leave a Preface | xiii

📄 Page 16

vacuum of expertise and talent. Like a house of cards, a single ambiguous label can lead the entire organizational strategy to crumble. Central to our discussion is the idea of concepts, shown in Figure P-1, serving as the foundation of our unifying methodology. While a deeper exploration awaits in later sections, for now you can think of concepts as the vital atoms whose unique configuration and combination creates the elements of our everyday experiences at the second level of granularity shown in Figure P-1: language, processes, and decision making. Consider data products, shown in Figure P-1, to be our metaphorical “handwashing” solution. Although our unifying principles help pinpoint ambiguity, knowledge gaps, and blind spots, it’s data products that, much like a sanitizing solution, actively cleanse and address these issues in practice. Imagine your data as a high-quality product on a store shelf. It should be well crafted, easy to use, and comprehensive. In this book, you’ll learn how to elevate your data to that level of quality. We’ll guide you through a standardized process that packages the structure, meaning, and context along with the data itself. Once your organization begins designing high-quality data products, the benefits of implementing data hygiene can be quite transformative, freeing up teams from putting out fires in a chronically troubled system and enabling them to focus on creating business value, enhancing efficiencies, and innovation excellence. Additionally, the principles and methodologies we’ve discussed so far set the founda‐ tion for something even more powerful: unified intelligence, which is applying the unifying methodology to human and machine learning system design. Chapter 15 introduces unified intelligence. However, before we can even begin to think of using the principles of unifying with AI, we need to get our data in good shape. Your AI Is Only as Good as Your Data The axiom “Your AI Is Only as Good as Your Data” serves as a critical pillar of this book, highlighting the inextricable link between data quality and AI efficacy. Our framework builds on the groundbreaking contributions of seminal figures in the field—Claude Shannon’s information theory, Alan Turing’s computational models, and Shane Legg and Marcus Hutter’s advancements in reinforcement learning. Their collective insights merge seamlessly into our comprehensive methodology, which we will explore in detail in Chapter 15. Data scientists leverage rigorous methodologies and empirical reasoning to dissect complex challenges and represent them in a structured format. This facilitates the deployment of machine learning algorithms and the construction of predictive mod‐ els. In this book, we introduce the concept of designing intelligence—a synthesized set xiv | Preface

📄 Page 17

of best practices aimed to equip both technical experts and managerial staff with a robust skill set in data-centric problem solving. Figure P-1. Three levels of granularity are shown in this illustration, each with issues that the unifying methodology will address at each level of granularity. The key activity you will be learning is to identify and minimize ambiguity, knowledge gaps, and blind spots to align all three levels. Preface | xv

📄 Page 18

Adopting these best practices doesn’t merely set the stage for successful AI initiatives; it transforms your entire organizational data culture, cultivating a fertile ground for data-centric innovation across your organization grounded in principles of designing intelligence. Aligning Problem-Solving Strategies, Data, and AI Reinforcement learning serves as a critical pillar in understanding principles of design‐ ing intelligent systems, guiding decision-making strategies that oscillate between exploration for new knowledge and exploitation of existing knowledge. As illustrated in Figure P-2, this dynamic reflects human and organizational tendencies to balance effort against reward, thereby shaping the innovation and efficiency strategies of companies. Figure P-2. This diagram presents a cycle of decision making based on outcomes from exploiting current data and exploring new data. Organizations or individuals can use this model to determine when to rely on existing knowledge (exploit) and when to seek out new information or try new approaches (explore). xvi | Preface

📄 Page 19

Too often, organizational leaders are ensnared in a narrow, top-down mindset that prioritizes exploitation strategies over exploration. This culminates in vague visions that rarely manifest into tangible innovation. When these approaches fall short, it’s usually the workforce that suffers the consequences, from blame and job loss to unsettling structural shifts. This book offers a suite of strategic and technical tools aimed at breaking this detrimental cycle, moving beyond short-term fixes to achieve sustainable progress. This book encapsulates our insights from personal exploration and exploitation jour‐ neys—knowledge we find crucial to share. We’re deeply grateful for your investment in this work. Our aspiration is that, by the end, the principles we unveil will resonate so deeply that their application becomes as intuitive and vital as washing your hands. A New Paradigm to Optimize Data Management and Business Strategy for the Age of AI Recognize that unlearning is the highest form of learning. —Rumi, Persian poet Unifying challenges conventional approaches with a cutting-edge approach: it uses principles from data science used in problem solving to optimize data and knowledge for creating business value. This strategy ensures that your organization will be maximally primed for success in AI endeavors. Whether you’re dealing with human decision making or computational systems, this book offers a practical blueprint for smarter operations: • Strategies and technologies unifying data management and business strategy are presented in Chapters 1–14. • The foundational theoretical principles from the fields of artificial intelligence, cognitive psychology, that were used to create the unifying methodology, are covered in Chapter 15. • Building upon your unified data management and business strategy and the principles of designing intelligent systems, Chapter 16 explores different ways to apply unifying with AI. In the pursuit of understanding and harnessing the power of data for business strat‐ egy, it’s crucial to keep an open mind—to entertain various hypotheses and embrace the uncertainty created when experiencing new ways of thinking. As Hala Nelson asserts in Essential Math for AI (O’Reilly, 2023), “Data is the fuel that powers most AI systems” and “What I did not know, and learned the hard way, was that getting real data was the biggest hurdle.” Preface | xvii

📄 Page 20

The methodology elucidated in this book empowers you to apply data science princi‐ ples and problem-solving strategies effectively without needing to be a data scientist, ensuring that the data you create and collect is not only more accurate and useful, but also a closer reflection of reality. By embracing the principles you will learn in this book, you will not just be able to solve existing problems better than ever before—you’ll preempt future ones from existing in the first place. The Origin Story of Unifying Driven by his work in AI within the edtech sector, Ron harbored an insatiable curios‐ ity to understand principles of designing intelligence that underpin both human and machine learning systems. He envisioned organizations not merely as static struc‐ tures, but as dynamic ecosystems where information networks intermingle much like the notes in a symphony. Enter Juan, a leading expert in JSON, JSON Schema, and data serialization. Juan wasn’t just technically proficient; he had the unique ability to take Ron’s grand vision and turn it into a finely tuned reality. Juan’s award-winning research in data serialization at the University of Oxford revealed he could apply the methodology all the way down to the binary level and all the way up to gold-standard protocols for a global-scale data specification. Our partnership was nothing short of magical—akin to a musical band discovering perfect harmony among its members. Together, we embarked on an unceasing jour‐ ney of growth and innovation, each challenging and enriching the other’s domain expertise. This book represents the zenith of our collaborative efforts, serving as a comprehensive guide that harmonizes overarching strategies with granular technical solutions for organizations. We wrote this book with a singular, transformative purpose in mind: to empower people with bold guiding principles and technical strategies that can cut through seemingly impossible problems by unifying people, processes, and data across multi‐ ple, and seemingly invisible, scales. We want to democratize this knowledge, to make it accessible and actionable for all, unleashing waves of creativity and ingenuity to transform the world for the better. The quest to explore and codify the principles of unifying led us into the realms of the mysterious and unknown. Sharing the wisdom we’ve garnered along this journey brings us the incomparable joy of serving a purpose far greater than ourselves. xviii | Preface