Network Programmability and Automation Skills for the Next-Generation Network Engineer, 2nd Edition (Matt Oswalt, Christian Adell, Scott S. Lowe etc.) (Z-Library)

📄 File Format: PDF

💾 File Size: 9.7 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

Matt Oswalt, Christian Adell, Scott S. Lowe & Jason Edelman Network Programmability & Automation Skills for the Next-Generation Network Engineer Second Edition

📄 Page 2

SECURIT Y “This book helps networking professionals and developers write clear, easily extensible network automation playbooks. It covers almost everything that you need to be at the helm of network automation.” —Akhil Behl Partner Manager, Red Hat Network Programmability & Automation Twitter: @oreillymedia linkedin.com/company/oreilly-media youtube.com/oreillymedia Network engineers are finding it harder than ever to rely solely on manual processes to get their jobs done. New protocols, technologies, delivery models, and the need for businesses to become more agile and flexible have made network automation essential. The updated second edition of this practical guide shows network engineers how to use a range of technologies and tools, covering the following topics: • New Linux-based networking technologies and cloud native environments, and how to use them to bootstrap development environments for your network projects • Programming skills with Python and Go • How templating can help you build consistent network device configurations • Data formats, encodings, and models: JSON, XML, YAML, Protobuf, and YANG • The role of application programming interfaces (APIs) in network automation, including HTTP APIs, NETCONF, RESTCONF, and gNMI • How to use version control systems like Git and development practices like continuous integration in your network automation projects • How to automate network devices and services using Ansible, Nornir, and Terraform • A holistic approach to architecting network automation solutions Matt Oswalt is a senior systems engineer at Cloudflare, where he works on distributed systems and internet and web protocols. Christian Adell is a principal architect at Network to Code, with experience building and delivering network automation solutions for on-premises and cloud environments. Scott Lowe works on the developer relations team at Pulumi, where he focuses on infrastructure as code and automating cloud infrastructure and services. Jason Edelman is the founder and CTO at Network to Code, a leading network automation solutions provider. Jason founded Network to Code in 2014. US $59.99 CAN $74.99 ISBN: 978-1-098-11083-3

📄 Page 3

Praise for Network Programmability and Automation I highly recommend this new edition of the book to anyone looking to improve their network automation skills. The authors do an outstanding job of explaining complex topics in a clear and concise way. They also provide a comprehensive overview of current technology trends in the field, as well as architectural design options and methodologies that you can use to achieve positive results for your organization. I found the book to be very helpful and informative, and I am confident that you will too. —Nicolas Leiva (Staff Solutions Architect at Red Hat and author of Network Automation with Go) Networking infrastructures have undergone significant evolution in various aspects of cloud computing, as well as cloud-native eras, including scale, complexity, functionalities, and virtualization depths. As a result, traditional network operation fashions are rapidly becoming obsolete. This book offers a comprehensive guide to addressing the challenge through network automation, covering fundamental programming languages, common automation tools, and the development of user-facing systems. By leveraging the techniques presented in this book, readers can establish a robust, flexible, and evolving automation platform using cutting-edge tools. —Arthur (Yanan Zhao), Senior Architect in Cloud Department, Trip.com This book helps networking professionals and developers write clear, easily extensible network automation playbooks. It covers almost everything that you need to be at the helm of network automation. —Akhil Behl, Partner Manager, Red Hat

📄 Page 4

Jason, Scott, and Matt have been key contributors in educating network engineers about both network automation and Linux networking. They have written and talked extensively about the importance of automation, on how automation impacts network engineers, and on the mechanics of automating networking devices. —Kirk Byers Creator of the Netmiko Python library Network automation is no longer just a proof of concept: it represents both the present and the future! Network Programmability and Automation provides the needed background for modern engineers, by widening the toolset for more consistent, stable, and reliable networks. —Mircea Ulinic Network Engineer, DigitalOcean Network automation is not hype anymore; it is a means to do your job faster, more consistently, and more reliably. However, network automation is not just a single discipline; it is a collection of protocols, tools, and processes that can be overwhelming to the uninitiated. This book does a great job covering everything you will need to get your automation up and running. —David Barroso Creator of NAPALM and Nornir

📄 Page 5

Matt Oswalt, Christian Adell, Scott S. Lowe, and Jason Edelman Network Programmability and Automation Skills for the Next-Generation Network Engineer SECOND EDITION Boston Farnham Sebastopol TokyoBeijing

📄 Page 6

978-1-098-11083-3 [LSI] Network Programmability and Automation by Matt Oswalt, Christian Adell, Scott S. Lowe, and Jason Edelman Copyright © 2023 Matt Oswalt, Scott S. Lowe, and Christian Adell. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (https://oreilly.com). For more information, contact our corporate/institu‐ tional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editors: Simina Calin and Jennifer Pollock Development Editor: Melissa Potter Production Editor: Kristen Brown Copyeditor: Sharon Wilkey Proofreader: Stephanie English Indexer: WordCo Indexing Services, Inc. Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea February 2018: First Edition August 2023: Second Edition Revision History for the Second Edition 2023-08-16: First Release See https://oreilly.com/catalog/errata.csp?isbn=9781098110833 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Network Programmability and Automa‐ tion, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

📄 Page 7

I dedicate this book to people chasing their dreams; keep it up! One of them was my beloved brother, Felip, with whom I shared my dreams and who will be with me forever. Special thanks to Jason for trusting me in this fantastic endeavor, and to Anna and Èlia for giving me the time and understanding to write it. —Christian Adell I dedicate this book to my wife, Crystal. Her love and her support propels me forward in everything I do, and I wouldn’t be who I am without her. I also thank God, whose gift of understanding (Proverbs 2:6) makes sharing knowledge with others possible. —Scott S. Lowe This book is dedicated to my children, who give me renewed purpose and hope every single day, and for whom I do everything. I also dedicate it to my wife, Jamie, for giving her time and energy to make it possible for me to participate in projects like this. —Matt Oswalt I dedicate this book to all network engineers starting their network automation journey. I sincerely hope it provides each of you with the knowledge needed to further enhance your career. I’d also like to thank Scott, Matt, Christian, and the whole O’Reilly team. Thanks to everyone for making it a reality and helping transform an industry. —Jason Edelman

📄 Page 8

(This page has no text content)

📄 Page 9

Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv 1. Network Industry Trends. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 The Rise of Software-Defined Networking 1 The Advent of OpenFlow 1 Why OpenFlow? 3 What Is Software-Defined Networking? 5 Opening Up the Data Plane 6 Network Functions Virtualization 8 Virtual Switching 10 Network Virtualization 11 Device APIs 12 Network Automation 13 Bare-Metal Switching 15 Data Center Network Fabrics 17 SD-WAN 18 Controller Networking 20 Cloud Native Networking 21 Summary 22 2. Network Automation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Why Network Automation? 24 Simplified Architectures 24 Deterministic Outcomes 25 Business Agility 25 Enhanced Security and Risk Reduction 26 Types of Network Automation 26 Device Provisioning 27 vii

📄 Page 10

Data Collection and Enrichment 29 Migrations 31 Configuration Management 32 Configuration Compliance 33 State Validation 34 Reporting 34 Troubleshooting 35 Evolving the Management Plane from SNMP to APIs 36 Application Programming Interfaces 37 Impact of Open Networking 40 Network Automation in the SDN Era 41 Summary 42 3. Linux. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Examining Linux in a Network Automation Context 43 Exploring a Brief History of Linux 44 Understanding Linux Distributions 45 Red Hat Enterprise Linux, Fedora, CentOS, and Amazon Linux 45 Debian, Ubuntu, and Other Derivatives 47 Other Linux Distributions 49 Interacting with Linux 49 Navigating the Filesystem 50 Manipulating Files and Directories 54 Running Programs 61 Working with Daemons 64 Networking in Linux 68 Working with Interfaces 68 Routing as an End Host 80 Routing as a Router 85 Bridging (Switching) 86 Automating Linux with bash Scripting 91 Evolving the Linux Kernel with eBPF and XDP 93 Summary 95 4. Cloud. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 Brief Definition of Cloud Computing 97 Networking Fundamentals in the Cloud 100 Cloud Networking Building Blocks 100 Cloud Network Topologies 102 Network Automation in the Cloud 108 Containers 112 What Do Containers Have to Do with the Cloud? 114 viii | Table of Contents

📄 Page 11

What Do Containers Have to Do with Networking? 114 Extending Linux Networking for Containers 115 Kubernetes 126 Key Kubernetes Concepts 127 Building Blocks of Networking in Kubernetes 128 Service Mesh 136 Summary 138 5. Network Developer Environments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 Text Editors 140 Syntax Highlighting 141 Customization 142 Intelligent Code Analysis 144 Text Editors Versus Integrated Development Environments 148 Development Tools 150 Virtualenv 151 Make 151 Docker 152 dyff 155 Emulation/Simulation Tools 157 VirtualBox 158 Vagrant 159 Containerlab 163 Other Tools 166 Summary 166 6. Python. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 Should Network Engineers Learn to Code? 168 Using the Python Interactive Interpreter 169 Understanding Python Data Types 172 Strings 173 Numbers 182 Booleans 184 Lists 187 Python Dictionaries 193 Sets and Tuples 198 Adding Conditional Logic to Your Code 200 Understanding Containment 202 Using Loops in Python 203 The while Loop 204 The for Loop 204 Using Python Functions 209 Table of Contents | ix

📄 Page 12

Working with Files 213 Reading from a File 213 Writing to a File 215 Creating Python Programs 218 Creating a Basic Python Script 218 Using Comments in Python 219 Migrating Code from the Python Interpreter to a Python Script 220 Working with Python Modules 222 Documenting Functions 224 Passing Arguments into a Python Script 225 Using pip and Installing Python Packages 227 Isolating Your Dependencies with Virtualenv 229 Understanding Python Classes 230 Using Classes 231 Building Your Own Classes 232 Embracing Failure with try/except 234 Parallelizing Your Python Programs 236 Summary 239 7. Go. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 Why Go? 243 Is Go Faster Than Python? 245 Is Go Harder than Python? 245 Fundamental Go Concepts 247 Types and Variables 248 Flow Control 253 Collection Types 258 Functions 269 Structs 275 Methods 277 Interfaces 282 Advanced Concepts 287 Concurrency 288 Generics 295 Packages and Modules 297 Standard Library Packages 299 Third-Party Modules and Packages 310 Summary 315 8. Data Formats and Models. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317 Benefits and Fundamentals of Structured Data Formats 318 When Structured Data Isn’t Available: Screen Scraping 321 x | Table of Contents

📄 Page 13

Types of Data 322 Documents Versus Data 322 Categories of Data Formats 323 Text-Based Data Formats 323 YAML 325 XML 329 JSON 335 Binary Data Formats 339 Protocol Buffers 343 Other Binary Data Formats 348 Data Modeling 350 YANG 353 JSON Schema 359 XML Schema Definition 364 Modeling and Validating Protocol Buffers 365 Summary 366 9. Templates. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367 The Rise of Modern Template Languages 368 Using Templates for Web Development 369 Expanding On the Use of Templates 370 The Value of Templates in Network Automation 370 Jinja 372 Rendering a Jinja Template File in Python 373 Using Conditionals and Loops 375 Using Jinja Filters 381 Template Inheritance in Jinja 384 Variable Creation in Jinja 386 Extensible Stylesheet Language Transformations 386 Go Templates 390 Summary 396 10. Working with Network APIs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 397 Understanding Network APIs 398 Getting Familiar with HTTP-Based APIs 399 Using NETCONF 410 Using RESTCONF 425 Using gRPC and gNMI 434 Comparing NETCONF, RESTCONF, and gNMI 446 Using Network APIs for Automation 451 The Python Requests Library 452 The Go net/http Package 464 Table of Contents | xi

📄 Page 14

The Python ncclient Library 468 The OpenConfig gNMIc Go Package 477 The Netmiko Python Library 485 Summary 493 11. Source Control with Git. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 495 Use Cases for Source Control 495 Benefits of Source Control 496 Change Tracking 496 Accountability 496 Process and Workflow 497 Benefits of Source Control for Networking 497 Enter Git 498 Brief History of Git 498 Git Terminology 499 Overview of Git’s Architecture 500 Working with Git 501 Installing Git 501 Creating a Repository 501 Adding Files to a Repository 503 Committing Changes to a Repository 504 Changing and Committing Tracked Files 507 Unstaging Files 510 Excluding Files from a Repository 513 Viewing More Information About a Repository 517 Distilling Differences Between Versions of Files 521 Tagging Commits in Git 524 Branching in Git 527 Renaming the Default Branch 532 Creating a Branch 533 Checking Out a Branch 534 Stashing Uncommitted Changes 536 Merging and Deleting Branches 539 Collaborating with Git 545 Collaborating Between Multiple Systems Running Git 545 Collaborating via Git-Based Online Services 561 Summary 566 12. Automation Tools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 567 Reviewing Automation Tools 568 Using Ansible 571 Discovering the Ansible Framework 571 xii | Table of Contents

📄 Page 15

Understanding How Ansible Works 573 Constructing an Inventory File 574 Executing an Ansible Playbook 581 Using Secrets 586 Using Variable Files 588 Writing Ansible Playbooks for Network Automation 589 Using Third-Party Collections and Modules 611 Ansible Summary 614 Automating with Nornir 615 Getting Started with Nornir 616 Using NAPALM with Nornir 624 Nornir Summary 631 Managing Dynamic Infrastructure with Terraform 632 Understanding Terraform Architecture 633 Provisioning Your First Resource with Terraform 637 Extending Terraform Execution 653 Managing Terraform at Scale 661 Terraform Out of Its Comfort Zone 673 Terraform Summary 679 Summary 679 13. Continuous Integration. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 681 Important Prerequisites 684 Simple Is Better 684 People, Process, and Technology 684 Learn to Code 685 Introduction to Continuous Integration 685 Basics of Continuous Integration 686 Continuous Delivery 687 Test-Driven Development 689 Why Continuous Integration for Networking? 690 A Continuous Integration Pipeline for Networking 691 Peer Review 693 Build Automation 699 Deployment Validation and Testing 705 Test/Dev/Staging Environment 707 Deployment Tools and Strategies 709 Summary 712 14. Network Automation Architecture. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 713 Introducing the Network Automation Architecture 714 Overview of the Architecture Components 717 Table of Contents | xiii

📄 Page 16

User Interactions 718 Source of Truth 727 Automation Engine 745 Telemetry and Observability 751 Orchestration 771 Understanding the Architecture with an Example 775 Determine the Operation’s Workflow 775 Translate the Workflow Steps to Automated Tasks 776 Map the Automated Tasks to Architecture Components 777 Choose the Tools to Implement Each Component’s Tasks 779 Summary 781 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 783 xiv | Table of Contents

📄 Page 17

Preface Welcome to Network Programmability and Automation! The networking industry is changing dramatically. The drive for organizations and networking professionals to embrace the ideas and concepts of network programm‐ ability and automation is greater now than perhaps it has ever been, fueled by a revolution in new protocols, new technologies, new delivery models, and a need for businesses to be more agile and more flexible in order to compete. But what is network programmability and automation? Let’s start this book by answering that question. What This Book Covers As its title indicates, this book is focused on network programmability and automa‐ tion. At its core, network programmability and automation is about simplifying the tasks involved in configuring, managing, and operating network equipment, network topologies, network services, and network connectivity. Many, many components are involved—including operating systems that are now seeing far broader use in networking than in the past, new methodologies like contin‐ uous integration, and tools that formerly might have fallen only in the realm of the system administrator (tools like source code control and configuration management systems). Because all of these play a part in the core definition of network programm‐ ability and automation, we cover all these topics. Our goal for this book is to enable you to establish a foundation of knowledge around network programmability and automation. xv

📄 Page 18

What’s New in This Edition This edition of the book has been updated from the original to include four totally new chapters: • Chapter 4, “Cloud” • Chapter 5, “Network Developer Environments” • Chapter 7, “Go” • Chapter 14, “Network Automation Architecture” We’ve also updated the existing chapters to include newer concepts such as these: • Google Protocol Buffers • gRPC/gNMI • Terraform • Nornir On top of all these new additions, we’ve refreshed and extended every chapter to reflect the advances and industry changes that have taken place since the first edition was published. We’re thrilled to have the opportunity to add these new, exciting topics to the book. Because of size constraints, some content from the first edition was not carried over into this one. However, rather than simply removing this content, we’ve made it freely available at https://oreilly-npa-book.github.io. How This Book Is Organized This book isn’t necessarily intended to be read from start to end; instead, we’ve broken up the topics so that you can easily find the content that most interests you. You may find it useful to start out sequentially reading the first three chapters, as they provide background information and set the stage for the rest of the book. From there, you’re welcome to jump to whatever topics are most useful or interesting to you. We’ve tried to keep the chapters relatively standalone, but—as with any technology— that’s not always possible. Wherever we can, we provide cross-references to help you find the information you need. xvi | Preface

📄 Page 19

Here’s a quick look at how we’ve organized the topics: Chapter 1, “Network Industry Trends” Provides an overview of the major events and trends that launched software- defined networking (SDN). As you’ll see, SDN was the genesis for an increased focus on network programmability and automation. Chapter 2, “Network Automation” Takes the SDN discussion from Chapter 1 and focuses specifically on network automation—the history of network automation, types of automation, tools and technologies involved in automation, and how automation affects operational models (and how operational models affect automation). Chapter 3, “Linux” Provides an overview of the Linux operating system. By no means a comprehen‐ sive discussion of Linux, this chapter aims to get networking professionals up to speed on Linux, basic Linux commands, and Linux networking concepts. Chapter 4, “Cloud” Introduces cloud computing from a networking perspective, and provides a jumping-off point for many relevant topics like containers and Kubernetes net‐ working. We also discuss how network engineers’ existing skills translate to a cloud-based environment. Chapter 5, “Network Developer Environments” Explores the tools and techniques for maintaining a network developer environ‐ ment. This is a crucial part of any network automation project, and it allows teams to collaborate on solutions more effectively, as well as efficiently move from development to testing, all the way to production. Chapter 6, “Python” Introduces networking professionals to Python. This programming language is frequently used in network programmability and automation contexts, and this chapter covers many of the basics of programming with Python: data types, conditionals, loops, working with files, functions, classes, and modules. Chapter 7, “Go” Adds a second option to the network automation professional’s programming language repertoire, by exploring Go. Having recently grown in popularity—in large part because of the cloud native ecosystem—Go is a valuable asset even for those just starting on their automation journey. Preface | xvii

📄 Page 20

Chapter 8, “Data Formats and Models” Digs into the formats and methods for transmitting, storing, and modeling network automation data. The ability to understand and work with data is a fundamental skill for any network automation professional, and in this chapter, we build a solid foundation for the chapters to follow. Chapter 9, “Templates” Looks at the use of templating languages to create network device configurations. While the primary focus of this chapter is on the Jinja templating language, as it integrates natively with Python, we also explore Go templates and Extensible Stylesheet Language Transformations (XSLT). Chapter 10, “Working with Network APIs” Covers the role of APIs in network programmability and automation. We explore key terms and technologies pertaining to APIs including HTTP, NETCONF, RESTCONF, and gNMI as examples to show how they can be used for network programmability and automation. We use both Python and Go libraries to auto‐ mate network devices using these APIs. Chapter 11, “Source Control with Git” Introduces Git, a popular and widely used tool for source code control. We talk about why source code control is important, how it is used in a network programmability and automation context, and how to work with popular online services such as GitHub. Chapter 12, “Automation Tools” Explores the use of open source automation tools such as Ansible, Nornir with NAPALM, and Terraform. You’ll learn how these tools can be used specifically for network programmability and automation, using the imperative and declara‐ tive approaches. Chapter 13, “Continuous Integration” Examines continuous integration (CI) and the key tools and technologies that are involved. We discuss the use of test-driven development (TDD), explore tools and frameworks like GitLab and Jenkins, and take a look at a sample network automation workflow that incorporates all these CI elements. Chapter 14, “Network Automation Architecture” Unifies all the concepts covered in prior chapters by proposing a reference archi‐ tecture for a holistic network automation solution. Brings together automation and orchestration, explores the idea of a source of truth, and shows how user interactions might work in such a system. xviii | Preface