Learning CoreDNS Configuring DNS for Cloud Native Environments (John Belamaric, Cricket Liu) (Z-Library)

John Belamaric & Cricket Liu Learning CoreDNS Configuring DNS for Cloud Native Environments

John Belamaric and Cricket Liu Learning CoreDNS Configuring DNS for Cloud Native Environments Boston Farnham Sebastopol TokyoBeijing

978-1-492-04796-4 [LSI] Learning CoreDNS by John Belamaric and Cricket Liu Copyright © 2019 John Belamaric and Cricket Liu. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: John Devins Development Editor: Melissa Potter Production Editor: Christopher Faucher Copyeditor: Octal Publishing, LLC Proofreader: Christina Edwards Indexer: Ellen Troutman-Zaig Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Rebecca Demarest September 2019: First Edition Revision History for the First Edition 2019-08-30: First Release See http://oreilly.com/catalog/errata.csp?isbn=9781492047964 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Learning CoreDNS, the cover image, and related trade dress are trademarksvof O’Reilly Media, Inc. The views expressed in this work are those of the authors, and do not represent the publisher’s views. While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.

Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii 1. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 What Is CoreDNS? 1 CoreDNS, Containers, and Microservices 2 CoreDNS Limitations 3 CoreDNS, Kubernetes, and the Cloud Native Computing Foundation 4 2. A DNS Refresher. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 What Is the Domain Name System? 5 Domain Names and the Namespace 6 Domains, Delegation, and Zones 7 Resource Records 9 DNS Servers and Authority 10 Resolvers 11 Resolution and Recursion 12 Caching 15 Resource Records 15 NAME 16 TTL 16 CLASS 17 Resource Record Types 17 The A Record 17 The AAAA Record 18 The CNAME Record 18 The MX Record 19 The NS Record 20 iii

The SRV Record 21 The PTR Record 23 The SOA Record 24 An Annotated Zone Data File 26 3. Configuring CoreDNS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 Getting CoreDNS 29 CoreDNS Command-Line Options 31 Corefile Syntax 32 Environment Variables 34 Reusable Snippets 35 Import 35 Server Blocks 35 Query Processing 37 Plug-ins 38 Root 39 File 39 Secondary 40 Forward 42 Cache 44 Errors 45 Log 47 Common Configuration Options 50 fallthrough 50 tls 50 transfer to 51 Sample DNS Server Configurations 51 Caching-Only DNS Server 51 Primary DNS Server 52 Secondary DNS Server 52 4. Managing Zone Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 The file Plug-in 55 The auto Plug-in 58 Using the auto Plug-in with Git 59 The hosts Plug-in 60 The route53 plug-in 62 5. Service Discovery. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 Introduction to Service Discovery 65 Solving the Service Discovery Problem 66 iv | Table of Contents

Service Discovery with CoreDNS and etcd 68 The etcd Plug-in 69 Other Service Discovery Options 74 Service Discovery and Container Orchestration 75 6. Kubernetes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 Basic Concepts 77 Kubernetes Networking 79 Cluster IP Services 80 Headless Services 81 Kubernetes DNS Specification 82 CoreDNS Integration 91 Default Configuration 93 Stub Domains and Federations 96 Cluster DNS Deployment Resources 98 Role-Based Access Control 98 Service 100 Deployment 101 Autoscaling 105 A Better Configuration 106 The kubernetes Plug-in 109 CoreDNS Extensions 111 Pod Options 111 Wildcard Queries 112 Autopath and the Dreaded ndots:5 113 Zone Transfer Support 115 Exposing Services Externally 116 Modifying the Available Records 117 7. Manipulating Queries and Responses. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 The template Plug-in 121 The rewrite Plug-in 124 Using the rewrite Plug-in for EDNS0 Options 127 Multiple rewrite Rules 128 The metadata Plug-in 129 Signing Responses with the DNS Security Extensions 130 Managing a DNSSEC-Signed Primary Zone 131 On-the-Fly DNSSEC Signing with the dnssec Plug-in 136 Case Study: Infoblox’s BloxOne Threat Defense 137 Identifying Users 138 Applying Policy 139 Table of Contents | v

8. Monitoring and Troubleshooting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 The prometheus Plug-in 141 The log Plug-in 143 The dnstap Plug-in 147 The errors Plug-in 150 The trace Plug-in 151 The debug Plug-in 154 9. Building a Custom Server. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 Compiling CoreDNS with an External Plug-in 157 Building Using Docker 158 Building on Your Workstation 159 Modifying plugin.cfg 161 Replacing main 164 Writing a Custom Plug-in 170 There Can Be Only One 173 Integrating with Metrics, Trace, and Metadata 178 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181 vi | Table of Contents

1 Which wins the award for most prosaic name, hands-down. Preface Why a New DNS Server? Upon seeing this book, the first question that might occur to you is, “Why does the world need another DNS server?” There are, after all, lots of implementations of DNS servers to choose from. For starters, there’s BIND, for Berkeley Internet Name Domain, the granddaddy of DNS servers. BIND has been around in some incarna‐ tion since the 1980s and supports just about every DNS standard written. There’s Microsoft’s DNS Server,1 which is widely used in Active Directory environments. NSD from NLnet Labs and Knot are excellent authoritative DNS servers, and Unbound, also from NLnet Labs, is a fast, lean recursive DNS server. So what does CoreDNS offer that these others don’t? To begin with, CoreDNS is written in Go, and Go is a memory-safe programming language. Why is that important? Well, if you’ve ever run a BIND-based DNS infra‐ structure and had to upgrade 100 DNS servers ASAP because of a buffer overrun, you know. A healthy proportion of vulnerabilities in DNS servers of all stripes (at least those written in C and C++) stem from buffer overflows or overruns and dangling pointers. Written in memory-safe Go, CoreDNS isn’t subject to these. Programs written in Go can also support concurrency, or parallel execution. This can be useful in wringing more performance out of multiprocessing or multitasking sys‐ tems. BIND’s performance somewhat notoriously doesn’t scale well on multiproces‐ sor systems, whereas CoreDNS’s performance scales nicely the more processors it has to work with. vii

2 Meaning that the same algorithm implemented the same way in Go, C, and C++ will probably run slightly faster in C and C++. Improving performance can be important because Go tends to run somewhat more slowly than C or C++,2 partly thanks to the overhead imposed by its many features. In most cases, however, this isn’t an issue: What’s important is that CoreDNS performs well enough to handle the workload you offer it, and in the vast majority of cases, it does, Go or no Go. Probably the most significant capability CoreDNS offers, though, is its ability to com‐ municate with container infrastructure and orchestration systems such as etcd and Kubernetes. Who Needs CoreDNS? The short answer: basically anyone running Kubernetes, and most folks running con‐ tainerized applications. The function CoreDNS fulfills in a containerized environment is that of a service directory, which we talk about in detail in this book. A service directory helps con‐ tainers determine the IP address or IP addresses where the containers that offer a particular service are running. For example, a container might look up a domain name that represents the database service for a specified application in order to retrieve some data. The service directory function is critical because, in the world of containers and microservices, applications are usually decomposed into many small services (hence, “microservices”!), and each service might be offered by several con‐ tainers, each running at a different IP address. But CoreDNS’s utility isn’t limited to containerized environments. CoreDNS’s plug- ins support advanced DNS functionality that even the big boys like BIND don’t sup‐ port. You can rewrite queries and responses on the fly, for example. You can automatically load zone data from GitHub or Amazon Route 53. And because Core‐ DNS itself is small and usually runs in a container, it’s suitable for use in scenarios in which a big DNS server such as BIND would not be. viii | Preface

Who This Book Is For This book is aimed at the following audiences: • Administrators of containerized environments that need DNS-based service dis‐ covery, particularly when those environments are managed by Kubernetes. • DNS administrators looking for a small, flexible DNS server that can run in a container. • DNS administrators looking for a DNS server that — Integrates with Route 53 — Supports flexible rewriting of queries and responses — Supports DNS over Transport Layer Security (TLS) and general-purpose Remote Procedure Call (gRPC) • Developers looking to implement custom DNS functionality by writing their own CoreDNS plug-ins. What You Will Learn Readers of this book will learn: • What distinguishes CoreDNS from other DNS servers • Basic DNS theory, including the DNS namespace, domain names, zones, resource records, recursion, caching, and forwarding • Basic CoreDNS configuration, including configuring common DNS servers such as primaries and secondaries and caching DNS servers • CoreDNS’s options for managing zone data, including advanced options such as loading from Git and Route 53 • How DNS-based service discovery works, and how to configure CoreDNS ser‐ vice discovery with etcd and Kubernetes • How to rewrite queries and responses • How to monitor and troubleshoot CoreDNS • How to build custom versions of CoreDNS and write new plug-ins Preface | ix

Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program ele‐ ments such as variable or function names, databases, data types, environment variables, statements, and keywords. This element signifies a general note. This element indicates a warning or caution. Using Code Examples This book is here to help you get your job done. In general, if example code is offered with this book, you may use it in your programs and documentation. You do not need to contact us for permission unless you’re reproducing a significant portion of the code. For example, writing a program that uses several chunks of code from this book does not require permission. Selling or distributing a CD-ROM of examples from O’Reilly books does require permission. Answering a question by citing this book and quoting example code does not require permission. Incorporating a signifi‐ cant amount of example code from this book into your product’s documentation does require permission. We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “Learning CoreDNS by John Belama‐ ric and Cricket Liu (O’Reilly). Copyright 2019 John Belamaric and Cricket Liu, 978-1-492-04796-4.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. x | Preface

O’Reilly Online Learning For almost 40 years, O’Reilly Media has provided technology and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, conferences, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in- depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, please visit http://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list errata, examples, and any additional information: https://oreil.ly/learning-coreDNS. To comment or ask technical questions about this book, please send an email to bookquestions@oreilly.com. For more information about our books, courses, conferences, and news, see our web‐ site at http://www.oreilly.com. Find us on Facebook: http://facebook.com/oreilly Follow us on Twitter: http://twitter.com/oreillymedia Watch us on YouTube: http://www.youtube.com/oreillymedia Preface | xi

Acknowledgments The authors would like to thank their able reviewers Miek Gieben, François Tur, and Michael Grosser for catching errors both subtle and egregious. They would also like to thank all the other members of the CoreDNS community for creating such an incredible product. John would like to thank his amazing wife, Robin, for her support, encouragement, and assistance. He couldn’t have done it without her. He also would like to acknowl‐ edge the support of his son, Owen, and daughter, Audrey, who have put up with all the nonsense that goes along with their dad writing a book. He gives thanks to Tim Hockin, Bowei Du, and the rest of the Kubernetes SIG-Network team for helping guide CoreDNS into Kubernetes, and to his former colleagues at Infoblox, particu‐ larly Chris O’Haver and Sandeep Rajan, who worked hard to make CoreDNS the right choice for Kubernetes. Finally, he would like to thank his former colleague Alan Conley, without whose support, CoreDNS would not be what it is today. Cricket would like to acknowledge his friends and colleagues at Infoblox, particularly his boss, Alan Conley. Without Alan’s regular harassment, this book would never have gotten off the ground. And he sends his love and thanks to Kristin, for her steadfast support; to his kids, Walt (née Walter B) and Greta (née Baby G), sources of amusement and amazement and no small amount of eye-rolling; and, finally, to Charlie and Jessie, who provided sisterly canine companionship through much of this project but, sadly, didn’t make it to see the end. xii | Preface

CHAPTER 1 Introduction This book is about CoreDNS, a new DNS server that’s been designed to work well with containers, such as Linux and Docker containers, and especially well in environ‐ ments managed by Kubernetes, the popular container orchestration system. This first chapter explains CoreDNS’s raison d'être, and how it differs from other DNS servers, including its limitations. The chapter also covers a little of the history of CoreDNS, such as its relationship to the Cloud Native Computing Foundation. What Is CoreDNS? CoreDNS is DNS server software that’s often used to support the service discovery function in containerized environments, particularly those managed by Kubernetes. Miek Gieben wrote the original version of CoreDNS in 2016. He’d previously written a DNS server called SkyDNS and a popular library of DNS functions in the Go lan‐ guage called Go DNS. Like its successor, CoreDNS, SkyDNS’s main purpose was to support service discovery. But Miek admired the architecture of a Go-based web server called Caddy, so he forked Caddy to create CoreDNS. CoreDNS thus inherited the major advantages of Caddy: its simple configuration syntax, its powerful plug-in- based architecture, and its foundation in Go. Compared to the syntax of, say, BIND’s configuration file, CoreDNS’s Corefile, as it’s called, is refreshingly simple. The Corefile for a basic CoreDNS-based DNS server is often just a few lines long and—relatively speaking—easy to read. CoreDNS uses plug-ins to provide DNS functionality. So there’s a plug-in for caching and a plug-in for forwarding, a plug-in for configuring a primary DNS server that reads zone data from a file and a plug-in for configuring a secondary DNS server. Not only is configuring each plug-in straightforward (see the previous paragraph), but if 1

you don’t need a plug-in, you don’t configure it and its code isn’t executed. That makes CoreDNS faster and more secure. Plug-ins are also fairly easy to develop. That’s important for two reasons. First, if you want to extend CoreDNS’s functionality, you can write your own plug-in; we cover that in Chapter 9. Second, because writing new plug-ins isn’t rocket science, many have been developed, and more are being written all the time. You might find one that provides functionality you need. The Go language is “memory-safe,” which means that it’s protected from “memory access errors” such as buffer overflows and dangling pointers. That’s particularly important for a DNS server such as CoreDNS, which anyone on the internet could conceivably access. A malicious actor might exploit a buffer overflow to crash a DNS server or even to gain control of the underlying operating system (OS). In fact, over the decades of its history, a substantial number of the serious vulnerabilities in BIND have been caused by memory access errors. With CoreDNS, you don’t need to worry about those. Probably the most significant advantage CoreDNS offers, though, is its ability to communicate with container infrastructure and orchestration systems such as etcd and Kubernetes. We discuss this in much more detail later in the book, but let’s take a quick look at this functionality here. CoreDNS, Containers, and Microservices If you’re in the tiny subset of humanity to whom this book appeals, you’ve probably heard of containers. If you haven’t, think of a container as a very lightweight, efficient virtual machine (VM). Whereas VMs can share a single hardware platform, courtesy of a hypervisor, containers provide execution environments that run under the same OS kernel but provide a similar level of isolation as VMs. Containers are much smaller than VMs and can be started and stopped much more quickly. Containers are often used in software based on a microservices architecture. With microservices, an application, often a complex one, is decomposed into many micro‐ services. Each microservice is responsible for providing a small but useful and clearly defined set of functionality. For example, one microservice might handle authentica‐ tion of users, whereas another manages authorization of those users. An application, in total, might comprise dozens or hundreds of microservices, communicating with one another over a network. In practice, each microservice might be provided by one or more containers. The authentication service, for example, might be implemented as a container. It’s so quick and easy to start and stop containers that the application—or a higher-level con‐ tainer orchestrator—might start and stop additional authentication containers dynam‐ ically as demand for authentication waxes and wanes. 2 | Chapter 1: Introduction

In such an environment, though, tracking where a particular service is running can be challenging. Say a container supporting the database service needs to communi‐ cate with the authorization service to determine whether a given user should be allowed to conduct a particular search. If the containers that implement the authori‐ zation service are being started and stopped dynamically to accommodate load, how do we get a list of all running authorization containers? The answer is most often DNS, the Domain Name System. Since the communications between containers is almost always based on IP, the Internet Protocol, and because developers have been using DNS to find the IP addresses of resources for literally decades, using DNS to identify containers that offer a given service is natural. It’s in this capacity that CoreDNS really shines. Not only is CoreDNS a flexible, secure DNS server, but it integrates directly with many container orchestration systems, including Kubernetes. This means that it’s easy for the administrators of container‐ ized applications to set up a DNS server to mediate and facilitate communications between containers. CoreDNS Limitations CoreDNS does currently have some significant limitations, though, and it won’t be suitable for every conceivable DNS server. Chief among these is that CoreDNS, at least in the latest version as of this writing, doesn’t support full recursion. In other words, CoreDNS can’t process a query by starting at the root of a DNS namespace, querying a root DNS server and following referrals until it gets an answer from one of the authoritative DNS servers. Instead, it relies on other DNS servers—usually called forwarders—for that. In Chapter 2, we talk more about recursion and forwarders. If you’re still on the fence about whether CoreDNS is the right choice for your partic‐ ular needs, Table 1-1 might help; it summarizes the key differences between CoreDNS’s functionality and BIND’s. Table 1-1. Key functional differences between CoreDNS and BIND CoreDNS BIND Full recursion No Yes Dynamic updates No Yes Integration with Kubernetes Yes No Integration with Amazon Route 53 Yes No Domain Name System Security Extensions (DNSSEC) support Limited Full Support for DNS over Transport Layer Security (DoT) Yes No What Is CoreDNS? | 3

If you’re unsure about what some of these terms mean, don’t worry, we cover them later in the book. Before we do, though, let’s talk briefly about the formal relationship between CoreDNS, Kubernetes, and something called the Cloud Native Computing Foundation. CoreDNS, Kubernetes, and the Cloud Native Computing Foundation Kubernetes, the container orchestration system with which CoreDNS integrates so nicely, was originally written at Google and then converted to an open source project in 2015. To manage the newly open sourced Kubernetes, Google partnered with The Linux Foundation to create the Cloud Native Computing Foundation, or CNCF for short. The CNCF has become the home for many technologies important to building cloud- based applications, including Prometheus, which supports collecting metrics and alerting, and Envoy, a service proxy. Projects managed by the CNCF move through various “maturity levels,” from “sandbox,” for early-stage projects; to “incubating,” for projects gaining acceptance; to “graduated,” for mature projects suitable for broad adoption. CoreDNS was submitted to the CNCF in 2017 and moved to “graduated” status in January 2019. As testament to CoreDNS’s criticality to Kubernetes environments, CoreDNS became the default DNS server shipped with Kubernetes with Kubernetes version 1.13, which was released in December 2018. Given that CoreDNS is now installed with almost every new Kubernetes implementation, and Kubernetes is a jug‐ gernaut in the world of containers (and containers themselves seem to be taking the world by storm), we expect the installed base of CoreDNS to explode. Enough of singing CoreDNS’s praises. We’ve talked about what CoreDNS is good for and what it isn’t, and how it’s had its fate lashed to Kubernetes. Next, we give you a whirlwind refresher on DNS theory so that we can begin talking about how to config‐ ure CoreDNS to do useful work! 4 | Chapter 1: Introduction

CHAPTER 2 A DNS Refresher So far, we’ve talked about practical matters like what CoreDNS is, what’s it’s good at (vis-à-vis DNS functionality) and what it’s not good at. Of course, that discussion had to include some DNS terminology—terminology that, in fairness, not everyone is familiar with. We deliberated for a while over how much DNS theory to include in this book. We could, of course, “Begin at the beginning, and go on till... the end, then stop,” but that’s been done in other books, including books we’ve written. Still, it didn’t seem fair to send you out into the world without at least a grounding in DNS. Our compromise is to try to give you just enough DNS theory to get by, and then to point you in the direction of, for example, DNS and BIND if you’re interested in more detail. (Hopefully that doesn’t seem too self-serving.) What Is the Domain Name System? The DNS is a naming system that maps names to other data, such as IP addresses, mail routing information, and more. And DNS isn’t just any naming system: it’s the internet’s standard naming system as well as one of the largest distributed databases in the world. DNS is also a client–server system, with DNS clients querying DNS servers to retrieve data stored in that distributed database. Because the database is distributed, DNS servers will often need to query one or more other DNS servers to find a given piece of data. DNS clients are often called resolvers, whereas DNS servers are sometimes 5

1 We’ll refer to them as DNS servers in this book, though in other books we referred to them as name servers. People change! 2 And yes, George Foreman is the canonical counterexample of this. But George’s five sons named “George” all have name suffixes (II, III, etc.) and nicknames to help tell them apart. called name servers.1 Resolvers ask DNS servers for information about particular indexes into the distributed database. Domain Names and the Namespace Those indexes into DNS’s distributed database are called domain names. These are the dotted names that should be familiar to you from internet email addresses and URLs. In an email address, the domain name appears to the right of the “@” sign. In a URL, the domain name appears after the “://” and before the next “/,” if any. So in the email address cricket@foo.example, “foo.example” is the domain name. In the URL http:// www.bar.example/, “www.bar.example” is the domain name. These domain names actually represent nodes in DNS’s namespace. DNS’s namespace is an inverted tree, with the root node at the top. Each node can have an arbitrarily large number of child nodes, and is usually depicted with links between it and its chil‐ dren. Each node also has a label, which can be up to 63 ASCII characters long. The root node has a special label: the null label, which has zero length. Only the root node has the null label. Beyond that, there aren’t many restrictions on labels—mainly that the child nodes of a single node must all have different labels. That makes sense: It helps avoid ambiguity and confusion, just as giving your children unique first names does.2 Figure 2-1 shows a portion of a fictional DNS namespace to help illustrate these concepts. Figure 2-1. A (semi-)fictional DNS namespace 6 | Chapter 2: A DNS Refresher