Docker Cookbook (Sébastien Goasguen)（Z-Library）

Page 1

(This page has no text content)

Page 2

Sebastien Goasguen Boston Docker Cookbook

Page 3

(This page has no text content)

Page 4

Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii 1. Getting Started with Docker. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 1.1 Installing Docker on Ubuntu 14.04 and CentOS 6.5 15 1.2 Setting Up a Local Docker Host Using Vagrant 17 1.3 Using boot2docker to Get a Docker Host on OSX 18 1.4 Running Boot2docker on Windows 8.1 Desktop 22 1.5 Starting a Docker Host in the Cloud Using Docker Machine 24 1.6 Running Hello World in Docker 28 1.7 Running a Docker Container in Detached Mode 30 1.8 Creating, Starting, Stopping, Removing Containers. 31 1.9 Sharing Host Data With Containers 33 1.10 Sharing Data Between Containers 34 1.11 Copying Data To And From Containers 35 1.12 Managing and Configuring the Docker Daemon 36 1.13 Running a Wordpress Blog Using Two Linked Containers 37 1.14 Backing up a Database Running in a Container 40 1.15 Using Supervisor to Run Wordpress in a Single Container 41 2. Image Creation and Sharing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 2.1 Keeping Changes Made to a Container by Committing to an Image. 45 2.2 Saving Images and Containers as Tar Files for Sharing. 46 2.3 Writing your First Dockerfile 48 2.4 Packaging a Flask Application inside a container 50 2.5 Versioning an Image with Tags 53 2.6 Migrating From Vagrant to Docker With the Docker Provider 54 2.7 Using Packer to Create a Docker Image 56 2.8 Publishing your image to Docker hub 60 iii

Page 5

2.9 Running a Private Registry 64 2.10 Setting Up an Automated Build on DockerHub for Continous Integration/Deployment 66 2.11 Setting up a Local Automated Build Using a Git Hook and a Private Registry 71 3. Docker Networking. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 3.1 Introducing Docker Containers Networking 73 3.2 Choosing a Container Networking Stack 76 3.3 Configuring the Docker Daemon IP tables and IP forwarding settings 78 3.4 Linking Containers in Docker 80 3.5 Using Pipework to Understand Container Networking 80 3.6 Setting up a Custom Bridge for Docker 85 3.7 Using OVS with Docker 86 3.8 Building a GRE Tunnel Between Docker Hosts 88 3.9 Networking Containers on Multiple Hosts with Docker Network 91 3.10 Diving Deeper Into The Docker Network Namespaces Configuration 95 3.11 Running Containers on a Weave Network 96 3.12 Running a Weave Network on AWS 97 3.13 Deploying flannel Overlay Between Docker Hosts 99 3.14 Using an Ambassador Container to Expose Services 101 4. Docker Configuration and Development. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 4.1 Compiling Your Own Docker Binary From Source 103 4.2 Running the Docker Test Suite for Docker Development 105 4.3 Replacing Your Current Docker Binary With a New One 106 4.4 Using nsenter 107 4.5 Introducing libcontainer 110 4.6 Using nsinit 110 4.7 Switching Execution Environment 110 4.8 Accessing the Docker Daemon Remotely 110 4.9 Exploring the Docker remote API to automate Docker tasks. 112 4.10 Securing the Docker Deamon for Remote Access 114 4.11 Using docker-py to Access the Docker Daemon Remotely 116 4.12 Using docker-py Securely 118 5. Kubernetes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 5.1 Understanding Kubernetes Architecture 123 5.2 Networking Pods for Container Connectivity 126 5.3 Using Labels for Container Placement and Application Management 129 5.4 Creating a Multi-node Kubernetes Cluster With Vagrant 129 5.5 Starting Containers on a Kubernetes Cluster with Pods 132 iv | Table of Contents

Page 6

5.6 Taking Advantage of Labels For Querying Kubernetes Objects 134 5.7 Using a Replication Controller to Manage the Number of Replicas of a Pod 135 5.8 Running Multiple Containers in a Pod 137 5.9 Using Service Proxies For Dynamic Linking of Containers 140 5.10 Defining Volumes in Pods 142 5.11 Creating a Single Node Kubernetes Cluster Using Docker Compose 143 5.12 Compiling Kubernetes to Create Your Own Release 146 5.13 Starting Kubernetes Components with hyperkube Binary 149 5.14 Exploring the Kubernetes API 150 5.15 Running the Kubernetes Dashboard 154 5.16 Switching to a New API Version 156 5.17 Configuring Authentication to a Kubernetes Cluster 158 5.18 Configuring the Kubernetes Client to Access Remote Clusters 159 6. Just Enough Operating System for Docker. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 6.1 Discovering the CoreOS Linux Distribution with Vagrant 161 6.2 Starting a Container on CoreOS via Cloud-init 164 6.3 Starting a CoreOS Cluster via Vagrant to Run Containers on Multiple Hosts 166 6.4 Using Fleet to Start Containers on a CoreOS Cluster 169 6.5 Deploying a Flannel Overlay Between CoreOS Instances 171 6.6 Running Docker Containers on RancherOS 174 6.7 Using Project Atomic to run Docker Containers 175 6.8 Starting and Atomic Instance on AWS to use Docker 176 6.9 Running Docker on Ubuntu Core Snappy in a Snap 177 6.10 Starting an Ubuntu Core Snappy Instance on AWS EC2 179 7. The Docker Ecosystem: Tools. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 7.1 Using Docker compose to Create a Wordpress Site 185 7.2 Using Docker compose to test Apache Mesos and Marathon on Docker 188 7.3 Looking at Docker Compose as a Replacement to Fig 190 7.4 Starting Containers on a Cluster with Docker Swarm 193 7.5 Using Docker Machine to Create a Swarm Cluster Across Cloud Providers 196 7.6 Managing Containers through Docker UI 198 7.7 Orchestrating Containers with Ansible Docker Module 200 7.8 Using Clocker 204 7.9 Using Deis 204 7.10 Using Rancher to Manage Containers on a Cluster of Docker Hosts 204 7.11 Running Containers Via Apache Mesos and Marathon 208 7.12 Using the Mesos Docker Containerizer on a Mesos Cluster 213 7.13 Discovering Docker Services with Registrator 215 Table of Contents | v

Page 7

8. Docker in the Cloud. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219 8.1 Accessing Public Clouds to Run Docker 220 8.2 Starting a Docker Host on AWS EC2 223 8.3 Starting a Docker Host on Google GCE 226 8.4 Starting a Docker Host on Microsoft Azure 229 8.5 Starting a Docker Host on Azure with Docker Machine 231 8.6 Running Cloud Providers CLI in Docker Containers 233 8.7 Using Google Container Registry to Store your Docker Images 235 8.8 Using Docker in GCE Google-Container Instances 237 8.9 Starting a Docker Host on AWS Using Docker Machine 240 8.10 Using Kubernetes in the Cloud via Google Container Engine 242 8.11 Managing Google Container Engine Resources Using kubecfg 244 8.12 Getting Setup to Use the EC2 Container Service 246 8.13 Creating a ECS Cluster 249 8.14 Starting Docker Containers on a ECS Cluster 252 8.15 Starting an Application in the Cloud Using Docker Support in AWS Beanstalk 256 8.16 Using AWS Elastic Container Service as a Beanstalk Environment 260 9. Monitoring containers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261 9.1 Getting Detailed Information About a Container With docker inspect 261 9.2 Obtaining Usage Statistics of a Running Container 263 9.3 Listening to Docker Events on Your Docker Hosts 264 9.4 Getting The Logs of a Container With docker logs 266 9.5 Using Logspout to Collect Container Logs 267 9.6 Managing logspout Routes to Store Container Logs 269 9.7 Using Elasticsearch and Kibana to Store and Visualize Container Logs 271 9.8 Using Collectd to Visualize Container Metrics 272 9.9 Accessing Container Logs Through Mounted Volumes 278 9.10 Using cAdvisor to Monitor Resource Usage in Containers 278 9.11 Monitoring Container Metrics With InfluxDB, Grafana and cAdvisor 280 9.12 Gaining Visibility Into Your Containers Layout with Weavescope 281 9.13 Monitoring a Kubernetes Cluster with Heapster 283 vi | Table of Contents

Page 8

Preface This book is not finished, it has not gone through technical review nor has it been edited for grammar, punctuation, typos etc. You are reading this book through the advanced release program of O’Reilly, please consider it a preview of a draft and be kind in your reviews. Feel free to send me any suggestions or comments to how2dock@gmail.com. You can also file a review in the O’Reilly portal. Happy reading. Why I Wrote This Book I have been working on Clouds at the IaaS layer for over ten years. With Amazon AWS, Google GCE and Microsoft Azure now providing large scale Cloud services for several years, it is fair to say that getting access to a server has never been that easy and that quick. The real value to me has been the availability of an API to access these services. We can now program to create an infrastructure and program to deploy an application. These programmable layers help us reach a higher level of automation, which for a business translates in faster time to market, more innovation and better user service. However, application packaging, configuration, composition of services in a dis‐ tributed environment has not progressed much despite a lot of work in configuration management and orchestration. Deploying and running a distributed application at scale and in a fault tolerant manner is still hard. I was not crazy about Docker until I tried it and understood what it brings to the table. Docker primarily brings a new user experience to linux containers. It is not about full virtualization versus containers, it is about the ease of packaging and run‐ ning an application. Once you start using Docker and enjoy this new experience, the side effect is that you will also start thinking automatically about composition and clustering. vii

Page 9

Containers help us think more in terms of functional isolation which in turn forces us to decompose our applications before stitching it back together for a distributed world. Yes, I have drunk the Kool-Aid but hopefully this book will show you why. How This Book Is Organized This cookbook is currently made of ten chapters. Each chapter is composed of recipes written in the standard O’Reilly recipes format (i.e Problem, Solution, Discussion). You can read it front to back or pick up a specific chapter/recipe. Each recipe is made to be independent of the others but when concepts needed in a recipe are needed, appropriate references are provided. • The Chapter 1 chapter goes through several Docker installation scenarios includ‐ ing Docker machine. It then presents the basic Docker commands to manage con‐ tainers, mount data volumes, link containers and so on. At the end of this chapter you should have a working Docker host and you should have started multiple containers as well as understood the lifecycle of containers. • In Chapter 2 we introduce the Dockerfile, Docker Hub and show how to build/tag/commit an image. We also show how to run your own Docker registry and setup automated builds. At the end of this chapter you will know how to cre‐ ate Docker images and share them privately or publicly and have some basic foundation to build continuous delivery pipelines. • Currently the Chapter 3 chapter only contains stubs of planned recipes. You will find information such as linking containers, using an ambassador container to expose services from different hosts, configuring a custom bridge for use in your Docker host. You will also learn about more advanced networking setups and tools, such as Weave, Flannel and Socketplane VXLAN overlays. Empty chapter right now. • The Chapter 4 chapter goes through some configuration of the Docker daemon, especially security settings and access to the Docker API remotely. It also covers a few basic problems, like compiling Docker from source, running its test suite and using a new Docker binary. A few recipes are meant to gain a better insight on linux namespaces and its use in containers. • Chapter 5 introduces the new container management platform from Google. Kubernetes provides a way to deploy multi container applications on a dis‐ tributed cluster. In addition it provides an automated way to expose services and create replicas of containers. We show how to deploy Kubernetes on your own infrastructure, starting with a local Vagrant cluster and subsequently on a set of machines started in the Cloud. We then present the key aspects of Kubernetes: pods, services and replication controllers. Empty chapter right now. viii | Preface

Page 10

• In the Chapter 6 chapter we cover three new linux distributions that are custom‐ ized to run containers: CoreOS, Project Atomic and Ubuntu core. These new dis‐ tributions provide just enough operating system to run and orchestrate docker containers. Recipes cover installation and access to machines that use these dis‐ tributions. We also introduce tools that are used with these distributions to ease container orchestration (e.g etcd, fleet, systemd) • One of Docker’s strength is its booming ecosystem. In Chapter 7 we introduce several tools that have been created over the last 18 months and that leverage Docker to ease application deployment, continuous integration, service discovery and orchestration. As an example, you will find recipes about Fig, Docker Swarm, Flynn and Apache Clocker. • The Docker daemon can be installed on a developer local machine, however, with Cloud computing providing easy access to on-demand servers it is fair to say that a lot of container based applications will be deployed in the Cloud. In Chapter 8 we present a few recipes to show how to access a Docker host on Ama‐ zon AWS, Google GCE and Microsoft Azure. We also introduce two new cloud services that use Docker: The AWS Elastic Container Service (ECS) and the Goo‐ gle Container Engine. This chapter is currently stubbed out. Empty chapter right now. • The Chapter 9 chapters aims to address some concerns about application moni‐ toring when using containers. Monitoring and visibility of the infrastructure and the application has been a huge focus in the DevOps community. As Docker becomes more pervasive as a development and operational mechanism, lessons learned need to be applied to container based applications. This chapter is cur‐ rently stubbed out. Empty chapter right now. • Finally, in the chapter we present end to end application deployment scenar‐ ios on both single host and clusters. While some basic application deployments are presented in earlier chapters. The recipes presented here aim to be closer to a production deployment setup. This is a more in depth chapter that puts the reader on the path towards designing more complex microservices. This chapter is currently stubbed out. Empty chapter right now. The book structure may still change and the order of the chapters may be modified. Finally, summarizes the book and provides some tips for further reading and investigation. Preface | ix

Page 11

Technology You Need to Understand This book is of an intermediate level and requires a minimum understanding of a few development and system administration concepts. Before diving into the book, you might want to review: Bash (Unix shell) This is the default Unix shell on Linux and OS X. Familiarity with the Unix shell, such as editing files, setting file permissions, moving files around the filesystems, user privileges, and some basic shell programming will be very beneficial. If you don’t know the Linux shell in general, consult books such as Cameron Newham’s Learning the Bash Shell or Carl Albing, JP Vossen, and Cameron Newham’s Bash Cookbook, both from O’Reilly. Package management The tools we will present in this book often have multiple dependencies that need to be met by installing some packages. Knowledge of the package management on your machine is therefore required. It could be apt on Ubuntu/Debian sys‐ tems, yum on CentOS/RHEL systems, port or brew on OS X. Whatever it is, make sure that you know how to install, upgrade, and remove packages. Git Git has established itself as the standard for distributed version control. If you are already familiar with CVS and SVN, but have not yet used Git, you should. Ver‐ sion Control with Git by Jon Loeliger and Matthew McCullough (O’Reilly) is a good start. Together with Git, the GitHub website is a great resource to get started with a hosted repository of your own. To learn GitHub, try http://train ing.github.com and the associated interactive tutorial. Python In addition to programming with C/C++ or Java, I always encourage students to pick up a scripting language of their choice. Perl used to rule the world, while these days, Ruby and Go seem to be prevalent. I personally use Python. Most examples in this book use Python but there are a few examples with Ruby, one even uses Clojure. O’Reilly offers an extensive collection of books on Python, including Introducing Python by Bill Lubanovic, Programming Python by Mark Lutz, and Python Cookbook by David Beazley and Brian K. Jones. Vagrant Vagrant has become one of the great tools for DevOps engineer to build and manage their virtual environments. It is best suited for testing and quickly provi‐ sioning virtual machines locally, but also has several plugins to connect to public cloud providers. In this book we use Vagrant to quickly deploy a virtual machine x | Preface

Page 12

instance that acts as a docker host. You might want to read Vagrant, Up and Run‐ ning from the author of Vagrant itself, Mitchell Hashimoto. Go Docker is written in Go. Over the last couple years go has established itself has the new programming language of choice in many startups. Docker is written in go, and while this cookbook is not about go programming, we will show how to compile a few go projects. If you want to know more, Go Up and Running is a good start. Online Content Code examples, Vagrantfile and other scripts used in this book are available at Git‐ Hub. You can clone this repository, go to the relevant chapter and recipe and use the code as is. For example to start an Ubuntu 14.04 virtual machine using Vagrant and install Docker do: $ git clone https://github.com/how2dock/docbook.git $ cd dockbook/ch01/ubuntu14.04/ $ vagrant up The examples in this repo are not made to represent optimized set‐ ups. There are the basic minimum required to run the examples in the recipes. Until the book is complete, expect frequent changes to this repo. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program ele‐ ments such as variable or function names, databases, data types, environment variables, statements, and keywords. Constant width bold Shows commands or other text that should be typed literally by the user. Constant width italic Shows text that should be replaced with user-supplied values or by values deter‐ mined by context. Preface | xi

Page 13

This element signifies a tip, suggestion, or general note. This element indicates a warning or caution. Safari® Books Online Safari Books Online is an on-demand digital library that deliv‐ ers expert content in both book and video form from the world’s leading authors in technology and business. Technology professionals, software developers, web designers, and business and crea‐ tive professionals use Safari Books Online as their primary resource for research, problem solving, learning, and certification training. Safari Books Online offers a range of plans and pricing for enterprise, government, education, and individuals. Members have access to thousands of books, training videos, and prepublication manuscripts in one fully searchable database from publishers like O’Reilly Media, Prentice Hall Professional, Addison-Wesley Professional, Microsoft Press, Sams, Que, Peachpit Press, Focal Press, Cisco Press, John Wiley & Sons, Syngress, Morgan Kauf‐ mann, IBM Redbooks, Packt, Adobe Press, FT Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett, Course Technology, and hundreds more. For more information about Safari Books Online, please visit us online. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at http://shop.oreilly.com/product/ 0636920034377.do. xii | Preface

Page 14

To comment or ask technical questions about this book, send email to bookques‐ tions@oreilly.com. For more information about our books, courses, conferences, and news, see our web‐ site at http://www.oreilly.com. Find us on Facebook: http://facebook.com/oreilly Follow us on Twitter: http://twitter.com/oreillymedia Watch us on YouTube: http://www.youtube.com/oreillymedia Acknowledgements Preface | xiii

Page 15

(This page has no text content)

Page 16

CHAPTER 1 Getting Started with Docker This chapter consists of introductory recipes. Readers should be able to go through the recipes to install a Docker Host, then pro‐ gressively discover the Docker CLI to manage containers and start a two container application. You can send me suggestions at how2dock@gmail.com There will be an introduction here. 1.1 Installing Docker on Ubuntu 14.04 and CentOS 6.5 Problem You want to use Docker on Ubuntu 14.04 or CentOS 6.5 Solution Install the docker.io package using your package management system and start the docker service. On Ubuntu 14.04 this is achieved in two lines of bash commands. The actual package to install is docker.io since there is a pre-existing docker package for Ubuntu which is not related to Docker. $ sudo apt-get update $ sudo apt-get install -y docker.io You can test that the installation worked fine by checking the version of docker: $ sudo docker --version Docker version 1.0.1, build 990021a 15

Page 17

You can stop, start, restart the service. For example, to restart it do: $ sudo service docker.io restart On CentOS 6.5, getting docker is achieved by grabing the docker-io package from the EPEL repository. $ sudo yum -y update $ sudo yum -y install epel-release $ sudo yum -y install docker-io $ sudo service docker start $ chkconfig docker on While on Ubuntu 14.04 the setup installed version 1.0.1, on CentOS 6.5 it installed version 1.3.1 # docker --version Docker version 1.3.1, build c78088f/1.3.1 If you want to use docker from a non root user, add the user account to the docker group $ sudo gpasswd -a <user> docker Discussion To install the latest version of Docker on fedora, Ubuntu, Debian, Linux Mint and Gentoo, there exists a simple bootstrap script that you can run instead. $ sudo curl -sSL https://get.docker.com/ | sudo sh $ sudo docker --version Docker version 1.4.1, build 5bc2ff8 See Also For installation of Docker on other operating systems see the official installation doc‐ umentation By Publishing time, the default version number and latest version number of Docker might change. 16 | Chapter 1: Getting Started with Docker

Page 18

1.2 Setting Up a Local Docker Host Using Vagrant Problem The operating system of your local machine is different than the operating system you want to use Docker on. For example you are running OSX and want to try Docker on Ubuntu. Solution Use Vagrant to start a virtual machine (VM) locally and bootstrap the VM using a shell provisioner in the Vagrantfile. With a working Virtual Box and Vagrant installation, create a Vagrantfile: VAGRANTFILE_API_VERSION = "2" Vagrant.configure(VAGRANTFILE_API_VERSION) do |config| config.vm.box = "ubuntu/trusty64" config.vm.network "private_network", ip: "192.168.33.10" config.vm.provider "virtualbox" do |vb| vb.customize ["modifyvm", :id, "--memory", "1024"] end config.vm.provision :shell, :path => "docker-bootstrap.sh" end And create a docker_bootstrap.sh script: #!/bin/bash sudo apt-get update sudo apt-get -y install docker.io sudo gpasswd -a vagrant docker sudo service docker.io restart You can then bring up the virtual machine. Vagrant will download the ubuntu/ trusty64 box from the Vagrant cloud, start an instance of it using virtual box and run the bootstrap script in the instance. You will then be able to ssh to the instance and use docker $ vagrant up $ vagrant ssh vagrant@vagrant-ubuntu-trusty-64:~$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1.2 Setting Up a Local Docker Host Using Vagrant | 17

Page 19

In this Vagrant setup, the vagrant user was added to the docker group. Hence docker commands can be issued even if you are not root. You can get these scripts from the how2dock repository in the ch01 directory. Discussion If you have never used Vagrant before, you will need to install it. The download page on the Vagrant website lists all major packages families. For example on Debian based systems grab the .deb package and install it like so: $ wget https://dl.bintray.com/mitchellh/vagrant/vagrant_1.6.5_x86_64.deb $ sudo dpkg -i vagrant_1.6.5_x86_64.deb $ sudo vagrant --version Vagrant 1.6.5 1.3 Using boot2docker to Get a Docker Host on OSX Problem The Docker daemon is not supported on OSX, but you want to use the Docker client seamlessly on your OSX host. Solution Use the boot2docker lightweight linux distribution. Boot2docker is based on Tiny Core Linux and configured specifically to act as a docker host. After installation, a boot2docker command will be available to you. You will use it to interact with a vir‐ tual machine started through virtual box that will act as a docker host. The docker client -which runs on OSX, unlike the daemon- will be setup on your local OSX machine. Let’s start by downloading and installing boot2docker. Go to the site where you will find several download links. From the release page, grab the latest release. Once the download is finished, launch the installer. 18 | Chapter 1: Getting Started with Docker

Page 20

Figure 1-1. Boot2docker installer wizard Once the installation is finished you are ready to use boot2docker. 1.3 Using boot2docker to Get a Docker Host on OSX | 19