Everyday Go - The Fast Track for Golang (Alex Ellis) (z-library.sk, 1lib.sk, z-lib.sk)

EVERYDAY GO BY ALEX ELLIS Learn tools, techniques and patterns from real tools used in production. YOUR FAST TRACK FOR GOLANG

Everyday Golang - The Fast Track Alex Ellis 2021

Contents 1 Introduction 3 How to use the book . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Help the book to evolve . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Changelog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Why Go? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Getting started with Go . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 A note for Windows users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Direct installation on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Packages and paths . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Vendoring and Go modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 2 Create your first Go program 10 Add an external dependency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Make use of flags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Define a separate package or library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 3 Cross-compile for different operating systems 19 Go build arguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 CGO and portability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 4 Fetch JSON from a HTTP endpoint 21 Get started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Learn to parse JSON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Retrieve the JSON via HTTP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Extracting additional data from the JSON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 5 Five ways to write an awesome CLI 28 Pick Go . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Parse flags & arguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 Automate everything . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Integrate with package managers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Accept contributions and gather feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Now, Go and write your CLI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 6 Writing unit-tests in Golang 33 Testing opinions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Testing the Sum method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Adding a test table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 Party tricks for “go test” . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 Statement coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 Generating an HTML coverage report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 Benchmarking your code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 Go doesn’t ship your tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 Stress testing your tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 Running tests in parallel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 1

Compile the tests to a binary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 Isolating dependencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Example of isolating dependencies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 Unit-testing a HTTP server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 7 Work concurrently work with Goroutines 53 A simple example with Goroutines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 Avoid a common mistake with Goroutines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 Error groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 Sharing data between Goroutines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 Using channels for synchronisation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 Using a context to handle timeouts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 Limiting concurrent work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 8 Store data with a database 70 9 Write your own config files in YAML 75 A practical example of YAML in action . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 Read and write your own YAML schema . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 Merging objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 10 Inject version information into your binaries 80 Capture useful variable(s) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80 Prepare your code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 Do it with Docker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 Examples in production . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 11 Embed data in your application 85 12 Create dynamic content with templates 87 Advanced templates: ranges and custom functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 Real-world templates - arkade . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 13 Write a HTTP server 93 Reading the request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 Advanced route handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 Writing HTTP middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 14 Add Prometheus metrics to your microservices 99 Tracking HTTP metrics for our microservice . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104 Adding custom metrics to your Prometheus data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 15 Building and releasing Go binaries 109 The release-it example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 Releasing your binaries with a GitHub Action . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110 16 Releasing a Docker container image 112 Going multi-arch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 Shipping a Docker container from GitHub Actions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 17 Your onward journey 116 Giving feedback . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 Contributing to Open Source . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 My other content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 2

Chapter 1 Introduction “Everyday Go” is the fast way to learn tools, techniques and patterns from real tools used in production. About the author: Alex is a software engineer and an avid technical writer. There’s nothing he likes more than learning a new technology, and then writing it up in a clear, practical way. He’s well known for his blog posts on containers, Kubernetes, cloud computing and Go. He founded the OpenFaaS project in 2016, and since then has gone on to publish training courses, eBooks and dozens of other tools for developers. How to use the book This book is a compilation of practical examples, lessons and techniques for Go developers. They are the kind of things that you may need to learn and apply in your everyday journey. The topics cover the software lifecycle from learning the fundamentals, to software testing, to distribution and monitoring. If you feel like you may have seen some of these techniques before, you would be right. I’ve been writing about Go since 2015 and wanted to take everything that I’ve needed on a regular basis, and to lay it out for you to benefit from. It’s not a reference book or a complete guide to the language, but if you like learning by example and aren’t afraid to get your feet wet, then this style is probably for you. The chapter “Writing unit-tests in Go” was originally part of my top ranking and performing blog post. It was even featured in the Kubernetes documentation to help new Go developers get up to speed. In Everyday Go, I’ve rewritten, updated and extended the tutorial with more content and techniques that you can use in your work. You could say that this is the fast track to get the results you’ll need in your everyday writing of applications in Go. • How to use the book • Why Go? • Getting started with Go • Managing packages and paths • Understand Go modules and vendoring • Creating your first program in Go • Adding external dependencies to your program • Cross-compile your code for different Operating Systems • Inject version information into your binaries • Merge Go objects together into one • Deal with HTTP and JSON parsing • Learn how to use concurrency with Goroutines, channels, Mutexes, error groups and WaitGroups • Avoid a common mistake with Goroutines • Write unit tests and understand everything that’s available to you • Learn party tricks of the testing in Go • Isolate dependencies and unit test a HTTP server • Embed data and files into your application • Create dynamic content with templates 3

• Build a CLI that you’ll enjoy using • Load and write to config files in YAML • Write your own HTTP microservices • Integrate with Prometheus for metrics • Learn to build and release Go binaries • Build Docker container images • Release your code on GitHub • Get set for your onward journey Help the book to evolve Just as my knowledge of Go has evolved over time, this book will also evolve with updates and new examples. This book should be a place that you can come to for a pattern or sample that you can use as a basis of a new program or feature. Check the final chapter for how to submit comments, suggestions or corrections. Changelog • 5 August 2021 - Clarify usage of RWMutex and locking behaviour • 19 July 2021 - Fix typo on X509 certificates. • 14 July 2021 - Correct git clone URL, explain Marshal and Unmarshal, remove an import. • 28 June 2021 - Update wording on concurrency vs parallelism, introduce new locking example and Mu- tex/RWMutex. • 27 June 2021 - Update to Go modules section and clarification of paths in package cache directory • 24 June 2021 - MIT license added for example apps • 24 June 2021 - Add section on multiple packages and new example app “multiple-packages” • 23 June 2021 - Add new section on Go modules usage and vendoring • 22 June 2021 - Update Go release date in intro, add “merge-objects” example app Why Go? Figure 1.1: Go’s new logo According to Wikipedia, Go (or Golang) is a programming language that was developed by a team at Google and first appeared in 2009. It’s synonymous with server-side development and tooling for infrastructure projects. “Go is an open source programming language that enables the production of simple, efficient and reliable software at scale” 4

• Go branding guidelines Here are some projects that come to mind, which are entirely written in Go: • Docker - a container engine written by Docker Inc • Kubernetes - a clustering system for containers built by Google and open source contributors • Traefik - a load-balancer built by Traefik Labs • Consul, Terraform - tools for managing infrastructure built by Hashicorp • Caddy - a high performance HTTP server written by Matt Holt • OpenFaaS - serverless framework for hosting your own functions, built by OpenFaaS Ltd and open source con- tributors These are just a few of hundreds of thousands of projects which have appeared on GitHub’s trending page for Go. As you can see from the above, Go is very popular for server-side projects, but it is also very popular forwriting Command Line Interfaces (CLIs) because the binaries can be statically linked. Statically-linked Go binaries are both fast and small, and do not require an additional runtime making them a very popular choice. My personal highlights for Go are: • portability - static compilation for easy distribution • cross-compilation - being able to build binaries for Linux, Windows, MacOS Operating Systems (OS) and Arm computers with a single command (go build) • concurrency model - Go has built-in mechanisms for concurrency that make it well suited to tasks both small and large • ecosystem - knowing Go means being able to contribute to almost any other open source Go project or product • tooling - Go includes its own unit-testing runner and profiler, both of which are simple and easy to use • opinions - Go has strong opinions on formatting, style and typing that make it easy to move between codebases Above everything else, Go has an impressive set of packages available within its standard libraries or “stdlib”. There are many useful packages and utilities that would need to be installed separately in a programming language like JavaScript, Ruby or Java. • crypto - libraries for X.509 certificates and cryptography • compress - work with zip files and archives • http - a very powerful and simple HTTP client and server package including things like reverse proxies • net - work directly with sockets, URLS and DNS • encoding/json - work directly with JSON files • text/template - a powerful templating engine for replacing tokens in files, and generating text and HTML • os - work with low level OS primitives You can find a complete package list here. Through the next few chapters we will explore some of the above highlights and also solve various practical problems. Even if you’re a seasoned Go developer, I’d recommend going through each of the exercises, because even after 5 years I still learn something new each week. Getting started with Go There are three ways to install Go. • Install Go using a system package - such as an msi (Windows) or pkg (MacOS) • Install Go using a package manager - such as apt-get (Debian), pacman (Arch Linux), brew (MacOS) • Download a tar archive and unpack it to a set location (Linux only) If you’re a new user and running on Windows or MacOS, then go ahead with the first option and download Go from the homepage. The version changes frequently, so download the latest stable version. If you like to use package managers, you should be aware that the version they offer often lags behind. You may be installing a very old version of Go, and some features may not be available. 5

A note for Windows users Whilst the Go commands can be run cmd.exe and PowerShell, some of the example commands expect a UNIX-style shell which is present on MacOS and Linux hosts. For ease of use, it’s recommended that you try the content using Windows Subsystem for Linux (WSL), if it is available for your system. Or if not, make use of Git Bash. Git bash provides a terminal where you can run bash, have access to the git command and various other UNIX-style commands such as export, ls, mkdir and curl. Direct installation on Linux As a Linux user, I often need to install Go on a server, within a Docker image, or on my Raspberry Pi. I use the third approach of downloading and unpacking a tar for this. For a PC: curl -fLS https://golang.org/dl/go1.16.linux-amd64.tar.gz \ -o /tmp/go.tgz For a Raspberry Pi: curl -fLS https://golang.org/dl/go1.16.linux-armv6l.tar.gz \ -o /tmp/go.tgz Then on either platform, unpack the archive: sudo mkdir -p /usr/local/go/ sudo tar -xvf /tmp/go.tgz --strip-components=1 -C /usr/local/go/ After the installation, you will need to set your PATH and GOPATH variables in your bash file, so that they load every time you log in. echo "export PATH=\$PATH:/usr/local/go/bin/" | tee -a ~/.bash_profile echo "export GOPATH=\$HOME/go" | tee -a ~/.bash_profile Now disconnect from your server or Raspberry Pi, or close the terminal and connect again. pi@raspberrypi:~$ go version go version go1.16 linux/arm Packages and paths Go is usually installed to /usr/local/go/, with the bin folder containing the main binary and any other built-in com- mands like gofmt. You should also make yourself familiar with the GOPATH variable. Whenever packages are fetched, they will usually be downloaded there. In the section on Go modules, we will clarify its use further. Whenever you run go install path/to/project, you’ll find the binaries there also. Imagine your GOPATH is $HOME/go/ • GOPATH/pkg/ - Go modules downloaded as dependencies • GOPATH/bin/ - any CLIs that you build or install via go get • GOPATH/src/ - the default location for source code Here is an example of how to download a HTTP server called hash-browns that I wrote to generate hashes based upon a HTTP POST body. 6

$ export GO111MODULE=on $ go get github.com/alexellis/hash-browns go: downloading github.com/alexellis/hash-browns v0.0.0-20200122201415-86d5dd6d7faa go: github.com/alexellis/hash-browns upgrade => v0.0.0-20200122201415-86d5dd6d7faa go: downloading github.com/prometheus/procfs v0.0.0-20180920065004-418d78d0b9a7 Note: For the rest of the book, you should assume that GO111MODULE is set to on and that Go modules are being used everywhere. The source for the specific package will be downloaded to: $GOPATH/pkg/mod/github.com/alexellis/hash-browns@v0.0.0-20200122201415-86d5dd6d7faa Note that the specific suffix after the @ could change, depending on the latest version of the package. WhenGoprograms are released with a v prefix, the import path will look more like v0.1.0 instead of v0.0.0- followed by a SHA. For example, the release for openfaas/faas-provider is prefixed with a v, therefore a “prettier” path is provided: $ go get github.com/openfaas/faas-provider And the path is set as: $GOPATH/pkg/mod/github.com/openfaas/faas-provider@v0.18.5 The other time that the more verbose path might be used, is if you are trying to reference a commit that isn’t attached to a release or a tag on GitHub. The dependencies for it will be downloaded as follows: $GOPATH/pkg/mod/github.com/prometheus/procfs@v0.0.0-20180920065004-418d78d0b9a7 Then it will be compiled and the binary will be created at: $GOPATH/bin/hash-browns You’ll see a similar technique used in project READMEs for tools written in Go, so you may want to add $GOPATH/bin/ to your PATH environment variable. Note: Whenever GO111MODULE is set to off, then the code is downloaded to $GOPATH/src/ instead of $GOPATH/pkg. Building without Go modules may be disabled in a future release of Go. Vendoring and Go modules In the early days of the Go ecosystem, the source code of any dependencies being used by your program was placed a vendor folder. The vendor folder would be committed alongside your application code, giving you a copy and back-up of anything required to build your application. In the previous example, hash-browns, the github.com/prometheus/procfs module was required, which would be cloned into the source tree as follows: $GOPATH/src/github.com/alexellis/hash-browns $GOPATH/src/github.com/alexellis/hash-browns/vendor/github.com/prometheus/procfs Various vendoring tools such as the now deprecated dep, were used to download and manage these dependencies. One file would specify which packages were required, and another “a lock file” would specify the versions. So at any time you could rm -rf the vendor folder and run something like dep ensure to get a fresh copy of all the various files. In Go 1.13 the concept of Go modules was introduced which largely mirrors vendoring, with one big change. The source for dependencies no longer needed to be cloned and committed to source control with your application. The transition for OpenFaaS’ 40+ projects was relatively smooth, with just a few projects causing headaches and needing to be deferred. 7

Another change with Go modules, was the ability to compile source code outside of the $GOPATH/src convention. For instance, you could have your source code at $HOME/dev/ instead of $HOME/go/src/ or wherever you’d set it. Whenever you create a new Go project, it’s best to run go mod init first. If your code falls within the conventional GOPATH structure, then it will have its name determined automatically, but if not, you can pass it as an argument. Compare the following: export GH_USERNAME=alexellis $ mkdir -p $GOPATH/src/github.com/$GH_USERNAME/hash-gen $ cd $GOPATH/src/github.com/$GH_USERNAME/hash-gen $ go mod init go: creating new go.mod: module github.com/alexellis/hash-gen $ cat go.mod module github.com/alexellis/hash-gen go 1.15 And when run outside of GOPATH: $ mkdir /tmp/hash-gen $ cd /tmp/hash-gen $ go mod init go: cannot determine module path for source directory /tmp/hash-gen (outside GOPATH, module path must be specified)↪ Example usage: 'go mod init example.com/m' to initialize a v0 or v1 module 'go mod init example.com/m/v2' to initialize a v2 module Run 'go help mod init' for more information. Try one of the following instead: $ go mod init hash-gen $ go mod init github.com/alexellis/hash-gen It’s also possible to use Go modules and vendoring at the same time: $ cd /tmp/ $ git clone https://github.com/alexellis/hash-browns $ cd hash-brown $ rm -rf vendor $ go mod vendor $ ls vendor/ Now see that the vendor folder has been created. You can ask Go to build using the vendor folder instead of its local packages cache at $GOPATH/pkg by adding -mod vendor to the go build command. $ go build -mod vendor So which approach should you use? Vendoring has the advantage of keeping all your source and dependencies in one place, it also doesn’t require anything 8

to be downloaded to execute a build. With some projects such as Kubernetes controllers this means saving time and bandwidth. If youhave any private dependencies, vendoring canwork out as the easiest option, especiallywhenbuilding inside a container or CI pipeline. Using modules without vendoring seems to be trendy, with many projects removing their vendor folder. It has the advantage ofmaking code reviews easier since there are far fewer files that change in a pull request or commit. Working with private Go modules can be a source of contention, so you have been warned. See also: Go modules reference docs 9

Chapter 2 Create your first Go program You can create a Go program from memory, but I find it more useful to use an IDE like Visual Studio Code. Visual Studio Code has a number of plugins that it can install to format your code, tidy it up and add any paths for modules or packages that you import. The automation is not as comprehensive as a paid IDE like Jetbrains’ GoLand, but is fast and free. Create a folder first-go then, save the following as main.go: package main func main() { } Now you have two ways to run the program. 1) Run go run main.go 2) Run go build and then run the resulting binary ./first-go Now print a console statement using fmt.Printf. This will import the fmt package at the top of the file, and VSCode will automatically run a number of Go plugins to update the main.go file for you. package main import ( "fmt" "os" ) func main() { name := os.Getenv("USER") fmt.Printf("Well done %s for having your first Go\n", name) } One of the programs run by VScode is gofmt and it’s an example of how Go’s opinionated design makes it easy to move between code-bases without learning new conventions. You can run it against a text file using gofmt -w -s main.go. Note the use of the os.Getenv call to look-up your username from the USER environment variable. Note for Windows users: try replacing “USER” with “USERNAME” Now, run the program again and you should see output as per above. 10

Figure 2.1: A screenshot of Visual Studio Code and our first-go app Add an external dependency Whilst the standard library is extensive, it’s likely that you will need to use an external dependency. Dependencies in Go are made available as source code that can be linked to your program, which differs from C# or Java where compiled assets (DLLs or JARs) are downloaded. For this reason, you’ll usually see that a Go package has a path to where it can be downloaded with git. You can use webhooks to receive events from third-party APIs such as GitHub.com or Stripe. Most of the time the webhooks are sent with a digest in the header, that you can use to verify that the message hasn’t been tampered with. HMAC uses a symmetric key that both sender/receiver share ahead of time. The sender will generate a hash when wanting to transmit amessage - this data is sent along with the payload. The recipient will then sign the payload with the shared key again. And if the hash matches then the payload is assumed to be from the sender. Quote from alexellis/hmac Let’s use my library to create a digest of our own message, and then show how to verify it after. To begin with, we should initialise a Go module for our application. The Go modules file go.mod lists external depen- dencies that we want our program to depend upon. export GH_USERNAME="alexellis" mkdir -p $GOPATH/src/github.com/$GH_USERNAME/validate-hmac/ go mod init go: creating new go.mod: module github.com/alexellis/validate-hmac Review the contents of go.mod: module github.com/alexellis/validate-hmac go 1.15 11

It shows the path or name of this module, along with the minimum Go version required. In the past all code needed to be placed with the $GOPATH i.e. $HOME/go/src/github.com/alexellis/validate-hmac, but with go modules, the go.mod file can make this explicit, instead of implicit (read from the path). Create main.go: package main import ( "fmt" "github.com/alexellis/hmac" ) func main() { input := []byte(`input message from API`) secret := []byte(`so secret`) digest := hmac.Sign(input, secret) fmt.Printf("Digest: %x\n", digest) } According to the Go docs, %x used in fmt.Printf() uses a base 16, with lower-case letters for a-f, and is a way that we can share the raw computed bytes in a human readable format. Now run the program: $ go run main.go go: finding module for package github.com/alexellis/hmac go: found github.com/alexellis/hmac in github.com/alexellis/hmac v0.0.0-20180624211220-5c52ab81c0de↪ Digest: 17074131772d763bc4a360a6e4cb1a5ad1a98764 The digest 17074131772d763bc4a360a6e4cb1a5ad1a98764 was printed which is a hash that can be sent with the original input. Any user with the secret can compute another hash and if they match, that user will know the message was from us. You’ll see that the Go module is resolved, then downloaded from GitHub, and updated in go.sum and go.mod: $ cat go.mod module github.com/alexellis/validate-hmac go 1.15 require github.com/alexellis/hmac v0.0.0-20180624211220-5c52ab81c0de // indirect $ cat go.sum github.com/alexellis/hmac v0.0.0-20180624211220-5c52ab81c0de h1:jiPEvtW8VT0KwJxRyjW2VAAvlssjj9SfecsQ3Vgv5tk=↪ github.com/alexellis/hmac v0.0.0-20180624211220-5c52ab81c0de/go.mod h1:uAbpy8G7sjNB4qYdY6ymf5OIQ+TLDPApBYiR0Vc3lhk=↪ The go.sum pins the specific version of the module for later use. Now let’s verify the digest we received using the library: 12

package main import ( "fmt" "github.com/alexellis/hmac" ) func main() { input := []byte(`input message from API`) secret := []byte(`so secret`) digest := hmac.Sign(input, secret) fmt.Printf("Digest: %x\n", digest) err := hmac.Validate(input, fmt.Sprintf("sha1=%x", digest), string(secret)) if err != nil { panic(err) } fmt.Printf("Digest validated.\n") } Run the program with go run main.go and you’ll see it pass as expected. Make use of flags Go has a built-in flag parsing library. We will explore an alternative library, Cobra, in a later chapter. I’ve used Cobra in several projects but even now, I always start with the simplest thing that will work. Then later, if it is warranted, I’ll move to a more complex, but powerful library than the standard library’s flags package. export GH_USERNAME="alexellis" mkdir -p $GOPATH/src/github.com/$GH_USERNAME/validate-hmac-flags/ go mod init go: creating new go.mod: module github.com/alexellis/validate-hmac-flags In our previous example we created hashes by hard-coding the input and secret, but that isn’t going to be useful for our users. Let’s use flags instead, so that we can run: go run main.go -message="my message" \ -secret="terces os" Adapt the solution to use flag.StringVar to define each flag, followed by flag.Parse: package main import ( "flag" "fmt" ) func main() { 13

var inputVar string var secretVar string flag.StringVar(&inputVar, "message", "", "message to create a digest from") flag.StringVar(&secretVar, "secret", "", "secret for the digest") flag.Parse() fmt.Printf("Computing hash for: %q\nSecret: %q\n", inputVar, secretVar) } This is a good time to let you know about the %q variable, which adds quotes around any string we print. I use it to make sure there are no extra lines or whitespace within a value. $ go run main.go -message="my message" -secret="terces os" Computing hash for: "my message" Secret: "terces os" Now let’s take the value from the flag, and pass it into the hash generation code. digest := hmac.Sign([]byte(inputVar), []byte(secretVar)) fmt.Printf("Digest: %x\n", digest) Run the whole program and see the digest generated: $ go run main.go -message="my message" -secret="terces os" Computing hash for: "my message" Secret: "terces os" Digest: 2aefe2f570c0e61bc346b9fd3b6fbb3107ebfdd9 Try the same message with a different secret: $ go run main.go -message="my message" -secret="so secret" Computing hash for: "my message" Secret: "terces os" Digest: d82655d25497765dc774e8e9cde8795858cb9f8e Now, what if no secret is given as a flag value? How are you going to validate that? if len(secretVar) == 0 { panic("--secret is required") } Try adding the above, which looks at the length of the string: $ go run main.go -message="my message" panic: --secret is required goroutine 1 [running]: main.main() /home/alex/go/src/github.com/alexellis/validate-hmac-flags/main.go:20 +0x3ed exit status 2 But we can get around this by setting the secret to ” ” whitespace, so how do you handle that? 14

if len(strings.TrimSpace(secretVar)) == 0 { panic("--secret is required") } We can simply wrap the value with strings.TrimSpace which will remove any whitespace and handle that case for us. Here’s the complete example: package main import ( "flag" "fmt" "strings" "github.com/alexellis/hmac" ) func main() { var inputVar string var secretVar string flag.StringVar(&inputVar, "message", "", "message to create a digest from") flag.StringVar(&secretVar, "secret", "", "secret for the digest") flag.Parse() if len(strings.TrimSpace(secretVar)) == 0 { panic("--secret is required") } fmt.Printf("Computing hash for: %q\nSecret: %q\n", inputVar, secretVar) digest := hmac.Sign([]byte(inputVar), []byte(secretVar)) fmt.Printf("Digest: %x\n", digest) } As an extra-work task, why don’t you add a mode, so that your CLI can be used for both generating and validating hashes? You could either add a flag with flag.BoolVar such as -generate=true/false, or a string mode like - mode=generate/validate. Define a separate package or library Until now, we have written code within a single package called main. The main package is required to make a program runnable on its own, however sometimes you’ll want to write a library so that code can be shared between programs. You may also have a project with a main package and an additional named package for code organisation. Packages can be tested separately. Generally, starting a variable, function or struct with a capital letter means that it will be available outside the package to other consumers. We’ll create a package that can perform a similar task to the dir or ls shell commands. 15

Create a new project: $ export GH_USERNAME="alexellis" $ mkdir -p $GOPATH/src/github.com/$GH_USERNAME/multiple-packages/ $ cd $GOPATH/src/github.com/$GH_USERNAME/multiple-packages/ $ go mod init go: creating new go.mod: module github.com/alexellis/multiple-packages Now create a folder for the package called cmd: $ mkdir -p $GOPATH/src/github.com/$GH_USERNAME/multiple-packages/cmd Then create ls.go inside it: package cmd Now you have defined a package called cmd, but it doesn’t have any functions or structs exported. package cmd func ExecuteLs(path string) (string, error) { return "", nil } The command can then be imported by a new main package using the path github.com/alexellis/multiple- packages/cmd. In some cases, like the HMAC library we used earlier, there is no need for the project to have its own main entrypoint because it will only be consumed by third-parties as a function. Create main.go in the root folder: package main import ( "fmt" "os" "github.com/alexellis/multiple-packages/cmd" ) func main() { wd, err := os.Getwd() if err != nil { fmt.Fprintf(os.Stderr, "unable to get working directory: %s", err.Error()) } res, err := cmd.ExecuteLs(wd) if err != nil { fmt.Fprintf(os.Stderr, "unable list files in %s, error: %s", wd, err.Error()) } fmt.Printf("%s\n", res) } Now populate the ls.go program with the rest of its code: 16

package cmd import ( "fmt" "os" ) func ExecuteLs(path string) (string, error) { entries, err := os.ReadDir(path) if err != nil { return "", err } output := fmt.Sprintf("Files in %s\n", path) output += "Name\tDirectory\t\n" for _, e := range entries { output += fmt.Sprintf("%s\t%v\n", e.Name(), e.IsDir()) } return output, nil } Did you know? Unit tests are written in the same package as the source code, so if you have tests for main, you’d place them in the root folder, and any tests for the cmd package would be placed within the cmd/ folder. In a later chapter, you’ll learn about unit testing in Go. And then try it out: $ go run . Files in /Users/alex/src/github.com/alexellis/multiple-packages Name Directory cmd true go.mod false main.go false multiple-packages false If you’d like to make use of this program on your system, you can copy it to your $PATH variable or run go install, to have it placed in $GOPATH/bin/ $ export GH_USERNAME="alexellis" $ cd $GOPATH/src/github.com/$GH_USERNAME/multiple-packages $ go install $ cd /tmp/ $ multiple-packages Files in /tmp Name Directory ... If you receive a “not found” error here, then you need to add the $GOPATH/bin/ to your $PATH variable on your system. Taking it further: • If you’d like to align the text into a table, why not try the TabWriter? • How would you use flags and packages to add other commands and execute them according to the user’s input? Derek is a GitHub bot that I wrote to make it easier to automate open source maintenance and to delegate permissions 17

to other contributors through comments. Derek is a good example of multiple packages, why not read the source code and see how his packages are exported and consumed in the main.go file. 18