Lets Go Further Advanced patterns for building APIs and web applications in Go (Alex Edwards) (z-library.sk, 1lib.sk, z-lib.sk)

📄 File Format: PDF

💾 File Size: 7.7 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

(This page has no text content)

📄 Page 2

Contents 1. Introduction 1.1. Prerequisites 2. Getting Started 2.1. Project Setup and Skeleton Structure 2.2. A Basic HTTP Server 2.3. API Endpoints and RESTful Routing 3. Sending JSON Responses 3.1. Fixed-Format JSON 3.2. JSON Encoding 3.3. Encoding Structs 3.4. Formatting and Enveloping Responses 3.5. Advanced JSON Customization 3.6. Sending Error Messages 4. Parsing JSON Requests 4.1. JSON Decoding 4.2. Managing Bad Requests 4.3. Restricting Inputs 4.4. Custom JSON Decoding 4.5. Validating JSON Input 5. Database Setup and Configuration 5.1. Setting up PostgreSQL 5.2. Connecting to PostgreSQL 5.3. Configuring the Database Connection Pool 6. SQL Migrations 6.1. An Overview of SQL Migrations 6.2. Working with SQL Migrations 7. CRUD Operations 7.1. Setting up the Movie Model 7.2. Creating a New Movie 7.3. Fetching a Movie 7.4. Updating a Movie

📄 Page 3

7.5. Deleting a Movie 8. Advanced CRUD Operations 8.1. Handling Partial Updates 8.2. Optimistic Concurrency Control 8.3. Managing SQL Query Timeouts 9. Filtering, Sorting, and Pagination 9.1. Parsing Query String Parameters 9.2. Validating Query String Parameters 9.3. Listing Data 9.4. Filtering Lists 9.5. Full-Text Search 9.6. Sorting Lists 9.7. Paginating Lists 9.8. Returning Pagination Metadata 10. Structured Logging and Error Handling 10.1. Structured JSON Log Entries 10.2. Panic Recovery 11. Rate Limiting 11.1. Global Rate Limiting 11.2. IP-based Rate Limiting 11.3. Configuring the Rate Limiters 12. Graceful Shutdown 12.1. Sending Shutdown Signals 12.2. Intercepting Shutdown Signals 12.3. Executing the Shutdown 13. User Model Setup and Registration 13.1. Setting up the Users Database Table 13.2. Setting up the Users Model 13.3. Registering a User 14. Sending Emails 14.1. SMTP Server Setup 14.2. Creating Email Templates 14.3. Sending a Welcome Email 14.4. Sending Background Emails

📄 Page 4

14.5. Graceful Shutdown of Background Tasks 15. User Activation 15.1. Setting up the Tokens Database Table 15.2. Creating Secure Activation Tokens 15.3. Sending Activation Tokens 15.4. Activating a User 16. Authentication 16.1. Authentication Options 16.2. Generating Authentication Tokens 16.3. Authenticating Requests 17. Permission-based Authorization 17.1. Requiring User Activation 17.2. Setting up the Permissions Database Table 17.3. Setting up the Permissions Model 17.4. Checking Permissions 17.5. Granting Permissions 18. Cross Origin Requests 18.1. An Overview of CORS 18.2. Demonstrating the Same-Origin Policy 18.3. Simple CORS Requests 18.4. Preflight CORS Requests 19. Metrics 19.1. Exposing Metrics with Expvar 19.2. Creating Custom Metrics 19.3. Request-level Metrics 19.4. Recording HTTP Status Codes 20. Building, Versioning and Quality Control 20.1. Creating and Using Makefiles 20.2. Managing Environment Variables 20.3. Quality Controlling Code 20.4. Module Proxies and Vendoring 20.5. Building Binaries 20.6. Managing and Automating Version Numbers 21. Deployment and Hosting

📄 Page 5

21.1. Creating a Digital Ocean Droplet 21.2. Server Configuration and Installing Software 21.3. Deployment and Executing Migrations 21.4. Running the API as a Background Service 21.5. Using Caddy as a Reverse Proxy 22. Appendices 22.1. Managing Password Resets 22.2. Creating Additional Activation Tokens 22.3. Authentication with JSON Web Tokens 22.4. JSON Encoding Nuances 22.5. JSON Decoding Nuances 22.6. Request Context Timeouts 23. Feedback

📄 Page 6

Chapter 1. Introduction In this book we’re going to work through the start-to-finish build of an application called Greenlight — a JSON API for retrieving and managing information about movies. You can think of the core functionality as being a bit like the Open Movie Database API. Ultimately, our Greenlight API will support the following endpoints and actions: Method URL Pattern Action GET /v1/healthcheck Show application health and version information GET /v1/movies Show the details of all movies POST /v1/movies Create a new movie GET /v1/movies/:id Show the details of a specific movie PATCH /v1/movies/:id Update the details of a specific movie DELETE /v1/movies/:id Delete a specific movie POST /v1/users Register a new user PUT /v1/users/activated Activate a specific user PUT /v1/users/password Update the password for a specific user POST /v1/tokens/authentication Generate a new authentication token POST /v1/tokens/password-reset Generate a new password-reset token GET /debug/vars Display application metrics To give you an idea of what the API will look like from a client’s point of view, by the end of this book the GET /v1/movies/:id endpoint will return a response similar this:

📄 Page 7

$ curl -H "Authorization: Bearer RIDBIAE3AMMK57T6IAEBUGA7ZQ" localhost:4000/v1/movies/1 { "movie": { "id": 1, "title": "Moana", "year": 2016, "runtime": "107 mins", "genres": [ "animation", "adventure" ], "version": 1 } } Behind the scenes, we’ll use PostgreSQL as the database for persistently storing all the data. And at the end of the book, we’ll deploy the finished API to a Linux server running on Digital Ocean. Conventions In this book, code blocks are shown with a silver background like the snippet below. If the code block is particularly long, parts that aren’t relevant may be replaced with an ellipsis. To make it easy to follow along, most code blocks also have a title bar at the top indicating the name of the file that we’re working on. File: hello.go package main ... // Indicates that some existing code has been omitted. func sayHello() { fmt.Println("Hello world!") } Terminal (command line) instructions are shown with a black background and generally start with a dollar symbol. These commands should work on any Unix-based operating system, including macOS and Linux. Sample output is shown in silver beneath the command, like so: $ echo "Hello world!" Hello world! If you’re using Windows, you should replace these commands with the DOS equivalent or carry out the action via the normal Windows GUI.

📄 Page 8

Some chapters in this book end with an additional information section. These sections contain information that isn’t relevant to our application build, but is still important (or sometimes, just interesting) to know about. About the author Hey, I’m Alex Edwards, a full-stack web developer and author. I live near Innsbruck, Austria. I’ve been working with Go for over 8 years, building production applications for myself and commercial clients, and helping people all around the world improve their Go skills. You can see more of my writing on my blog (where I publish detailed tutorials), some of my open-source work on GitHub, and you can also follow me on Instagram and Twitter. Copyright and disclaimer Let’s Go Further. Copyright © 2021 Alex Edwards. Last updated 2021-08-27 08:41:48 UTC. Version 1.1.0. The Go gopher was designed by Renee French and is used under the Creative Commons 3.0 Attributions license. Cover gopher adapted from vectors by Egon Elbre. The information provided within this book is for general informational purposes only. While the author and publisher have made every effort to ensure that the accuracy of the information within this book was correct at time of publication there are no representations or warranties, express or implied, about the completeness, accuracy, reliability, suitability or availability with respect to the information, products, services, or related graphics contained in this book for any purpose. Any use of this information is at your own risk.

📄 Page 9

Chapter 1.1. Prerequisites Background knowledge This book is written as a follow up to Let’s Go, and we’ll leverage a lot of the information and code patterns from that book again here. If you’ve already read and enjoyed Let’s Go, then this book should be a good fit for you and the ideal next step in your learning. If you haven’t, then I highly recommend starting with Let’s Go first — especially if you’re a newcomer to Go. You can read this as a standalone book, but please be aware that it is somewhat advanced — it doesn’t explain the fundamentals in detail, and some topics (like testing) don’t feature at all because they were covered heavily in the previous book. But if you’re comfortable using Go and already have a decent amount of experience behind you, then this book may also be a good fit for you. Feel free to jump straight in. Go 1.17 The information in this book is correct for the latest major release of Go (version 1.17), and you should install this if you’d like to code-along with the application build. If you’ve already got Go installed, you can check the version number from your terminal by using the go version command. The output should look similar to this: $ go version go version go1.17 linux/amd64 If you need to upgrade your version of Go, then please go ahead and do that now. The instructions for your operating system can be found here. Other software There are a few other bits of software that you should make sure are available on your computer if you want to follow along fully. They are:

📄 Page 10

The curl tool for working with HTTP requests and responses from your terminal. On MacOS and Linux machines it should be pre-installed or available in your software repositories. Otherwise, you can download the latest version from here. The hey tool for carrying out some basic load tests. So long as you have Go 1.16 or newer on your computer, you can install hey with the go install command: $ go install github.com/rakyll/hey@latest The git version control system. Installation instructions for all operating systems can be found here. A web browser with good developer tools. I’ll be using Firefox in this book, but Chromium, Chrome or Microsoft Edge will work too. Your favorite text editor

📄 Page 11

Chapter 2. Getting Started In this first section of the book, we’re going to set up a project directory and lay the groundwork for building our Greenlight API. We will: Create a skeleton directory structure for the project and explain at a high-level how our Go code and other assets will be organized. Establish a HTTP server to listen for incoming HTTP requests. Introduce a sensible pattern for managing configuration settings (via command-line flags) and using dependency injection to make dependencies available to our handlers. Use the httprouter package to help implement a standard RESTful structure for the API endpoints.

📄 Page 12

Chapter 2.1. Project Setup and Skeleton Structure Let’s kick things off by creating a greenlight directory to act as the top-level ‘home’ for this project. I’m going to create my project directory at $HOME/Projects/greenlight , but feel free to choose a different location if you wish. $ mkdir -p $HOME/Projects/greenlight Then change into this directory and use the go mod init command to enable modules for the project. When running this command you’ll need to specify a module path, which is essentially a unique identifier for your project. In this book I’ll use greenlight.alexedwards.net as my module path, but if you’re following along you should ideally swap this for something that is unique to you instead. $ cd $HOME/Projects/greenlight $ go mod init greenlight.alexedwards.net go: creating new go.mod: module greenlight.alexedwards.net At this point you’ll see that a go.mod file has been created in the root of your project directory. If you open it up, it should look similar to this: File: go.mod module greenlight.alexedwards.net go 1.17 We talked about modules in detail as part of the first Let’s Go book, but as a quick refresher let’s recap the main points here. When there is a valid go.mod file in the root of your project directory, your project is a module. When you’re working inside your project directory and download a dependency with go get , then the exact version of the dependency will be recorded in the go.mod file. Because the exact version is known, this makes it much easier to ensure reproducible builds across different machines and environments.

📄 Page 13

When you run or build the code in your project, Go will use the exact dependencies listed in the go.mod file. If the necessary dependencies aren’t already on your local machine, then Go will automatically download them for you — along with any recursive dependencies too. The go.mod file also defines the module path (which is greenlight.alexedwards.net in my case). This is essentially the identifier that will be used as the root import path for the packages in your project. It’s good practice to make the module path unique to you and your project. A common convention in the Go community is to base it on a URL that you own. Hint: If you feel unsure about any aspect of modules or how they work, the official Go Modules Wiki is an excellent resource and contains answers to a wide range of FAQs — although please be aware that it hasn’t yet been fully updated for Go 1.17 at the time of writing. Generating the skeleton directory structure Alright, now that our project directory has been created and we have a go.mod file, you can go ahead and run the following commands to generate a high-level skeleton structure for the project: $ mkdir -p bin cmd/api internal migrations remote $ touch Makefile $ touch cmd/api/main.go At this point your project directory should look exactly like this: . ├── bin ├── cmd │ └── api │ └── main.go ├── internal ├── migrations ├── remote ├── go.mod └── Makefile Let’s take a moment to talk through these files and folders and explain the purpose that they’ll serve in our finished project.

📄 Page 14

The bin directory will contain our compiled application binaries, ready for deployment to a production server. The cmd/api directory will contain the application-specific code for our Greenlight API application. This will include the code for running the server, reading and writing HTTP requests, and managing authentication. The internal directory will contain various ancillary packages used by our API. It will contain the code for interacting with our database, doing data validation, sending emails and so on. Basically, any code which isn’t application-specific and can potentially be reused will live in here. Our Go code under cmd/api will import the packages in the internal directory (but never the other way around). The migrations directory will contain the SQL migration files for our database. The remote directory will contain the configuration files and setup scripts for our production server. The go.mod file will declare our project dependencies, versions and module path. The Makefile will contain recipes for automating common administrative tasks — like auditing our Go code, building binaries, and executing database migrations. It’s important to point out that the directory name internal carries a special meaning and behavior in Go: any packages which live under this directory can only be imported by code inside the parent of the internal directory. In our case, this means that any packages which live in internal can only be imported by code inside our greenlight project directory. Or, looking at it the other way, this means that any packages under internal cannot be imported by code outside of our project. This is useful because it prevents other codebases from importing and relying on the (potentially unversioned and unsupported) packages in our internal directory — even if the project code is publicly available somewhere like GitHub. Hello world! Before we continue, let’s quickly check that everything is setup correctly. Open the cmd/api/main.go file in your text editor and add the following code:

📄 Page 15

File: cmd/api/main.go package main import "fmt" func main() { fmt.Println("Hello world!") } Save this file, then use the go run command in your terminal to compile and execute the code in the cmd/api package. All being well, you will see the following output: $ go run ./cmd/api Hello world!

📄 Page 16

Chapter 2.2. A Basic HTTP Server Now that the skeleton structure for our project is in place, let’s focus our attention on getting a HTTP server up and running. To start with, we’ll configure our server to have just one endpoint: /v1/healthcheck . This endpoint will return some basic information about our API, including its current version number and operating environment (development, staging, production, etc.). URL Pattern Handler Action /v1/healthcheck healthcheckHandler Show application information If you’re following along, open up the cmd/api/main.go file and replace the ‘hello world’ application with the following code: File: main.go package main import ( "flag" "fmt" "log" "net/http" "os" "time" ) // Declare a string containing the application version number. Later in the book we'll // generate this automatically at build time, but for now we'll just store the version // number as a hard-coded global constant. const version = "1.0.0" // Define a config struct to hold all the configuration settings for our application. // For now, the only configuration settings will be the network port that we want the // server to listen on, and the name of the current operating environment for the // application (development, staging, production, etc.). We will read in these // configuration settings from command-line flags when the application starts. type config struct { port int env string } // Define an application struct to hold the dependencies for our HTTP handlers, helpers, // and middleware. At the moment this only contains a copy of the config struct and a // logger, but it will grow to include a lot more as our build progresses. type application struct { config config logger *log.Logger }

📄 Page 17

} func main() { // Declare an instance of the config struct. var cfg config // Read the value of the port and env command-line flags into the config struct. We // default to using the port number 4000 and the environment "development" if no // corresponding flags are provided. flag.IntVar(&cfg.port, "port", 4000, "API server port") flag.StringVar(&cfg.env, "env", "development", "Environment (development|staging|production)") flag.Parse() // Initialize a new logger which writes messages to the standard out stream, // prefixed with the current date and time. logger := log.New(os.Stdout, "", log.Ldate | log.Ltime) // Declare an instance of the application struct, containing the config struct and // the logger. app := &application{ config: cfg, logger: logger, } // Declare a new servemux and add a /v1/healthcheck route which dispatches requests // to the healthcheckHandler method (which we will create in a moment). mux := http.NewServeMux() mux.HandleFunc("/v1/healthcheck", app.healthcheckHandler) // Declare a HTTP server with some sensible timeout settings, which listens on the // port provided in the config struct and uses the servemux we created above as the // handler. srv := &http.Server{ Addr: fmt.Sprintf(":%d", cfg.port), Handler: mux, IdleTimeout: time.Minute, ReadTimeout: 10 * time.Second, WriteTimeout: 30 * time.Second, } // Start the HTTP server. logger.Printf("starting %s server on %s", cfg.env, srv.Addr) err := srv.ListenAndServe() logger.Fatal(err) } Important: If any of the terminology or patterns in the code above are unfamiliar to you, then I strongly recommend reading the first Let’s Go book which explains them in detail. Creating the healthcheck handler The next thing we need to do is create the healthcheckHandler method for responding to HTTP requests. For now, we’ll keep the logic in this handler really simple and have it return a plain-text response containing three pieces of information:

📄 Page 18

A fixed "status: available" string. The API version from the hard-coded version constant. The operating environment name from the env command-line flag. Go ahead and create a new cmd/api/healthcheck.go file: $ touch cmd/api/healthcheck.go And then add the following code: File: cmd/api/healthcheck.go package main import ( "fmt" "net/http" ) // Declare a handler which writes a plain-text response with information about the // application status, operating environment and version. func (app *application) healthcheckHandler(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, "status: available") fmt.Fprintf(w, "environment: %s\n", app.config.env) fmt.Fprintf(w, "version: %s\n", version) } The important thing to point out here is that healthcheckHandler is implemented as a method on our application struct. This is an effective and idiomatic way to make dependencies available to our handlers without resorting to global variables or closures — any dependency that the healthcheckHandler needs can simply be included as a field in the application struct when we initialize it in main(). We can see this pattern already being used in the code above, where the operating environment name is retrieved from the application struct by calling app.config.env. Demonstration OK, let’s try this out. Make sure that all your changes are saved, then use the go run command again to execute the code in the cmd/api package. You should see a log message confirming that the HTTP server is running, similar to this:

📄 Page 19

$ go run ./cmd/api 2021/04/05 19:42:50 starting development server on :4000 While the server is running, go ahead and try visiting localhost:4000/v1/healthcheck in your web browser. You should get a response from the healthcheckHandler which looks like this: Or alternatively, you can use curl to make the request from your terminal: $ curl -i localhost:4000/v1/healthcheck HTTP/1.1 200 OK Date: Mon, 05 Apr 2021 17:46:14 GMT Content-Length: 58 Content-Type: text/plain; charset=utf-8 status: available environment: development version: 1.0.0 Note: The -i flag in the command above instructs curl to display the HTTP response headers as well as the response body. If you want, you can also verify that the command-line flags are working correctly by

📄 Page 20

specifying alternative port and env values when starting the application. When you do this, you should see the contents of the log message change accordingly. For example: $ go run ./cmd/api -port=3030 -env=production 2021/04/05 19:48:34 starting production server on :3030 Additional Information API versioning APIs which support real-world businesses and users often need to change their functionality and endpoints over time — sometimes in a backwards-incompatible way. So, to avoid problems and confusion for clients, it’s a good idea to always implement some form of API versioning. There are two common approaches to doing this: 1. By prefixing all URLs with your API version, like /v1/healthcheck or /v2/healthcheck . 2. By using custom Accept and Content-Type headers on requests and responses to convey the API version, like Accept: application/vnd.greenlight-v1 . From a HTTP semantics point of view, using headers to convey the API version is the ‘purer’ approach. But from a user-experience point of view, using a URL prefix is arguably better. It makes it possible for developers to see which version of the API is being used at a glance, and it also means that the API can still be explored using a regular web browser (which is harder if custom headers are required). Throughout this book we’ll version our API by prefixing all the URL paths with /v1/ — just like we did with the /v1/healthcheck endpoint in this chapter.