Effective Go Recipes Fast Solutions to Common Tasks (Miki Tebeka) (z-library.sk, 1lib.sk, z-lib.sk)

📄 File Format: PDF

💾 File Size: 3.8 MB

📖 Read Online ⬇️ Download

Registered users can read the full content for free

Register as a Gaohf Library member to read the complete e-book online for free and enjoy a better reading experience.

📄 Page 1

(This page has no text content)

📄 Page 2

(This page has no text content)

📄 Page 3

Early Praise for Effective Go Recipes I have known Miki for years, and he is one of the very few authors that have ma- terial I can trust without question. Miki takes the time to research each topic and finds the right people to review the content to guarantee the material is accurate and practical. Miki’s experience working on development teams and products is what sets his material apart from many other authors who tend to write books with only a cursory knowledge of engineering principles. If you are looking for real, practical development training from an experienced software developer, Miki is your guy. ➤ Bill Kennedy Managing Partner, Ardan Labs It’s such a refreshing change to see a highly technical book that brings a human element alongside technical excellence. It really does feel like Miki is speaking and teaching you, personally. I need to congratulate Miki and his editors for bringing such a personal tone on the page. ➤ Adelina Simion Education Engineer, Spectro Cloud

📄 Page 4

As a senior engineer with experience in both open source and proprietary software, I’ve tackled numerous challenges in building and maintaining projects. Effective Go Recipes: Fast Solutions to Common Tasks is an invaluable resource that distills the collective wisdom of the Go community. While Go may appear simple, it’s en- riched with patterns and practices that have evolved over time. This book is a concise guide to these patterns, offering a ready reference for both budding and seasoned Go developers. I firmly believe every Go enthusiast and any organization utilizing Go should have a copy on their shelf. ➤ Yoni Davidson Core Engineer, Tabnine I have worked with Go for almost a decade on open source and closed source projects. Finding some well-written common practice for those projects, and especially around ramping up junior engineers, was always a challenge. I have found Effective Go Recipes: Fast Solutions to Common Tasks to be a Swiss Army knife for such recipes. A very clear and down-to-earth guide on how to construct your own Go program. I would recommend it to everyone who writes production Go code. ➤ Aviv Laufer Senior Principal Software Engineer, Rokt

📄 Page 5

Effective Go Recipes Fast Solutions to Common Tasks Miki Tebeka The Pragmatic Bookshelf Dallas, Texas

📄 Page 6

For our complete catalog of hands-on, practical, and Pragmatic content for software developers, please visit https://pragprog.com. Contact support@pragprog.com for sales, volume licensing, and support. For international rights, please contact rights@pragprog.com. The team that produced this book includes: Dave ThomasPublisher: Janet FurlowCOO: Susannah DavidsonExecutive Editor: Margaret EldridgeDevelopment Editor: L. Sakhi MacMillanCopy Editor: Potomac Indexing, LLCIndexing: Gilson GraphicsLayout: Copyright © 2024 The Pragmatic Programmers, LLC. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form, or by any means, electronic, mechanical, photocopying, recording, or otherwise, without the prior consent of the publisher. When we are aware that a term used in this book is claimed as a trademark, the designation is printed with an initial capital letter or in all capitals. The Pragmatic Starter Kit, The Pragmatic Programmer, Pragmatic Programming, Pragmatic Bookshelf, PragProg and the linking g device are trademarks of The Pragmatic Programmers, LLC. Every precaution was taken in the preparation of this book. However, the publisher assumes no responsibility for errors or omissions, or for damages that may result from the use of information (including program listings) contained herein. ISBN-13: 978-1-68050-846-8 Encoded using recycled binary digits. Book version: P1.0—April 2024

📄 Page 7

Contents Preface . . . . . . . . . . . . . . ix 1. Reading and Writing (I/O) . . . . . . . . . . 1 Using In-Memory Readers and Writers to Support []byte 1 Recipe 1. Recipe 2. Compressing Old Log Files 3 Recipe 3. Using bytes.Buffer to Generate SQL 7 Recipe 4. Conditionally Decompressing Files 8 Recipe 5. Implementing io.Writer for Frequency Calculation 11 Recipe 6. Using os.Pipe for Dynamic Data Generation 13 Recipe 7. Searching in a Memory Mapped File 16 Final Thoughts 19 2. Serializing Data . . . . . . . . . . . . 21 Streaming Events with encoding/gob 21Recipe 8. Recipe 9. Parsing Complex JSON Documents 24 Recipe 10. Streaming JSON 26 Recipe 11. Handling Missing Values in JSON 28 Recipe 12. Serializing Custom Types 30 Recipe 13. Unmarshaling Dynamic Types 32 Recipe 14. Parsing Struct Tags 34 Final Thoughts 36 3. Utilizing HTTP . . . . . . . . . . . . 37 Making GET Requests 37Recipe 15. Recipe 16. Streaming POST Requests 41 Recipe 17. Writing Middleware to Monitor Performance 43 Recipe 18. Setting Server Timeouts 44

📄 Page 8

Recipe 19. Supporting Several API Versions in the Same HTTP Server 45 Final Thoughts 47 4. Working with Text . . . . . . . . . . . 49 Using Formatting Verbs for Better Output 49Recipe 20. Recipe 21. Adding String Representation to Your Types 52 Recipe 22. Detecting Encoding 54 Recipe 23. Using Regular Expressions to Convert camelCase to lower_with_underscore 56 Recipe 24. Folding Strings for Case-Insensitive Comparison 58 Recipe 25. Using Unicode Normalization for Comparison 60 Final Thoughts 63 5. Working with Functions . . . . . . . . . . 65 Using a Function Registry 65Recipe 26. Recipe 27. Using Functions as Options 70 Recipe 28. Using Closures to Provide Options with Arguments 72 Recipe 29. Passing Notifications with Functions 74 Recipe 30. Accessing Unexported Functions 76 Final Thoughts 79 6. Working with Basic Types . . . . . . . . . 81 Using the comma, ok Paradigm 82Recipe 31. Recipe 32. Using a Slice to Implement a Stack 85 Recipe 33. Calculating Cumulative Sum 89 Recipe 34. Serializing Time to/from JSON 93 Recipe 35. Using Composite Keys in Maps 96 Recipe 36. Parsing Time Strings 99 Final Thoughts 103 7. Working with Structs, Methods, and Interfaces . . . . 105 Using Ad Hoc Interfaces 105Recipe 37. Recipe 38. Wrapping the http.ResponseWriter Interface 109 Recipe 39. Using Generics to Reduce Code Size 111 Recipe 40. Using Generics for Type-Safe Data Structures 113 Recipe 41. Using Generics for Better Type Safety 116 Final Thoughts 117 8. Working with Errors . . . . . . . . . . 119 Handling and Returning Errors 119Recipe 42. Contents • vi

📄 Page 9

Recipe 43. Handling Panics 121 Recipe 44. Handling Panics in Goroutines 123 Recipe 45. Checking Errors 126 Recipe 46. Wrapping Errors 128 Final Thoughts 130 9. Using Goroutines, Channels, and Context for Concurrency . 133 Converting Sequential Code to Parallel 134Recipe 47. Recipe 48. Limiting the Number of Goroutines with a Buffered Channel 137 Recipe 49. Using a Worker Pool with Channels 140 Recipe 50. Using context.Context for Timeouts 143 Recipe 51. Passing Logger with Request ID in Context 147 Final Thoughts 150 10. Lower-Level Concurrency . . . . . . . . . 151 Writing Idempotent Functions with sync.Once 151Recipe 52. Recipe 53. Waiting for Job Completion with sync.WaitGroup 153 Recipe 54. Allowing Multiple Readers with sync.RWMutex 156 Recipe 55. Using the Race Detector 157 Recipe 56. Using sync/atomic for a Faster Now 162 Final Thoughts 164 11. Working with Sockets . . . . . . . . . . 165 Accepting Files over TCP Sockets 165Recipe 57. Recipe 58. Sending Files over TCP Sockets 168 Recipe 59. Writing a Serverless Platform 170 Recipe 60. Reading Time with NTP over UDP 173 Final Thoughts 176 12. Communicating with Non-Go Code . . . . . . . 179 Using os/exec to Ping Servers 180Recipe 61. Recipe 62. Calling C Functions to AlignText 183 Recipe 63. Redirecting Subprocess stdin and stdout to Prototype a Calculator 185 Recipe 64. Stemming Words Using a C Library 188 Final Thoughts 191 13. Testing Your Code . . . . . . . . . . . 193 Conditionally Running Continuous Integration Tests 193 Recipe 65. Recipe 66. Reading Test Cases from a File 195 Contents • vii

📄 Page 10

Recipe 67. Fuzzing Bugs Away 199 Recipe 68. Mocking HTTP Client Calls 202 Recipe 69. Writing Global Setup/Teardown Functions 205 Recipe 70. Running Services in Testing 207 Recipe 71. Writing a Linter 210 Final Thoughts 213 14. Building Applications . . . . . . . . . . 215 Embedding Assets in Your Binary 215Recipe 72. Recipe 73. Injecting Version to Your Executable 217 Recipe 74. Ensuring Static Builds 220 Recipe 75. Using Build Tags for Conditional Builds 223 Recipe 76. Building Executables for Different Platforms 224 Recipe 77. Generating Code 227 Final Thoughts 229 15. Shipping Your Code . . . . . . . . . . 231 Configuring Your Application 231Recipe 78. Recipe 79. Patching Dependencies 234 Recipe 80. Packaging Applications in Docker 237 Recipe 81. Catching Signals for Graceful Shutdown 239 Recipe 82. Writing Logs 241 Recipe 83. Using Metrics as Eyes to Production 243 Recipe 84. Debugging Running Services 246 Final Thoughts 249 Index . . . . . . . . . . . . . . 251 Contents • viii

📄 Page 11

Preface Cooking with Go Go is a simple language. The Go 1.20 language specification1 is about 105 pages; in contrast, the Java 20 language specification2 comes to 854 pages. But even though Go is simple, it doesn’t mean you can’t do a lot with it. As any TV chef will repeatedly tell you, you can make amazing dishes with a few basic, simple ingredients. The same goes for software—you can build sleek, fast, and useful applications using a simple language like Go. A good way to learn to cook is by following recipes. These recipes might be from professionals or handed down from generation to generation. At the beginning, you’ll follow recipes to the letter. But once you get more experience, you’ll start to diverge and add your own touch. The same goes for programming recipes. If you’re a novice, you’ll probably want to follow the recipes almost to the letter. But once you gain experience, you’ll develop your own way of doing things. Hopefully, once at the expert level, you’ll share your own recipes. Who Are You? This book is written for Go developers (doh!). The recipes in this book vary in difficulty. Some recipes will be a good fit for a Go beginner, while others will help more advanced developers. This book won’t teach you Go—you’ll find many good books, tutorials, and courses out there for that (I teach some of them). To use this book, you should also be familiar with working in the terminal and know some client-server terms and technologies, such as HTTP and sockets. 1. https://go.dev/ref/spec 2. https://docs.oracle.com/javase/specs/jls/se20/jls20.pdf report erratum • discuss

📄 Page 12

Reading This Book This book is a collection of recipes. For each recipe, I tried to set the stage for when it is applicable, provide some context, light some candles … (strike the last one, it’s not that kind of book). The recipes are split into sections, but there is overlap—you can talk about errors without knowing about interfaces. In each section, the recipes are sorted according to their difficulty level. Feel free to roam around; what I think as difficult might be easy for you. A Foolish Consistency Is the Hobgoblin of Little Minds We use some conventions in this book. Code samples and terminal sessions look like this: fmt.Println("Hello Gophers ♡") Modern terminals are much wider than the common page width. In some cases I had to truncate output using ... or break long lines using \. For example: $ go test -run NONE -fuzz . -fuzztime 10s fuzz: elapsed: 0s, gathering baseline coverage: 0/2 completed fuzz: elapsed: 0s, gathering baseline coverage: 2/2 completed, \ now fuzzing with 12 workers ... The code and examples show output from Go version 1.20. Some of the output might change in the future, but Go has a compatibility promise—things shouldn’t change that much. Wait! There’s More! (online) At the book’s website3 you can find the full source code used in this book. You’ll also find a link to the errata page—please help your fellow readers and report any issues you find. And finally, you’ll find a link to a discussion forum about the book. Let’s get cooking! 3. https://pragprog.com/titles/mtgo Preface • x report erratum • discuss

📄 Page 13

CHAPTER 1 Reading and Writing (I/O) Your program communicates with the world using input and output (I/O for short). Go provides a great abstraction over I/O with the io.Reader and io.Writer interfaces. These interfaces are implemented by files, sockets, HTTP response bodies, and more. While io.Reader and io.Writer give a standard API to I/O, there are many imple- mentation-specific details you need to consider when working with I/O. In this chapter we’ll look at in-memory I/O, compression, and memory mapped files for faster searching. We’ll also see how you can implement these interfaces in your types and why you should do so. Since files are a big part of I/O, we’ll look at file paths (for example, /var/log/httpd.log) and how to manipulate them. Recipe 1 Using In-Memory Readers and Writers to Support []byte Task You’re working on the back end for a ride-sharing site. The back end is com- posed of several processes that communicate with each other. The existing implementation has code to serialize data, but it works with io.Reader and io.Writer. You want to explore shared memory as a medium to exchange data, and in shared memory you need to work with []byte. report erratum • discuss

📄 Page 14

Solution You start by looking at the current code. You see a Ride struct: io/marshal/marshal.go // Ride is a ride record. type Ride struct { ID int Time time.Time Duration time.Duration Distance float64 Price float64 } Then you look at the current NewDecoder, which returns a *Decoder: io/marshal/marshal.go // NewDecoder returns a new Decoder. func NewDecoder(r io.Reader) *Decoder { return &Decoder{json.NewDecoder(r)} } What you need to do is to convert a []byte to an io.Reader. This is where bytes.NewReader comes in handy: io/marshal/marshal.go // UnmarshalRide returns a Ride from serialized data. func UnmarshalRide(data []byte, ride *Ride) error { r := bytes.NewReader(data) return NewDecoder(r).DecodeRide(ride) } Discussion bytes.NewReader and its cousins strings.NewReader, bytes.Buffer, and others create in-memory io.Reader and io.Writer around concrete types. They’re helpful when dealing with APIs that only work with io.Reader (such as encoding/gob). I use these in-memory readers and writers a lot while testing. I start with a file with a list of concrete examples, then read each case and pass it, wrapped in an in-memory reader, to a decoder and test it. To learn more about testing, head over to Chapter 13, Testing Your Code, on page 193, and to learn more about memory mapped files, see Recipe 7, Searching in a Memory Mapped File, on page 16. Chapter 1. Reading and Writing (I/O) • 2 report erratum • discuss

📄 Page 15

Recipe 2 Compressing Old Log Files Task You’re “on loan” to the operations team—you’re going to help them by writing some utilities. The operations team lead sends you the following email: Welcome to the team! Your first task, should you choose to accept it, is to write a “janitor” utility. This utility should go over a given directory and compress any log file (having .log suffix) that is over a month old. After compressing, you should compare the content of the compressed file with the original file (say, using SHA1) and if it matches, delete the original file. This message won’t destruct in five seconds. After some back-and-forth, you agree that “a month” equals thirty days and that you’ll use gzip compression. Shell Magic I know you can complete this task with a combination of the Unix find and gzip utilities. However, the goal here is to learn how to work with files and not to practice Unix command-line tricks. Solution You write a gzCompress function that gets a source file and compresses it using gzip to the destination file: files/janitor/janitor.go // gzCompress compresses src to dest with gzip compression.Line 1 func gzCompress(src, dest string) error {- file, err := os.Open(src)- if err != nil {- return err5 }- defer file.Close()- - out, err := os.Create(dest)- if err != nil {10 return err- }- report erratum • discuss Compressing Old Log Files • 3

📄 Page 16

defer out.Close()- - w := gzip.NewWriter(out)15 defer w.Close()- - // Update metadata, must be before io.Copy- w.Name = src- info, err := file.Stat()20 if err == nil {- w.ModTime = info.ModTime()- }- - if _, err := io.Copy(w, file); err != nil {25 os.Remove(dest)- return err- }- - return nil30 }- Then you write shouldCompress, which checks if a file is older than a given time span: files/janitor/janitor.go func shouldCompress(path string, maxAge time.Duration) bool {Line 1 info, err := os.Stat(path)- if err != nil {- log.Printf("warning: %q: can't get info: %s", path, err)- return false5 }- - if info.IsDir() {- return false- }10 - return time.Since(info.ModTime()) >= maxAge- }- And then you write filesToCompress, which will return a list of files that are older than a given time span in a directory: files/janitor/janitor.go func filesToCompress(dir string, maxAge time.Duration) ([]string, error) { root := os.DirFS(dir) logFiles, err := fs.Glob(root, "*.log") if err != nil { return nil, err } Chapter 1. Reading and Writing (I/O) • 4 report erratum • discuss

📄 Page 17

var names []string for _, src := range logFiles { name := path.Join(dir, src) if shouldCompress(name, maxAge) { names = append(names, name) } } return names, nil } Then, to check that a file was compressed without issues, you write code to compare the file SHA1 signature: files/janitor/janitor.go func fileSHA1(fileName string) (string, error) { file, err := os.Open(fileName) if err != nil { return "", nil } defer file.Close() var r io.Reader = file if path.Ext(fileName) == ".gz" { var err error r, err = gzip.NewReader(r) if err != nil { return "", err } } w := sha1.New() if _, err := io.Copy(w, r); err != nil { return "", err } sig := fmt.Sprintf("%x", w.Sum(nil)) return sig, nil } func sameSig(file1, file2 string) (bool, error) { sig1, err := fileSHA1(file1) if err != nil { return false, err } sig2, err := fileSHA1(file2) if err != nil { return false, err } return sig1 == sig2, nil } report erratum • discuss Compressing Old Log Files • 5

📄 Page 18

Finally, you write compressFiles, which will compress only files older than the time span: files/janitor/janitor.go func compressFiles(rootDir string, maxAge time.Duration) error { files, err := filesToCompress(rootDir, maxAge) if err != nil { return err } for _, src := range files { dest := src + ".gz" if err := gzCompress(src, dest); err != nil { return fmt.Errorf("%q: %w", src, err) } match, err := sameSig(src, dest) if err != nil { return err } if !match { return fmt.Errorf("%q <-> %q: signature don't match", src, dest) } if err := os.Remove(src); err != nil { log.Printf("warning: %q: can't delete - %s", src, err) } } return nil } Discussion Reading is by far the most common operation you’ll do with files. os.Open is a convenient, short way to open a file for reading. On line 7, you use defer to make sure the file is closed. The operating system defines a limit on the number of open files a process can hold. On Unix sys- tems, you can use the ulimit utility to see this limit. On line 15, you wrap the output file with gzip.NewWriter. gzip.NewWriter also implements io.Writer, meaning it can be used by io.Copy. *os.File implements many of the interfaces defined in the io package—io.Reader, io.Writer, and io.Closer—and you can pass it to any function accepting one of these interfaces. On line 2, you use the os.Stat function to get information about the file. Chapter 1. Reading and Writing (I/O) • 6 report erratum • discuss

📄 Page 19

You can do a lot of system-level work using functions from the os and io—read the docs and get familiar with these functions. Recipe 3 Using bytes.Buffer to Generate SQL Task The ride-sharing application you’re working on stores its data in a SQL database. The company wants to allow access to its data from an API. You don’t want to allow users to pass arbitrary SQL statements—it’s a secu- rity risk (SQL injection), and it ties the database design to the API design. First, you need to provide a function that, given a table name and a list of columns, returns a SQL query that will select this data from the database. For a table name rides and the columns id, time, and duration, you generate the following output: SELECT id, time, duration FROM rides; Solution You write a genSelect function that accepts a table name and a slice of columns and returns SQL as a string: io/sql/sql.go func genSelect(table string, columns []string) (string, error) { var buf bytes.Buffer if len(columns) == 0 { return "", fmt.Errorf("empty select") } fmt.Fprintln(&buf, "SELECT") for i, col := range columns { suffix := "," if i == len(columns)-1 { suffix = "" // No trailing comma in SQL } report erratum • discuss Using bytes.Buffer to Generate SQL • 7

📄 Page 20

fmt.Fprintf(&buf, " %s%s\n", col, suffix) } fmt.Fprintf(&buf, "FROM %s;", table) return buf.String(), nil } Discussion genSelect uses bytes.Buffer as an io.Writer interface. This way, the fmt function emits data into the buffer and not to the standard output. At the end of the function, you use the String method to return the accumulated string. Using bytes.Buffer as in-memory io.Writer gives us two advantages: • It simplifies our code generation algorithm • It allows us to use the fmt package You’d be surprised how many times using an algorithm that generates data on the fly, such as printing, will make your code simpler. By using an in- memory io.Writer, we’re able to emit data as it comes and not accumulate it and then return it. For fancier code or text generation, check out the text/template and html/template built-in packages. Recipe 4 Conditionally Decompressing Files Task You recently changed the layout of your company’s website. To be backward compatible, you emit an HTTP redirect status code to tell clients accessing the old pages where to find the new location. After a while, you want to check how many clients are still trying to access the old pages. If nobody is accessing the old pages, you can delete some legacy code. The HTTP server log files are saved to the logs directory, and some of them are compressed. Let’s see what’s there: Chapter 1. Reading and Writing (I/O) • 8 report erratum • discuss