# Table of Contents - [Effective Go - The Go Programming Language](#effective-go-the-go-programming-language) - [Case Studies - The Go Programming Language](#case-studies-the-go-programming-language) - [The Go Programming Language](#the-go-programming-language) - [Use Cases - The Go Programming Language](#use-cases-the-go-programming-language) - [A Tour of Go](#a-tour-of-go) - [American Express Uses Go for Payments & Rewards - The Go Programming Language](#american-express-uses-go-for-payments-rewards-the-go-programming-language) - [Command-line Interfaces (CLIs) - The Go Programming Language](#command-line-interfaces-clis-the-go-programming-language) - [PayPal Taps Go to Modernize and Scale - The Go Programming Language](#paypal-taps-go-to-modernize-and-scale-the-go-programming-language) - [Go for Web Development - The Go Programming Language](#go-for-web-development-the-go-programming-language) - [Development Operations & Site Reliability Engineering - The Go Programming Language](#development-operations-site-reliability-engineering-the-go-programming-language) - [Go for Cloud & Network Services - The Go Programming Language](#go-for-cloud-network-services-the-go-programming-language) - [MercadoLibre Grows with Go - The Go Programming Language](#mercadolibre-grows-with-go-the-go-programming-language) - [Download and install - The Go Programming Language](#download-and-install-the-go-programming-language) - [Using Go at Google - The Go Programming Language](#using-go-at-google-the-go-programming-language) - [A Tour of Go](#a-tour-of-go) - [Tutorial: Get started with Go - The Go Programming Language](#tutorial-get-started-with-go-the-go-programming-language) - [Security Best Practices for Go Developers - The Go Programming Language](#security-best-practices-for-go-developers-the-go-programming-language) - [talks/2009 - The Go Programming Language](#talks-2009-the-go-programming-language) - [Unknown](#unknown) - [Tutorial: Create a Go module - The Go Programming Language](#tutorial-create-a-go-module-the-go-programming-language) - [Go Vulnerability Management - The Go Programming Language](#go-vulnerability-management-the-go-programming-language) - [Tutorial: Find and fix vulnerable dependencies with govulncheck - The Go Programming Language](#tutorial-find-and-fix-vulnerable-dependencies-with-govulncheck-the-go-programming-language) - [Vulnerability Scanning in IDEs - The Go Programming Language](#vulnerability-scanning-in-ides-the-go-programming-language) - [FIPS 140-3 Compliance - The Go Programming Language](#fips-140-3-compliance-the-go-programming-language) - [Tutorial: Find and fix vulnerable dependencies with VS Code Go - The Go Programming Language](#tutorial-find-and-fix-vulnerable-dependencies-with-vs-code-go-the-go-programming-language) - [Go 1.25 is released - The Go Programming Language](#go-1-25-is-released-the-go-programming-language) - [talks/static - The Go Programming Language](#talks-static-the-go-programming-language) - [talks/2019 - The Go Programming Language](#talks-2019-the-go-programming-language) - [talks/2011 - The Go Programming Language](#talks-2011-the-go-programming-language) - [Using prepared statements - The Go Programming Language](#using-prepared-statements-the-go-programming-language) - [Managing connections - The Go Programming Language](#managing-connections-the-go-programming-language) - [The Green Tea Garbage Collector - The Go Programming Language](#the-green-tea-garbage-collector-the-go-programming-language) - [Unknown](#unknown) - [Avoiding SQL injection risk - The Go Programming Language](#avoiding-sql-injection-risk-the-go-programming-language) - [Editor plugins and IDEs - The Go Programming Language](#editor-plugins-and-ides-the-go-programming-language) - [Canceling in-progress operations - The Go Programming Language](#canceling-in-progress-operations-the-go-programming-language) - [Executing SQL statements that don't return data - The Go Programming Language](#executing-sql-statements-that-don-t-return-data-the-go-programming-language) - [Managing module source - The Go Programming Language](#managing-module-source-the-go-programming-language) - [Managing dependencies - The Go Programming Language](#managing-dependencies-the-go-programming-language) - [Organizing a Go module - The Go Programming Language](#organizing-a-go-module-the-go-programming-language) - [Profile-guided optimization - The Go Programming Language](#profile-guided-optimization-the-go-programming-language) - [Developing a major version update - The Go Programming Language](#developing-a-major-version-update-the-go-programming-language) - [Managing Go installations - The Go Programming Language](#managing-go-installations-the-go-programming-language) - [The FIPS 140-3 Go Cryptographic Module - The Go Programming Language](#the-fips-140-3-go-cryptographic-module-the-go-programming-language) - [Codewalk: First-Class Functions in Go - The Go Programming Language](#codewalk-first-class-functions-in-go-the-go-programming-language) - [Executing transactions - The Go Programming Language](#executing-transactions-the-go-programming-language) - [Opening a database handle - The Go Programming Language](#opening-a-database-handle-the-go-programming-language) - [go.mod file reference - The Go Programming Language](#go-mod-file-reference-the-go-programming-language) - [Developing and publishing modules - The Go Programming Language](#developing-and-publishing-modules-the-go-programming-language) - [Accessing relational databases - The Go Programming Language](#accessing-relational-databases-the-go-programming-language) - [Module version numbering - The Go Programming Language](#module-version-numbering-the-go-programming-language) - [Go's Declaration Syntax - The Go Programming Language](#go-s-declaration-syntax-the-go-programming-language) - [Module release and versioning workflow - The Go Programming Language](#module-release-and-versioning-workflow-the-go-programming-language) - [Chrome Content Optimization Service Runs on Go - The Go Programming Language](#chrome-content-optimization-service-runs-on-go-the-go-programming-language) - [Codewalk: Share Memory By Communicating - The Go Programming Language](#codewalk-share-memory-by-communicating-the-go-programming-language) - [Tutorial: Getting started with multi-module workspaces - The Go Programming Language](#tutorial-getting-started-with-multi-module-workspaces-the-go-programming-language) - [Querying for data - The Go Programming Language](#querying-for-data-the-go-programming-language) - [Diagnostics - The Go Programming Language](#diagnostics-the-go-programming-language) - [The Laws of Reflection - The Go Programming Language](#the-laws-of-reflection-the-go-programming-language) - [Actuating Google Production: How Google’s Site Reliability Engineering Team Uses Go - The Go Programming Language](#actuating-google-production-how-google-s-site-reliability-engineering-team-uses-go-the-go-programming-language) - [Publishing a module - The Go Programming Language](#publishing-a-module-the-go-programming-language) - [JSON and Go - The Go Programming Language](#json-and-go-the-go-programming-language) - [How the Firebase Hosting Team Scaled With Go - The Go Programming Language](#how-the-firebase-hosting-team-scaled-with-go-the-go-programming-language) - [A Quick Guide to Go's Assembler - The Go Programming Language](#a-quick-guide-to-go-s-assembler-the-go-programming-language) - [The Go Memory Model - The Go Programming Language](#the-go-memory-model-the-go-programming-language) - [Coverage profiling support for integration tests - The Go Programming Language](#coverage-profiling-support-for-integration-tests-the-go-programming-language) - [JSON-RPC: a tale of interfaces - The Go Programming Language](#json-rpc-a-tale-of-interfaces-the-go-programming-language) - [Go Slices: usage and internals - The Go Programming Language](#go-slices-usage-and-internals-the-go-programming-language) - [Debugging Go Code with GDB - The Go Programming Language](#debugging-go-code-with-gdb-the-go-programming-language) - [How Google's Core Data Solutions Team Uses Go - The Go Programming Language](#how-google-s-core-data-solutions-team-uses-go-the-go-programming-language) - [Go Concurrency Patterns: Timing out, moving on - The Go Programming Language](#go-concurrency-patterns-timing-out-moving-on-the-go-programming-language) - [Tutorial: Accessing a relational database - The Go Programming Language](#tutorial-accessing-a-relational-database-the-go-programming-language) - [Defer, Panic, and Recover - The Go Programming Language](#defer-panic-and-recover-the-go-programming-language) - [About the go command - The Go Programming Language](#about-the-go-command-the-go-programming-language) - [Go Wiki: GoUserGroups - The Go Programming Language](#go-wiki-gousergroups-the-go-programming-language) - [Command Documentation - The Go Programming Language](#command-documentation-the-go-programming-language) - [Go Wiki: NonEnglish - The Go Programming Language](#go-wiki-nonenglish-the-go-programming-language) - [Container-aware GOMAXPROCS - The Go Programming Language](#container-aware-gomaxprocs-the-go-programming-language) - [Flight Recorder in Go 1.25 - The Go Programming Language](#flight-recorder-in-go-1-25-the-go-programming-language) - [talks/2017 - The Go Programming Language](#talks-2017-the-go-programming-language) - [Migrating to Go Modules - The Go Programming Language](#migrating-to-go-modules-the-go-programming-language) - [A GIF decoder: an exercise in Go interfaces - The Go Programming Language](#a-gif-decoder-an-exercise-in-go-interfaces-the-go-programming-language) - [talks/2016 - The Go Programming Language](#talks-2016-the-go-programming-language) - [Writing Web Applications - The Go Programming Language](#writing-web-applications-the-go-programming-language) - [Go Wiki: Go talks - The Go Programming Language](#go-wiki-go-talks-the-go-programming-language) - [Contribution Guide - The Go Programming Language](#contribution-guide-the-go-programming-language) - [Tutorial: Getting started with generics - The Go Programming Language](#tutorial-getting-started-with-generics-the-go-programming-language) - [Codewalk: Generating arbitrary text: a Markov chain algorithm - The Go Programming Language](#codewalk-generating-arbitrary-text-a-markov-chain-algorithm-the-go-programming-language) - [Gobs of data - The Go Programming Language](#gobs-of-data-the-go-programming-language) - [The Go image package - The Go Programming Language](#the-go-image-package-the-go-programming-language) - [Go Wiki: Learn - The Go Programming Language](#go-wiki-learn-the-go-programming-language) - [A Guide to the Go Garbage Collector - The Go Programming Language](#a-guide-to-the-go-garbage-collector-the-go-programming-language) - [Go Vulnerability Database - The Go Programming Language](#go-vulnerability-database-the-go-programming-language) - [It's survey time! How has Go has been working out for you? - The Go Programming Language](#it-s-survey-time-how-has-go-has-been-working-out-for-you-the-go-programming-language) - [Testing Time (and other asynchronicities) - The Go Programming Language](#testing-time-and-other-asynchronicities-the-go-programming-language) - [The Go image/draw package - The Go Programming Language](#the-go-image-draw-package-the-go-programming-language) - [Using Go Modules - The Go Programming Language](#using-go-modules-the-go-programming-language) - [Tutorial: Developing a RESTful API with Go and Gin - The Go Programming Language](#tutorial-developing-a-restful-api-with-go-and-gin-the-go-programming-language) - [Go Modules: v2 and Beyond - The Go Programming Language](#go-modules-v2-and-beyond-the-go-programming-language) - [Publishing Go Modules - The Go Programming Language](#publishing-go-modules-the-go-programming-language) - [Profiling Go Programs - The Go Programming Language](#profiling-go-programs-the-go-programming-language) - [Tutorial: Getting started with fuzzing - The Go Programming Language](#tutorial-getting-started-with-fuzzing-the-go-programming-language) - [C? Go? Cgo! - The Go Programming Language](#c-go-cgo-the-go-programming-language) - [Error handling and Go - The Go Programming Language](#error-handling-and-go-the-go-programming-language) - [talks/2013 - The Go Programming Language](#talks-2013-the-go-programming-language) - [Generic interfaces - The Go Programming Language](#generic-interfaces-the-go-programming-language) - [Introducing the Go Race Detector - The Go Programming Language](#introducing-the-go-race-detector-the-go-programming-language) - [Go Fuzzing - The Go Programming Language](#go-fuzzing-the-go-programming-language) - [A new experimental Go API for JSON - The Go Programming Language](#a-new-experimental-go-api-for-json-the-go-programming-language) - [Go 1.3 Release Notes - The Go Programming Language](#go-1-3-release-notes-the-go-programming-language) - [Go Security Policy - The Go Programming Language](#go-security-policy-the-go-programming-language) - [Installing Go from source - The Go Programming Language](#installing-go-from-source-the-go-programming-language) - [Gopls: The language server for Go - The Go Programming Language](#gopls-the-language-server-for-go-the-go-programming-language) - [Go 1.13 Release Notes - The Go Programming Language](#go-1-13-release-notes-the-go-programming-language) - [Go 1.1 Release Notes - The Go Programming Language](#go-1-1-release-notes-the-go-programming-language) - [Go 1.9 Release Notes - The Go Programming Language](#go-1-9-release-notes-the-go-programming-language) - [Go 1.11 Release Notes - The Go Programming Language](#go-1-11-release-notes-the-go-programming-language) - [Keeping Your Modules Compatible - The Go Programming Language](#keeping-your-modules-compatible-the-go-programming-language) - [talks/2015 - The Go Programming Language](#talks-2015-the-go-programming-language) - [Go 1.12 Release Notes - The Go Programming Language](#go-1-12-release-notes-the-go-programming-language) - [Go 1.15 Release Notes - The Go Programming Language](#go-1-15-release-notes-the-go-programming-language) - [Go 1.16 Release Notes - The Go Programming Language](#go-1-16-release-notes-the-go-programming-language) - [Go 1.17 Release Notes - The Go Programming Language](#go-1-17-release-notes-the-go-programming-language) - [Go 1 Release Notes - The Go Programming Language](#go-1-release-notes-the-go-programming-language) - [Go 1.14 Release Notes - The Go Programming Language](#go-1-14-release-notes-the-go-programming-language) - [talks/2014 - The Go Programming Language](#talks-2014-the-go-programming-language) - [Go 1.10 Release Notes - The Go Programming Language](#go-1-10-release-notes-the-go-programming-language) - [Pre-Go 1 Release History - The Go Programming Language](#pre-go-1-release-history-the-go-programming-language) - [Go 1.4 Release Notes - The Go Programming Language](#go-1-4-release-notes-the-go-programming-language) - [Go 1.6 Release Notes - The Go Programming Language](#go-1-6-release-notes-the-go-programming-language) - [Go 1.2 Release Notes - The Go Programming Language](#go-1-2-release-notes-the-go-programming-language) - [Go 1.7 Release Notes - The Go Programming Language](#go-1-7-release-notes-the-go-programming-language) - [Go 1.19 Release Notes - The Go Programming Language](#go-1-19-release-notes-the-go-programming-language) - [Go 1.22 Release Notes - The Go Programming Language](#go-1-22-release-notes-the-go-programming-language) - [Go 1.21 Release Notes - The Go Programming Language](#go-1-21-release-notes-the-go-programming-language) - [Go 1.20 Release Notes - The Go Programming Language](#go-1-20-release-notes-the-go-programming-language) - [Go 1.8 Release Notes - The Go Programming Language](#go-1-8-release-notes-the-go-programming-language) - [Go 1.5 Release Notes - The Go Programming Language](#go-1-5-release-notes-the-go-programming-language) - [Tutorials - The Go Programming Language](#tutorials-the-go-programming-language) - [A Tour of Go](#a-tour-of-go) - [Unknown](#unknown) - [Data Race Detector - The Go Programming Language](#data-race-detector-the-go-programming-language) - [Call your code from another module - The Go Programming Language](#call-your-code-from-another-module-the-go-programming-language) - [Return a random greeting - The Go Programming Language](#return-a-random-greeting-the-go-programming-language) - [Return and handle an error - The Go Programming Language](#return-and-handle-an-error-the-go-programming-language) - [Return greetings for multiple people - The Go Programming Language](#return-greetings-for-multiple-people-the-go-programming-language) - [Compile and install the application - The Go Programming Language](#compile-and-install-the-application-the-go-programming-language) - [Add a test - The Go Programming Language](#add-a-test-the-go-programming-language) - [A Tour of Go](#a-tour-of-go) - [Go CNA Policy - The Go Programming Language](#go-cna-policy-the-go-programming-language) - [Unknown](#unknown) - [Unknown](#unknown) - [Go 1.24 is released! - The Go Programming Language](#go-1-24-is-released-the-go-programming-language) - [Goodbye core types - Hello Go as we know and love it! - The Go Programming Language](#goodbye-core-types-hello-go-as-we-know-and-love-it-the-go-programming-language) - [Vulnerability Management for Go - The Go Programming Language](#vulnerability-management-for-go-the-go-programming-language) - [More predictable benchmarking with testing.B.Loop - The Go Programming Language](#more-predictable-benchmarking-with-testing-b-loop-the-go-programming-language) - [Go, Backwards Compatibility, and GODEBUG - The Go Programming Language](#go-backwards-compatibility-and-godebug-the-go-programming-language) - [Lexical Scanning in Go](#lexical-scanning-in-go) - [Go Cryptography Security Audit - The Go Programming Language](#go-cryptography-security-audit-the-go-programming-language) - [Go Wiki: All Wiki Pages - The Go Programming Language](#go-wiki-all-wiki-pages-the-go-programming-language) - [Faster Go maps with Swiss Tables - The Go Programming Language](#faster-go-maps-with-swiss-tables-the-go-programming-language) - [talks/2019/playground-v3 - The Go Programming Language](#talks-2019-playground-v3-the-go-programming-language) - [Go at Google: Language Design in the Service of Software Engineering - The Go Programming Language](#go-at-google-language-design-in-the-service-of-software-engineering-the-go-programming-language) - [Range Over Function Types - The Go Programming Language](#range-over-function-types-the-go-programming-language) - [talks/2011/lex - The Go Programming Language](#talks-2011-lex-the-go-programming-language) - [Go Wiki: Why Go - The Go Programming Language](#go-wiki-why-go-the-go-programming-language) - [Gopls release v0.20.0 - The Go Programming Language](#gopls-release-v0-20-0-the-go-programming-language) - [Go for C programmers](#go-for-c-programmers) - [Go Tech Talk](#go-tech-talk) - [Go on Android](#go-on-android) - [gRPC Go](#grpc-go) - [10 things you (probably) don't know about Go](#10-things-you-probably-don-t-know-about-go) - [Go (January 12, 2010)](#go-january-12-2010-) - [Go, Networked (January 21, 2010)](#go-networked-january-21-2010-) - [Go: code that grows with grace](#go-code-that-grows-with-grace) - [Gopls: Using Vim or Neovim - The Go Programming Language](#gopls-using-vim-or-neovim-the-go-programming-language) - [Go becomes more stable - The Go Programming Language](#go-becomes-more-stable-the-go-programming-language) - [Go Wiki: PGO Tools - The Go Programming Language](#go-wiki-pgo-tools-the-go-programming-language) - [Go Wiki: Editors and IDEs for Go - The Go Programming Language](#go-wiki-editors-and-ides-for-go-the-go-programming-language) - [Gopls release v0.17.0 - The Go Programming Language](#gopls-release-v0-17-0-the-go-programming-language) - [Go Wiki: SQL Database Drivers - The Go Programming Language](#go-wiki-sql-database-drivers-the-go-programming-language) - [Go Toolchains - The Go Programming Language](#go-toolchains-the-go-programming-language) - [Go Wiki: Spectre - The Go Programming Language](#go-wiki-spectre-the-go-programming-language) - [Go Wiki: CoreDumpDebugging - The Go Programming Language](#go-wiki-coredumpdebugging-the-go-programming-language) - [Go Wiki: Asking Questions - The Go Programming Language](#go-wiki-asking-questions-the-go-programming-language) - [Go Wiki: Go success stories from around the web - The Go Programming Language](#go-wiki-go-success-stories-from-around-the-web-the-go-programming-language) - [Gopls release v0.18.0 - The Go Programming Language](#gopls-release-v0-18-0-the-go-programming-language) - [Go Wiki: Contributing - The Go Programming Language](#go-wiki-contributing-the-go-programming-language) - [Generating code - The Go Programming Language](#generating-code-the-go-programming-language) - [Traversal-resistant file APIs - The Go Programming Language](#traversal-resistant-file-apis-the-go-programming-language) - [ - The Go Programming Language](#-the-go-programming-language) - [talks/2013/go1.1 - The Go Programming Language](#talks-2013-go1-1-the-go-programming-language) - [Go 1 and the Future of Go Programs - The Go Programming Language](#go-1-and-the-future-of-go-programs-the-go-programming-language) - [Everything You Always Wanted to Know About Type Inference - And a Little Bit More - The Go Programming Language](#everything-you-always-wanted-to-know-about-type-inference-and-a-little-bit-more-the-go-programming-language) - [Go Playground - The Go Programming Language](#go-playground-the-go-programming-language) - [Setting up and using gccgo - The Go Programming Language](#setting-up-and-using-gccgo-the-go-programming-language) - [Go Concurrency Patterns](#go-concurrency-patterns) - [Go Telemetry - The Go Programming Language](#go-telemetry-the-go-programming-language) - [Gopls release v0.19.0 - The Go Programming Language](#gopls-release-v0-19-0-the-go-programming-language) - [Go Wiki: Rangefunc Experiment - The Go Programming Language](#go-wiki-rangefunc-experiment-the-go-programming-language) - [Go Concurrency Patterns: Context - The Go Programming Language](#go-concurrency-patterns-context-the-go-programming-language) - [The cover story - The Go Programming Language](#the-cover-story-the-go-programming-language) - [talks/2010/io - The Go Programming Language](#talks-2010-io-the-go-programming-language) - [Telemetry in Go 1.23 and beyond - The Go Programming Language](#telemetry-in-go-1-23-and-beyond-the-go-programming-language) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [ - The Go Programming Language](#-the-go-programming-language) - [Unknown](#unknown) - [Go Playground - The Go Programming Language](#go-playground-the-go-programming-language) - [Forward Compatibility and Toolchain Management in Go 1.21 - The Go Programming Language](#forward-compatibility-and-toolchain-management-in-go-1-21-the-go-programming-language) - [ - The Go Programming Language](#-the-go-programming-language) - [Introducing the Go Playground - The Go Programming Language](#introducing-the-go-playground-the-go-programming-language) - [ - The Go Programming Language](#-the-go-programming-language) - [ - The Go Programming Language](#-the-go-programming-language) - [Strings, bytes, runes and characters in Go - The Go Programming Language](#strings-bytes-runes-and-characters-in-go-the-go-programming-language) - [Go Playground - The Go Programming Language](#go-playground-the-go-programming-language) - [ - The Go Programming Language](#-the-go-programming-language) - [ - The Go Programming Language](#-the-go-programming-language) - [ - The Go Programming Language](#-the-go-programming-language) - [ - The Go Programming Language](#-the-go-programming-language) - [ - The Go Programming Language](#-the-go-programming-language) - [ - The Go Programming Language](#-the-go-programming-language) - [ - The Go Programming Language](#-the-go-programming-language) - [Go on App Engine: tools, tests, and concurrency - The Go Programming Language](#go-on-app-engine-tools-tests-and-concurrency-the-go-programming-language) - [ - The Go Programming Language](#-the-go-programming-language) - [ - The Go Programming Language](#-the-go-programming-language) - [ - The Go Programming Language](#-the-go-programming-language) - [ - The Go Programming Language](#-the-go-programming-language) - [ - The Go Programming Language](#-the-go-programming-language) - [ - The Go Programming Language](#-the-go-programming-language) - [Unknown](#unknown) - [Unknown](#unknown) - [ - The Go Programming Language](#-the-go-programming-language) - [ - The Go Programming Language](#-the-go-programming-language) - [Gopls: Settings - The Go Programming Language](#gopls-settings-the-go-programming-language) - [ - The Go Programming Language](#-the-go-programming-language) - [Backward Compatibility, Go 1.21, and Go 2 - The Go Programming Language](#backward-compatibility-go-1-21-and-go-2-the-go-programming-language) - [ - The Go Programming Language](#-the-go-programming-language) - [ - The Go Programming Language](#-the-go-programming-language) - [ - The Go Programming Language](#-the-go-programming-language) - [Unknown](#unknown) - [Gopls: Code lenses - The Go Programming Language](#gopls-code-lenses-the-go-programming-language) - [Unknown](#unknown) - [Go 1.21 is released! - The Go Programming Language](#go-1-21-is-released-the-go-programming-language) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) --- # Effective Go - The Go Programming Language 1. [Documentation](https://go.dev/doc/) 2. [Effective Go](https://go.dev/doc/effective_go) Effective Go ============ Introduction[¶](https://go.dev/doc/effective_go#introduction) -------------------------------------------------------------- Go is a new language. Although it borrows ideas from existing languages, it has unusual properties that make effective Go programs different in character from programs written in its relatives. A straightforward translation of a C++ or Java program into Go is unlikely to produce a satisfactory result—Java programs are written in Java, not Go. On the other hand, thinking about the problem from a Go perspective could produce a successful but quite different program. In other words, to write Go well, it's important to understand its properties and idioms. It's also important to know the established conventions for programming in Go, such as naming, formatting, program construction, and so on, so that programs you write will be easy for other Go programmers to understand. This document gives tips for writing clear, idiomatic Go code. It augments the [language specification](https://go.dev/ref/spec) , the [Tour of Go](https://go.dev/tour/) , and [How to Write Go Code](https://go.dev/doc/code.html) , all of which you should read first. Note added January, 2022: This document was written for Go's release in 2009, and has not been updated significantly since. Although it is a good guide to understand how to use the language itself, thanks to the stability of the language, it says little about the libraries and nothing about significant changes to the Go ecosystem since it was written, such as the build system, testing, modules, and polymorphism. There are no plans to update it, as so much has happened and a large and growing set of documents, blogs, and books do a fine job of describing modern Go usage. Effective Go continues to be useful, but the reader should understand it is far from a complete guide. See [issue 28782](https://go.dev/issue/28782) for context. ### Examples[¶](https://go.dev/doc/effective_go#examples) The [Go package sources](https://go.dev/src/) are intended to serve not only as the core library but also as examples of how to use the language. Moreover, many of the packages contain working, self-contained executable examples you can run directly from the [go.dev](https://go.dev/) web site, such as [this one](https://go.dev/pkg/strings/#example-Map) (if necessary, click on the word "Example" to open it up). If you have a question about how to approach a problem or how something might be implemented, the documentation, code and examples in the library can provide answers, ideas and background. Formatting[¶](https://go.dev/doc/effective_go#formatting) ---------------------------------------------------------- Formatting issues are the most contentious but the least consequential. People can adapt to different formatting styles but it's better if they don't have to, and less time is devoted to the topic if everyone adheres to the same style. The problem is how to approach this Utopia without a long prescriptive style guide. With Go we take an unusual approach and let the machine take care of most formatting issues. The `gofmt` program (also available as `go fmt`, which operates at the package level rather than source file level) reads a Go program and emits the source in a standard style of indentation and vertical alignment, retaining and if necessary reformatting comments. If you want to know how to handle some new layout situation, run `gofmt`; if the answer doesn't seem right, rearrange your program (or file a bug about `gofmt`), don't work around it. As an example, there's no need to spend time lining up the comments on the fields of a structure. `Gofmt` will do that for you. Given the declaration type T struct { name string // name of the object value int // its value } `gofmt` will line up the columns: type T struct { name string // name of the object value int // its value } All Go code in the standard packages has been formatted with `gofmt`. Some formatting details remain. Very briefly: Indentation We use tabs for indentation and `gofmt` emits them by default. Use spaces only if you must. Line length Go has no line length limit. Don't worry about overflowing a punched card. If a line feels too long, wrap it and indent with an extra tab. Parentheses Go needs fewer parentheses than C and Java: control structures (`if`, `for`, `switch`) do not have parentheses in their syntax. Also, the operator precedence hierarchy is shorter and clearer, so x<<8 + y<<16 means what the spacing implies, unlike in the other languages. Commentary[¶](https://go.dev/doc/effective_go#commentary) ---------------------------------------------------------- Go provides C-style `/* */` block comments and C++-style `//` line comments. Line comments are the norm; block comments appear mostly as package comments, but are useful within an expression or to disable large swaths of code. Comments that appear before top-level declarations, with no intervening newlines, are considered to document the declaration itself. These “doc comments” are the primary documentation for a given Go package or command. For more about doc comments, see “[Go Doc Comments](https://go.dev/doc/comment) ”. Names[¶](https://go.dev/doc/effective_go#names) ------------------------------------------------ Names are as important in Go as in any other language. They even have semantic effect: the visibility of a name outside a package is determined by whether its first character is upper case. It's therefore worth spending a little time talking about naming conventions in Go programs. ### Package names[¶](https://go.dev/doc/effective_go#package-names) When a package is imported, the package name becomes an accessor for the contents. After import "bytes" the importing package can talk about `bytes.Buffer`. It's helpful if everyone using the package can use the same name to refer to its contents, which implies that the package name should be good: short, concise, evocative. By convention, packages are given lower case, single-word names; there should be no need for underscores or mixedCaps. Err on the side of brevity, since everyone using your package will be typing that name. And don't worry about collisions _a priori_. The package name is only the default name for imports; it need not be unique across all source code, and in the rare case of a collision the importing package can choose a different name to use locally. In any case, confusion is rare because the file name in the import determines just which package is being used. Another convention is that the package name is the base name of its source directory; the package in `src/encoding/base64` is imported as `"encoding/base64"` but has name `base64`, not `encoding_base64` and not `encodingBase64`. The importer of a package will use the name to refer to its contents, so exported names in the package can use that fact to avoid repetition. (Don't use the `import .` notation, which can simplify tests that must run outside the package they are testing, but should otherwise be avoided.) For instance, the buffered reader type in the `bufio` package is called `Reader`, not `BufReader`, because users see it as `bufio.Reader`, which is a clear, concise name. Moreover, because imported entities are always addressed with their package name, `bufio.Reader` does not conflict with `io.Reader`. Similarly, the function to make new instances of `ring.Ring`—which is the definition of a _constructor_ in Go—would normally be called `NewRing`, but since `Ring` is the only type exported by the package, and since the package is called `ring`, it's called just `New`, which clients of the package see as `ring.New`. Use the package structure to help you choose good names. Another short example is `once.Do`; `once.Do(setup)` reads well and would not be improved by writing `once.DoOrWaitUntilDone(setup)`. Long names don't automatically make things more readable. A helpful doc comment can often be more valuable than an extra long name. ### Getters[¶](https://go.dev/doc/effective_go#Getters) Go doesn't provide automatic support for getters and setters. There's nothing wrong with providing getters and setters yourself, and it's often appropriate to do so, but it's neither idiomatic nor necessary to put `Get` into the getter's name. If you have a field called `owner` (lower case, unexported), the getter method should be called `Owner` (upper case, exported), not `GetOwner`. The use of upper-case names for export provides the hook to discriminate the field from the method. A setter function, if needed, will likely be called `SetOwner`. Both names read well in practice: owner := obj.Owner() if owner != user { obj.SetOwner(user) } ### Interface names[¶](https://go.dev/doc/effective_go#interface-names) By convention, one-method interfaces are named by the method name plus an -er suffix or similar modification to construct an agent noun: `Reader`, `Writer`, `Formatter`, `CloseNotifier` etc. There are a number of such names and it's productive to honor them and the function names they capture. `Read`, `Write`, `Close`, `Flush`, `String` and so on have canonical signatures and meanings. To avoid confusion, don't give your method one of those names unless it has the same signature and meaning. Conversely, if your type implements a method with the same meaning as a method on a well-known type, give it the same name and signature; call your string-converter method `String` not `ToString`. ### MixedCaps[¶](https://go.dev/doc/effective_go#mixed-caps) Finally, the convention in Go is to use `MixedCaps` or `mixedCaps` rather than underscores to write multiword names. Semicolons[¶](https://go.dev/doc/effective_go#semicolons) ---------------------------------------------------------- Like C, Go's formal grammar uses semicolons to terminate statements, but unlike in C, those semicolons do not appear in the source. Instead the lexer uses a simple rule to insert semicolons automatically as it scans, so the input text is mostly free of them. The rule is this. If the last token before a newline is an identifier (which includes words like `int` and `float64`), a basic literal such as a number or string constant, or one of the tokens break continue fallthrough return ++ -- ) } the lexer always inserts a semicolon after the token. This could be summarized as, “if the newline comes after a token that could end a statement, insert a semicolon”. A semicolon can also be omitted immediately before a closing brace, so a statement such as go func() { for { dst <- <-src } }() needs no semicolons. Idiomatic Go programs have semicolons only in places such as `for` loop clauses, to separate the initializer, condition, and continuation elements. They are also necessary to separate multiple statements on a line, should you write code that way. One consequence of the semicolon insertion rules is that you cannot put the opening brace of a control structure (`if`, `for`, `switch`, or `select`) on the next line. If you do, a semicolon will be inserted before the brace, which could cause unwanted effects. Write them like this if i < f() { g() } not like this if i < f() // wrong! { // wrong! g() } Control structures[¶](https://go.dev/doc/effective_go#control-structures) -------------------------------------------------------------------------- The control structures of Go are related to those of C but differ in important ways. There is no `do` or `while` loop, only a slightly generalized `for`; `switch` is more flexible; `if` and `switch` accept an optional initialization statement like that of `for`; `break` and `continue` statements take an optional label to identify what to break or continue; and there are new control structures including a type switch and a multiway communications multiplexer, `select`. The syntax is also slightly different: there are no parentheses and the bodies must always be brace-delimited. ### If[¶](https://go.dev/doc/effective_go#if) In Go a simple `if` looks like this: if x > 0 { return y } Mandatory braces encourage writing simple `if` statements on multiple lines. It's good style to do so anyway, especially when the body contains a control statement such as a `return` or `break`. Since `if` and `switch` accept an initialization statement, it's common to see one used to set up a local variable. if err := file.Chmod(0664); err != nil { log.Print(err) return err } In the Go libraries, you'll find that when an `if` statement doesn't flow into the next statement—that is, the body ends in `break`, `continue`, `goto`, or `return`—the unnecessary `else` is omitted. f, err := os.Open(name) if err != nil { return err } codeUsing(f) This is an example of a common situation where code must guard against a sequence of error conditions. The code reads well if the successful flow of control runs down the page, eliminating error cases as they arise. Since error cases tend to end in `return` statements, the resulting code needs no `else` statements. f, err := os.Open(name) if err != nil { return err } d, err := f.Stat() if err != nil { f.Close() return err } codeUsing(f, d) ### Redeclaration and reassignment[¶](https://go.dev/doc/effective_go#redeclaration) An aside: The last example in the previous section demonstrates a detail of how the `:=` short declaration form works. The declaration that calls `os.Open` reads, f, err := os.Open(name) This statement declares two variables, `f` and `err`. A few lines later, the call to `f.Stat` reads, d, err := f.Stat() which looks as if it declares `d` and `err`. Notice, though, that `err` appears in both statements. This duplication is legal: `err` is declared by the first statement, but only _re-assigned_ in the second. This means that the call to `f.Stat` uses the existing `err` variable declared above, and just gives it a new value. In a `:=` declaration a variable `v` may appear even if it has already been declared, provided: * this declaration is in the same scope as the existing declaration of `v` (if `v` is already declared in an outer scope, the declaration will create a new variable §), * the corresponding value in the initialization is assignable to `v`, and * there is at least one other variable that is created by the declaration. This unusual property is pure pragmatism, making it easy to use a single `err` value, for example, in a long `if-else` chain. You'll see it used often. § It's worth noting here that in Go the scope of function parameters and return values is the same as the function body, even though they appear lexically outside the braces that enclose the body. ### For[¶](https://go.dev/doc/effective_go#for) The Go `for` loop is similar to—but not the same as—C's. It unifies `for` and `while` and there is no `do-while`. There are three forms, only one of which has semicolons. // Like a C for for init; condition; post { } // Like a C while for condition { } // Like a C for(;;) for { } Short declarations make it easy to declare the index variable right in the loop. sum := 0 for i := 0; i < 10; i++ { sum += i } If you're looping over an array, slice, string, or map, or reading from a channel, a `range` clause can manage the loop. for key, value := range oldMap { newMap\[key\] = value } If you only need the first item in the range (the key or index), drop the second: for key := range m { if key.expired() { delete(m, key) } } If you only need the second item in the range (the value), use the _blank identifier_, an underscore, to discard the first: sum := 0 for \_, value := range array { sum += value } The blank identifier has many uses, as described in [a later section](https://go.dev/doc/effective_go#blank) . For strings, the `range` does more work for you, breaking out individual Unicode code points by parsing the UTF-8. Erroneous encodings consume one byte and produce the replacement rune U+FFFD. (The name (with associated builtin type) `rune` is Go terminology for a single Unicode code point. See [the language specification](https://go.dev/ref/spec#Rune_literals) for details.) The loop for pos, char := range "日本\\x80語" { // \\x80 is an illegal UTF-8 encoding fmt.Printf("character %#U starts at byte position %d\\n", char, pos) } prints character U+65E5 '日' starts at byte position 0 character U+672C '本' starts at byte position 3 character U+FFFD '�' starts at byte position 6 character U+8A9E '語' starts at byte position 7 Finally, Go has no comma operator and `++` and `--` are statements not expressions. Thus if you want to run multiple variables in a `for` you should use parallel assignment (although that precludes `++` and `--`). // Reverse a for i, j := 0, len(a)-1; i < j; i, j = i+1, j-1 { a\[i\], a\[j\] = a\[j\], a\[i\] } ### Switch[¶](https://go.dev/doc/effective_go#switch) Go's `switch` is more general than C's. The expressions need not be constants or even integers, the cases are evaluated top to bottom until a match is found, and if the `switch` has no expression it switches on `true`. It's therefore possible—and idiomatic—to write an `if`\-`else`\-`if`\-`else` chain as a `switch`. func unhex(c byte) byte { switch { case '0' <= c && c <= '9': return c - '0' case 'a' <= c && c <= 'f': return c - 'a' + 10 case 'A' <= c && c <= 'F': return c - 'A' + 10 } return 0 } There is no automatic fall through, but cases can be presented in comma-separated lists. func shouldEscape(c byte) bool { switch c { case ' ', '?', '&', '=', '#', '+', '%': return true } return false } Although they are not nearly as common in Go as some other C-like languages, `break` statements can be used to terminate a `switch` early. Sometimes, though, it's necessary to break out of a surrounding loop, not the switch, and in Go that can be accomplished by putting a label on the loop and "breaking" to that label. This example shows both uses. Loop: for n := 0; n < len(src); n += size { switch { case src\[n\] < sizeOne: if validateOnly { break } size = 1 update(src\[n\]) case src\[n\] < sizeTwo: if n+1 >= len(src) { err = errShortInput break Loop } if validateOnly { break } size = 2 update(src\[n\] + src\[n+1\]< b func Compare(a, b \[\]byte) int { for i := 0; i < len(a) && i < len(b); i++ { switch { case a\[i\] > b\[i\]: return 1 case a\[i\] < b\[i\]: return -1 } } switch { case len(a) > len(b): return 1 case len(a) < len(b): return -1 } return 0 } ### Type switch[¶](https://go.dev/doc/effective_go#type_switch) A switch can also be used to discover the dynamic type of an interface variable. Such a _type switch_ uses the syntax of a type assertion with the keyword `type` inside the parentheses. If the switch declares a variable in the expression, the variable will have the corresponding type in each clause. It's also idiomatic to reuse the name in such cases, in effect declaring a new variable with the same name but a different type in each case. var t interface{} t = functionOfSomeType() switch t := t.(type) { default: fmt.Printf("unexpected type %T\\n", t) // %T prints whatever type t has case bool: fmt.Printf("boolean %t\\n", t) // t has type bool case int: fmt.Printf("integer %d\\n", t) // t has type int case \*bool: fmt.Printf("pointer to boolean %t\\n", \*t) // t has type \*bool case \*int: fmt.Printf("pointer to integer %d\\n", \*t) // t has type \*int } Functions[¶](https://go.dev/doc/effective_go#functions) -------------------------------------------------------- ### Multiple return values[¶](https://go.dev/doc/effective_go#multiple-returns) One of Go's unusual features is that functions and methods can return multiple values. This form can be used to improve on a couple of clumsy idioms in C programs: in-band error returns such as `-1` for `EOF` and modifying an argument passed by address. In C, a write error is signaled by a negative count with the error code secreted away in a volatile location. In Go, `Write` can return a count _and_ an error: “Yes, you wrote some bytes but not all of them because you filled the device”. The signature of the `Write` method on files from package `os` is: func (file \*File) Write(b \[\]byte) (n int, err error) and as the documentation says, it returns the number of bytes written and a non-nil `error` when `n` `!=` `len(b)`. This is a common style; see the section on error handling for more examples. A similar approach obviates the need to pass a pointer to a return value to simulate a reference parameter. Here's a simple-minded function to grab a number from a position in a byte slice, returning the number and the next position. func nextInt(b \[\]byte, i int) (int, int) { for ; i < len(b) && !isDigit(b\[i\]); i++ { } x := 0 for ; i < len(b) && isDigit(b\[i\]); i++ { x = x\*10 + int(b\[i\]) - '0' } return x, i } You could use it to scan the numbers in an input slice `b` like this: for i := 0; i < len(b); { x, i = nextInt(b, i) fmt.Println(x) } ### Named result parameters[¶](https://go.dev/doc/effective_go#named-results) The return or result "parameters" of a Go function can be given names and used as regular variables, just like the incoming parameters. When named, they are initialized to the zero values for their types when the function begins; if the function executes a `return` statement with no arguments, the current values of the result parameters are used as the returned values. The names are not mandatory but they can make code shorter and clearer: they're documentation. If we name the results of `nextInt` it becomes obvious which returned `int` is which. func nextInt(b \[\]byte, pos int) (value, nextPos int) { Because named results are initialized and tied to an unadorned return, they can simplify as well as clarify. Here's a version of `io.ReadFull` that uses them well: func ReadFull(r Reader, buf \[\]byte) (n int, err error) { for len(buf) > 0 && err == nil { var nr int nr, err = r.Read(buf) n += nr buf = buf\[nr:\] } return } ### Defer[¶](https://go.dev/doc/effective_go#defer) Go's `defer` statement schedules a function call (the _deferred_ function) to be run immediately before the function executing the `defer` returns. It's an unusual but effective way to deal with situations such as resources that must be released regardless of which path a function takes to return. The canonical examples are unlocking a mutex or closing a file. // Contents returns the file's contents as a string. func Contents(filename string) (string, error) { f, err := os.Open(filename) if err != nil { return "", err } defer f.Close() // f.Close will run when we're finished. var result \[\]byte buf := make(\[\]byte, 100) for { n, err := f.Read(buf\[0:\]) result = append(result, buf\[0:n\]...) // append is discussed later. if err != nil { if err == io.EOF { break } return "", err // f will be closed if we return here. } } return string(result), nil // f will be closed if we return here. } Deferring a call to a function such as `Close` has two advantages. First, it guarantees that you will never forget to close the file, a mistake that's easy to make if you later edit the function to add a new return path. Second, it means that the close sits near the open, which is much clearer than placing it at the end of the function. The arguments to the deferred function (which include the receiver if the function is a method) are evaluated when the _defer_ executes, not when the _call_ executes. Besides avoiding worries about variables changing values as the function executes, this means that a single deferred call site can defer multiple function executions. Here's a silly example. for i := 0; i < 5; i++ { defer fmt.Printf("%d ", i) } Deferred functions are executed in LIFO order, so this code will cause `4 3 2 1 0` to be printed when the function returns. A more plausible example is a simple way to trace function execution through the program. We could write a couple of simple tracing routines like this: func trace(s string) { fmt.Println("entering:", s) } func untrace(s string) { fmt.Println("leaving:", s) } // Use them like this: func a() { trace("a") defer untrace("a") // do something.... } We can do better by exploiting the fact that arguments to deferred functions are evaluated when the `defer` executes. The tracing routine can set up the argument to the untracing routine. This example: func trace(s string) string { fmt.Println("entering:", s) return s } func un(s string) { fmt.Println("leaving:", s) } func a() { defer un(trace("a")) fmt.Println("in a") } func b() { defer un(trace("b")) fmt.Println("in b") a() } func main() { b() } prints entering: b in b entering: a in a leaving: a leaving: b For programmers accustomed to block-level resource management from other languages, `defer` may seem peculiar, but its most interesting and powerful applications come precisely from the fact that it's not block-based but function-based. In the section on `panic` and `recover` we'll see another example of its possibilities. Data[¶](https://go.dev/doc/effective_go#data) ---------------------------------------------- ### Allocation with `new`[¶](https://go.dev/doc/effective_go#allocation_new) Go has two allocation primitives, the built-in functions `new` and `make`. They do different things and apply to different types, which can be confusing, but the rules are simple. Let's talk about `new` first. It's a built-in function that allocates memory, but unlike its namesakes in some other languages it does not _initialize_ the memory, it only _zeros_ it. That is, `new(T)` allocates zeroed storage for a new item of type `T` and returns its address, a value of type `*T`. In Go terminology, it returns a pointer to a newly allocated zero value of type `T`. Since the memory returned by `new` is zeroed, it's helpful to arrange when designing your data structures that the zero value of each type can be used without further initialization. This means a user of the data structure can create one with `new` and get right to work. For example, the documentation for `bytes.Buffer` states that "the zero value for `Buffer` is an empty buffer ready to use." Similarly, `sync.Mutex` does not have an explicit constructor or `Init` method. Instead, the zero value for a `sync.Mutex` is defined to be an unlocked mutex. The zero-value-is-useful property works transitively. Consider this type declaration. type SyncedBuffer struct { lock sync.Mutex buffer bytes.Buffer } Values of type `SyncedBuffer` are also ready to use immediately upon allocation or just declaration. In the next snippet, both `p` and `v` will work correctly without further arrangement. p := new(SyncedBuffer) // type \*SyncedBuffer var v SyncedBuffer // type SyncedBuffer ### Constructors and composite literals[¶](https://go.dev/doc/effective_go#composite_literals) Sometimes the zero value isn't good enough and an initializing constructor is necessary, as in this example derived from package `os`. func NewFile(fd int, name string) \*File { if fd < 0 { return nil } f := new(File) f.fd = fd f.name = name f.dirinfo = nil f.nepipe = 0 return f } There's a lot of boilerplate in there. We can simplify it using a _composite literal_, which is an expression that creates a new instance each time it is evaluated. func NewFile(fd int, name string) \*File { if fd < 0 { return nil } f := File{fd, name, nil, 0} return &f } Note that, unlike in C, it's perfectly OK to return the address of a local variable; the storage associated with the variable survives after the function returns. In fact, taking the address of a composite literal allocates a fresh instance each time it is evaluated, so we can combine these last two lines. return &File{fd, name, nil, 0} The fields of a composite literal are laid out in order and must all be present. However, by labeling the elements explicitly as _field_`:`_value_ pairs, the initializers can appear in any order, with the missing ones left as their respective zero values. Thus we could say return &File{fd: fd, name: name} As a limiting case, if a composite literal contains no fields at all, it creates a zero value for the type. The expressions `new(File)` and `&File{}` are equivalent. Composite literals can also be created for arrays, slices, and maps, with the field labels being indices or map keys as appropriate. In these examples, the initializations work regardless of the values of `Enone`, `Eio`, and `Einval`, as long as they are distinct. a := \[...\]string {Enone: "no error", Eio: "Eio", Einval: "invalid argument"} s := \[\]string {Enone: "no error", Eio: "Eio", Einval: "invalid argument"} m := map\[int\]string{Enone: "no error", Eio: "Eio", Einval: "invalid argument"} ### Allocation with `make`[¶](https://go.dev/doc/effective_go#allocation_make) Back to allocation. The built-in function `make(T,` _args_`)` serves a purpose different from `new(T)`. It creates slices, maps, and channels only, and it returns an _initialized_ (not _zeroed_) value of type `T` (not `*T`). The reason for the distinction is that these three types represent, under the covers, references to data structures that must be initialized before use. A slice, for example, is a three-item descriptor containing a pointer to the data (inside an array), the length, and the capacity, and until those items are initialized, the slice is `nil`. For slices, maps, and channels, `make` initializes the internal data structure and prepares the value for use. For instance, make(\[\]int, 10, 100) allocates an array of 100 ints and then creates a slice structure with length 10 and a capacity of 100 pointing at the first 10 elements of the array. (When making a slice, the capacity can be omitted; see the section on slices for more information.) In contrast, `new([]int)` returns a pointer to a newly allocated, zeroed slice structure, that is, a pointer to a `nil` slice value. These examples illustrate the difference between `new` and `make`. var p \*\[\]int = new(\[\]int) // allocates slice structure; \*p == nil; rarely useful var v \[\]int = make(\[\]int, 100) // the slice v now refers to a new array of 100 ints // Unnecessarily complex: var p \*\[\]int = new(\[\]int) \*p = make(\[\]int, 100, 100) // Idiomatic: v := make(\[\]int, 100) Remember that `make` applies only to maps, slices and channels and does not return a pointer. To obtain an explicit pointer allocate with `new` or take the address of a variable explicitly. ### Arrays[¶](https://go.dev/doc/effective_go#arrays) Arrays are useful when planning the detailed layout of memory and sometimes can help avoid allocation, but primarily they are a building block for slices, the subject of the next section. To lay the foundation for that topic, here are a few words about arrays. There are major differences between the ways arrays work in Go and C. In Go, * Arrays are values. Assigning one array to another copies all the elements. * In particular, if you pass an array to a function, it will receive a _copy_ of the array, not a pointer to it. * The size of an array is part of its type. The types `[10]int` and `[20]int` are distinct. The value property can be useful but also expensive; if you want C-like behavior and efficiency, you can pass a pointer to the array. func Sum(a \*\[3\]float64) (sum float64) { for \_, v := range \*a { sum += v } return } array := \[...\]float64{7.0, 8.5, 9.1} x := Sum(&array) // Note the explicit address-of operator But even this style isn't idiomatic Go. Use slices instead. ### Slices[¶](https://go.dev/doc/effective_go#slices) Slices wrap arrays to give a more general, powerful, and convenient interface to sequences of data. Except for items with explicit dimension such as transformation matrices, most array programming in Go is done with slices rather than simple arrays. Slices hold references to an underlying array, and if you assign one slice to another, both refer to the same array. If a function takes a slice argument, changes it makes to the elements of the slice will be visible to the caller, analogous to passing a pointer to the underlying array. A `Read` function can therefore accept a slice argument rather than a pointer and a count; the length within the slice sets an upper limit of how much data to read. Here is the signature of the `Read` method of the `File` type in package `os`: func (f \*File) Read(buf \[\]byte) (n int, err error) The method returns the number of bytes read and an error value, if any. To read into the first 32 bytes of a larger buffer `buf`, _slice_ (here used as a verb) the buffer. n, err := f.Read(buf\[0:32\]) Such slicing is common and efficient. In fact, leaving efficiency aside for the moment, the following snippet would also read the first 32 bytes of the buffer. var n int var err error for i := 0; i < 32; i++ { nbytes, e := f.Read(buf\[i:i+1\]) // Read one byte. n += nbytes if nbytes == 0 || e != nil { err = e break } } The length of a slice may be changed as long as it still fits within the limits of the underlying array; just assign it to a slice of itself. The _capacity_ of a slice, accessible by the built-in function `cap`, reports the maximum length the slice may assume. Here is a function to append data to a slice. If the data exceeds the capacity, the slice is reallocated. The resulting slice is returned. The function uses the fact that `len` and `cap` are legal when applied to the `nil` slice, and return 0. func Append(slice, data \[\]byte) \[\]byte { l := len(slice) if l + len(data) > cap(slice) { // reallocate // Allocate double what's needed, for future growth. newSlice := make(\[\]byte, (l+len(data))\*2) // The copy function is predeclared and works for any slice type. copy(newSlice, slice) slice = newSlice } slice = slice\[0:l+len(data)\] copy(slice\[l:\], data) return slice } We must return the slice afterwards because, although `Append` can modify the elements of `slice`, the slice itself (the run-time data structure holding the pointer, length, and capacity) is passed by value. The idea of appending to a slice is so useful it's captured by the `append` built-in function. To understand that function's design, though, we need a little more information, so we'll return to it later. ### Two-dimensional slices[¶](https://go.dev/doc/effective_go#two_dimensional_slices) Go's arrays and slices are one-dimensional. To create the equivalent of a 2D array or slice, it is necessary to define an array-of-arrays or slice-of-slices, like this: type Transform \[3\]\[3\]float64 // A 3x3 array, really an array of arrays. type LinesOfText \[\]\[\]byte // A slice of byte slices. Because slices are variable-length, it is possible to have each inner slice be a different length. That can be a common situation, as in our `LinesOfText` example: each line has an independent length. text := LinesOfText{ \[\]byte("Now is the time"), \[\]byte("for all good gophers"), \[\]byte("to bring some fun to the party."), } Sometimes it's necessary to allocate a 2D slice, a situation that can arise when processing scan lines of pixels, for instance. There are two ways to achieve this. One is to allocate each slice independently; the other is to allocate a single array and point the individual slices into it. Which to use depends on your application. If the slices might grow or shrink, they should be allocated independently to avoid overwriting the next line; if not, it can be more efficient to construct the object with a single allocation. For reference, here are sketches of the two methods. First, a line at a time: // Allocate the top-level slice. picture := make(\[\]\[\]uint8, YSize) // One row per unit of y. // Loop over the rows, allocating the slice for each row. for i := range picture { picture\[i\] = make(\[\]uint8, XSize) } And now as one allocation, sliced into lines: // Allocate the top-level slice, the same as before. picture := make(\[\]\[\]uint8, YSize) // One row per unit of y. // Allocate one large slice to hold all the pixels. pixels := make(\[\]uint8, XSize\*YSize) // Has type \[\]uint8 even though picture is \[\]\[\]uint8. // Loop over the rows, slicing each row from the front of the remaining pixels slice. for i := range picture { picture\[i\], pixels = pixels\[:XSize\], pixels\[XSize:\] } ### Maps[¶](https://go.dev/doc/effective_go#maps) Maps are a convenient and powerful built-in data structure that associate values of one type (the _key_) with values of another type (the _element_ or _value_). The key can be of any type for which the equality operator is defined, such as integers, floating point and complex numbers, strings, pointers, interfaces (as long as the dynamic type supports equality), structs and arrays. Slices cannot be used as map keys, because equality is not defined on them. Like slices, maps hold references to an underlying data structure. If you pass a map to a function that changes the contents of the map, the changes will be visible in the caller. Maps can be constructed using the usual composite literal syntax with colon-separated key-value pairs, so it's easy to build them during initialization. var timeZone = map\[string\]int{ "UTC": 0\*60\*60, "EST": -5\*60\*60, "CST": -6\*60\*60, "MST": -7\*60\*60, "PST": -8\*60\*60, } Assigning and fetching map values looks syntactically just like doing the same for arrays and slices except that the index doesn't need to be an integer. offset := timeZone\["EST"\] An attempt to fetch a map value with a key that is not present in the map will return the zero value for the type of the entries in the map. For instance, if the map contains integers, looking up a non-existent key will return `0`. A set can be implemented as a map with value type `bool`. Set the map entry to `true` to put the value in the set, and then test it by simple indexing. attended := map\[string\]bool{ "Ann": true, "Joe": true, ... } if attended\[person\] { // will be false if person is not in the map fmt.Println(person, "was at the meeting") } Sometimes you need to distinguish a missing entry from a zero value. Is there an entry for `"UTC"` or is that 0 because it's not in the map at all? You can discriminate with a form of multiple assignment. var seconds int var ok bool seconds, ok = timeZone\[tz\] For obvious reasons this is called the “comma ok” idiom. In this example, if `tz` is present, `seconds` will be set appropriately and `ok` will be true; if not, `seconds` will be set to zero and `ok` will be false. Here's a function that puts it together with a nice error report: func offset(tz string) int { if seconds, ok := timeZone\[tz\]; ok { return seconds } log.Println("unknown time zone:", tz) return 0 } To test for presence in the map without worrying about the actual value, you can use the [blank identifier](https://go.dev/doc/effective_go#blank) (`_`) in place of the usual variable for the value. \_, present := timeZone\[tz\] To delete a map entry, use the `delete` built-in function, whose arguments are the map and the key to be deleted. It's safe to do this even if the key is already absent from the map. delete(timeZone, "PDT") // Now on Standard Time ### Printing[¶](https://go.dev/doc/effective_go#printing) Formatted printing in Go uses a style similar to C's `printf` family but is richer and more general. The functions live in the `fmt` package and have capitalized names: `fmt.Printf`, `fmt.Fprintf`, `fmt.Sprintf` and so on. The string functions (`Sprintf` etc.) return a string rather than filling in a provided buffer. You don't need to provide a format string. For each of `Printf`, `Fprintf` and `Sprintf` there is another pair of functions, for instance `Print` and `Println`. These functions do not take a format string but instead generate a default format for each argument. The `Println` versions also insert a blank between arguments and append a newline to the output while the `Print` versions add blanks only if the operand on neither side is a string. In this example each line produces the same output. fmt.Printf("Hello %d\\n", 23) fmt.Fprint(os.Stdout, "Hello ", 23, "\\n") fmt.Println("Hello", 23) fmt.Println(fmt.Sprint("Hello ", 23)) The formatted print functions `fmt.Fprint` and friends take as a first argument any object that implements the `io.Writer` interface; the variables `os.Stdout` and `os.Stderr` are familiar instances. Here things start to diverge from C. First, the numeric formats such as `%d` do not take flags for signedness or size; instead, the printing routines use the type of the argument to decide these properties. var x uint64 = 1<<64 - 1 fmt.Printf("%d %x; %d %x\\n", x, x, int64(x), int64(x)) prints 18446744073709551615 ffffffffffffffff; -1 -1 If you just want the default conversion, such as decimal for integers, you can use the catchall format `%v` (for “value”); the result is exactly what `Print` and `Println` would produce. Moreover, that format can print _any_ value, even arrays, slices, structs, and maps. Here is a print statement for the time zone map defined in the previous section. fmt.Printf("%v\\n", timeZone) // or just fmt.Println(timeZone) which gives output: map\[CST:-21600 EST:-18000 MST:-25200 PST:-28800 UTC:0\] For maps, `Printf` and friends sort the output lexicographically by key. When printing a struct, the modified format `%+v` annotates the fields of the structure with their names, and for any value the alternate format `%#v` prints the value in full Go syntax. type T struct { a int b float64 c string } t := &T{ 7, -2.35, "abc\\tdef" } fmt.Printf("%v\\n", t) fmt.Printf("%+v\\n", t) fmt.Printf("%#v\\n", t) fmt.Printf("%#v\\n", timeZone) prints &{7 -2.35 abc def} &{a:7 b:-2.35 c:abc def} &main.T{a:7, b:-2.35, c:"abc\\tdef"} map\[string\]int{"CST":-21600, "EST":-18000, "MST":-25200, "PST":-28800, "UTC":0} (Note the ampersands.) That quoted string format is also available through `%q` when applied to a value of type `string` or `[]byte`. The alternate format `%#q` will use backquotes instead if possible. (The `%q` format also applies to integers and runes, producing a single-quoted rune constant.) Also, `%x` works on strings, byte arrays and byte slices as well as on integers, generating a long hexadecimal string, and with a space in the format (`% x`) it puts spaces between the bytes. Another handy format is `%T`, which prints the _type_ of a value. fmt.Printf("%T\\n", timeZone) prints map\[string\]int If you want to control the default format for a custom type, all that's required is to define a method with the signature `String() string` on the type. For our simple type `T`, that might look like this. func (t \*T) String() string { return fmt.Sprintf("%d/%g/%q", t.a, t.b, t.c) } fmt.Printf("%v\\n", t) to print in the format 7/-2.35/"abc\\tdef" (If you need to print _values_ of type `T` as well as pointers to `T`, the receiver for `String` must be of value type; this example used a pointer because that's more efficient and idiomatic for struct types. See the section below on [pointers vs. value receivers](https://go.dev/doc/effective_go#pointers_vs_values) for more information.) Our `String` method is able to call `Sprintf` because the print routines are fully reentrant and can be wrapped this way. There is one important detail to understand about this approach, however: don't construct a `String` method by calling `Sprintf` in a way that will recur into your `String` method indefinitely. This can happen if the `Sprintf` call attempts to print the receiver directly as a string, which in turn will invoke the method again. It's a common and easy mistake to make, as this example shows. type MyString string func (m MyString) String() string { return fmt.Sprintf("MyString=%s", m) // Error: will recur forever. } It's also easy to fix: convert the argument to the basic string type, which does not have the method. type MyString string func (m MyString) String() string { return fmt.Sprintf("MyString=%s", string(m)) // OK: note conversion. } In the [initialization section](https://go.dev/doc/effective_go#initialization) we'll see another technique that avoids this recursion. Another printing technique is to pass a print routine's arguments directly to another such routine. The signature of `Printf` uses the type `...interface{}` for its final argument to specify that an arbitrary number of parameters (of arbitrary type) can appear after the format. func Printf(format string, v ...interface{}) (n int, err error) { Within the function `Printf`, `v` acts like a variable of type `[]interface{}` but if it is passed to another variadic function, it acts like a regular list of arguments. Here is the implementation of the function `log.Println` we used above. It passes its arguments directly to `fmt.Sprintln` for the actual formatting. // Println prints to the standard logger in the manner of fmt.Println. func Println(v ...interface{}) { std.Output(2, fmt.Sprintln(v...)) // Output takes parameters (int, string) } We write `...` after `v` in the nested call to `Sprintln` to tell the compiler to treat `v` as a list of arguments; otherwise it would just pass `v` as a single slice argument. There's even more to printing than we've covered here. See the `godoc` documentation for package `fmt` for the details. By the way, a `...` parameter can be of a specific type, for instance `...int` for a min function that chooses the least of a list of integers: func Min(a ...int) int { min := int(^uint(0) >> 1) // largest int for \_, i := range a { if i < min { min = i } } return min } ### Append[¶](https://go.dev/doc/effective_go#append) Now we have the missing piece we needed to explain the design of the `append` built-in function. The signature of `append` is different from our custom `Append` function above. Schematically, it's like this: func append(slice \[\]_T_, elements ..._T_) \[\]_T_ where _T_ is a placeholder for any given type. You can't actually write a function in Go where the type `T` is determined by the caller. That's why `append` is built in: it needs support from the compiler. What `append` does is append the elements to the end of the slice and return the result. The result needs to be returned because, as with our hand-written `Append`, the underlying array may change. This simple example x := \[\]int{1,2,3} x = append(x, 4, 5, 6) fmt.Println(x) prints `[1 2 3 4 5 6]`. So `append` works a little like `Printf`, collecting an arbitrary number of arguments. But what if we wanted to do what our `Append` does and append a slice to a slice? Easy: use `...` at the call site, just as we did in the call to `Output` above. This snippet produces identical output to the one above. x := \[\]int{1,2,3} y := \[\]int{4,5,6} x = append(x, y...) fmt.Println(x) Without that `...`, it wouldn't compile because the types would be wrong; `y` is not of type `int`. Initialization[¶](https://go.dev/doc/effective_go#initialization) ------------------------------------------------------------------ Although it doesn't look superficially very different from initialization in C or C++, initialization in Go is more powerful. Complex structures can be built during initialization and the ordering issues among initialized objects, even among different packages, are handled correctly. ### Constants[¶](https://go.dev/doc/effective_go#constants) Constants in Go are just that—constant. They are created at compile time, even when defined as locals in functions, and can only be numbers, characters (runes), strings or booleans. Because of the compile-time restriction, the expressions that define them must be constant expressions, evaluatable by the compiler. For instance, `1<<3` is a constant expression, while `math.Sin(math.Pi/4)` is not because the function call to `math.Sin` needs to happen at run time. In Go, enumerated constants are created using the `iota` enumerator. Since `iota` can be part of an expression and expressions can be implicitly repeated, it is easy to build intricate sets of values. type ByteSize float64 const ( \_ = iota // ignore first value by assigning to blank identifier KB ByteSize = 1 << (10 \* iota) MB GB TB PB EB ZB YB ) The ability to attach a method such as `String` to any user-defined type makes it possible for arbitrary values to format themselves automatically for printing. Although you'll see it most often applied to structs, this technique is also useful for scalar types such as floating-point types like `ByteSize`. func (b ByteSize) String() string { switch { case b >= YB: return fmt.Sprintf("%.2fYB", b/YB) case b >= ZB: return fmt.Sprintf("%.2fZB", b/ZB) case b >= EB: return fmt.Sprintf("%.2fEB", b/EB) case b >= PB: return fmt.Sprintf("%.2fPB", b/PB) case b >= TB: return fmt.Sprintf("%.2fTB", b/TB) case b >= GB: return fmt.Sprintf("%.2fGB", b/GB) case b >= MB: return fmt.Sprintf("%.2fMB", b/MB) case b >= KB: return fmt.Sprintf("%.2fKB", b/KB) } return fmt.Sprintf("%.2fB", b) } The expression `YB` prints as `1.00YB`, while `ByteSize(1e13)` prints as `9.09TB`. The use here of `Sprintf` to implement `ByteSize`'s `String` method is safe (avoids recurring indefinitely) not because of a conversion but because it calls `Sprintf` with `%f`, which is not a string format: `Sprintf` will only call the `String` method when it wants a string, and `%f` wants a floating-point value. ### Variables[¶](https://go.dev/doc/effective_go#variables) Variables can be initialized just like constants but the initializer can be a general expression computed at run time. var ( home = os.Getenv("HOME") user = os.Getenv("USER") gopath = os.Getenv("GOPATH") ) ### The init function[¶](https://go.dev/doc/effective_go#init) Finally, each source file can define its own niladic `init` function to set up whatever state is required. (Actually each file can have multiple `init` functions.) And finally means finally: `init` is called after all the variable declarations in the package have evaluated their initializers, and those are evaluated only after all the imported packages have been initialized. Besides initializations that cannot be expressed as declarations, a common use of `init` functions is to verify or repair correctness of the program state before real execution begins. func init() { if user == "" { log.Fatal("$USER not set") } if home == "" { home = "/home/" + user } if gopath == "" { gopath = home + "/go" } // gopath may be overridden by --gopath flag on command line. flag.StringVar(&gopath, "gopath", gopath, "override default GOPATH") } Methods[¶](https://go.dev/doc/effective_go#methods) ---------------------------------------------------- ### Pointers vs. Values[¶](https://go.dev/doc/effective_go#pointers_vs_values) As we saw with `ByteSize`, methods can be defined for any named type (except a pointer or an interface); the receiver does not have to be a struct. In the discussion of slices above, we wrote an `Append` function. We can define it as a method on slices instead. To do this, we first declare a named type to which we can bind the method, and then make the receiver for the method a value of that type. type ByteSlice \[\]byte func (slice ByteSlice) Append(data \[\]byte) \[\]byte { // Body exactly the same as the Append function defined above. } This still requires the method to return the updated slice. We can eliminate that clumsiness by redefining the method to take a _pointer_ to a `ByteSlice` as its receiver, so the method can overwrite the caller's slice. func (p \*ByteSlice) Append(data \[\]byte) { slice := \*p // Body as above, without the return. \*p = slice } In fact, we can do even better. If we modify our function so it looks like a standard `Write` method, like this, func (p \*ByteSlice) Write(data \[\]byte) (n int, err error) { slice := \*p // Again as above. \*p = slice return len(data), nil } then the type `*ByteSlice` satisfies the standard interface `io.Writer`, which is handy. For instance, we can print into one. var b ByteSlice fmt.Fprintf(&b, "This hour has %d days\\n", 7) We pass the address of a `ByteSlice` because only `*ByteSlice` satisfies `io.Writer`. The rule about pointers vs. values for receivers is that value methods can be invoked on pointers and values, but pointer methods can only be invoked on pointers. This rule arises because pointer methods can modify the receiver; invoking them on a value would cause the method to receive a copy of the value, so any modifications would be discarded. The language therefore disallows this mistake. There is a handy exception, though. When the value is addressable, the language takes care of the common case of invoking a pointer method on a value by inserting the address operator automatically. In our example, the variable `b` is addressable, so we can call its `Write` method with just `b.Write`. The compiler will rewrite that to `(&b).Write` for us. By the way, the idea of using `Write` on a slice of bytes is central to the implementation of `bytes.Buffer`. Interfaces and other types[¶](https://go.dev/doc/effective_go#interfaces_and_types) ------------------------------------------------------------------------------------ ### Interfaces[¶](https://go.dev/doc/effective_go#interfaces) Interfaces in Go provide a way to specify the behavior of an object: if something can do _this_, then it can be used _here_. We've seen a couple of simple examples already; custom printers can be implemented by a `String` method while `Fprintf` can generate output to anything with a `Write` method. Interfaces with only one or two methods are common in Go code, and are usually given a name derived from the method, such as `io.Writer` for something that implements `Write`. A type can implement multiple interfaces. For instance, a collection can be sorted by the routines in package `sort` if it implements `sort.Interface`, which contains `Len()`, `Less(i, j int) bool`, and `Swap(i, j int)`, and it could also have a custom formatter. In this contrived example `Sequence` satisfies both. type Sequence \[\]int // Methods required by sort.Interface. func (s Sequence) Len() int { return len(s) } func (s Sequence) Less(i, j int) bool { return s\[i\] < s\[j\] } func (s Sequence) Swap(i, j int) { s\[i\], s\[j\] = s\[j\], s\[i\] } // Copy returns a copy of the Sequence. func (s Sequence) Copy() Sequence { copy := make(Sequence, 0, len(s)) return append(copy, s...) } // Method for printing - sorts the elements before printing. func (s Sequence) String() string { s = s.Copy() // Make a copy; don't overwrite argument. sort.Sort(s) str := "\["\ for i, elem := range s { // Loop is O(N²); will fix that in next example.\ if i > 0 {\ str += " "\ }\ str += fmt.Sprint(elem)\ }\ return str + "\]" } ### Conversions[¶](https://go.dev/doc/effective_go#conversions) The `String` method of `Sequence` is recreating the work that `Sprint` already does for slices. (It also has complexity O(N²), which is poor.) We can share the effort (and also speed it up) if we convert the `Sequence` to a plain `[]int` before calling `Sprint`. func (s Sequence) String() string { s = s.Copy() sort.Sort(s) return fmt.Sprint(\[\]int(s)) } This method is another example of the conversion technique for calling `Sprintf` safely from a `String` method. Because the two types (`Sequence` and `[]int`) are the same if we ignore the type name, it's legal to convert between them. The conversion doesn't create a new value, it just temporarily acts as though the existing value has a new type. (There are other legal conversions, such as from integer to floating point, that do create a new value.) It's an idiom in Go programs to convert the type of an expression to access a different set of methods. As an example, we could use the existing type `sort.IntSlice` to reduce the entire example to this: type Sequence \[\]int // Method for printing - sorts the elements before printing func (s Sequence) String() string { s = s.Copy() sort.IntSlice(s).Sort() return fmt.Sprint(\[\]int(s)) } Now, instead of having `Sequence` implement multiple interfaces (sorting and printing), we're using the ability of a data item to be converted to multiple types (`Sequence`, `sort.IntSlice` and `[]int`), each of which does some part of the job. That's more unusual in practice but can be effective. ### Interface conversions and type assertions[¶](https://go.dev/doc/effective_go#interface_conversions) [Type switches](https://go.dev/doc/effective_go#type_switch) are a form of conversion: they take an interface and, for each case in the switch, in a sense convert it to the type of that case. Here's a simplified version of how the code under `fmt.Printf` turns a value into a string using a type switch. If it's already a string, we want the actual string value held by the interface, while if it has a `String` method we want the result of calling the method. type Stringer interface { String() string } var value interface{} // Value provided by caller. switch str := value.(type) { case string: return str case Stringer: return str.String() } The first case finds a concrete value; the second converts the interface into another interface. It's perfectly fine to mix types this way. What if there's only one type we care about? If we know the value holds a `string` and we just want to extract it? A one-case type switch would do, but so would a _type assertion_. A type assertion takes an interface value and extracts from it a value of the specified explicit type. The syntax borrows from the clause opening a type switch, but with an explicit type rather than the `type` keyword: value.(typeName) and the result is a new value with the static type `typeName`. That type must either be the concrete type held by the interface, or a second interface type that the value can be converted to. To extract the string we know is in the value, we could write: str := value.(string) But if it turns out that the value does not contain a string, the program will crash with a run-time error. To guard against that, use the "comma, ok" idiom to test, safely, whether the value is a string: str, ok := value.(string) if ok { fmt.Printf("string value is: %q\\n", str) } else { fmt.Printf("value is not a string\\n") } If the type assertion fails, `str` will still exist and be of type string, but it will have the zero value, an empty string. As an illustration of the capability, here's an `if`\-`else` statement that's equivalent to the type switch that opened this section. if str, ok := value.(string); ok { return str } else if str, ok := value.(Stringer); ok { return str.String() } ### Generality[¶](https://go.dev/doc/effective_go#generality) If a type exists only to implement an interface and will never have exported methods beyond that interface, there is no need to export the type itself. Exporting just the interface makes it clear the value has no interesting behavior beyond what is described in the interface. It also avoids the need to repeat the documentation on every instance of a common method. In such cases, the constructor should return an interface value rather than the implementing type. As an example, in the hash libraries both `crc32.NewIEEE` and `adler32.New` return the interface type `hash.Hash32`. Substituting the CRC-32 algorithm for Adler-32 in a Go program requires only changing the constructor call; the rest of the code is unaffected by the change of algorithm. A similar approach allows the streaming cipher algorithms in the various `crypto` packages to be separated from the block ciphers they chain together. The `Block` interface in the `crypto/cipher` package specifies the behavior of a block cipher, which provides encryption of a single block of data. Then, by analogy with the `bufio` package, cipher packages that implement this interface can be used to construct streaming ciphers, represented by the `Stream` interface, without knowing the details of the block encryption. The `crypto/cipher` interfaces look like this: type Block interface { BlockSize() int Encrypt(dst, src \[\]byte) Decrypt(dst, src \[\]byte) } type Stream interface { XORKeyStream(dst, src \[\]byte) } Here's the definition of the counter mode (CTR) stream, which turns a block cipher into a streaming cipher; notice that the block cipher's details are abstracted away: // NewCTR returns a Stream that encrypts/decrypts using the given Block in // counter mode. The length of iv must be the same as the Block's block size. func NewCTR(block Block, iv \[\]byte) Stream `NewCTR` applies not just to one specific encryption algorithm and data source but to any implementation of the `Block` interface and any `Stream`. Because they return interface values, replacing CTR encryption with other encryption modes is a localized change. The constructor calls must be edited, but because the surrounding code must treat the result only as a `Stream`, it won't notice the difference. ### Interfaces and methods[¶](https://go.dev/doc/effective_go#interface_methods) Since almost anything can have methods attached, almost anything can satisfy an interface. One illustrative example is in the `http` package, which defines the `Handler` interface. Any object that implements `Handler` can serve HTTP requests. type Handler interface { ServeHTTP(ResponseWriter, \*Request) } `ResponseWriter` is itself an interface that provides access to the methods needed to return the response to the client. Those methods include the standard `Write` method, so an `http.ResponseWriter` can be used wherever an `io.Writer` can be used. `Request` is a struct containing a parsed representation of the request from the client. For brevity, let's ignore POSTs and assume HTTP requests are always GETs; that simplification does not affect the way the handlers are set up. Here's a trivial implementation of a handler to count the number of times the page is visited. // Simple counter server. type Counter struct { n int } func (ctr \*Counter) ServeHTTP(w http.ResponseWriter, req \*http.Request) { ctr.n++ fmt.Fprintf(w, "counter = %d\\n", ctr.n) } (Keeping with our theme, note how `Fprintf` can print to an `http.ResponseWriter`.) In a real server, access to `ctr.n` would need protection from concurrent access. See the `sync` and `atomic` packages for suggestions. For reference, here's how to attach such a server to a node on the URL tree. import "net/http" ... ctr := new(Counter) http.Handle("/counter", ctr) But why make `Counter` a struct? An integer is all that's needed. (The receiver needs to be a pointer so the increment is visible to the caller.) // Simpler counter server. type Counter int func (ctr \*Counter) ServeHTTP(w http.ResponseWriter, req \*http.Request) { \*ctr++ fmt.Fprintf(w, "counter = %d\\n", \*ctr) } What if your program has some internal state that needs to be notified that a page has been visited? Tie a channel to the web page. // A channel that sends a notification on each visit. // (Probably want the channel to be buffered.) type Chan chan \*http.Request func (ch Chan) ServeHTTP(w http.ResponseWriter, req \*http.Request) { ch <- req fmt.Fprint(w, "notification sent") } Finally, let's say we wanted to present on `/args` the arguments used when invoking the server binary. It's easy to write a function to print the arguments. func ArgServer() { fmt.Println(os.Args) } How do we turn that into an HTTP server? We could make `ArgServer` a method of some type whose value we ignore, but there's a cleaner way. Since we can define a method for any type except pointers and interfaces, we can write a method for a function. The `http` package contains this code: // The HandlerFunc type is an adapter to allow the use of // ordinary functions as HTTP handlers. If f is a function // with the appropriate signature, HandlerFunc(f) is a // Handler object that calls f. type HandlerFunc func(ResponseWriter, \*Request) // ServeHTTP calls f(w, req). func (f HandlerFunc) ServeHTTP(w ResponseWriter, req \*Request) { f(w, req) } `HandlerFunc` is a type with a method, `ServeHTTP`, so values of that type can serve HTTP requests. Look at the implementation of the method: the receiver is a function, `f`, and the method calls `f`. That may seem odd but it's not that different from, say, the receiver being a channel and the method sending on the channel. To make `ArgServer` into an HTTP server, we first modify it to have the right signature. // Argument server. func ArgServer(w http.ResponseWriter, req \*http.Request) { fmt.Fprintln(w, os.Args) } `ArgServer` now has the same signature as `HandlerFunc`, so it can be converted to that type to access its methods, just as we converted `Sequence` to `IntSlice` to access `IntSlice.Sort`. The code to set it up is concise: http.Handle("/args", http.HandlerFunc(ArgServer)) When someone visits the page `/args`, the handler installed at that page has value `ArgServer` and type `HandlerFunc`. The HTTP server will invoke the method `ServeHTTP` of that type, with `ArgServer` as the receiver, which will in turn call `ArgServer` (via the invocation `f(w, req)` inside `HandlerFunc.ServeHTTP`). The arguments will then be displayed. In this section we have made an HTTP server from a struct, an integer, a channel, and a function, all because interfaces are just sets of methods, which can be defined for (almost) any type. The blank identifier[¶](https://go.dev/doc/effective_go#blank) --------------------------------------------------------------- We've mentioned the blank identifier a couple of times now, in the context of [`for` `range` loops](https://go.dev/doc/effective_go#for) and [maps](https://go.dev/doc/effective_go#maps) . The blank identifier can be assigned or declared with any value of any type, with the value discarded harmlessly. It's a bit like writing to the Unix `/dev/null` file: it represents a write-only value to be used as a place-holder where a variable is needed but the actual value is irrelevant. It has uses beyond those we've seen already. ### The blank identifier in multiple assignment[¶](https://go.dev/doc/effective_go#blank_assign) The use of a blank identifier in a `for` `range` loop is a special case of a general situation: multiple assignment. If an assignment requires multiple values on the left side, but one of the values will not be used by the program, a blank identifier on the left-hand-side of the assignment avoids the need to create a dummy variable and makes it clear that the value is to be discarded. For instance, when calling a function that returns a value and an error, but only the error is important, use the blank identifier to discard the irrelevant value. if \_, err := os.Stat(path); os.IsNotExist(err) { fmt.Printf("%s does not exist\\n", path) } Occasionally you'll see code that discards the error value in order to ignore the error; this is terrible practice. Always check error returns; they're provided for a reason. // Bad! This code will crash if path does not exist. fi, \_ := os.Stat(path) if fi.IsDir() { fmt.Printf("%s is a directory\\n", path) } ### Unused imports and variables[¶](https://go.dev/doc/effective_go#blank_unused) It is an error to import a package or to declare a variable without using it. Unused imports bloat the program and slow compilation, while a variable that is initialized but not used is at least a wasted computation and perhaps indicative of a larger bug. When a program is under active development, however, unused imports and variables often arise and it can be annoying to delete them just to have the compilation proceed, only to have them be needed again later. The blank identifier provides a workaround. This half-written program has two unused imports (`fmt` and `io`) and an unused variable (`fd`), so it will not compile, but it would be nice to see if the code so far is correct. package main import ( "fmt" "io" "log" "os" ) func main() { fd, err := os.Open("test.go") if err != nil { log.Fatal(err) } // TODO: use fd. } To silence complaints about the unused imports, use a blank identifier to refer to a symbol from the imported package. Similarly, assigning the unused variable `fd` to the blank identifier will silence the unused variable error. This version of the program does compile. package main import ( "fmt" "io" "log" "os" ) var \_ = fmt.Printf // For debugging; delete when done. var \_ io.Reader // For debugging; delete when done. func main() { fd, err := os.Open("test.go") if err != nil { log.Fatal(err) } // TODO: use fd. \_ = fd } By convention, the global declarations to silence import errors should come right after the imports and be commented, both to make them easy to find and as a reminder to clean things up later. ### Import for side effect[¶](https://go.dev/doc/effective_go#blank_import) An unused import like `fmt` or `io` in the previous example should eventually be used or removed: blank assignments identify code as a work in progress. But sometimes it is useful to import a package only for its side effects, without any explicit use. For example, during its `init` function, the `[net/http/pprof](https://go.dev/pkg/net/http/pprof/) ` package registers HTTP handlers that provide debugging information. It has an exported API, but most clients need only the handler registration and access the data through a web page. To import the package only for its side effects, rename the package to the blank identifier: import \_ "net/http/pprof" This form of import makes clear that the package is being imported for its side effects, because there is no other possible use of the package: in this file, it doesn't have a name. (If it did, and we didn't use that name, the compiler would reject the program.) ### Interface checks[¶](https://go.dev/doc/effective_go#blank_implements) As we saw in the discussion of [interfaces](https://go.dev/doc/effective_go#interfaces_and_types) above, a type need not declare explicitly that it implements an interface. Instead, a type implements the interface just by implementing the interface's methods. In practice, most interface conversions are static and therefore checked at compile time. For example, passing an `*os.File` to a function expecting an `io.Reader` will not compile unless `*os.File` implements the `io.Reader` interface. Some interface checks do happen at run-time, though. One instance is in the `[encoding/json](https://go.dev/pkg/encoding/json/) ` package, which defines a `[Marshaler](https://go.dev/pkg/encoding/json/#Marshaler) ` interface. When the JSON encoder receives a value that implements that interface, the encoder invokes the value's marshaling method to convert it to JSON instead of doing the standard conversion. The encoder checks this property at run time with a [type assertion](https://go.dev/doc/effective_go#interface_conversions) like: m, ok := val.(json.Marshaler) If it's necessary only to ask whether a type implements an interface, without actually using the interface itself, perhaps as part of an error check, use the blank identifier to ignore the type-asserted value: if \_, ok := val.(json.Marshaler); ok { fmt.Printf("value %v of type %T implements json.Marshaler\\n", val, val) } One place this situation arises is when it is necessary to guarantee within the package implementing the type that it actually satisfies the interface. If a type—for example, `[json.RawMessage](https://go.dev/pkg/encoding/json/#RawMessage) `—needs a custom JSON representation, it should implement `json.Marshaler`, but there are no static conversions that would cause the compiler to verify this automatically. If the type inadvertently fails to satisfy the interface, the JSON encoder will still work, but will not use the custom implementation. To guarantee that the implementation is correct, a global declaration using the blank identifier can be used in the package: var \_ json.Marshaler = (\*RawMessage)(nil) In this declaration, the assignment involving a conversion of a `*RawMessage` to a `Marshaler` requires that `*RawMessage` implements `Marshaler`, and that property will be checked at compile time. Should the `json.Marshaler` interface change, this package will no longer compile and we will be on notice that it needs to be updated. The appearance of the blank identifier in this construct indicates that the declaration exists only for the type checking, not to create a variable. Don't do this for every type that satisfies an interface, though. By convention, such declarations are only used when there are no static conversions already present in the code, which is a rare event. Embedding[¶](https://go.dev/doc/effective_go#embedding) -------------------------------------------------------- Go does not provide the typical, type-driven notion of subclassing, but it does have the ability to “borrow” pieces of an implementation by _embedding_ types within a struct or interface. Interface embedding is very simple. We've mentioned the `io.Reader` and `io.Writer` interfaces before; here are their definitions. type Reader interface { Read(p \[\]byte) (n int, err error) } type Writer interface { Write(p \[\]byte) (n int, err error) } The `io` package also exports several other interfaces that specify objects that can implement several such methods. For instance, there is `io.ReadWriter`, an interface containing both `Read` and `Write`. We could specify `io.ReadWriter` by listing the two methods explicitly, but it's easier and more evocative to embed the two interfaces to form the new one, like this: // ReadWriter is the interface that combines the Reader and Writer interfaces. type ReadWriter interface { Reader Writer } This says just what it looks like: A `ReadWriter` can do what a `Reader` does _and_ what a `Writer` does; it is a union of the embedded interfaces. Only interfaces can be embedded within interfaces. The same basic idea applies to structs, but with more far-reaching implications. The `bufio` package has two struct types, `bufio.Reader` and `bufio.Writer`, each of which of course implements the analogous interfaces from package `io`. And `bufio` also implements a buffered reader/writer, which it does by combining a reader and a writer into one struct using embedding: it lists the types within the struct but does not give them field names. // ReadWriter stores pointers to a Reader and a Writer. // It implements io.ReadWriter. type ReadWriter struct { \*Reader // \*bufio.Reader \*Writer // \*bufio.Writer } The embedded elements are pointers to structs and of course must be initialized to point to valid structs before they can be used. The `ReadWriter` struct could be written as type ReadWriter struct { reader \*Reader writer \*Writer } but then to promote the methods of the fields and to satisfy the `io` interfaces, we would also need to provide forwarding methods, like this: func (rw \*ReadWriter) Read(p \[\]byte) (n int, err error) { return rw.reader.Read(p) } By embedding the structs directly, we avoid this bookkeeping. The methods of embedded types come along for free, which means that `bufio.ReadWriter` not only has the methods of `bufio.Reader` and `bufio.Writer`, it also satisfies all three interfaces: `io.Reader`, `io.Writer`, and `io.ReadWriter`. There's an important way in which embedding differs from subclassing. When we embed a type, the methods of that type become methods of the outer type, but when they are invoked the receiver of the method is the inner type, not the outer one. In our example, when the `Read` method of a `bufio.ReadWriter` is invoked, it has exactly the same effect as the forwarding method written out above; the receiver is the `reader` field of the `ReadWriter`, not the `ReadWriter` itself. Embedding can also be a simple convenience. This example shows an embedded field alongside a regular, named field. type Job struct { Command string \*log.Logger } The `Job` type now has the `Print`, `Printf`, `Println` and other methods of `*log.Logger`. We could have given the `Logger` a field name, of course, but it's not necessary to do so. And now, once initialized, we can log to the `Job`: job.Println("starting now...") The `Logger` is a regular field of the `Job` struct, so we can initialize it in the usual way inside the constructor for `Job`, like this, func NewJob(command string, logger \*log.Logger) \*Job { return &Job{command, logger} } or with a composite literal, job := &Job{command, log.New(os.Stderr, "Job: ", log.Ldate)} If we need to refer to an embedded field directly, the type name of the field, ignoring the package qualifier, serves as a field name, as it did in the `Read` method of our `ReadWriter` struct. Here, if we needed to access the `*log.Logger` of a `Job` variable `job`, we would write `job.Logger`, which would be useful if we wanted to refine the methods of `Logger`. func (job \*Job) Printf(format string, args ...interface{}) { job.Logger.Printf("%q: %s", job.Command, fmt.Sprintf(format, args...)) } Embedding types introduces the problem of name conflicts but the rules to resolve them are simple. First, a field or method `X` hides any other item `X` in a more deeply nested part of the type. If `log.Logger` contained a field or method called `Command`, the `Command` field of `Job` would dominate it. Second, if the same name appears at the same nesting level, it is usually an error; it would be erroneous to embed `log.Logger` if the `Job` struct contained another field or method called `Logger`. However, if the duplicate name is never mentioned in the program outside the type definition, it is OK. This qualification provides some protection against changes made to types embedded from outside; there is no problem if a field is added that conflicts with another field in another subtype if neither field is ever used. Concurrency[¶](https://go.dev/doc/effective_go#concurrency) ------------------------------------------------------------ ### Share by communicating[¶](https://go.dev/doc/effective_go#sharing) Concurrent programming is a large topic and there is space only for some Go-specific highlights here. Concurrent programming in many environments is made difficult by the subtleties required to implement correct access to shared variables. Go encourages a different approach in which shared values are passed around on channels and, in fact, never actively shared by separate threads of execution. Only one goroutine has access to the value at any given time. Data races cannot occur, by design. To encourage this way of thinking we have reduced it to a slogan: > Do not communicate by sharing memory; instead, share memory by communicating. This approach can be taken too far. Reference counts may be best done by putting a mutex around an integer variable, for instance. But as a high-level approach, using channels to control access makes it easier to write clear, correct programs. One way to think about this model is to consider a typical single-threaded program running on one CPU. It has no need for synchronization primitives. Now run another such instance; it too needs no synchronization. Now let those two communicate; if the communication is the synchronizer, there's still no need for other synchronization. Unix pipelines, for example, fit this model perfectly. Although Go's approach to concurrency originates in Hoare's Communicating Sequential Processes (CSP), it can also be seen as a type-safe generalization of Unix pipes. ### Goroutines[¶](https://go.dev/doc/effective_go#goroutines) They're called _goroutines_ because the existing terms—threads, coroutines, processes, and so on—convey inaccurate connotations. A goroutine has a simple model: it is a function executing concurrently with other goroutines in the same address space. It is lightweight, costing little more than the allocation of stack space. And the stacks start small, so they are cheap, and grow by allocating (and freeing) heap storage as required. Goroutines are multiplexed onto multiple OS threads so if one should block, such as while waiting for I/O, others continue to run. Their design hides many of the complexities of thread creation and management. Prefix a function or method call with the `go` keyword to run the call in a new goroutine. When the call completes, the goroutine exits, silently. (The effect is similar to the Unix shell's `&` notation for running a command in the background.) go list.Sort() // run list.Sort concurrently; don't wait for it. A function literal can be handy in a goroutine invocation. func Announce(message string, delay time.Duration) { go func() { time.Sleep(delay) fmt.Println(message) }() // Note the parentheses - must call the function. } In Go, function literals are closures: the implementation makes sure the variables referred to by the function survive as long as they are active. These examples aren't too practical because the functions have no way of signaling completion. For that, we need channels. ### Channels[¶](https://go.dev/doc/effective_go#channels) Like maps, channels are allocated with `make`, and the resulting value acts as a reference to an underlying data structure. If an optional integer parameter is provided, it sets the buffer size for the channel. The default is zero, for an unbuffered or synchronous channel. ci := make(chan int) // unbuffered channel of integers cj := make(chan int, 0) // unbuffered channel of integers cs := make(chan \*os.File, 100) // buffered channel of pointers to Files Unbuffered channels combine communication—the exchange of a value—with synchronization—guaranteeing that two calculations (goroutines) are in a known state. There are lots of nice idioms using channels. Here's one to get us started. In the previous section we launched a sort in the background. A channel can allow the launching goroutine to wait for the sort to complete. c := make(chan int) // Allocate a channel. // Start the sort in a goroutine; when it completes, signal on the channel. go func() { list.Sort() c <- 1 // Send a signal; value does not matter. }() doSomethingForAWhile() <-c // Wait for sort to finish; discard sent value. Receivers always block until there is data to receive. If the channel is unbuffered, the sender blocks until the receiver has received the value. If the channel has a buffer, the sender blocks only until the value has been copied to the buffer; if the buffer is full, this means waiting until some receiver has retrieved a value. A buffered channel can be used like a semaphore, for instance to limit throughput. In this example, incoming requests are passed to `handle`, which sends a value into the channel, processes the request, and then receives a value from the channel to ready the “semaphore” for the next consumer. The capacity of the channel buffer limits the number of simultaneous calls to `process`. var sem = make(chan int, MaxOutstanding) func handle(r \*Request) { sem <- 1 // Wait for active queue to drain. process(r) // May take a long time. <-sem // Done; enable next request to run. } func Serve(queue chan \*Request) { for { req := <-queue go handle(req) // Don't wait for handle to finish. } } Once `MaxOutstanding` handlers are executing `process`, any more will block trying to send into the filled channel buffer, until one of the existing handlers finishes and receives from the buffer. This design has a problem, though: `Serve` creates a new goroutine for every incoming request, even though only `MaxOutstanding` of them can run at any moment. As a result, the program can consume unlimited resources if the requests come in too fast. We can address that deficiency by changing `Serve` to gate the creation of the goroutines: func Serve(queue chan \*Request) { for req := range queue { sem <- 1 go func() { process(req) <-sem }() } } (Note that in Go versions before 1.22 this code has a bug: the loop variable is shared across all goroutines. See the [Go wiki](https://go.dev/wiki/LoopvarExperiment) for details.) Another approach that manages resources well is to start a fixed number of `handle` goroutines all reading from the request channel. The number of goroutines limits the number of simultaneous calls to `process`. This `Serve` function also accepts a channel on which it will be told to exit; after launching the goroutines it blocks receiving from that channel. func handle(queue chan \*Request) { for r := range queue { process(r) } } func Serve(clientRequests chan \*Request, quit chan bool) { // Start handlers for i := 0; i < MaxOutstanding; i++ { go handle(clientRequests) } <-quit // Wait to be told to exit. } ### Channels of channels[¶](https://go.dev/doc/effective_go#chan_of_chan) One of the most important properties of Go is that a channel is a first-class value that can be allocated and passed around like any other. A common use of this property is to implement safe, parallel demultiplexing. In the example in the previous section, `handle` was an idealized handler for a request but we didn't define the type it was handling. If that type includes a channel on which to reply, each client can provide its own path for the answer. Here's a schematic definition of type `Request`. type Request struct { args \[\]int f func(\[\]int) int resultChan chan int } The client provides a function and its arguments, as well as a channel inside the request object on which to receive the answer. func sum(a \[\]int) (s int) { for \_, v := range a { s += v } return } request := &Request{\[\]int{3, 4, 5}, sum, make(chan int)} // Send request clientRequests <- request // Wait for response. fmt.Printf("answer: %d\\n", <-request.resultChan) On the server side, the handler function is the only thing that changes. func handle(queue chan \*Request) { for req := range queue { req.resultChan <- req.f(req.args) } } There's clearly a lot more to do to make it realistic, but this code is a framework for a rate-limited, parallel, non-blocking RPC system, and there's not a mutex in sight. ### Parallelization[¶](https://go.dev/doc/effective_go#parallel) Another application of these ideas is to parallelize a calculation across multiple CPU cores. If the calculation can be broken into separate pieces that can execute independently, it can be parallelized, with a channel to signal when each piece completes. Let's say we have an expensive operation to perform on a vector of items, and that the value of the operation on each item is independent, as in this idealized example. type Vector \[\]float64 // Apply the operation to v\[i\], v\[i+1\] ... up to v\[n-1\]. func (v Vector) DoSome(i, n int, u Vector, c chan int) { for ; i < n; i++ { v\[i\] += u.Op(v\[i\]) } c <- 1 // signal that this piece is done } We launch the pieces independently in a loop, one per CPU. They can complete in any order but it doesn't matter; we just count the completion signals by draining the channel after launching all the goroutines. const numCPU = 4 // number of CPU cores func (v Vector) DoAll(u Vector) { c := make(chan int, numCPU) // Buffering optional but sensible. for i := 0; i < numCPU; i++ { go v.DoSome(i\*len(v)/numCPU, (i+1)\*len(v)/numCPU, u, c) } // Drain the channel. for i := 0; i < numCPU; i++ { <-c // wait for one task to complete } // All done. } Rather than create a constant value for numCPU, we can ask the runtime what value is appropriate. The function `[runtime.NumCPU](https://go.dev/pkg/runtime#NumCPU) ` returns the number of hardware CPU cores in the machine, so we could write var numCPU = runtime.NumCPU() There is also a function `[runtime.GOMAXPROCS](https://go.dev/pkg/runtime#GOMAXPROCS) `, which reports (or sets) the user-specified number of cores that a Go program can have running simultaneously. It defaults to the value of `runtime.NumCPU` but can be overridden by setting the similarly named shell environment variable or by calling the function with a positive number. Calling it with zero just queries the value. Therefore if we want to honor the user's resource request, we should write var numCPU = runtime.GOMAXPROCS(0) Be sure not to confuse the ideas of concurrency—structuring a program as independently executing components—and parallelism—executing calculations in parallel for efficiency on multiple CPUs. Although the concurrency features of Go can make some problems easy to structure as parallel computations, Go is a concurrent language, not a parallel one, and not all parallelization problems fit Go's model. For a discussion of the distinction, see the talk cited in [this blog post](https://go.dev/blog/concurrency-is-not-parallelism) . ### A leaky buffer[¶](https://go.dev/doc/effective_go#leaky_buffer) The tools of concurrent programming can even make non-concurrent ideas easier to express. Here's an example abstracted from an RPC package. The client goroutine loops receiving data from some source, perhaps a network. To avoid allocating and freeing buffers, it keeps a free list, and uses a buffered channel to represent it. If the channel is empty, a new buffer gets allocated. Once the message buffer is ready, it's sent to the server on `serverChan`. var freeList = make(chan \*Buffer, 100) var serverChan = make(chan \*Buffer) func client() { for { var b \*Buffer // Grab a buffer if available; allocate if not. select { case b = <-freeList: // Got one; nothing more to do. default: // None free, so allocate a new one. b = new(Buffer) } load(b) // Read next message from the net. serverChan <- b // Send to server. } } The server loop receives each message from the client, processes it, and returns the buffer to the free list. func server() { for { b := <-serverChan // Wait for work. process(b) // Reuse buffer if there's room. select { case freeList <- b: // Buffer on free list; nothing more to do. default: // Free list full, just carry on. } } } The client attempts to retrieve a buffer from `freeList`; if none is available, it allocates a fresh one. The server's send to `freeList` puts `b` back on the free list unless the list is full, in which case the buffer is dropped on the floor to be reclaimed by the garbage collector. (The `default` clauses in the `select` statements execute when no other case is ready, meaning that the `selects` never block.) This implementation builds a leaky bucket free list in just a few lines, relying on the buffered channel and the garbage collector for bookkeeping. Errors[¶](https://go.dev/doc/effective_go#errors) -------------------------------------------------- Library routines must often return some sort of error indication to the caller. As mentioned earlier, Go's multivalue return makes it easy to return a detailed error description alongside the normal return value. It is good style to use this feature to provide detailed error information. For example, as we'll see, `os.Open` doesn't just return a `nil` pointer on failure, it also returns an error value that describes what went wrong. By convention, errors have type `error`, a simple built-in interface. type error interface { Error() string } A library writer is free to implement this interface with a richer model under the covers, making it possible not only to see the error but also to provide some context. As mentioned, alongside the usual `*os.File` return value, `os.Open` also returns an error value. If the file is opened successfully, the error will be `nil`, but when there is a problem, it will hold an `os.PathError`: // PathError records an error and the operation and // file path that caused it. type PathError struct { Op string // "open", "unlink", etc. Path string // The associated file. Err error // Returned by the system call. } func (e \*PathError) Error() string { return e.Op + " " + e.Path + ": " + e.Err.Error() } `PathError`'s `Error` generates a string like this: open /etc/passwx: no such file or directory Such an error, which includes the problematic file name, the operation, and the operating system error it triggered, is useful even if printed far from the call that caused it; it is much more informative than the plain "no such file or directory". When feasible, error strings should identify their origin, such as by having a prefix naming the operation or package that generated the error. For example, in package `image`, the string representation for a decoding error due to an unknown format is "image: unknown format". Callers that care about the precise error details can use a type switch or a type assertion to look for specific errors and extract details. For `PathErrors` this might include examining the internal `Err` field for recoverable failures. for try := 0; try < 2; try++ { file, err = os.Create(filename) if err == nil { return } if e, ok := err.(\*os.PathError); ok && e.Err == syscall.ENOSPC { deleteTempFiles() // Recover some space. continue } return } The second `if` statement here is another [type assertion](https://go.dev/doc/effective_go#interface_conversions) . If it fails, `ok` will be false, and `e` will be `nil`. If it succeeds, `ok` will be true, which means the error was of type `*os.PathError`, and then so is `e`, which we can examine for more information about the error. ### Panic[¶](https://go.dev/doc/effective_go#panic) The usual way to report an error to a caller is to return an `error` as an extra return value. The canonical `Read` method is a well-known instance; it returns a byte count and an `error`. But what if the error is unrecoverable? Sometimes the program simply cannot continue. For this purpose, there is a built-in function `panic` that in effect creates a run-time error that will stop the program (but see the next section). The function takes a single argument of arbitrary type—often a string—to be printed as the program dies. It's also a way to indicate that something impossible has happened, such as exiting an infinite loop. // A toy implementation of cube root using Newton's method. func CubeRoot(x float64) float64 { z := x/3 // Arbitrary initial value for i := 0; i < 1e6; i++ { prevz := z z -= (z\*z\*z-x) / (3\*z\*z) if veryClose(z, prevz) { return z } } // A million iterations has not converged; something is wrong. panic(fmt.Sprintf("CubeRoot(%g) did not converge", x)) } This is only an example but real library functions should avoid `panic`. If the problem can be masked or worked around, it's always better to let things continue to run rather than taking down the whole program. One possible counterexample is during initialization: if the library truly cannot set itself up, it might be reasonable to panic, so to speak. var user = os.Getenv("USER") func init() { if user == "" { panic("no value for $USER") } } ### Recover[¶](https://go.dev/doc/effective_go#recover) When `panic` is called, including implicitly for run-time errors such as indexing a slice out of bounds or failing a type assertion, it immediately stops execution of the current function and begins unwinding the stack of the goroutine, running any deferred functions along the way. If that unwinding reaches the top of the goroutine's stack, the program dies. However, it is possible to use the built-in function `recover` to regain control of the goroutine and resume normal execution. A call to `recover` stops the unwinding and returns the argument passed to `panic`. Because the only code that runs while unwinding is inside deferred functions, `recover` is only useful inside deferred functions. One application of `recover` is to shut down a failing goroutine inside a server without killing the other executing goroutines. func server(workChan <-chan \*Work) { for work := range workChan { go safelyDo(work) } } func safelyDo(work \*Work) { defer func() { if err := recover(); err != nil { log.Println("work failed:", err) } }() do(work) } In this example, if `do(work)` panics, the result will be logged and the goroutine will exit cleanly without disturbing the others. There's no need to do anything else in the deferred closure; calling `recover` handles the condition completely. Because `recover` always returns `nil` unless called directly from a deferred function, deferred code can call library routines that themselves use `panic` and `recover` without failing. As an example, the deferred function in `safelyDo` might call a logging function before calling `recover`, and that logging code would run unaffected by the panicking state. With our recovery pattern in place, the `do` function (and anything it calls) can get out of any bad situation cleanly by calling `panic`. We can use that idea to simplify error handling in complex software. Let's look at an idealized version of a `regexp` package, which reports parsing errors by calling `panic` with a local error type. Here's the definition of `Error`, an `error` method, and the `Compile` function. // Error is the type of a parse error; it satisfies the error interface. type Error string func (e Error) Error() string { return string(e) } // error is a method of \*Regexp that reports parsing errors by // panicking with an Error. func (regexp \*Regexp) error(err string) { panic(Error(err)) } // Compile returns a parsed representation of the regular expression. func Compile(str string) (regexp \*Regexp, err error) { regexp = new(Regexp) // doParse will panic if there is a parse error. defer func() { if e := recover(); e != nil { regexp = nil // Clear return value. err = e.(Error) // Will re-panic if not a parse error. } }() return regexp.doParse(str), nil } If `doParse` panics, the recovery block will set the return value to `nil`—deferred functions can modify named return values. It will then check, in the assignment to `err`, that the problem was a parse error by asserting that it has the local type `Error`. If it does not, the type assertion will fail, causing a run-time error that continues the stack unwinding as though nothing had interrupted it. This check means that if something unexpected happens, such as an index out of bounds, the code will fail even though we are using `panic` and `recover` to handle parse errors. With error handling in place, the `error` method (because it's a method bound to a type, it's fine, even natural, for it to have the same name as the builtin `error` type) makes it easy to report parse errors without worrying about unwinding the parse stack by hand: if pos == 0 { re.error("'\*' illegal at start of expression") } Useful though this pattern is, it should be used only within a package. `Parse` turns its internal `panic` calls into `error` values; it does not expose `panics` to its client. That is a good rule to follow. By the way, this re-panic idiom changes the panic value if an actual error occurs. However, both the original and new failures will be presented in the crash report, so the root cause of the problem will still be visible. Thus this simple re-panic approach is usually sufficient—it's a crash after all—but if you want to display only the original value, you can write a little more code to filter unexpected problems and re-panic with the original error. That's left as an exercise for the reader. A web server[¶](https://go.dev/doc/effective_go#web_server) ------------------------------------------------------------ Let's finish with a complete Go program, a web server. This one is actually a kind of web re-server. Google provides a service at `chart.apis.google.com` that does automatic formatting of data into charts and graphs. It's hard to use interactively, though, because you need to put the data into the URL as a query. The program here provides a nicer interface to one form of data: given a short piece of text, it calls on the chart server to produce a QR code, a matrix of boxes that encode the text. That image can be grabbed with your cell phone's camera and interpreted as, for instance, a URL, saving you typing the URL into the phone's tiny keyboard. Here's the complete program. An explanation follows. package main import ( "flag" "html/template" "log" "net/http" ) var addr = flag.String("addr", ":1718", "http service address") // Q=17, R=18 var templ = template.Must(template.New("qr").Parse(templateStr)) func main() { flag.Parse() http.Handle("/", http.HandlerFunc(QR)) err := http.ListenAndServe(\*addr, nil) if err != nil { log.Fatal("ListenAndServe:", err) } } func QR(w http.ResponseWriter, req \*http.Request) { templ.Execute(w, req.FormValue("s")) } const templateStr = \` QR Link Generator {{if .}}
{{.}}

{{end}}
\` The pieces up to `main` should be easy to follow. The one flag sets a default HTTP port for our server. The template variable `templ` is where the fun happens. It builds an HTML template that will be executed by the server to display the page; more about that in a moment. The `main` function parses the flags and, using the mechanism we talked about above, binds the function `QR` to the root path for the server. Then `http.ListenAndServe` is called to start the server; it blocks while the server runs. `QR` just receives the request, which contains form data, and executes the template on the data in the form value named `s`. The template package `html/template` is powerful; this program just touches on its capabilities. In essence, it rewrites a piece of HTML text on the fly by substituting elements derived from data items passed to `templ.Execute`, in this case the form value. Within the template text (`templateStr`), double-brace-delimited pieces denote template actions. The piece from `{{if .}}` to `{{end}}` executes only if the value of the current data item, called `.` (dot), is non-empty. That is, when the string is empty, this piece of the template is suppressed. The two snippets `{{.}}` say to show the data presented to the template—the query string—on the web page. The HTML template package automatically provides appropriate escaping so the text is safe to display. The rest of the template string is just the HTML to show when the page loads. If this is too quick an explanation, see the [documentation](https://go.dev/pkg/html/template/) for the template package for a more thorough discussion. And there you have it: a useful web server in a few lines of code plus some data-driven HTML text. Go is powerful enough to make a lot happen in a few lines. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Case Studies - The Go Programming Language 1. [Why Go](https://go.dev/solutions/) 2. [Case Studies](https://go.dev/solutions/case-studies) * ![Using Go at Google](https://go.dev/images/go_google_case_study_carousel.png) RECENTLY UPDATED Using Go at Google ------------------ Go was created at Google in 2007, and since then, engineering teams across Google have adopted Go to build products and services at massive scale. [Learn more _arrow\_forward_](https://go.dev/solutions/google/) * ![PayPal Taps Go to Modernize and Scale](https://go.dev/images/go_paypal_case_study.png) RECENTLY UPDATED PayPal Taps Go to Modernize and Scale ------------------------------------- Go’s value in producing clean, efficient code that readily scales as software deployment scales made the language a strong fit to support PayPal’s goals. [Learn more _arrow\_forward_](https://go.dev/solutions/paypal) * ![American Express Uses Go for Payments & Rewards](https://go.dev/images/go_amex_case_study.png) RECENTLY UPDATED American Express Uses Go for Payments & Rewards ----------------------------------------------- Go provides American Express with the speed and scalability it needs for both its payment and rewards networks. [Learn more _arrow\_forward_](https://go.dev/solutions/americanexpress) Opens in new window. _navigate\_before_ _navigate\_next_ * [![Allegro](https://go.dev/images/logos/allegro_dark.svg) ![Allegro](https://go.dev/images/logos/allegro_light.svg)\ \ Allegro – Writing a very fast cache service with millions of entries in Go\ --------------------------------------------------------------------------\ \ “Finally, we sped up our application from more than 2.5 seconds to less than 250 milliseconds for the longest request.”\ \ View Case Study](https://blog.allegro.tech/2016/03/writing-fast-cache-service-in-go.html) * [![American Express](https://go.dev/images/logos/american-express.svg) ![American Express](https://go.dev/images/logos/american-express.svg)\ \ American Express Uses Go for Payments & Rewards\ -----------------------------------------------\ \ Go provides American Express with the speed and scalability it needs for both its payment and rewards networks.\ \ View Case Study](https://go.dev/solutions/americanexpress) * [![Armut](https://go.dev/images/logos/armut_dark.png) ![Armut](https://go.dev/images/logos/armut_light.png)\ \ How Armut Labs use Go\ ---------------------\ \ Learn about how Armut Labs reduced resource consumption and API response time after moving from C# and .net core to Go.\ \ View Case Study](https://labs.armut.com/how-we-decreased-one-of-our-apis-response-time-by-87-and-used-less-resources-ce847e83308) * [![Bitly](https://go.dev/images/logos/bitly.svg) ![Bitly](https://go.dev/images/logos/bitly.svg)\ \ Bitly - Why We Write Everything in Go\ -------------------------------------\ \ In 2014, we wrote a little open source project called NSQ (nsq.io) and put a promising new language called Go through its paces. We liked what we saw so much that we started writing everything new in Go, and soon thereafter we began porting all legacy services to Go as well.\ \ View Case Study](https://bitly.com/blog/why-we-write-everything-in-go/?utm_source=go-dev&utm_medium=referral&utm_campaign=go-dev&utm_content=case-study) * [![ByteDance](https://go.dev/images/logos/bytedance_dark.svg) ![ByteDance](https://go.dev/images/logos/bytedance_light.svg)\ \ Massive practice in Go at ByteDance\ -----------------------------------\ \ Go was introduced to ByteDance in 2014, and since then engineering teams across ByteDance have adopted Go to build products and services on a massive scale. As we went deeper, relatively mature microservice best practices under Go were developed and summarized, which then were open-sourced and named CloudWeGo since 2021. Now 70% of microservices within ByteDance are written by Go.\ \ View Case Study](https://www.cloudwego.io/) * [![Capital One](https://go.dev/images/logos/capitalone_dark.svg) ![Capital One](https://go.dev/images/logos/capitalone_light.svg)\ \ Capital One - A Serverless and Go Journey\ -----------------------------------------\ \ At the time, no single team member knew Go, but within a month, everyone was writing in Go and we were building out the endpoints. It was the flexibility, how easy it was to use, and the really cool concept behind Go (how Go handles native concurrency, garbage collection, and of course safety+speed.) that helped engage us during the build. Also, who can beat that cute mascot!\ \ View Case Study](https://medium.com/capital-one-tech/a-serverless-and-go-journey-credit-offers-api-74ef1f9fde7f) * [![Cloudflare](https://go.dev/images/logos/cloudflare_dark.svg) ![Cloudflare](https://go.dev/images/logos/cloudflare_light.svg)\ \ Graceful upgrades in Go\ -----------------------\ \ Cloudflare speeds up and protects millions of websites, APIs, SaaS services, and other properties connected to the Internet. “Go is at the heart of CloudFlare’s services including handling compression for high-latency HTTP connections, our entire DNS infrastructure, SSL, load testing and more.”\ \ View Case Study](https://blog.cloudflare.com/graceful-upgrades-in-go/) * [![Cockroach Labs](https://go.dev/images/logos/cockroach.svg) ![Cockroach Labs](https://go.dev/images/logos/cockroach.svg)\ \ Cockroach Labs - Why We Chose to Build Our Database with Go\ -----------------------------------------------------------\ \ Go's performance benefits, garbage collection, and low barrier to entry made it a great fit for CockroachDB.\ \ View Case Study](https://www.cockroachlabs.com/blog/why-go-was-the-right-choice-for-cockroachdb/) * [![Curve](https://go.dev/images/logos/curve.png) ![Curve](https://go.dev/images/logos/curve.png)\ \ How Curve is getting ahead with Golang\ --------------------------------------\ \ Curve shares how Go's efficiency, standard library, and thriving community help them move banking to the cloud.\ \ View Case Study](https://jaxenter.com/golang-curve-163187.html) * [![Dropbox](https://go.dev/images/logos/dropbox.png) ![Dropbox](https://go.dev/images/logos/dropbox.png)\ \ Dropbox - Open sourcing our Go libraries\ ----------------------------------------\ \ About a year ago, we decided to migrate our performance-critical backends from Python to Go to leverage better concurrency support and faster execution speed. ... At this point, we have successfully moved major parts of our infrastructure to Go.\ \ View Case Study](https://dropbox.tech/infrastructure/open-sourcing-our-go-libraries) * [![Facebook](https://go.dev/images/logos/meta_dark.svg) ![Facebook](https://go.dev/images/logos/meta_light.svg)\ \ How Facebook built an entity framework in Go\ --------------------------------------------\ \ Learn about a Facebook engineering team's decision to write a new entity framework (ORM) in Go.\ \ View Case Study](https://entgo.io/blog/2019/10/03/introducing-ent/) * [![Google](https://go.dev/images/logos/google.svg) ![Google](https://go.dev/images/logos/google.svg)\ \ Using Go at Google\ ------------------\ \ Go was created at Google in 2007, and since then, engineering teams across Google have adopted Go to build products and services at massive scale.\ \ View Case Study](https://go.dev/solutions/google/) * [![GRAIL](https://go.dev/images/logos/grail_dark.png) ![GRAIL](https://go.dev/images/logos/grail_light.png)\ \ Bigslice - A cluster computing system in Go\ -------------------------------------------\ \ At GRAIL, we use the Go programming language for most of our bioinformatics, data processing, and machine learning tasks. Go’s simplicity makes it easy for newcomers to learn; its transparent runtime semantics makes it easy to reason about performance; and its ability to control data layout and allocation makes it possible to write highly performant data processing code.\ \ View Case Study](https://medium.com/grail-eng/bigslice-a-cluster-computing-system-for-go-7e03acd2419b) * [![MercadoLibre](https://go.dev/images/logos/mercadolibre_dark.svg) ![MercadoLibre](https://go.dev/images/logos/mercadolibre_light.svg)\ \ MercadoLibre Grows with Go\ --------------------------\ \ Go provides clean, efficient code that readily scales as MercadoLibre’s online commerce grows, and increases developer productivity by allowing their engineers to serve their ever-increasing audience while writing less code.\ \ View Case Study](https://go.dev/solutions/mercadolibre) * [![Microsoft](https://go.dev/images/logos/microsoft_dark.svg) ![Microsoft](https://go.dev/images/logos/microsoft_light.svg)\ \ How Microsoft Embraces Go\ -------------------------\ \ Learn about how Microsoft has helped support Go and how it uses Go to power pieces of its cloud infrastructure.\ \ View Case Study](https://cloudblogs.microsoft.com/opensource/2018/02/21/go-lang-brian-ketelsen-explains-fast-growth/) * [![Monzo](https://go.dev/images/logos/monzo_dark.svg) ![Monzo](https://go.dev/images/logos/monzo_light.svg)\ \ Monzo – Building a Bank with Golang, Microservices and Containers\ -----------------------------------------------------------------\ \ “Go is a perfect language for creating microservice architectures, and the concurrency features, and the language in general, has allowed the easy creation of small and simple networked services at Monzo that are focused around the ‘single responsibility principle’.”\ \ View Case Study](https://www.infoq.com/news/2017/03/monzo-bank-golang/) * [![Netflix](https://go.dev/images/logos/netflix.svg) ![Netflix](https://go.dev/images/logos/netflix.svg)\ \ Netflix - Application data caching using SSDs\ ---------------------------------------------\ \ The decision to use Go was deliberate, because we needed something that had lower latency than Java (where garbage collection pauses are an issue) and is more productive for developers than C, while also handling tens of thousands of client connections. Go fits this space well.\ \ View Case Study](https://medium.com/netflix-techblog/application-data-caching-using-ssds-5bf25df851ef) * [![PayPal](https://go.dev/images/logos/paypal.svg) ![PayPal](https://go.dev/images/logos/paypal.svg)\ \ PayPal Taps Go to Modernize and Scale\ -------------------------------------\ \ Go’s value in producing clean, efficient code that readily scales as software deployment scales made the language a strong fit to support PayPal’s goals.\ \ View Case Study](https://go.dev/solutions/paypal) * [![Riot Games](https://go.dev/images/logos/riot_dark.svg) ![Riot Games](https://go.dev/images/logos/riot_light.svg)\ \ Riot Games - Leveraging Golang for Game Development and Operations\ ------------------------------------------------------------------\ \ Learn how Riot uses Go to develop, deploy, and operate backend microservices at scale–globally. They share their experience across use cases, with specific examples, and speak to the value of the gopher community.\ \ View Case Study](https://technology.riotgames.com/news/leveraging-golang-game-development-and-operations) * [![Salesforce](https://go.dev/images/logos/salesforce.svg) ![Salesforce](https://go.dev/images/logos/salesforce.svg)\ \ Salesforce - From Python/C to Go\ --------------------------------\ \ One of the big advantages is that Go's cross-platform features make porting code easy.\ \ View Case Study](https://www.zdnet.com/article/salesforce-why-we-ditched-python-for-googles-go-language-in-einstein-analytics/) * [![SIXT](https://go.dev/images/logos/sixt_dark.svg) ![SIXT](https://go.dev/images/logos/sixt_light.svg)\ \ Find out more about Golang at SIXT\ ----------------------------------\ \ “We have been doing Golang at SIXT since 2015. Back then there was not that many people here in our area which were doing Golang in production mode, mostly side projects. So it was really a bold move from our side but it proved to be quite successful. Fast forward to 2019 we have over 15 teams doing Golang. Many of the applications they have built are basically foundation for most of our mobility product offer including Rent, Ride and Share.”\ \ View Case Study](https://www.facebook.com/sixtkarriere/posts/find-out-more-about-golang-at-sixt-to-become-a-godeveloper-mfd-at-sixt-click-her/2049632898495842/) * [![Stream](https://go.dev/images/logos/getstream_dark.svg) ![Stream](https://go.dev/images/logos/getstream_light.svg)\ \ Stream – Why We Switched from Python to Go\ ------------------------------------------\ \ Go’s combination of a great ecosystem, easy onboarding for new developers, fast performance, solid support for concurrency and a productive programming environment make it a great choice. It allowed a small development team at Stream to power feeds and chat for over 500 million end users.\ \ View Case Study](https://getstream.io/blog/switched-python-go/) * [![Trivago](https://go.dev/images/logos/trivago_dark.svg) ![Trivago](https://go.dev/images/logos/trivago_light.svg)\ \ Trivago – Why We Chose Go\ -------------------------\ \ “Go’s simplicity and its sophisticated tooling let us scale not only our service but more importantly, the process of software engineering itself. Reducing the friction of onboarding and training someone has a significant impact on the company’s productivity, even more so in a constantly moving environment like trivago.”\ \ View Case Study](https://tech.trivago.com/2020/03/02/why-we-chose-go/) * [![Twitch](https://go.dev/images/logos/twitch.svg) ![Twitch](https://go.dev/images/logos/twitch.svg)\ \ Twitch - Go’s march to low latency GC\ -------------------------------------\ \ We use Go at Twitch for many of our busiest systems. Its simplicity, safety, performance, and readability make it a good tool for the problems we encounter with serving live video and chat to our millions of users.\ \ View Case Study](https://blog.twitch.tv/en/2016/07/05/gos-march-to-low-latency-gc-a6fa96f06eb7/) * [![Uber](https://go.dev/images/logos/uber_dark.svg) ![Uber](https://go.dev/images/logos/uber_light.svg)\ \ Uber - GPU-power analytics engine in Go\ ---------------------------------------\ \ AresDB \[,written in Go,\] is widely used at Uber to power our real-time data analytics dashboards, enabling us to make data-driven decisions at scale about myriad aspects of our business.\ \ View Case Study](https://eng.uber.com/aresdb/) * [![Wildlife Studios](https://go.dev/images/logos/wildlife_dark.svg) ![Wildlife Studios](https://go.dev/images/logos/wildlife_light.svg)\ \ How Wildlife Studios builds backend systems in Go\ -------------------------------------------------\ \ Wildlife is a Brazilian native global company focused on mobile gaming. We aim to develop games that will make billions of people happy. We have almost 40 million daily active users, and we rely on Go as the main language for our core platform, given its features to scale our backend services.\ \ View Case Study](https://medium.com/tech-at-wildlife-studios/pitaya-wildlifes-golang-go-af57865f7a11) * [![X](https://go.dev/images/logos/x.png) ![X](https://go.dev/images/logos/x.png)\ \ X - 5 billion sessions a day in realtime\ ----------------------------------------\ \ We now see about five billion sessions per day, and growing. Hundreds of millions of devices send millions of events every second to the Answers endpoint. During the time that it took you to read to here, the Answers back-end will have received and processed about 10,000,000 analytics events.\ \ View Case Study](https://blog.x.com/engineering/en_us/a/2015/handling-five-billion-sessions-a-day-in-real-time.html) go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # The Go Programming Language Build simple, secure, scalable systems with Go ============================================== * An open-source programming language supported by Google * Easy to learn and great for teams * Built-in concurrency and a robust standard library * Large ecosystem of partners, communities, and tools [Get Started](https://go.dev/learn/) [Download](https://go.dev/dl) Opens a new window with the Get Started guide. Opens a new window to download Go. Download packages for Windows 64-bit, macOS, Linux, and [more](https://go.dev/dl/) The `go` command by default downloads and authenticates modules using the Go module mirror and Go checksum database run by Google. [Learn more.](https://go.dev/dl) Opens in new window. ![Go Gopher climbing a ladder.](https://go.dev/images/gophers/ladder.svg) Companies using Go ------------------ Organizations in every industry use Go to power their software and services [View all stories](https://go.dev/solutions/) * [![](https://go.dev/images/logos/google.svg)\ \ View case study](https://go.dev/solutions/google/) * [![](https://go.dev/images/logos/paypal.svg)\ \ View case study](https://go.dev/solutions/paypal) * [![](https://go.dev/images/logos/american-express.svg)\ \ View case study](https://go.dev/solutions/americanexpress) * [![](https://go.dev/images/logos/mercadolibre_light.svg)\ \ View case study](https://go.dev/solutions/mercadolibre) * [![](https://go.dev/images/logos/bitly.svg)](https://bitly.com/blog/why-we-write-everything-in-go/?utm_source=go-dev&utm_medium=referral&utm_campaign=go-dev&utm_content=case-study) * [![](https://go.dev/images/logos/capitalone_light.svg)](https://medium.com/capital-one-tech/a-serverless-and-go-journey-credit-offers-api-74ef1f9fde7f) * [![](https://go.dev/images/logos/cockroach.svg)](https://www.cockroachlabs.com/blog/why-go-was-the-right-choice-for-cockroachdb/) * [![](https://go.dev/images/logos/dropbox.png)](https://dropbox.tech/infrastructure/open-sourcing-our-go-libraries) * [![](https://go.dev/images/logos/cloudflare_light.svg)](https://blog.cloudflare.com/graceful-upgrades-in-go/) * [![](https://go.dev/images/logos/meta_light.svg)](https://entgo.io/blog/2019/10/03/introducing-ent/) * [![](https://go.dev/images/logos/microsoft_light.svg)](https://cloudblogs.microsoft.com/opensource/2018/02/21/go-lang-brian-ketelsen-explains-fast-growth/) * [![](https://go.dev/images/logos/wildlife_light.svg)](https://medium.com/tech-at-wildlife-studios/pitaya-wildlifes-golang-go-af57865f7a11) * [![](https://go.dev/images/logos/bytedance_light.svg)](https://www.cloudwego.io/) * [![](https://go.dev/images/logos/netflix.svg)](https://medium.com/netflix-techblog/application-data-caching-using-ssds-5bf25df851ef) * [![](https://go.dev/images/logos/riot_light.svg)](https://technology.riotgames.com/news/leveraging-golang-game-development-and-operations) * [![](https://go.dev/images/logos/salesforce.svg)](https://www.zdnet.com/article/salesforce-why-we-ditched-python-for-googles-go-language-in-einstein-analytics/) * [![](https://go.dev/images/logos/twitch.svg)](https://blog.twitch.tv/en/2016/07/05/gos-march-to-low-latency-gc-a6fa96f06eb7/) * [![](https://go.dev/images/logos/uber_light.svg)](https://eng.uber.com/aresdb/) * [![](https://go.dev/images/logos/x.png)](https://blog.x.com/engineering/en_us/a/2015/handling-five-billion-sessions-a-day-in-real-time.html) * “At the time, no single team member knew Go, but **within a month, everyone was writing in Go** and we were building out the endpoints. It was the flexibility, how easy it was to use, and the really cool concept behind Go (how Go handles native concurrency, garbage collection, and of course safety+speed.) that helped engage us during the build. Also, who can beat that cute mascot!” — Jaime Enrique Garcia Lopez, Senior Software Development Manager at Capital One * "**A small language that compiles fast makes for a happy developer.** The Go language is small, compiles really fast, and as a result it lets your mind focus on the actual problem and less on the tool you are using to solve it. Code, test, debug cycles are so quick that you forget you are not working with an interpreted language. Looking at our code, you see **less boilerplate and more business logic.**" — Clayton Coleman, Lead Engineer, Open Shift at RedHat * “**Go has excellent characteristics for scalability and services written using it typically have very small memory footprints.** Because code is compiled into a single static binary, services can also be containerised with ease, making it much simpler to build and deploy. These attributes make **Go an ideal choice for companies building microservices**, as you can easily deploy into a highly available and scalable environment such as Kubernetes.” — Matt Boyle, Lead Software Engineer at Curve * "In our tightly managed environments where we run Go code, **we have seen a CPU reduction of approximately 10%** with cleaner and maintainable code." — Bala Natarajan, Sr. Director of Engineering, Developer Experience at PayPal * "Tooling has always been a problem with our legacy code base... but we have found that Go has excellent tooling, plus built-in testing, benchmarking, and profiling frameworks. It is easy to write efficient and resilient applications. **After working on Go, most of our developers don't want to go back to other languages.**" — Benjamin Cane, Vice President and Principal Engineer at American Express * "...when a programming language is designed for exactly the environment most of us use right now—scalable, cloud-based servers that are optimized for performance—a lot can go right." — John Biggs and Ben Popper, at Stack Overflow _navigate\_before_ _navigate\_next_ Try Go ------ Press Esc to move out of the editor. // You can edit this code! // Click here and start typing. package main import "fmt" func main() { fmt.Println("Hello, 世界") } Press Esc to move out of the editor. Waiting for remote server... Hello, World! Conway's Game of Life Fibonacci Closure Peano Integers Concurrent pi Concurrent Prime Sieve Peg Solitaire Solver Tree Comparison Run Share [Tour](https://go.dev/tour/ "Tour Go from your browser") What’s possible with Go ----------------------- Use Go for a variety of software development purposes * ![Sphere](https://go.dev/images/icons/sphere-dark.svg) ![Sphere](https://go.dev/images/icons/sphere.svg) ### Cloud & Network Services With a strong ecosystem of tools and APIs on major cloud providers, it is easier than ever to build services with Go. ![Packages.](https://go.dev/images/icons/package.svg) Popular Packages: * [cloud.google.com/go](https://cloud.google.com/go/home) * [aws/client](https://aws.amazon.com/sdk-for-go/) * [Azure/azure-sdk-for-go](https://github.com/Azure/azure-sdk-for-go) [Learn More _arrow\_forward_](https://go.dev/solutions/cloud/) * ![Command Line](https://go.dev/images/icons/command-folder-dark.svg) ![Command Line](https://go.dev/images/icons/command-folder.svg) ### Command-line Interfaces With popular open source packages and a robust standard library, use Go to create fast and elegant CLIs. ![Packages.](https://go.dev/images/icons/package.svg) Popular Packages: * [spf13/cobra](https://github.com/spf13/cobra) * [spf13/viper](https://github.com/spf13/viper) * [urfave/cli](https://github.com/urfave/cli) * [delve](https://github.com/go-delve/delve) * [chzyer/readline](https://github.com/chzyer/readline) [Learn More _arrow\_forward_](https://go.dev/solutions/clis/) * ![Code](https://go.dev/images/icons/code-dark.svg) ![Code](https://go.dev/images/icons/code.svg) ### Web Development With enhanced memory performance and support for several IDEs, Go powers fast and scalable web applications. ![Packages.](https://go.dev/images/icons/package.svg) Popular Packages: * [net/http](https://go.dev/pkg/net/http/) * [html/template](https://go.dev/pkg/html/template/) * [flosch/pongo2](https://github.com/flosch/pongo2) * [database/sql](https://go.dev/pkg/database/sql/) * [elastic/go-elasticsearch](https://github.com/elastic/go-elasticsearch) [Learn More _arrow\_forward_](https://go.dev/solutions/webdev/) * ![Sphere](https://go.dev/images/icons/gear-dark.svg) ![Sphere](https://go.dev/images/icons/gear.svg) ### DevOps & Site Reliability With fast build times, lean syntax, an automatic formatter and doc generator, Go is built to support both DevOps and SRE. ![Packages.](https://go.dev/images/icons/package.svg) Popular Packages: * [open-telemetry/opentelemetry-go](https://github.com/open-telemetry/opentelemetry-go) * [istio/istio](https://github.com/istio/istio) * [urfave/cli](https://github.com/urfave/cli) [Learn More _arrow\_forward_](https://go.dev/solutions/devops/) * ![Go Gopher is skateboarding.](https://go.dev/images/gophers/biplane.svg) [More use cases _arrow\_forward_](https://go.dev/solutions/use-cases) Get started with Go ------------------- Explore a wealth of learning resources, including guided journeys, courses, books, and more. [Get Started](https://go.dev/learn/) [Download Go](https://go.dev/doc/install/) * Resources to start on your own * [Guided learning journeys](https://go.dev/learn#guided-learning-journeys) Step-by-step tutorials to get your feet wet * [Online learning](https://go.dev/learn#online-learning) Browse resources and learn at your own pace * [Featured books](https://go.dev/learn#featured-books) Read through structured chapters and theories * [Cloud Self-paced labs](https://go.dev/learn#self-paced-labs) Jump in to deploying Go apps on GCP * In-Person Trainings * [Ardan Labs](https://www.ardanlabs.com/) Offering customized on-site live training classes. * [Gopher Guides](https://www.gopherguides.com/) Customized In-person, remote, and online training classes. Training for Developers by Developers. * [Boss Sauce Creative](https://bosssauce.it/services/training) Personalized or track-based Go training for teams. * [Shiju Varghese](https://github.com/shijuvar/gokit/tree/master/training) On-site classroom training on Go and consulting on distributed systems architectures, in India. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Use Cases - The Go Programming Language 1. [Why Go](https://go.dev/solutions/) 2. [Use Cases](https://go.dev/solutions/use-cases) Use Cases ========= * [![cloud icon](https://go.dev/solutions/cloud-green.svg) ![cloud icon](https://go.dev/solutions/cloud-white.svg)\ \ ### Cloud & Network Services\ \ With a strong ecosystem of tools and APIs on major cloud providers, it is easier than ever to build services with Go.\ \ Learn More](https://go.dev/solutions/cloud) * [![CLI icon](https://go.dev/solutions/clis-green.svg) ![CLI icon](https://go.dev/solutions/clis-white.svg)\ \ ### Command-line Interfaces (CLIs)\ \ With popular open source packages and a robust standard library, use Go to create fast and elegant CLIs.\ \ Learn More](https://go.dev/solutions/clis) * [![web dev icon](https://go.dev/solutions/webdev-green.svg) ![web dev icon](https://go.dev/solutions/webdev-white.svg)\ \ ### Web Development\ \ With enhanced memory performance and support for several IDEs, Go powers fast and scalable web applications.\ \ Learn More](https://go.dev/solutions/webdev) * [![ops icon](https://go.dev/solutions/devops-green.svg) ![ops icon](https://go.dev/solutions/devops-white.svg)\ \ ### Development Operations & Site Reliability Engineering\ \ With fast build times, lean syntax, an automatic formatter and doc generator, Go is built to support both DevOps and SRE.\ \ Learn More](https://go.dev/solutions/devops) Opens in new window. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # A Tour of Go [![](https://go.dev/images/go-logo-white.svg)](https://go.dev/) [A Tour of Go](https://go.dev/tour/list) ![System theme](https://go.dev/images/icons/brightness_6_gm_grey_24dp.svg) ![Dark theme](https://go.dev/images/icons/brightness_2_gm_grey_24dp.svg) ![Light theme](https://go.dev/images/icons/light_mode_gm_grey_24dp.svg) --- # American Express Uses Go for Payments & Rewards - The Go Programming Language 1. [Why Go](https://go.dev/solutions/) 2. [American Express Uses Go for Payments & Rewards](https://go.dev/solutions/americanexpress) American Express Uses Go for Payments & Rewards =============================================== 19 December 2019 ![American Express](https://go.dev/images/go_amex_case_study_logo.png) ![Quotation mark.](https://go.dev/images/quote.svg) What makes Go different from other programming languages is cognitive load. You can do more with less code, which makes it easier to reason about and understand the code that you do end up writing. The majority of Go code ends up looking quite similar, so, even if you’re working with a completely new codebase, you can get up and running pretty quickly. — Glen Balliet ,  Engineering Director of loyalty platforms  at American Express Go Improves Microservices and Speeds Productivity ------------------------------------------------- Founded in 1850, American Express is a globally integrated payments company offering charge and credit card products, merchant acquisition and processing services, network services, and travel-related services. American Express’ payment processing systems have been developed over its long history and have been updated across multiple architectural evolutions. Foremost in any update, payment processing needs to be fast, especially at very large transaction volumes, with resilience built across systems that must all be compliant with security and regulatory standards. With Go, American Express gains the speed and scalability it needs for both its payment and rewards networks. ### Modernizing American Express systems American Express understands that the programming language landscape is changing drastically. The company’s existing systems were purpose-built for high concurrency and low latency, but knowing that those systems would be re-platformed in the near future. The payments platform team decided to take the time to identify what languages were ideal for American Express’s evolving needs. The payments and rewards platform teams at American Express were among the first to start evaluating Go. These teams were focused on microservices, transaction routing, and load-balancing use cases, and they needed to modernize their architecture. Many American Express developers were familiar with the language’s capabilities and wanted to pilot Go for their high concurrency and low latency applications (such as custom transactional load balancers). With this goal in mind, the teams began lobbying senior leadership to deploy Go on the American Express payment platform. “We wanted to find the optimal language for writing fast and efficient applications for payment processing,” says Benjamin Cane, vice president and principal engineer at American Express. “To do so, we started an internal programming language showdown with the goal of seeing which language best fit our design and performance needs.” ### Comparing languages For their assessment, Cane’s team chose to build a microservice in four different programming languages. They then compared the four languages for speed/performance, tooling, testing, and ease of development. For the service, they decided on an ISO8583 to JSON converter. ISO8583 is an international standard for financial transactions, and it’s commonly used within American Express’s payment network. For the programming languages, they chose to compare C++, Go, Java and Node.js. With the exception of Go, all of these languages were already in use within American Express. From a speed perspective, Go achieved the second-best performance at 140,000 requests per second. Go showed that it excels when used for backend microservices. While Go may not have been the fastest language tested, its powerful tooling helped bolster its overall results. Go’s built-in testing framework, profiling capabilities, and benchmarking tools impressed the team. “It is easy to write effective tests in Go,” says Cane. “The benchmarking and profiling features make it simple to tune our application. Coupled with its fast build times, Go makes it easy to write well-tested and optimized code.” Ultimately, Go was selected by the team as the preferred language for building high-performance microservices. The tooling, testing frameworks, performance, and language simplicity were all key contributors. ### Go for infrastructure “Many of our services are running in Docker containers within our Kubernetes-based internal cloud platform” says Cane. Kubernetes is an open-source container-orchestration system written in Go. It provides clusters of hosts to run container based workloads, most notably Docker containers. Docker is a software product, also written in Go, that uses operating system level virtualization to provide portable software runtimes called containers. American Express also collects application metrics via Prometheus, an open-source monitoring and alerting toolkit written in Go. Prometheus collects and aggregates real-time events and metrics for monitoring and alerts. This triumvirate of Go solutions—Kubernetes, Docker, and Prometheus—has helped modernize American Express infrastructure. ### Improving performance with Go Today, scores of developers are programming with Go at American Express, with most working on platforms designed for high availability and performance. “Tooling has always been a critical area of need for our legacy code base,” says Cane. “We have found that Go has excellent tooling, plus built-in testing, benchmarking, and profiling frameworks. It is easy to write efficient and resilient applications.” “ After working on Go, most of our developers don’t want to go back to other languages. ” — Benjamin Cane ,  Vice President and Principal Engineer  at American Express American Express is just beginning to see the benefits of Go. For example, Go was designed from the ground up with concurrency in mind – using lightweight “goroutines” rather than heavier-weight operating system threads – making it practical to create hundreds of thousands of goroutines in the same address space. Using goroutines, American Express has seen improved performance numbers in its real-time transaction processing. Go’s garbage collection is also a major improvement over other languages, both in terms of performance and ease of development. “We saw far better results of garbage collection in Go than we did in other languages, and garbage collection for real time transaction processing is a big deal.” says Cane. “Tuning garbage collection in other languages can be very complicated. With Go you don’t tune anything.” To learn more, read [“Choosing Go at American Express”](https://americanexpress.io/choosing-go/) which goes into more depth about American Express’s Go adoption. ### Getting your enterprise started with Go Just as American Express is using Go to modernize its payment and rewards networks, dozens of other large enterprises are adopting Go as well. There are over one million developers using Go worldwide—spanning banking and commerce, gaming and media, technology, and other industries, at enterprises as diverse as [PayPal](https://go.dev/solutions/paypal) , [Mercado Libre](https://go.dev/solutions/mercadolibre) , Capital One, Dropbox, IBM, Mercado Libre, Monzo, New York Times, Salesforce, Square, Target, Twitch, Uber, and of course Google. To learn more about how Go can help your enterprise build reliable, scalable software as it does at American Express, visit [go.dev](https://go.dev/) today. ![American Express](https://go.dev/images/logos/american-express.svg) ![American Express](https://go.dev/images/logos/american-express.svg) ### About American Express Go provides American Express with the speed and scalability it needs for both its payment and rewards networks. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Command-line Interfaces (CLIs) - The Go Programming Language 1. [Why Go](https://go.dev/solutions/) 2. [Command-line Interfaces (CLIs)](https://go.dev/solutions/clis) Command-line Interfaces (CLIs) ============================== 4 October 2019 Overview -------- ### CLI developers prefer Go for portability, performance, and ease of creation Command line interfaces (CLIs), unlike graphical user interfaces (GUIs), are text-only. Cloud and infrastructure applications are primarily CLI-based due to their easy automation and remote capabilities. Key benefits ------------ ### Leverage fast compile times to build programs that start quickly and run on any system Developers of CLIs find Go to be ideal for designing their applications. Go compiles very quickly into a single binary, works across platforms with a consistent style, and brings a strong development community. From a single Windows or Mac laptop, developers can build a Go program for every one of the dozens of architectures and operating systems Go supports in a matter of seconds, no complicated build farms are needed. No other compiled language can be built as portably or quickly. Go applications are built into a single self contained binary making installing Go applications trivial. Specifically, **programs written in Go run on any system without requiring any existing libraries, runtimes, or dependencies**. And **programs written in Go have an immediate startup time**—similar to C or C++ but unobtainable with other programming languages. Use Case -------- ### Use Go for building elegant CLIs [“\ \ I was tasked with building our CLI tool and found two really great projects, Cobra and Viper, which make building CLI’s easy. Individually they are very powerful, very flexible and very good at what they do. But together they will help you show your next CLI who is boss!\ \ ”\ \ — Steve Domino ,  senior engineer and architect at Strala](https://medium.com/@skdomino/writing-better-clis-one-snake-at-a-time-d22e50e60056) [“\ \ Cobra is a great product to write small tools or even large ones. It’s more of a framework than a library, because when you call the binary that would create a skeleton, then you would be adding code in between.”\ \ ”\ \ — Francesc Campoy ,  VP of product at DGraph Labs and producer of Just For Func videos](https://www.youtube.com/watch?v=WvWPGVKLvR4) When developing CLIs in Go, two tools are widely used: Cobra & Viper. [Cobra](https://pkg.go.dev/github.com/spf13/cobra?tab=overview) is both a library for creating powerful modern CLI applications and a program to generate applications and CLI applications in Go. Cobra powers most of the popular Go applications including CoreOS, Delve, Docker, Dropbox, Git Lfs, Hugo, Kubernetes, and [many more](https://pkg.go.dev/github.com/spf13/cobra?tab=importedby) . With integrated command help, autocomplete and documentation “\[it\] makes documenting each command really simple,” says [Alex Ellis](https://blog.alexellis.io/5-keys-to-a-killer-go-cli/) , founder of OpenFaaS. [Viper](https://pkg.go.dev/github.com/spf13/viper?tab=overview) is a complete configuration solution for Go applications, designed to work within an app to handle configuration needs and formats. Cobra and Viper are designed to work together. Viper [supports nested structures](https://scene-si.org/2017/04/20/managing-configuration-with-viper/) in the configuration, allowing CLI developers to manage the configuration for multiple parts of a large application. Viper also provides all of the tooling need to easily build twelve factor apps. “If you don’t want to pollute your command line, or if you’re working with sensitive data which you don’t want to show up in the history, it’s a good idea to work with environment variables. To do this, you can use Viper,” [suggests Geudens](https://ordina-jworks.github.io/development/2018/10/20/make-your-own-cli-with-golang-and-cobra.html) . Featured users -------------- | Customer | Brief introduction | Projects using go | | --- | --- | --- | | ![Comcast](https://go.dev/images/logos/comcast.svg) ![Comcast](https://go.dev/images/logos/comcast.svg) | ![Comcast](https://go.dev/images/logos/comcast.svg) Comcast uses Go for a CLI client used to publish and subscribe to its high-traffic sites. The company also supports an open source client library which is written in Go - designed for working with Apache Pulsar. | * [Client library for Apache Pulsar](https://github.com/Comcast/pulsar-client-go)

* [Pulsar CLI Client](https://github.com/Comcast/pulsar-client-go/blob/master/cli/main.go) | | ![GitHub](https://go.dev/images/logos/github.svg) ![GitHub](https://go.dev/images/logos/github.svg) | ![GitHub](https://go.dev/images/logos/github.svg) GitHub uses Go for a command-line tool that makes it easier to work with GitHub, wrapping git in order to extend it with extra features and commands. | * [GitHub command-line tool](https://github.com/cli/cli) | | ![Hugo](https://go.dev/images/logos/hugo.svg) ![Hugo](https://go.dev/images/logos/hugo.svg) | ![Hugo](https://go.dev/images/logos/hugo.svg) Hugo is one of the most popular Go CLI applications powering thousands of sites, including this one. One reason for its popularity is its ease of install thanks to Go. Hugo author Bjørn Erik Pedersen writes “The single binary takes most of the pain out of installation and upgrades.” | * [Hugo Website](https://gohugo.io/) | | ![Kubernetes](https://go.dev/images/logos/kubernetes.svg) ![Kubernetes](https://go.dev/images/logos/kubernetes.svg) | ![Kubernetes](https://go.dev/images/logos/kubernetes.svg) Kubernetes is one of the most popular Go CLI applications. Kubernetes Creator, Joe Beda, said that for writing Kubernetes, “Go was the only logical choice”. Calling Go “the sweet spot” between low level languages like C++ and high level languages like Python. | * [Kubernetes + Go](https://blog.gopheracademy.com/birthday-bash-2014/kubernetes-go-crazy-delicious/) | | ![MongoDB](https://go.dev/images/logos/mongodb.svg) ![MongoDB](https://go.dev/images/logos/mongodb.svg) | ![MongoDB](https://go.dev/images/logos/mongodb.svg) MongoDB chose to implement their Backup CLI Tool in Go citing Go’s “C-like syntax, strong standard library, the resolution of concurrency problems via goroutines, and painless multi-platform distribution” as reasons. | * [MongoDB Backup Service](https://www.mongodb.com/blog/post/go-agent-go) | | ![Netflix](https://go.dev/images/logos/netflix.svg) ![Netflix](https://go.dev/images/logos/netflix.svg) | ![Netflix](https://go.dev/images/logos/netflix.svg) Netflix uses Go to build the CLI application ChaosMonkey, an application responsible for randomly terminating instances in production to ensure that engineers implement their services to be resilient to instance failures. | * [Netflix Techblog Article](https://medium.com/netflix-techblog/application-data-caching-using-ssds-5bf25df851ef) | | ![Stripe](https://go.dev/images/logos/stripe.svg) ![Stripe](https://go.dev/images/logos/stripe.svg) | ![Stripe](https://go.dev/images/logos/stripe.svg) Stripe uses Go for the Stripe CLI aimed to help build, test, and manage a Stripe integration right from the terminal. | * [Stripe CLI](https://github.com/stripe/stripe-cli) | | ![Uber](https://go.dev/images/logos/uber.svg) ![Uber](https://go.dev/images/logos/uber.svg) | ![Uber](https://go.dev/images/logos/uber.svg) Uber uses Go for several CLI tools, including the CLI API for Jaeger, a distributed tracing system used for monitoring microservice distributed systems. | * [CLI API for Jaeger](https://www.jaegertracing.io/docs/1.14/cli/) | More projects Get Started ----------- ### Go books for creating CLIs * [![Powerful Command-Line Applications in Go thumbnail.](https://go.dev/images/books/powerful-command-line-applications-in-go.jpg) Powerful Command-Line Applications in Go](https://www.amazon.com/Powerful-Command-Line-Applications-Go-Maintainable/dp/168050696X) * [![Go in Action thumbnail.](https://go.dev/images/books/go-in-action.jpg) Go in Action](https://www.amazon.com/Go-Action-William-Kennedy/dp/1617291781) * [![The Go Programming Language thumbnail.](https://go.dev/images/learn/go-programming-language-book.png) The Go Programming Language](https://www.gopl.io/) * [![Go Programming Blueprints thumbnail.](https://go.dev/images/learn/go-programming-blueprints.png) Go Programming Blueprints](https://github.com/matryer/goblueprints) ### CLI Libraries * [spf13/cobra](https://pkg.go.dev/github.com/spf13/cobra?tab=overview) A library for creating powerful modern CLI applications and a program to generate applications and CLI applications in Go * [spf13/viper](https://pkg.go.dev/github.com/spf13/viper?tab=overview) A complete configuration solution for Go applications, designed to work within an app to handle configuration needs and formats * [urfave/cli](https://pkg.go.dev/github.com/urfave/cli?tab=overview) A minimal framework for creating and organizing command line Go applications * [delve](https://pkg.go.dev/github.com/go-delve/delve?tab=overview) A simple and powerful tool built for programmers used to using a source-level debugger in a compiled language * [chzyer/readline](https://pkg.go.dev/github.com/chzyer/readline?tab=overview) A pure Golang implementation that provides most features in GNU Readline (under MIT license) * [dixonwille/wmenu](https://pkg.go.dev/github.com/dixonwille/wmenu?tab=overview) An easy-to-use menu structure for CLI applications that prompts users to make choices * [spf13/pflag](https://pkg.go.dev/github.com/spf13/pflag?tab=overview) A drop-in replacement for Go’s flag package, implementing POSIX/GNU-style flags * [golang/glog](https://pkg.go.dev/github.com/golang/glog?tab=overview) Leveled execution logs for Go * [go-prompt](https://pkg.go.dev/github.com/c-bata/go-prompt?tab=overview) A library for building powerful interactive prompts, making it easier to build cross-platform command line tools using Go. [View More](https://pkg.go.dev/search?q=command%20line%20OR%20CLI) go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # PayPal Taps Go to Modernize and Scale - The Go Programming Language 1. [Why Go](https://go.dev/solutions/) 2. [PayPal Taps Go to Modernize and Scale](https://go.dev/solutions/paypal) PayPal Taps Go to Modernize and Scale ===================================== 1 June 2020 ![PayPal](https://go.dev/images/go_paypal_case_study_logo.png) ![Quotation mark.](https://go.dev/images/quote.svg) Since our NoSQL and DB proxy used quite a bit of system details in a multi-threaded mode, the code got complex managing the different conditions, given that Go provides channels and routines to deal with complexity, we were able to structure the code to meet our requirements. — Bala Natarajan ,  Sr. Director of Engineering, Developer Experience  at PayPal New code infrastructure built on Go ----------------------------------- PayPal was created to democratize financial services and empower people and businesses to join and thrive in the global economy. Central to this effort is PayPal’s Payments Platform, which uses a combination of proprietary and third-party technologies to efficiently and securely facilitate transactions between millions of merchants and consumers worldwide. As the Payments Platform grew larger and more complicated, PayPal sought to modernize its systems and reduce time-to-market for new applications. Go’s value in producing clean, efficient code that readily scales as software deployment scales made the language a strong fit to support PayPal’s goals. Central to the Payment Processing Platform is a proprietary NoSQL database that PayPal had developed in C++. The complexity of the code, however, was substantially decreasing its developers’ ability to evolve the platform. Go’s simple code layouts, goroutines (lightweight threads of execution) and channels (which serve as the pipes that connect concurrent goroutines), made Go a natural choice for the NoSQL development team to simplify and modernize the platform. As a proof of concept, a development team spent six months learning Go and reimplementing the NoSQL system from the ground up in Go, during which they also provided insights on how Go could be implemented more broadly at PayPal. As of today, thirty percent of the clusters have been migrated to use the new NoSQL database. Using Go to simplify for scale ------------------------------ As PayPal’s platform becomes more intricate, Go provides a way to readily simplify the complexity of creating and running software at scale. The language provides PayPal with great libraries and fast tools, plus concurrency, garbage collection, and type safety. With Go, PayPal enables its developers to spend more time looking at code and thinking strategically, by freeing them from the noise of C++ and Java development. After the success of this newly re-written NoSQL system, more platform and content teams within PayPal began adopting Go. Natarajan’s current team is responsible for PayPal’s build, test, and release pipelines—all built in Go. The company has a large build and test farm which is completely managed using Go infrastructure to support builds-as-a-service (and tests-as-a-service) for developers across the company. ![Go gopher factory](https://go.dev/images/gophers/factory.png) Modernizing PayPal systems with Go ---------------------------------- With the distributed computing capabilities required by PayPal, Go was the right language to refresh their systems. PayPal needed programming that is concurrent and parallel, compiled for high performance and highly portable, and that brings developers the benefits of a modular, composable open-source architecture—Go has delivered all that and more to help PayPal modernize its systems. Security and supportability are key matters at PayPal, and the company’s operational pipelines are increasingly dominated by Go because the language’s cleanliness and modularity help them achieve these goals. PayPal’s deployment of Go engenders a platform of creativity for developers, allowing them to produce simple, efficient, and reliable software at scale for PayPal’s worldwide markets. As PayPal continues to modernize their software-defined networking (SDN) infrastructure with Go, they are seeing performance benefits in addition to more maintainable code. For example, Go now powers routers, load balances, and an increasing number of production systems. “ In our tightly managed environments where we run Go code, we have seen a CPU reduction of approximately ten percent with cleaner and maintainable code. ” — Bala Natarajan ,  Sr. Director of Engineering Go increases developer productivity ----------------------------------- As a global operation, PayPal needs its development teams to be effective at managing two kinds of scale: production scale, especially concurrent systems interacting with many other servers (such as cloud services); and development scale, especially large codebases developed by many programmers in coordination (such as open-source development) PayPal leverages Go to address these issues of scale. The company’s developers benefit from Go’s ability to combine the ease of programming of an interpreted, dynamically typed language with the efficiency and safety of a statically typed, compiled language. As PayPal modernizes its system, support for networked and multicore computing is critical. Go not only delivers such support but delivers quickly—it takes at most a few seconds to compile a large executable on a single computer. There are currently over 100 Go developers at PayPal, and future developers who choose to adopt Go will have an easier time getting the language approved thanks to the many successful implementations already in production at the company. Most importantly, PayPal developers have increased their productivity with Go. Go’s concurrency mechanisms have made it easy to write programs that get the most out of PayPal’s multicore and networked machines. Developers using Go also benefit from the fact that it compiles quickly to machine code and their apps gain the convenience of garbage collection and the power of run-time reflection. Speeding PayPal’s time to market -------------------------------- The first-class languages at PayPal today are Java and Node, with Go primarily used as an infrastructure language. While Go may never replace Node.js for certain applications, Natarajan is pushing to make Go a first-class language at PayPal. Through his efforts, PayPal is also evaluating moving to the Google Kubernetes Engine (GKE) to speed their new products’ time-to-market. The GKE is a managed, production-ready environment for deploying containerized applications, and brings Google’s latest innovations in developer productivity, automated operations, and open source flexibility. For PayPal, deploying to GKE would enable rapid development and iteration by making it easier for PayPal to deploy, update, and manage its applications and services. Plus PayPal will find it easier to run Machine Learning, General Purpose GPU, High-Performance Computing, and other workloads that benefit from specialized hardware accelerators supported by the GKE. Most importantly for PayPal, the combination of Go development and the GKE allows the company to scale effortless to meet demand, as Kubernetes autoscaling will allow PayPal to handle increased user demand for services—keeping them available when it matters most, then scale back in the quiet periods to save money. Getting your enterprise started with Go --------------------------------------- PayPal’s story is not unique; dozens of other large enterprises are discovering how Go can help them ship reliable software faster. There are over one million developers using Go worldwide—spanning banking and commerce, gaming and media, technology, and other industries, at enterprises as diverse as [American Express](https://go.dev/solutions/americanexpress) , [Mercado Libre](https://go.dev/solutions/mercadolibre) , Capital One, Dropbox, IBM, Monzo, New York Times, Salesforce, Square, Target, Twitch, Uber, and of course Google. To learn more about how Go can help your enterprise build reliable, scalable software as it does at PayPal, visit [go.dev](https://go.dev/) today. ![PayPal](https://go.dev/images/logos/paypal.svg) ![PayPal](https://go.dev/images/logos/paypal.svg) ### About PayPal Go’s value in producing clean, efficient code that readily scales as software deployment scales made the language a strong fit to support PayPal’s goals. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Go for Web Development - The Go Programming Language 1. [Why Go](https://go.dev/solutions/) 2. [Go for Web Development](https://go.dev/solutions/webdev) Go for Web Development ====================== 4 October 2019 Overview -------- ### Go delivers speed, security, and developer-friendly tools for Web Applications Go is designed to enable developers to rapidly develop scalable and secure web applications. Go ships with an easy to use, secure and performant web server and includes it own web templating library. Go has excellent support for all of the latest technologies from [HTTP/2](https://pkg.go.dev/net/http) , to databases like [MySQL](https://pkg.go.dev/mod/github.com/go-sql-driver/mysql) , [MongoDB](https://pkg.go.dev/mod/go.mongodb.org/mongo-driver) and [Elasticsearch](https://pkg.go.dev/mod/github.com/elastic/go-elasticsearch/v8) , to the latest encryption standards including [TLS 1.3](https://pkg.go.dev/crypto/tls) . Go web applications run natively on [Google App Engine](https://cloud.google.com/appengine/) and [Google Cloud Run](https://cloud.google.com/run/) (for easy scaling) or on any environment, cloud, or operating system thanks to Go’s extreme portability. Key Benefits ------------ ### Deploy across platforms in record speed For enterprises, Go is preferred for providing rapid cross-platform deployment. With its goroutines, native compilation, and the URI-based package namespacing, Go code compiles to a single, small binary—with zero dependencies—making it very fast. ### Leverage Go’s out-of-the-box performance to scale with ease Tigran Bayburtsyan, Co-Founder and CTO at Hexact Inc., summarizes five key reasons his company switched to Go: * **Compiles into a single binary** — “Using static linking, Go actually combining all dependency libraries and modules into one single binary file based on OS type and architecture.” * **Static type system** — “Type system is really important for large scale applications.” * **Performance** — “Go performed better because of its concurrency model and CPU scalability. Whenever we need to process some internal request, we are doing it with separate Goroutines which are 10x cheaper in resources than Python Threads.” * **No need for a web framework** — “In most of the cases you really don’t need any third-party library.” * **Great IDE support and debugging** — “After rewriting all projects to Go, we got 64 percent less code than we had earlier.” Featured users -------------- | Customer | Brief introduction | Projects using go | | --- | --- | --- | | ![Caddy](https://go.dev/images/logos/caddy.svg) ![Caddy](https://go.dev/images/logos/caddy.svg) | ![Caddy](https://go.dev/images/logos/caddy.svg) Caddy 2 is a powerful, enterprise-ready, open source web server with automatic HTTPS written in Go. Caddy offers greater memory safety than servers written in C. A hardened TLS stack powered by the Go standard library serves a significant portion of all Internet traffic. | * [Caddy 2](https://caddyserver.com/) | | ![Cloudflare](https://go.dev/images/logos/cloudflare-icon.svg) ![Cloudflare](https://go.dev/images/logos/cloudflare-icon.svg) | ![Cloudflare](https://go.dev/images/logos/cloudflare-icon.svg) Cloudflare speeds up and protects millions of websites, APIs, SaaS services, and other properties connected to the Internet. “Go is at the heart of CloudFlare’s services including handling compression for high-latency HTTP connections, our entire DNS infrastructure, SSL, load testing and more.” | * [Cloudflare and Go](https://blog.cloudflare.com/what-weve-been-doing-with-go/) | | ![gov.uk](https://go.dev/images/logos/govuk_dark.svg) ![gov.uk](https://go.dev/images/logos/govuk_light.svg) | ![gov.uk](https://go.dev/images/logos/govuk_light.svg) The simplicity and safety of the Go language were a good fit for the United Kingdom’s government’s HTTP infrastructure, and some brief experiments with the excellent net/http package convinced web developers they were on the right track. “In particular, Go’s concurrency model makes it absurdly easy to build performant I/O-bound applications.” | * [Building a new router for gov.uk](https://technology.blog.gov.uk/2013/12/05/building-a-new-router-for-gov-uk/)

* [Using Go in government](https://technology.blog.gov.uk/2014/11/14/using-go-in-government/) | | ![Hugo](https://go.dev/images/logos/hugo.svg) ![Hugo](https://go.dev/images/logos/hugo.svg) | ![Hugo](https://go.dev/images/logos/hugo.svg) Hugo is a fast and modern website engine written in Go, and designed to make website creation fun again. Websites built with Hugo are extremely fast and secure and can be hosted anywhere without any dependencies. | * [Hugo](https://gohugo.io/) | | ![Mattermost](https://go.dev/images/logos/mattermost_dark.svg) ![Mattermost](https://go.dev/images/logos/mattermost_light.svg) | ![Mattermost](https://go.dev/images/logos/mattermost_light.svg) Mattermost is a flexible, open source messaging platform that enables secure team collaboration. It’s written in Go and React. | * [Mattermost](https://mattermost.com/) | | ![Medium](https://go.dev/images/logos/medium_dark.svg) ![Medium](https://go.dev/images/logos/medium_light.svg) | ![Medium](https://go.dev/images/logos/medium_light.svg) Medium uses Go to power their social graph, their image server and several auxiliary services. “We’ve found Go very easy to build, package, and deploy. We like the type-safety without the verbosity and JVM tuning of Java.” | * [Medium's Go Services](https://medium.engineering/how-medium-goes-social-b7dbefa6d413) | | ![The Economist](https://go.dev/images/logos/economist.svg) ![The Economist](https://go.dev/images/logos/economist.svg) | ![The Economist](https://go.dev/images/logos/economist.svg) The Economist needed more flexibility to deliver content to increasingly diverse digital channels. Services written in Go were a key component of the new system that would enable The Economist to deliver scalable, high performing services and quickly iterate new products. “Overall, it was determined that Go was the language best designed for usability and efficiency in a distributed, cloud-based system.” | * [The Economist's Go microservices](https://www.infoq.com/articles/golang-the-economist/) | More projects Get Started ----------- ### Go books on web development * [![Web Development with Go thumbnail.](https://go.dev/images/books/web-development-with-go.jpg) Web Development with Go](https://www.amazon.com/Web-Development-Go-Building-Scalable-ebook/dp/B01JCOC6Z6) * [![Go Web Programming thumbnail.](https://go.dev/images/books/go-web-programming.jpg) Go Web Programming](https://www.amazon.com/Web-Programming-Sau-Sheong-Chang/dp/1617292567) * [![Web Development Cookbook: Build full-stack web applications with Go thumbnail.](https://go.dev/images/books/go-web-development-cookbook.jpg) Web Development Cookbook: Build full-stack web applications with Go](https://www.amazon.com/Web-Development-Cookbook-full-stack-applications-ebook/dp/B077TVQ28W) * [![Building RESTful Web services with Go thumbnail.](https://go.dev/images/books/building-restful-web-services-with-go.jpg) Building RESTful Web services with Go](https://www.amazon.com/Building-RESTful-Web-services-gracefully-ebook/dp/B072QB8KL1) * [![Mastering Go Web Services thumbnail.](https://go.dev/images/books/mastering-go-web-services.jpg) Mastering Go Web Services](https://www.amazon.com/Mastering-Web-Services-Nathan-Kozyra-ebook/dp/B00W5GUKL6) ### Web frameworks * [Echo](https://echo.labstack.com/) A high performance, extensible, and minimalist Go web framework * [Flamingo](https://www.flamingo.me/) A fast open-source framework based on Go with clean and scalable architecture * [Gin](https://gin-gonic.com/) A web framework written in Go, with a martini-like API. * [Gorilla](https://www.gorillatoolkit.org/) A web toolkit for the Go programming language. [View More](https://pkg.go.dev/search?q=web+framework) ### Routers * [net/http](https://pkg.go.dev/net/http) A standard library HTTP package * [julienschmidt/httprouter](https://pkg.go.dev/github.com/julienschmidt/httprouter?tab=overview) A lightweight high performance HTTP request router * [gorilla/mux](https://pkg.go.dev/github.com/gorilla/mux?tab=overview) A powerful HTTP router and URL matcher for building Go web servers with 🦍 * [Chi](https://pkg.go.dev/github.com/go-chi/chi?tab=overview) A lightweight, idiomatic and composable router for building Go HTTP services. [View More](https://pkg.go.dev/search?q=http%20router) ### Template Engines * [html/template](https://pkg.go.dev/html/template) A standard library HTML template engine * [flosch/pongo2](https://pkg.go.dev/github.com/flosch/pongo2?tab=overview) A Django-syntax like templating-language [View More](https://pkg.go.dev/search?q=templates) ### Databases & Drivers * [database/sql](https://pkg.go.dev/database/sql) A standard library interface with driver support for MySQL, Postgres, Oracle, MS SQL, BigQuery and most SQL databases * [mongo-driver/mongo](https://pkg.go.dev/go.mongodb.org/mongo-driver/mongo?tab=overview) The MongoDB supported driver for Go * [elastic/go-elasticsearch](https://pkg.go.dev/github.com/elastic/go-elasticsearch/v8?tab=overview) An Elasticsearch client for Go * [GORM](https://gorm.io/) An ORM library for Go * [Bleve](https://blevesearch.com/) Full-text search and indexing for Go * [CockroachDB](https://www.cockroachlabs.com/) An evolution of the database—architected for the cloud to deliver resilient, consistent, distributed SQL at scale [View More](https://pkg.go.dev/search?q=database%20OR%20sql) ### Web Libraries * [markbates/goth](https://pkg.go.dev/github.com/markbates/goth?tab=overview) Authentication for web apps * [jinzhu/gorm](https://pkg.go.dev/github.com/jinzhu/gorm?tab=overview) An ORM library for Go * [dgrijalva/jwt-go](https://pkg.go.dev/github.com/dgrijalva/jwt-go?tab=overview) A Go implementation of json web tokens [View More](https://pkg.go.dev/search?q=web) ### Other Projects * [gopherjs](https://pkg.go.dev/github.com/gopherjs/gopherjs?tab=overview) A compiler from Go to JavaScript allowing developers to write front-end code in Go which will run in all browsers. [View More](https://go.dev/solutions/webdev) ### Courses * [Learn to Create Web Applications using Go](https://www.usegolang.com/) , a paid online course ### Projects * [gopherjs](https://pkg.go.dev/github.com/gopherjs/gopherjs?tab=overview) , a compiler from Go to JavaScript allowing developers to write front-end code in Go which will run in all browsers. * [Hugo](https://gohugo.io/) , The world’s fastest framework for building websites * [Mattermost](https://mattermost.com/) , a flexible, open source messaging platform that enables secure team collaboration * [Caddy](https://caddyserver.com/) , a powerful, enterprise-ready, open source web server with automatic HTTPS written in Go go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Development Operations & Site Reliability Engineering - The Go Programming Language 1. [Why Go](https://go.dev/solutions/) 2. [Development Operations & Site Reliability Engineering](https://go.dev/solutions/devops) Development Operations & Site Reliability Engineering ===================================================== 3 October 2019 Overview -------- ### Go helps enterprises automate and scale Development Operations (DevOps) teams help engineering organizations automate tasks and improve their continuous integration and continuous delivery and deployment (CI/CD) process. DevOps can topple developmental silos and implement tooling and automation to enhance software development, deployment, and support. Site Reliability Engineering (SRE) was born at Google to make the company’s “large-scale sites more reliable, efficient, and scalable,” [writes Silvia Fressard](https://opensource.com/article/18/10/what-site-reliability-engineer) , an independent DevOps consultant. “And the practices they developed responded so well to Google’s needs that other big tech companies, such as Amazon and Netflix, also adopted them.” SRE requires a mix of development and operations skills, and “[empowers software developers](https://stackify.com/site-reliability-engineering/) to own the ongoing daily operation of their applications in production.” Go serves both siblings, DevOps and SRE, from its fast build times and lean syntax to its security and reliability support. Go’s concurrency and networking features also make it ideal for tools that manage cloud deployment—readily supporting automation while scaling for speed and code maintainability as development infrastructure grows over time. DevOps/SRE teams write software ranging from small scripts, to command-line interfaces (CLI), to complex automation and services, and Go’s feature set has benefits for every situation. Key Benefits ------------ ### Easily build small scripts with Go’s robust standard library and static typing Go’s fast build and startup times. Go’s extensive standard library—including packages for common needs like HTTP, file I/O, time, regular expressions, exec, and JSON/CSV formats—lets DevOps/SREs get right into their business logic. Plus, Go’s static type system and explicit error handling make even small scripts more robust. ### Quickly deploy CLIs with Go’s fast build times Every site reliability engineer has written “one-time use” scripts that turned into CLIs used by dozens of other engineers every day. And small deployment automation scripts turn into rollout management services. With Go, DevOps/SREs are in a great position to be successful when software scope inevitably creeps. Starting with Go puts you in a great position to be successful when that happens. ### Scale and maintain larger applications with Go’s low memory footprint and doc generator Go’s garbage collector means DevOps/SRE teams don’t have to worry about memory management. And Go’s automatic documentation generator (godoc) makes code self-documenting–lowering maintenance overhead and establishing best practices from the get-go. Featured users -------------- | Customer | Brief introduction | Projects using go | | --- | --- | --- | | ![Docker](https://go.dev/images/logos/docker.svg) ![Docker](https://go.dev/images/logos/docker.svg) | ![Docker](https://go.dev/images/logos/docker.svg) Docker is a software-as-a-service (SaaS) product, written in Go, that DevOps/SRE teams leverage to “drive secure automation and deployment at massive scale,” supporting their CI/CD efforts. | * [Docker CI/CD](https://www.docker.com/solutions/cicd) | | ![Drone](https://go.dev/images/logos/drone.svg) ![Drone](https://go.dev/images/logos/drone.svg) | ![Drone](https://go.dev/images/logos/drone.svg) Drone is a Continuous Delivery system built on container technology, written in Go, that uses a simple YAML configuration file, a superset of docker-compose, to define and execute Pipelines inside Docker containers. | * [Drone](https://github.com/drone) | | ![etcd](https://go.dev/images/logos/etcd.svg) ![etcd](https://go.dev/images/logos/etcd.svg) | ![etcd](https://go.dev/images/logos/etcd.svg) etcd is a strongly consistent, distributed key-value store that provides a reliable way to store data that needs to be accessed by a distributed system or cluster of machines, and it's written in Go. | * [etcd](https://github.com/etcd-io/etcd) | | ![IBM](https://go.dev/images/logos/ibm.svg) ![IBM](https://go.dev/images/logos/ibm.svg) | ![IBM](https://go.dev/images/logos/ibm.svg) IBM’s DevOps teams use Go through Docker and Kubernetes, plus other DevOps and CI/CD tools written in Go. The company also supports connection to it’s messaging middleware through a Go-specific API. | * [IBM Applications in Golang](https://developer.ibm.com/messaging/2019/02/05/simplified-ibm-mq-applications-golang/) | | ![Netflix](https://go.dev/images/logos/netflix.svg) ![Netflix](https://go.dev/images/logos/netflix.svg) | ![Netflix](https://go.dev/images/logos/netflix.svg) Netflix uses Go to handle large scale data caching, with a service called Rend, which manages globally replicated storage for personalization data. | * [Application Data Caching](https://medium.com/netflix-techblog/application-data-caching-using-ssds-5bf25df851ef)

* [Rend](https://github.com/netflix/rend) | | ![Microsoft](https://go.dev/images/logos/microsoft_dark.svg) ![Microsoft](https://go.dev/images/logos/microsoft_light.svg) | ![Microsoft](https://go.dev/images/logos/microsoft_light.svg) Microsoft uses Go in Azure Red Hat OpenShift services. This Microsoft solution provides DevOps teams with OpenShift clusters to maintain regulatory compliance and focus on application development. | * [OpenShift](https://azure.microsoft.com/en-us/services/openshift/) | | ![Terraform](https://go.dev/images/logos/terraform-icon.svg) ![Terraform](https://go.dev/images/logos/terraform-icon.svg) | ![Terraform](https://go.dev/images/logos/terraform-icon.svg) Terraform is a tool for building, changing, and versioning infrastructure safely and efficiently. It supports a number of cloud providers such as AWS, IBM Cloud, GCP, and Microsoft Azure - and it’s written in Go. | * [Terraform](https://www.terraform.io/intro/index.html) | | ![Prometheus](https://go.dev/images/logos/prometheus.svg) ![Prometheus](https://go.dev/images/logos/prometheus.svg) | ![Prometheus](https://go.dev/images/logos/prometheus.svg) Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. Most Prometheus components are written in Go, making them easy to build and deploy as static binaries. | * [Prometheus](https://github.com/prometheus/prometheus) | | ![YouTube](https://go.dev/images/logos/youtube.svg) ![YouTube](https://go.dev/images/logos/youtube.svg) | ![YouTube](https://go.dev/images/logos/youtube.svg) YouTube uses Go with Vitess (now part of PlanetScale), its database clustering system for horizontal scaling of MySQL through generalized sharding. Since 2011 it’s been a core component of YouTube’s database infrastructure, and has grown to encompass tens of thousands of MySQL nodes. | * [Vitess](https://github.com/vitessio/vitess) | More projects Get Started ----------- ### Go books on DevOps & SRE * [![Go Programming for Network Operations thumbnail.](https://go.dev/images/books/go-programming-for-network-operations.jpg) Go Programming for Network Operations](https://www.amazon.com/Go-Programming-Network-Operations-Automation-ebook/dp/B07JKKN34L/ref=sr_1_16) * [![Go Programming Blueprints thumbnail.](https://go.dev/images/learn/go-programming-blueprints.png) Go Programming Blueprints](https://github.com/matryer/goblueprints) * [![Go in Action thumbnail.](https://go.dev/images/books/go-in-action.jpg) Go in Action](https://www.amazon.com/Go-Action-William-Kennedy/dp/1617291781) * [![The Go Programming Language thumbnail.](https://go.dev/images/learn/go-programming-language-book.png) The Go Programming Language](https://www.gopl.io/) ### Monitoring and tracing * [open-telemetry/opentelemetry-go](https://pkg.go.dev/go.opentelemetry.io/otel) Vendor-neutral APIs and instrumentation for monitoring and distributed tracing * [jaegertracing/jaeger-client-go](https://pkg.go.dev/github.com/jaegertracing/jaeger-client-go?tab=overview) An open source distributed tracing system developed by Uber formats * [grafana/grafana](https://pkg.go.dev/github.com/grafana/grafana?tab=overview) An open-source platform for monitoring and observability * [istio/istio](https://pkg.go.dev/github.com/istio/istio?tab=overview) An open-source service mesh and integratable platform [View More](https://pkg.go.dev/search?q=tracing) ### CLI Libraries * [spf13/cobra](https://pkg.go.dev/github.com/spf13/cobra?tab=overview) A library for creating powerful modern CLI applications and a program to generate applications and CLI applications in Go * [spf13/viper](https://pkg.go.dev/github.com/spf13/viper?tab=overview) A complete configuration solution for Go applications, designed to work within an app to handle configuration needs and formats * [urfave/cli](https://pkg.go.dev/github.com/urfave/cli?tab=overview) A minimal framework for creating and organizing command line Go applications [View More](https://pkg.go.dev/search?q=command%20line%20OR%20CLI) ### Other projects * [golang-migrate/migrate](https://pkg.go.dev/github.com/golang-migrate/migrate?tab=overview) A database migration tool written in Go [View More](https://go.dev/solutions/devops) go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Go for Cloud & Network Services - The Go Programming Language 1. [Why Go](https://go.dev/solutions/) 2. [Go for Cloud & Network Services](https://go.dev/solutions/cloud) Go for Cloud & Network Services =============================== 4 October 2019 Overview -------- ### Go helps enterprises build and scale cloud computing systems As applications and processing move to the cloud, concurrency becomes a very big issue. Cloud computing systems, by their very nature, share and scale resources. Coordinating access to shared resources is an issue that impacts every application processing in the cloud, and requires programming languages “explicitly geared to develop highly reliable concurrent applications.” Go makes it very easy to scale as a company. This is very important because, as our engineering team grows, each service can be managed by a different unit. [— Ruchi Malik, developer at Choozle](https://builtin.com/software-engineering-perspectives/golang-advantages) Key Benefits ------------ ### Address tradeoff between development cycle time and server performance Go was created to address exactly these concurrency needs for scaled applications, microservices, and cloud development. In fact, over 75 percent of projects in the Cloud Native Computing Foundation are written in Go. Go helps reduce the need to make this tradeoff, with its fast build times that enable iterative development, lower memory and CPU utilization. Servers built with Go experience instant start up times and are cheaper to run in pay-as-you-go and serverless deployments. ### Address challenges with the modern cloud, delivering standard idiomatic APIs Go addresses many challenges developers face with the modern cloud, delivering standard idiomatic APIs, and built in concurrency to take advantage of multicore processors. Go’s low-latency and “no knob” tuning make Go a great balance between performance and productivity - granting engineering teams the power to choose and the power to move. Use Case -------- ### Use Go for Cloud Computing Go’s strengths shine when it comes to building services. Its speed and built-in support for concurrency results in fast and efficient services, while static typing, robust tooling, and emphasis on simplicity and readability help build reliable and maintainable code. Go has a strong ecosystem supporting service development. The [standard library](https://go.dev/pkg/) includes packages for common needs like HTTP servers and clients, JSON/XML parsing, SQL databases, and a range of security/encryption functionality, while the Go runtime includes tools for [race detection](https://go.dev/doc/articles/race_detector.html) , [benchmarking](https://go.dev/pkg/testing/#hdr-Benchmarks) /profiling, code generation, and static code analysis. The major Cloud providers ([GCP](https://cloud.google.com/go/home) , [AWS](https://aws.amazon.com/sdk-for-go/) , [Azure](https://docs.microsoft.com/en-us/azure/go/) ) have Go APIs for their services, and popular open source libraries provide support for API tooling ([Swagger](https://github.com/go-swagger/go-swagger) ), transport ([protocol buffers](https://github.com/golang/protobuf) , [gRPC](https://grpc.io/docs/quickstart/go/) ), monitoring ([OpenCensus](https://godoc.org/go.opencensus.io) ), Object-Relational Mapping ([gORM](https://gorm.io/) ), and authentication ([JWT](https://github.com/dgrijalva/jwt-go) ). The open source community has also provided several service frameworks, including [Go Kit](https://gokit.io/) , [Go Micro](https://micro.mu/docs/go-micro.html) , and [Gizmo](https://github.com/nytimes/gizmo) , which can be a great way to get started quickly. ### Go tools for Cloud Computing [![Docker](https://go.dev/images/logos/docker.svg) Docker](https://www.docker.com/) Docker is a platform-as-a-service that delivers software in containers. Containers bundle software, libraries, and config files, are hosted by a Docker Engine, and are run by a single operating-system kernel (utilizing less system resources than virtual machines). Cloud developers use Docker to manage their Go code and support multiple platforms, as Docker supports the development workflow and deployment process. [![Kubernetes](https://go.dev/images/logos/kubernetes.svg) Kubernetes](https://kubernetes.io/) Kubernetes is an open-source container-orchestration system, written in Go, for automating web app deployment. Web apps are often built using containers (as noted above) packaged with their dependencies and configurations. Kubernetes helps deploying and managing those containers at scale. Cloud programmers use Kubernetes to build, deliver, and scale containerized apps quickly—managing the growing complexity via APIs that controls how the containers will run. Featured users -------------- | Customer | Brief introduction | Projects using go | | --- | --- | --- | | ![Google](https://go.dev/images/logos/google-cloud.svg) ![Google](https://go.dev/images/logos/google-cloud.svg) | ![Google](https://go.dev/images/logos/google-cloud.svg) Google Cloud uses Go across its ecosystem of products and tools, including Kubernetes, gVisor, Knative, Istio, and Anthos. Go is fully supported on Google Cloud across all APIs and runtimes. | * [Go on Google Cloud Platform](https://cloud.google.com/go) | | ![Capital One](https://go.dev/images/logos/capitalone_dark.svg) ![Capital One](https://go.dev/images/logos/capitalone_light.svg) | ![Capital One](https://go.dev/images/logos/capitalone_light.svg) Capital One uses Go to power the Credit Offers API, a critical service. The engineering team is also building their serverless architecture with Go, citing Go’s speed and simplicity, and mentioning that “\[they\] didn’t want to go serverless without Go.” | * [Credit Offers API](https://medium.com/capital-one-tech/a-serverless-and-go-journey-credit-offers-api-74ef1f9fde7f) | | ![Dropbox](https://go.dev/images/logos/dropbox.svg) ![Dropbox](https://go.dev/images/logos/dropbox.svg) | ![Dropbox](https://go.dev/images/logos/dropbox.svg) Dropbox was built on Python, but in 2013 decided to migrate their performance-critical backends to Go. Today, most of the company’s infrastructure is written in Go. | * [Dropbox libraries](https://dropbox.tech/infrastructure/open-sourcing-our-go-libraries) | | ![Mercado Libre](https://go.dev/images/logos/mercadolibre_dark.svg) ![Mercado Libre](https://go.dev/images/logos/mercadolibre_light.svg) | ![Mercado Libre](https://go.dev/images/logos/mercadolibre_light.svg) MercadoLibre uses Go to scale its eCommerce platform. Go produces efficient code that readily scales as MercadoLibre’s online commerce grows. Go improves their productivity while streamlining and expanding MercadoLibre services. | * [MercadoLibre & Go](https://go.dev/solutions/mercadolibre) | | ![The New York Times](https://go.dev/images/logos/the-new-york-times-icon.svg) ![The New York Times](https://go.dev/images/logos/the-new-york-times-icon.svg) | ![The New York Times](https://go.dev/images/logos/the-new-york-times-icon.svg) The New York Times adopted Go “to build better back-end services”. As the usage of Go expanded with in the company they felt the need to create a toolkit to “to help developers quickly configure and build microservice APIs and pubsub daemons”, which they have open sourced. | * [NYTimes - Gizmo](https://open.nytimes.com/introducing-gizmo-aa7ea463b208)

* [Gizmo GitHub](https://github.com/nytimes/gizmo) | | ![Twitch](https://go.dev/images/logos/twitch.svg) ![Twitch](https://go.dev/images/logos/twitch.svg) | ![Twitch](https://go.dev/images/logos/twitch.svg) Twitch uses Go to power many of its busiest systems that serve live video and chat to millions of users. | * [Go’s march to low-latency GC](https://blog.twitch.tv/en/2016/07/05/gos-march-to-low-latency-gc-a6fa96f06eb7/) | | ![Uber](https://go.dev/images/logos/uber_dark.svg) ![Uber](https://go.dev/images/logos/uber_light.svg) | ![Uber](https://go.dev/images/logos/uber_light.svg) Uber uses Go to power several of its critical services that impact the experience of millions of drivers and passengers around the world. From their real-time analytics engine, AresDB, to their microservice for Geo-querying, Geofence, and their resource scheduler, Peloton. | * [AresDB](https://eng.uber.com/aresdb/)

* [Geofence](https://eng.uber.com/go-geofence/)

* [Peloton](https://eng.uber.com/open-sourcing-peloton/) | More projects Get Started ----------- ### Go books for cloud computing * [![Building Microservices with Go thumbnail.](https://go.dev/images/books/building-microservices-with-go.jpg) Building Microservices with Go](https://www.amazon.com/Building-Microservices-Go-efficient-microservices/dp/1786468662/) * [![Hands-On Software Architecture with Golang thumbnail.](https://go.dev/images/books/hands-on-software-architecture-with-golang.jpg) Hands-On Software Architecture with Golang](https://www.amazon.com/dp/1788622596/ref=cm_sw_r_tw_dp_U_x_-aZWDbS8PD7R4) * [![Building RESTful Web services with Go thumbnail.](https://go.dev/images/books/building-restful-web-services-with-go.jpg) Building RESTful Web services with Go](https://www.amazon.com/Building-RESTful-Web-services-gracefully-ebook/dp/B072QB8KL1) * [![Mastering Go Web Services thumbnail.](https://go.dev/images/books/mastering-go-web-services.jpg) Mastering Go Web Services](https://www.amazon.com/Mastering-Web-Services-Nathan-Kozyra/dp/178398130X) ### Web frameworks * [Echo](https://echo.labstack.com/) A high performance, extensible, and minimalist Go web framework * [Flamingo](https://www.flamingo.me/) A fast open-source framework based on Go with clean and scalable architecture * [Gin](https://gin-gonic.com/) A web framework written in Go, with a martini-like API. * [Gorilla](https://www.gorillatoolkit.org/) A web toolkit for the Go programming language. [View More](https://pkg.go.dev/search?q=web+framework) ### Routers * [net/http](https://pkg.go.dev/net/http) A standard library HTTP package * [julienschmidt/httprouter](https://pkg.go.dev/github.com/julienschmidt/httprouter?tab=overview) A lightweight high performance HTTP request router * [gorilla/mux](https://pkg.go.dev/github.com/gorilla/mux?tab=overview) A powerful HTTP router and URL matcher for building Go web servers with 🦍 * [Chi](https://pkg.go.dev/github.com/go-chi/chi?tab=overview) A lightweight, idiomatic and composable router for building Go HTTP services. [View More](https://pkg.go.dev/search?q=http%20router) ### Template Engines * [html/template](https://pkg.go.dev/html/template) A standard library HTML template engine * [flosch/pongo2](https://pkg.go.dev/github.com/flosch/pongo2?tab=overview) A Django-syntax like templating-language [View More](https://pkg.go.dev/search?q=templates) ### Databases & Drivers * [database/sql](https://pkg.go.dev/database/sql) A standard library interface with driver support for MySQL, Postgres, Oracle, MS SQL, BigQuery and most SQL databases * [mongo-driver/mongo](https://pkg.go.dev/go.mongodb.org/mongo-driver/mongo?tab=overview) The MongoDB supported driver for Go * [elastic/go-elasticsearch](https://pkg.go.dev/github.com/elastic/go-elasticsearch/v8?tab=overview) An Elasticsearch client for Go * [GORM](https://gorm.io/) An ORM library for Go * [Bleve](https://blevesearch.com/) Full-text search and indexing for Go * [CockroachDB](https://www.cockroachlabs.com/) An evolution of the database—architected for the cloud to deliver resilient, consistent, distributed SQL at scale [View More](https://pkg.go.dev/search?q=database%20OR%20sql) ### Web Libraries * [markbates/goth](https://pkg.go.dev/github.com/markbates/goth?tab=overview) Authentication for web apps * [jinzhu/gorm](https://pkg.go.dev/github.com/jinzhu/gorm?tab=overview) An ORM library for Go * [dgrijalva/jwt-go](https://pkg.go.dev/github.com/dgrijalva/jwt-go?tab=overview) A Go implementation of json web tokens [View More](https://pkg.go.dev/search?q=web) ### Other Projects * [gopherjs](https://pkg.go.dev/github.com/gopherjs/gopherjs?tab=overview) A compiler from Go to JavaScript allowing developers to write front-end code in Go which will run in all browsers. [View More](https://go.dev/solutions/cloud) go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # MercadoLibre Grows with Go - The Go Programming Language 1. [Why Go](https://go.dev/solutions/) 2. [MercadoLibre Grows with Go](https://go.dev/solutions/mercadolibre) MercadoLibre Grows with Go ========================== 10 November 2019 ![MercadoLibre](https://go.dev/images/go_mercadolibre_case_study_logo.png) ![Quotation mark.](https://go.dev/images/quote.svg) I think that **the tour of Go is by far the best introduction to a language that I’ve seen**, It’s really simple and it gives you a fair overview of probably 80 percent of the language. When we want to get developers to learn Go, and to get to production fast, we tell them to start with the tour of Go. — Eric Kohan ,  Software Engineering Manager  at MercadoLibre Go helps integrated ecosystem attract developers and scale eCommerce -------------------------------------------------------------------- MercadoLibre, Inc. hosts the largest online commerce ecosystem in Latin America and is present in 18 countries. Founded in 1999 and headquartered in Argentina, the company has turned to Go to help it scale and modernize its ecosystem. Go provides clean, efficient code that readily scales as MercadoLibre’s online commerce grows, and increases developer productivity by allowing their engineers to serve their ever-increasing audience while writing less code. ### MercadoLibre taps Go for scale Back in 2015, there was a growing sense within MercadoLibre that their existing API framework, on Groovy and Grails, was reaching its limits and the company needed a different platform to continue scaling. MercadoLibre’s platform was (and continues) to expand exponentially, which created a lot of extra work for its developers: Both Groovy and Grails require a lot of decisions from developers and Groovy is a dynamic programming language. This was not a good combination for quickly scaling growth, as MercadoLibre needed very experienced developers in this very resource intensive environment to develop and tune to achieve desired performance. Test execution times were slow, and build and deploy times were slow. Thus, the need for code efficiency and scalability became as important as the need for speed in code development. ### Go improves system efficiency As one example of Go’s contributions to network efficiency, the core API team builds and maintains the largest APIs at the center of the company’s microservices solutions. This team creates user APIs, which in turn are used by the MercadoLibre Marketplace, by the MercadoPago FinTech platform, by MercadoLibre’s shipping and logistics solutions, and other hosted solutions. With the high service levels demanded by these solutions—the average user API has between eight and ten million requests per minute—the team employs Go to serve them at less than ten milliseconds per request. The API team also deploys Docker containers—a software-as-a-service (SaaS) product, also written in Go—to virtualize their development and readily deploy their microservices via the Docker Engine. This system supports larger, mission-critical APIs that handle **more than 20 million requests per minute in Go.** One API made important use of Go’s concurrency primitives to efficiently multiplex IDs from several services. The team was able to accomplish this with just a few lines of Go code, and the success of this API convinced the core API team to migrate more and more microservices to Go. The end result for MercadoLibre has been improved cost-efficiencies and system response times. ### Go for scalability Historically, much of the company’s stack was based on Grails and Groovy backed by relational databases. However this big framework with multiple layers was soon found encountering scalability issues. Converting that legacy architecture to Go as a new, very thin framework for building APIs streamlined those intermediate layers and yielded great performance benefits. For example, one large Go service is now able to **run 70,000 requests per machine with just 20 MB of RAM.** “ Go was just marvelous for us. It’s very powerful and very easy to learn, and with backend infrastructure, has been great for us in terms of scalability. ” — Eric Kohan ,  Software Engineering Manager  at MercadoLibre Using **Go allowed MercadoLibre to cut the number of servers** they use for this service to one-eighth the original number (from 32 servers down to four), plus each server can operate with less power (originally four CPU cores, now down to two CPU cores). With Go, the company **obviated 88 percent of their servers and cut CPU on the remaining ones in half**—producing a tremendous cost-savings. Sitting between developers and the cloud providers, MercadoLibre uses a platform called Fury—a platform-as-a-service tool for building, deploying, monitoring, and managing services in a cloud-agnostic way. As a result, any team that wants to create a new service in Go has access to proven templates for a variety of service types, and can quickly spin up a repository in GitHub with starter code, a Docker image for the service, and a deployment pipeline. The end result is a system that allows engineers to focus on building innovative services while avoiding the tedious stages of setting up a new project—all while effectively standardizing the build and deployment pipelines. Today, **roughly half of Mercadolibre’s traffic is handled by Go applications.** ### MercadoLibre uses Go for developers The programming _lingua francas_ for MercadoLibre’s infrastructure are currently Go and Java. Every app, every program, every microservice is hosted on its own GitHub repository, plus the company uses an additional GitHub repository of toolkits to solve new problems and allow clients to interact with its services. These extensive and well-curated Go and Java toolkits allow programmers to develop new apps quickly and with great support. Plus, in a community of more than 2,800 developers, MercadoLibre has multiple internal groups available for chat and guidance on deploying Go, whether across different development centers or different countries. The company also fosters internal working groups to provide training sessions for new MercadoLibre Go developers, and hosts Go meetups for external developers to help build a broader community of Latin American Go developers. ### Go as a recruiting tool MercadoLibre’s Go advocacy has also become a strong recruiting tool for the company. MercadoLibre was among the first companies using Go in Argentina, and is perhaps the largest in Latin America using the language so widely in production. Headquartered in Buenos Aires, with many start-ups and emerging technology companies nearby, MercadoLibre’s adoption of Go has shaped the market for developers across the Pampas. “ We really see eye-to-eye with the larger philosophy of the language. We love Go’s simplicity, and we find that having its very explicit error handling has been a gain for developers because it results in safer, more stable code in production. ” — Eric Kohan ,  Software Engineering Manager  at MercadoLibre Buenos Aires is today a very competitive market for programmers, offering computer programmers many employment options, and the high demand for technology in the region drives great salaries, great benefits, and the ability to be selective when choosing an employer. As such, MercadoLibre—like all employers of engineers and programmers in the region—strives to provide an exciting workplace and strong career path. Go has proven to be a key differentiator for MercadoLibre: the company organizes Go workshops for external developers so they can come and learn Go, and when they enjoy what they are doing and the people they talk to, they quickly recognize MercadoLibre as an enticing place to work. ### Go enabling developers MercadoLibre employs Go for its simplicity with systems at scale, but that simplicity is also why the company’s developers love Go. The company also uses web pages like [Go by Example](https://gobyexample.com/) and [Effective Go](https://go.dev/doc/effective_go.html) to educate new programmers, and shares representative internal APIs written in Go to speed understanding and proficiency. MercadoLibre developers get the resources they need to embrace the language, then leverage their own skills and enthusiasm to start programming. “ Go has been great for writing business logic, and we are the team that writes those APIs. ” — Federico Martin Roasio ,  Technical Project Lead  at MercadoLibre MercadoLibre leverages Go’s expressive and clean syntax to make it easier for developers to write programs that run efficiently on modern cloud platforms. And while speed in development yields cost efficiency for the company, developers individually benefit from the swift learning curve Go delivers. Not only are MercadoLibre’s experienced engineers able to build highly critical applications very quickly with Go, but even entry-level engineers have been able to write services that, in other languages, MercadoLibre would only trust to more senior developers. For example, a key set of user APIs—handling almost ten million requests per minute—were developed by entry-level software engineers, many of whom only knew about programming from recent courses at university. Similarly, MercadoLibre has seen developers already proficient with other programming languages (such as Java or .NET or Ruby) learn Go fast enough start writing production services in just a few weeks. With Go, MercadoLibre’s **build times are three times (3x) faster** and their **test suite runs an amazing 24 times faster**. This means the company’s developers can make a change, then build and test that change much faster than they could before. And dropping MercadoLibre’s test suite runtimes from 90-seconds to **just 3-seconds with Go** was a huge boon for its developers—allowing them to keep focus (and context) while the much faster tests complete. Leveraging this success, MercadoLibre is committed not only to ongoing education for its programmers, but ongoing Go education. The company sends key engineering leaders to GopherCon and other Go events each year, MercadoLibre’s infrastructure and security teams encourage all the development teams to keep Go versions up to date, and the company has a team developing a _Go-meli-toolkit_: A complete Go library to interface all the services provided by Fury. ### Getting your enterprise started with Go Just as MercadoLibre started with a proof-of-concept project to implement Go, dozens of other large enterprises are adopting Go as well. There are over one million developers using Go worldwide—spanning banking and commerce, gaming and media, technology, and other industries, at enterprises as diverse as [American Express](https://go.dev/solutions/americanexpress) , [PayPal](https://go.dev/solutions/paypal) , Capital One, Dropbox, IBM, Monzo, New York Times, Salesforce, Square, Target, Twitch, Uber, and of course Google. To learn more about how Go can help your enterprise build reliable, scalable software as it does at MercadoLibre, visit [go.dev](https://go.dev/) today. ![MercadoLibre](https://go.dev/images/logos/mercadolibre_dark.svg) ![MercadoLibre](https://go.dev/images/logos/mercadolibre_light.svg) ### About MercadoLibre Go provides clean, efficient code that readily scales as MercadoLibre’s online commerce grows, and increases developer productivity by allowing their engineers to serve their ever-increasing audience while writing less code. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Download and install - The Go Programming Language 1. [Documentation](https://go.dev/doc/) 2. [Download and install](https://go.dev/doc/install) Download and install ==================== Download and install Go quickly with the steps described here. For other content on installing, you might be interested in: * [Managing Go installations](https://go.dev/doc/manage-install) -- How to install multiple versions and uninstall. * [Installing Go from source](https://go.dev/doc/install/source) -- How to check out the sources, build them on your own machine, and run them. [Download](https://go.dev/dl/) Go installation --------------- Select the tab for your computer's operating system below, then follow its installation instructions. Linux ![](https://go.dev/images/icons/underline.svg) Mac ![](https://go.dev/images/icons/underline.svg) Windows ![](https://go.dev/images/icons/underline.svg) 1. **Remove any previous Go installation** by deleting the /usr/local/go folder (if it exists), then extract the archive you just downloaded into /usr/local, creating a fresh Go tree in /usr/local/go: $ rm -rf /usr/local/go && tar -C /usr/local -xzf go1.14.3.linux-amd64.tar.gz ![](https://go.dev/images/icons/copy-paste.svg) ![](https://go.dev/images/icons/copy-paste-dark.svg) (You may need to run each command separately with the necessary permissions, as root or through `sudo`.) **Do not** untar the archive into an existing /usr/local/go tree. This is known to produce broken Go installations. 2. Add /usr/local/go/bin to the `PATH` environment variable. You can do this by adding the following line to your $HOME/.profile or /etc/profile (for a system-wide installation): export PATH=$PATH:/usr/local/go/bin ![](https://go.dev/images/icons/copy-paste.svg) ![](https://go.dev/images/icons/copy-paste-dark.svg) **Note:** Changes made to a profile file may not apply until the next time you log into your computer. To apply the changes immediately, just run the shell commands directly or execute them from the profile using a command such as `source $HOME/.profile`. 3. Verify that you've installed Go by opening a command prompt and typing the following command: $ go version ![](https://go.dev/images/icons/copy-paste.svg) ![](https://go.dev/images/icons/copy-paste-dark.svg) 4. Confirm that the command prints the installed version of Go. 1. Open the package file you downloaded and follow the prompts to install Go. The package installs the Go distribution to /usr/local/go. The package should put the /usr/local/go/bin directory in your `PATH` environment variable. You may need to restart any open Terminal sessions for the change to take effect. 2. Verify that you've installed Go by opening a command prompt and typing the following command: $ go version ![](https://go.dev/images/icons/copy-paste.svg) ![](https://go.dev/images/icons/copy-paste-dark.svg) 3. Confirm that the command prints the installed version of Go. 1. Open the MSI file you downloaded and follow the prompts to install Go. By default, the installer will install Go to `Program Files` or `Program Files (x86)`. You can change the location as needed. After installing, you will need to close and reopen any open command prompts so that changes to the environment made by the installer are reflected at the command prompt. 2. Verify that you've installed Go. 1. In **Windows**, click the **Start** menu. 2. In the menu's search box, type `cmd`, then press the **Enter** key. 3. In the Command Prompt window that appears, type the following command: $ go version ![](https://go.dev/images/icons/copy-paste.svg) ![](https://go.dev/images/icons/copy-paste-dark.svg) 4. Confirm that the command prints the installed version of Go. You're all set! --------------- Visit the [Getting Started tutorial](https://go.dev/doc/tutorial/getting-started.html) to write some simple Go code. It takes about 10 minutes to complete. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Using Go at Google - The Go Programming Language 1. [Why Go](https://go.dev/solutions/) 2. [Using Go at Google](https://go.dev/solutions/google/) Using Go at Google ================== 27 August 2020 ![Google](https://go.dev/images/go_core_data_case_study.png) ![Quotation mark.](https://go.dev/images/quote.svg) Go started in September 2007 when Robert Griesemer, Ken Thompson, and I began discussing a new language to address the engineering challenges we and our colleagues at Google were facing in our daily work. When we first released Go to the public in November 2009, we didn’t know if the language would be widely adopted or if it might influence future languages. Looking back from 2020, Go has succeeded in both ways: it is widely used both inside and outside Google, and its approaches to network concurrency and software engineering have had a noticeable effect on other languages and their tools. Go has turned out to have a much broader reach than we had ever expected. Its growth in the industry has been phenomenal, and it has powered many projects at Google. — Rob Pike The following stories are a small sample of the many ways that Go is used at Google. ### How Google’s Core Data Solutions Team Uses Go Google’s mission is “to organize the world’s information and make it universally accessible and useful.” One of the teams responsible for organizing that information is Google’s Core Data Solutions team. The team, among other things, maintains services to index web pages across the globe. These web indexing services help support products like Google Search by keeping search results updated and comprehensive, and they’re written in Go. [Learn more](https://go.dev/solutions/google/coredata/) * * * ### Chrome Content Optimization Service Runs on Go When the product Chrome comes to mind, you probably think solely of the user-installed browser. But behind the scenes, Chrome has an extensive fleet of backends. Among these is the Chrome Optimization Guide service. This service forms an important basis for Chrome’s user experience strategy, operating in the critical path for users, and is implemented in Go. [Learn more](https://go.dev/solutions/google/chrome/) * * * ### How the Firebase Hosting Team Scaled With Go The Firebase Hosting team provides static web hosting services for Google Cloud customers. They provide a static web host that sits behind a global content delivery network, and offer users tools that are easy to use. The team also develops features that range from uploading site files to registering domains to tracking usage. [Learn more](https://go.dev/solutions/google/firebase/) * * * ### Actuating Google Production: How Google’s Site Reliability Engineering Team Uses Go Google runs a small number of very large services. Those services are powered by a global infrastructure covering everything one needs: storage systems, load balancers, network, logging, monitoring, and many more. Nevertheless, it is not a static system - it cannot be. Architecture evolves, new products and ideas are created, new versions must be rolled out, configs pushed, database schema updated, and more. We end up deploying changes to our systems dozens of times per second. [Learn more](https://go.dev/solutions/google/sitereliability/) ![Google](https://go.dev/images/logos/google.svg) ![Google](https://go.dev/images/logos/google.svg) ### About Google Google is a technology company whose mission is to organize the world’s information and make it universally accessible and useful. Go was created at Google in 2007 to improve programming productivity in an era of multi-core networked machines and large codebases. Today, over 10 years since its public announcement in 2009, Go’s use inside Google has grown tremendously. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # A Tour of Go [![](https://go.dev/images/go-logo-white.svg)](https://go.dev/) [A Tour of Go](https://go.dev/tour/list) ![System theme](https://go.dev/images/icons/brightness_6_gm_grey_24dp.svg) ![Dark theme](https://go.dev/images/icons/brightness_2_gm_grey_24dp.svg) ![Light theme](https://go.dev/images/icons/light_mode_gm_grey_24dp.svg) --- # Tutorial: Get started with Go - The Go Programming Language 1. [Documentation](https://go.dev/doc/) 2. [Tutorials](https://go.dev/doc/tutorial/) 3. [Tutorial: Get started with Go](https://go.dev/doc/tutorial/getting-started) Tutorial: Get started with Go ============================= In this tutorial, you'll get a brief introduction to Go programming. Along the way, you will: * Install Go (if you haven't already). * Write some simple "Hello, world" code. * Use the `go` command to run your code. * Use the Go package discovery tool to find packages you can use in your own code. * Call functions of an external module. Prerequisites ------------- * **Some programming experience.** The code here is pretty simple, but it helps to know something about functions. * **A tool to edit your code.** Any text editor you have will work fine. Most text editors have good support for Go. The most popular are VSCode (free), GoLand (paid), and Vim (free). * **A command terminal.** Go works well using any terminal on Linux and Mac, and on PowerShell or cmd in Windows. Install Go ---------- Just use the [Download and install](https://go.dev/doc/install) steps. Write some code --------------- Get started with Hello, World. 1. Open a command prompt and cd to your home directory. On Linux or Mac: cd On Windows: cd %HOMEPATH% 2. Create a hello directory for your first Go source code. For example, use the following commands: mkdir hello cd hello 3. Enable dependency tracking for your code. When your code imports packages contained in other modules, you manage those dependencies through your code's own module. That module is defined by a go.mod file that tracks the modules that provide those packages. That go.mod file stays with your code, including in your source code repository. To enable dependency tracking for your code by creating a go.mod file, run the [`go mod init`](https://go.dev/ref/mod#go-mod-init) command, giving it the name of the module your code will be in. The name is the module's module path. In actual development, the module path will typically be the repository location where your source code will be kept. For example, the module path might be `github.com/mymodule`. If you plan to publish your module for others to use, the module path _must_ be a location from which Go tools can download your module. For more about naming a module with a module path, see [Managing dependencies](https://go.dev/doc/modules/managing-dependencies#naming_module) . For the purposes of this tutorial, just use `example/hello`. $ go mod init example/hello go: creating new go.mod: module example/hello 4. In your text editor, create a file hello.go in which to write your code. 5. Paste the following code into your hello.go file and save the file. package main import "fmt" func main() { fmt.Println("Hello, World!") } This is your Go code. In this code, you: * Declare a `main` package (a package is a way to group functions, and it's made up of all the files in the same directory). * Import the popular [`fmt` package](https://pkg.go.dev/fmt/) , which contains functions for formatting text, including printing to the console. This package is one of the [standard library](https://pkg.go.dev/std) packages you got when you installed Go. * Implement a `main` function to print a message to the console. A `main` function executes by default when you run the `main` package. 6. Run your code to see the greeting. $ go run . Hello, World! The [`go run`](https://go.dev/cmd/go/#hdr-Compile_and_run_Go_program) command is one of many `go` commands you'll use to get things done with Go. Use the following command to get a list of the others: $ go help Call code in an external package -------------------------------- When you need your code to do something that might have been implemented by someone else, you can look for a package that has functions you can use in your code. 1. Make your printed message a little more interesting with a function from an external module. 1. Visit pkg.go.dev and [search for a "quote" package](https://pkg.go.dev/search?q=quote) . 2. In the search results, locate and click on the v1 of the `rsc.io/quote` package (it should be listed with the "Other major versions" of `rsc.io/quote/v4`). 3. In the **Documentation** section, under **Index**, note the list of functions you can call from your code. You'll use the `Go` function. 4. At the top of this page, note that package `quote` is included in the `rsc.io/quote` module. You can use the pkg.go.dev site to find published modules whose packages have functions you can use in your own code. Packages are published in modules -- like `rsc.io/quote` -- where others can use them. Modules are improved with new versions over time, and you can upgrade your code to use the improved versions. 2. In your Go code, import the `rsc.io/quote` package and add a call to its `Go` function. After adding the highlighted lines, your code should include the following: package main import "fmt" import "rsc.io/quote" func main() { fmt.Println(quote.Go()) } 3. Add new module requirements and sums. Go will add the `quote` module as a requirement, as well as a go.sum file for use in authenticating the module. For more, see [Authenticating modules](https://go.dev/ref/mod#authenticating) in the Go Modules Reference. $ go mod tidy go: finding module for package rsc.io/quote go: found rsc.io/quote in rsc.io/quote v1.5.2 4. Run your code to see the message generated by the function you're calling. $ go run . Don't communicate by sharing memory, share memory by communicating. Notice that your code calls the `Go` function, printing a clever message about communication. When you ran `go mod tidy`, it located and downloaded the `rsc.io/quote` module that contains the package you imported. By default, it downloaded the latest version -- v1.5.2. Write more code --------------- With this quick introduction, you got Go installed and learned some of the basics. To write some more code with another tutorial, take a look at [Create a Go module](https://go.dev/doc/tutorial/create-module.html) . go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Security Best Practices for Go Developers - The Go Programming Language Security Best Practices for Go Developers ========================================= [Back to Go Security](https://go.dev/security) This page provides Go developers with best practices for prioritizing the security of their projects. From automating testing with fuzzing to easily checking for race conditions, these tips can help make your codebase more secure and reliable. Scan source code and binaries for vulnerabilities ------------------------------------------------- Regularly scanning your code and binaries for vulnerabilities helps identify potential security risks early. You can use [govulncheck](https://pkg.go.dev/golang.org/x/vuln/cmd/govulncheck) , backed by the [Go vulnerability database](https://pkg.go.dev/) , to scan your code for vulnerabilities and analyze which ones actually affect you. Get started with [the govulncheck tutorial](https://go.dev/doc/tutorial/govulncheck) . Govulncheck can also be integrated into CI/CD flows. The Go team provides a [GitHub Action for govulncheck](https://github.com/marketplace/actions/golang-govulncheck-action) on the GitHub Marketplace. Govulncheck also supports a `-json` flag to help developers integrate vulnerability scanning with other CI/CD systems. You can also scan for vulnerabilities directly in your code editor by using the [Go extension for Visual Studio Code](https://go.dev/security/vuln/editor) . Get started with [this tutorial](https://go.dev/doc/tutorial/govulncheck-ide) . Keep your Go version and dependencies up to date ------------------------------------------------ Keeping your [Go version up-to-date](https://go.dev/doc/install) offers access to the latest language features, performance improvements and patches for known security vulnerabilities. An updated Go version also ensures compatibility with newer versions of dependencies, helping to avoid potential integration issues. Review the [Go release history](https://go.dev/doc/devel/release) to see what changes have been made to Go between releases. The Go team issues point releases throughout the release cycle to address security bugs. Be sure to update to the latest minor Go version to ensure you have the latest security fixes. Maintaining up-to-date third-party dependencies is also crucial for software security, performance, and compliance with the latest standards in the Go ecosystem. However, updating to the latest versions without thorough review [can also be risky](https://research.swtch.com/npm-colors) , potentially introducing new bugs, incompatible changes, or even malicious code. Therefore, while it’s essential to update dependencies for the latest security patches and improvements, each update should be carefully reviewed and tested. Test with fuzzing to uncover edge-case exploits ----------------------------------------------- [Fuzzing](https://go.dev/security/fuzz) is a type of automated testing that uses coverage guidance to manipulate random inputs and walk through code to find and report potential vulnerabilities like SQL injections, buffer overflows, denial or service and cross-site scripting attacks. Fuzzing can often reach edge cases that programmers miss, or deem too improbable to test. Get started with [this tutorial](https://go.dev/doc/tutorial/fuzz) . Check for race conditions with Go’s race detector ------------------------------------------------- Race conditions occur when two or more [goroutines](https://go.dev/tour/concurrency/1) access the same resource concurrently, and at least one of those accesses is a write. This can lead to unpredictable, difficult-to-diagnose issues in your software. Identify potential race conditions in your Go code using the built-in [race detector](https://go.dev/doc/articles/race_detector) , which can help you ensure the safety and reliability of your concurrent programs. The race detector finds races that occur at runtime, however, so it will not find races in code paths that are not executed. To use the race detector, add the `-race` flag when running your tests or building your application, for example, `go test -race`. This will compile your code with the race detector enabled and report any race conditions it detects at runtime. When the race detector finds a data race in the program, it will [print a report](https://go.dev/doc/articles/race_detector#report-format) containing stack traces for conflicting accesses, and stacks where the involved goroutines were created. Use Vet to examine suspicious constructs ---------------------------------------- Go’s [vet command](https://pkg.go.dev/cmd/vet) is designed to analyze your source code and flag potential issues that might not necessarily be syntax errors, but could lead to problems during runtime. These include suspicious constructs, such as unreachable code, unused variables, and common mistakes around goroutines. By catching these issues early in the development process, go vet helps maintain code quality, reduces debugging time, and enhances overall software reliability. To run go vet for a specified project, run: go vet ./... Subscribe to golang-announce for notification of security releases ------------------------------------------------------------------ Go releases containing security fixes are pre-announced to the low-volume mailing list [golang-announce@googlegroups.com](https://groups.google.com/group/golang-announce) . If you want to know when security fixes to Go itself are on the way, subscribe. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # talks/2009 - The Go Programming Language Go talks ======== talks/2009 ---------- #### Files: [go\_talk-20091030.pdf](https://go.dev/talks/2009/go_talk-20091030.pdf) Opens in new window. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Unknown The Go Programming Language Rob Pike golang.org Oct 30, 2009 http://golang.org Monday, November 2, 2009 Go New Experimental Concurrent Garbage-collected Systems Language Monday, November 2, 2009 Hello, world package main import "fmt" func main() { fmt.Printf("Hello, \\n"); } Monday, November 2, 2009 Who Robert Griesemer, Ken Thompson, and Rob Pike started the project in late 2007. By mid 2008 the language was mostly designed and the implementation (compiler, run-time) starting to work. Ian Lance Taylor and Russ Cox joined in 2008. Lots of help from many others. Monday, November 2, 2009 Why Go fast! Make programming fun again. Monday, November 2, 2009 Our changing world No new major systems language in a decade. But much has changed: - sprawling libraries & dependency chains - dominance of networking - client/server focus - massive clusters - the rise of multi-core CPUs Major systems languages were not designed with all these factors in mind. Monday, November 2, 2009 Construction speed It takes too long to build software. The tools are slow and are getting slower. Dependencies are uncontrolled. Machines have stopped getting faster. Yet software still grows and grows. If we stay as we are, before long software construction will be unbearably slow. Monday, November 2, 2009 Type system tyranny Robert Griesemer: “Clumsy type systems drive people to dynamically typed languages.” Clunky typing: Taints good idea with bad implementation. Makes programming harder (think of C's const: well-intentioned but awkward in practice). Hierarchy is too stringent: Types in large programs do not easily fall into hierarchies. Programmers spend too much time deciding tree structure and rearranging inheritance. You can be productive or safe, not both. Monday, November 2, 2009 Why a new language? These problems are endemic and linguistic. New libraries won’t help. (Adding anything is going in the wrong direction.) Need to start over, thinking about the way programs are written and constructed. Monday, November 2, 2009 A New Language Monday, November 2, 2009 Goals The efficiency of a statically-typed compiled language with the ease of programming of a dynamic language. Safety: type-safe and memory-safe. Good support for concurrency and communication. Efficient, latency-free garbage collection. High-speed compilation. Monday, November 2, 2009 As xkcd observes... http://xkcd.com/303/ The image is licensed under a Creative Commons Attribution-NonCommercial 2.5 License. Monday, November 2, 2009 Compilation demo Monday, November 2, 2009 Design principles Keep concepts orthogonal. A few orthogonal features work better than a lot of overlapping ones. Keep the grammar regular and simple. Few keywords, parsable without a symbol table. Reduce typing. Let the language work things out. No stuttering; don't want to see foo.Foo \*myFoo = new foo.Foo(foo.FOO\_INIT) Avoid bookkeeping. But keep things safe. Reduce typing. Keep the type system clear. No type hierarchy. Too clumsy to write code by constructing type hierarchies. It can still be object-oriented. Monday, November 2, 2009 The big picture Fundamentals: Clean, concise syntax. Lightweight type system. No implicit conversions: keep things explicit. Untyped unsized constants: no more 0x80ULL. Strict separation of interface and implementation. Run-time: Garbage collection. Strings, maps, communication channels. Concurrency. Package model: Explicit dependencies to enable faster builds. Monday, November 2, 2009 New approach: Dependencies Construction speed depends on managing dependencies. Explicit dependencies in source allow: - fast compilation - fast linking The Go compiler pulls transitive dependency type info from the object file - but only what it needs. If A.go depends on B.go depends on C.go: - compile C.go, B.go, then A.go. - to compile A.go, compiler reads B.o not C.o. At scale, this can be a huge speedup. Monday, November 2, 2009 New approach: Concurrency Go provides a way to write systems and servers as concurrent, garbage-collected processes (goroutines) with support from the language and run-time. Language takes care of goroutine management, memory management. Growing stacks, multiplexing of goroutines onto threads is done automatically. Concurrency is hard without garbage collection. Garbage collection is hard without the right language. Monday, November 2, 2009 ... quickly Monday, November 2, 2009 Basics const N = 1024 // just a number const str = “this is a string\\n” var x, y \*float var ch = '\\u1234' /\* Define and use a type, T. \*/ type T struct { a, b int } var t0 \*T = new(T); t1 := new(T); // type taken from expr // Control structures: // (no parens, always braces) if len(str) > 0 { ch = str\[0\] } Monday, November 2, 2009 Program structure package main import "os" import "flag" var nFlag = flag.Bool("n", false, \`no \\n\`) func main() { !flag.Parse(); !s := ""; !for i := 0; i < flag.NArg(); i++ { ! ! if i > 0 { s += " " } ! ! s += flag.Arg(i) !} !if !\*nFlag { s += "\\n" } !os.Stdout.WriteString(s); } Monday, November 2, 2009 Constants type TZ int const ( UTC TZ = 0\*60\*60; EST TZ = -5\*60\*60; // and so on ) // iota enumerates: const ( bit0, mask0 uint32 = 1<](https://go.dev/doc/tutorial/call-module-code.html) go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Go Vulnerability Management - The Go Programming Language Go Vulnerability Management =========================== [Back to Go Security](https://go.dev/security) Overview -------- Go helps developers detect, assess, and resolve errors or weaknesses that are at risk of being exploited by attackers. Behind the scenes, the Go team runs a pipeline to curate reports about vulnerabilities, which are stored in the Go vulnerability database. Various libraries and tools can read and analyze those reports to understand how specific user projects may be affected. This functionality is integrated into the [Go package discovery site](https://pkg.go.dev/) and a new CLI tool, govulncheck. This project is a work in progress and under active development. We welcome your [feedback](https://go.dev/doc/security/vuln/#feedback) to help us improve! **NOTE**: To report a vulnerability in the Go project, please see the [Go Security Policy](https://go.dev/security/policy) . Architecture ------------ ![Go Vulnerability Management Architecture](https://go.dev/doc/security/vuln/architecture.png) Vulnerability management in Go consists of the following high-level pieces: 1. A **data pipeline** collects vulnerability information from various sources, including the [National Vulnerability Database (NVD)](https://nvd.nist.gov/) , the [GitHub Advisory Database](https://github.com/advisories) , and [directly from Go package maintainers](https://go.dev/s/vulndb-report-new) . 2. A **vulnerability database** is populated with reports using information from the data pipeline. All reports in the database are reviewed and curated by the Go Security team. Reports are formatted in the [Open Source Vulnerability (OSV) format](https://ossf.github.io/osv-schema/) and accessible through the [API](https://go.dev/security/vuln/database#api) . 3. **Integrations** with [pkg.go.dev](https://pkg.go.dev/) and govulncheck to enable developers to find vulnerabilities in their projects. The [govulncheck command](https://pkg.go.dev/golang.org/x/vuln/cmd/govulncheck) analyzes your codebase and only surfaces vulnerabilities that actually affect you, based on which functions in your code are transitively calling vulnerable functions. Govulncheck provides a low-noise, reliable way to find known vulnerabilities in your projects. Resources --------- ### Go Vulnerability Database The [Go vulnerability database](https://vuln.go.dev/) contains information from many existing sources in addition to direct reports by Go package maintainers to the Go security team. Each entry in the database is reviewed to ensure that the vulnerability’s description, package and symbol information, and version details are accurate. See [go.dev/security/vuln/database](https://go.dev/security/vuln/database) for more information about the Go vulnerability database, and [pkg.go.dev/vuln](https://pkg.go.dev/vuln) to view vulnerabilities in the database in your browser. We encourage package maintainers to [contribute](https://go.dev/doc/security/vuln/#feedback) information about public vulnerabilities in their own projects and [send us suggestions](https://go.dev/s/vuln-feedback) on how to reduce friction. ### Vulnerability Detection for Go Go’s vulnerability detection aims to provide a low-noise, reliable way for Go users to learn about known vulnerabilities that may affect their projects. Vulnerability checking is integrated into Go’s tools and services, including a new command line tool, [govulncheck](https://pkg.go.dev/golang.org/x/vuln/cmd/govulncheck) , the [Go package discovery site](https://pkg.go.dev/) , [major editors](https://go.dev/security/vuln/editor) like VS Code with the Go extension. To start using govulncheck, run the following from your project: $ go install golang.org/x/vuln/cmd/govulncheck@latest $ govulncheck ./... To enable vulnerability detection in your editor, see the instruction in the [editor integration](https://go.dev/security/vuln/editor) page. ### Go CNA The Go security team is a [CVE Numbering Authority](https://www.cve.org/ProgramOrganization/CNAs) . See [go.dev/security/vuln/cna](https://go.dev/security/vuln/cna) for more information. Feedback -------- We would love for you to contribute and help us make improvements in the following ways: * [Contribute new](https://go.dev/s/vulndb-report-new) and [update existing](https://go.dev/s/vulndb-report-feedback) information about public vulnerabilities for Go packages that you maintain * [Take this survey](https://go.dev/s/govulncheck-feedback) to share your experience using govulncheck * [Send us feedback](https://go.dev/s/vuln-feedback) about issues and feature requests FAQs ---- **How do I report a vulnerability in the Go project?** Report all security bugs in the Go project by email to [security@golang.org](mailto:security@golang.org) . Read [Go’s Security Policy](https://go.dev/security/policy) for more information about our processes. **How do I add a public vulnerability to the Go vulnerability database?** To request addition of a public vulnerability to the Go vulnerability database, [fill out this form](https://go.dev/s/vulndb-report-new) . A vulnerability is considered public if it has already been disclosed publicly, or if it exists in a package you maintain (and you are ready to disclose it). The form is only for public vulnerabilities in importable Go packages that are not maintained by the Go Team (anything outside the Go standard library, Go toolchain, and golang.org modules). The form can also be used to request a new CVE ID. [Read more here](https://go.dev/security/vuln/cna) about the Go CVE Numbering Authority. **How do I suggest an edit to a vulnerability?** To suggest an edit to an existing report in the Go vulnerability database, [fill out the form here](https://go.dev/s/vulndb-report-feedback) . **How do I report an issue or give feedback about govulncheck?** Submit your issue or feedback [on the Go issue tracker](https://go.dev/s/vuln-feedback) . **I found this vulnerability in another database. Why is it not in the Go vulnerability database?** Reports may be excluded from the Go vulnerability database for various reasons, including the relevant vulnerability not being present in a Go package, the vulnerability being in an installable command instead of an importable package, or the vulnerability being subsumed by another vulnerability that is already present in the database. You can learn more about the Go Security team’s [reasons for excluding reports here](https://go.dev/security/vuln/database#excluded-reports) . If you think that a report was incorrectly excluded from vuln.go.dev, [please let us know](https://go.dev/s/vulndb-report-feedback) . **Why does the Go vulnerability database not use severity labels?** Most vulnerability reporting formats use severity labels such as “LOW,” “MEDIUM”, and “CRITICAL” to indicate the impact of different vulnerabilities and to help developers prioritize security issues. For several reasons, however, Go avoids using such labels. The impact of a vulnerability is rarely universal, which means that severity indicators can often be deceptive. For example, a crash in a parser may be a critical severity issue if it is used to parse user-supplied input and can be leveraged in a DoS attack, but if the parser is used to parse local configuration files, even calling the severity “low” might be an overstatement. Labeling severity is also necessarily subjective. This is true even for [the CVE program](https://www.cve.org/About/Overview) , which posits a formula to break down relevant aspects of a vulnerability, such as attack vector, complexity, and exploitability. All of these, however, require subjective evaluation. We believe good descriptions of vulnerabilities are more useful than severity indicators. A good description can break down what an issue is, how it can be triggered, and what consumers should consider when determining the impact on their own software. Feel free to [file an issue](https://go.dev/s/vuln-feedback) if you would like to share your thoughts with us on this topic. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Tutorial: Find and fix vulnerable dependencies with govulncheck - The Go Programming Language 1. [Documentation](https://go.dev/doc/) 2. [Tutorials](https://go.dev/doc/tutorial/) 3. [Tutorial: Find and fix vulnerable dependencies with govulncheck](https://go.dev/doc/tutorial/govulncheck) Tutorial: Find and fix vulnerable dependencies with govulncheck =============================================================== Govulncheck is a low-noise tool that helps you find and fix vulnerable dependencies in your Go projects. It does this by scanning your project’s dependencies for known vulnerabilities and then identifying any direct or indirect calls to those vulnerabilities in your code. In this tutorial, you will learn how to use govulncheck to scan a simple program for vulnerabilities. You will also learn how to prioritize and evaluate vulnerabilities so that you can focus on fixing the most important ones first. To learn more about govulncheck, see the [govulncheck documentation](https://pkg.go.dev/golang.org/x/vuln/cmd/govulncheck) , and this [blog post on vulnerability management](https://go.dev/blog/vuln) for Go. We’d also love to [hear your feedback](https://go.dev/s/govulncheck-feedback) . Prerequisites ------------- * **Go.** We recommend using the latest version of Go to follow this tutorial. (For installation instructions, see [Installing Go](https://go.dev/doc/install) .) * **A code editor.** Any editor you have will work fine. * **A command terminal.** Go works well using any terminal on Linux and Mac, and on PowerShell or cmd in Windows. The tutorial will take you through the following steps: 1. Create a sample Go module with a vulnerable dependency 2. Install and run govulncheck 3. Evaluate vulnerabilities 4. Upgrade vulnerable dependencies Create a sample Go module with a vulnerable dependency ------------------------------------------------------ **Step 1.** To begin, create a new folder called `vuln-tutorial` and initialize a Go module. (If you are new to Go modules, check out [go.dev/doc/tutorial/create-module](https://go.dev/doc/tutorial/create-module) . For example, from your home directory, run the following: $ mkdir vuln-tutorial $ cd vuln-tutorial $ go mod init vuln.tutorial **Step 2.** Create a file called `main.go` within the `vuln-tutorial` folder, and copy the following code into it: package main import ( "fmt" "os" "golang.org/x/text/language" ) func main() { for _, arg := range os.Args[1:] { tag, err := language.Parse(arg) if err != nil { fmt.Printf("%s: error: %v\n", arg, err) } else if tag == language.Und { fmt.Printf("%s: undefined\n", arg) } else { fmt.Printf("%s: tag %s\n", arg, tag) } } } This sample program takes a list of language tags as command line arguments and prints a message for each tag indicating if it was parsed successfully, the tag is undefined, or whether there was an error while parsing the tag. **Step 3.** Run `go mod tidy`, which will populate the `go.mod` file with all the dependencies required by the code you added to `main.go` in the previous step. From the `vuln-tutorial` folder, run: $ go mod tidy You should see this output: go: finding module for package golang.org/x/text/language go: downloading golang.org/x/text v0.9.0 go: found golang.org/x/text/language in golang.org/x/text v0.9.0 **Step 4.** Open your `go.mod` file to verify that it looks like this: module vuln.tutorial go 1.20 require golang.org/x/text v0.9.0 **Step 5.** Downgrade the version of `golang.org/x/text` to v0.3.5, which contains known vulnerabilities. Run: $ go get golang.org/x/text@v0.3.5 You should see this output: go: downgraded golang.org/x/text v0.9.0 => v0.3.5 The `go.mod` file should now read: module vuln.tutorial go 1.20 require golang.org/x/text v0.3.5 Now, let’s see govulncheck in action. Install and run govulncheck --------------------------- **Step 6.** Install govulncheck with the `go install` command: $ go install golang.org/x/vuln/cmd/govulncheck@latest **Step 7.** From the folder you want to analyze (in this case, `vuln-tutorial`). Run: $ govulncheck ./... You should see this output: govulncheck is an experimental tool. Share feedback at https://go.dev/s/govulncheck-feedback. Using go1.20.3 and govulncheck@v0.0.0 with vulnerability data from https://vuln.go.dev (last modified 2023-04-18 21:32:26 +0000 UTC). Scanning your code and 46 packages across 1 dependent module for known vulnerabilities... Your code is affected by 1 vulnerability from 1 module. Vulnerability #1: GO-2021-0113 Due to improper index calculation, an incorrectly formatted language tag can cause Parse to panic via an out of bounds read. If Parse is used to process untrusted user inputs, this may be used as a vector for a denial of service attack. More info: https://pkg.go.dev/vuln/GO-2021-0113 Module: golang.org/x/text Found in: golang.org/x/text@v0.3.5 Fixed in: golang.org/x/text@v0.3.7 Call stacks in your code: main.go:12:29: vuln.tutorial.main calls golang.org/x/text/language.Parse === Informational === Found 1 vulnerability in packages that you import, but there are no call stacks leading to the use of this vulnerability. You may not need to take any action. See https://pkg.go.dev/golang.org/x/vuln/cmd/govulncheck for details. Vulnerability #1: GO-2022-1059 An attacker may cause a denial of service by crafting an Accept-Language header which ParseAcceptLanguage will take significant time to parse. More info: https://pkg.go.dev/vuln/GO-2022-1059 Found in: golang.org/x/text@v0.3.5 Fixed in: golang.org/x/text@v0.3.8 ### Interpreting the output \*Note: If you are not using the latest version of Go, you may see additional vulnerabilities from the standard library. Our code is affected by one vulnerability, [GO-2021-0113](https://pkg.go.dev/vuln/GO-2021-0113) , because it directly calls the `Parse` function of `golang.org/x/text/language` at a vulnerable version (v0.3.5). Another vulnerability, [GO-2022-1059](https://pkg.go.dev/vuln/GO-2022-1059) , exists in the `golang.org/x/text` module at v0.3.5. However, it is reported as “Informational” because our code never (directly or indirectly) calls any of its vulnerable functions. Now, let’s evaluate the vulnerabilities and determine an action to take. ### Evaluate vulnerabilities a. Evaluate vulnerabilities. First, read the description of the vulnerability and determine if it actually applies to your code and your use case. If you need more information, visit the “More info” link. Based on the description, vulnerability GO-2021-0113 can cause a panic when `Parse` is used to process untrusted user inputs. Let’s suppose that we intend our program to withstand untrusted inputs, and we are concerned about denial of service, so the vulnerability likely applies. GO-2022-1059 likely does not affect our code, because our code does not call any vulnerable functions from that report. b. Decide on an action. To mitigate GO-2021-0113, we have a few options: * **Option 1: Upgrade to a fixed version.** If there is a fix available, we can remove a vulnerable dependency by upgrading to a fixed version of the module. * **Option 2: Stop using the vulnerable symbol(s).** We could choose to remove all calls to the vulnerable function in our code. We would need to find an alternative or implement it ourselves. In this case, a fix is available, and the `Parse` function is integral to our program. Let’s upgrade our dependency to the “fixed in” version, v0.3.7. We decided to deprioritize fixing the informational vulnerability, GO-2022-1059, but because it is in the same module as GO-2021-0113, and because the fixed in version for it is v0.3.8, we can easily remove both at the same time by upgrading to v0.3.8. Upgrade vulnerable dependencies ------------------------------- Luckily, upgrading vulnerable dependencies is quite simple. **Step 8.** Upgrade `golang.org/x/text` to v0.3.8: $ go get golang.org/x/text@v0.3.8 You should see this output: go: upgraded golang.org/x/text v0.3.5 => v0.3.8 (Note that we could have also chosen to upgrade to `latest`, or any other version after v0.3.8). **Step 9.** Now run govulncheck again: $ govulncheck ./... You will now see this output: govulncheck is an experimental tool. Share feedback at https://go.dev/s/govulncheck-feedback. Using go1.20.3 and govulncheck@v0.0.0 with vulnerability data from https://vuln.go.dev (last modified 2023-04-06 19:19:26 +0000 UTC). Scanning your code and 46 packages across 1 dependent module for known vulnerabilities... No vulnerabilities found. Finally, govulncheck confirms that there are no vulnerabilities found. By regularly scanning your dependencies with command govulncheck, you can safeguard your codebase by identifying, prioritizing, and addressing vulnerabilities. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Vulnerability Scanning in IDEs - The Go Programming Language Vulnerability Scanning in IDEs ============================== [Back to Go Security](https://go.dev/security) Editors integrated with the [Go language server](https://pkg.go.dev/golang.org/x/tools/cmd/gopls) , such as [VS Code with the Go extension](https://marketplace.visualstudio.com/items?itemName=golang.go) , can detect vulnerabilities in your dependencies. There are two modes for detecting vulnerabilities in dependencies. Both are backed by the [Go vulnerability database](https://vuln.go.dev/) and complement each other. * Imports-based analysis: in this mode, editors report vulnerabilities by scanning the set of packages imported in the workspace, and surface the findings as diagnostics in the `go.mod` files. This is fast, but may report false positives in case your code imports the packages that contain vulnerable symbols but the functions with the vulnerability are not reachable. This mode can be enabled by the [`"vulncheck": "Imports"`](https://github.com/golang/tools/blob/master/gopls/doc/settings.md#vulncheck-enum) gopls setting. * `Govulncheck` analysis: this is based on the [`govulncheck`](https://pkg.go.dev/golang.org/x/vuln/cmd/govulncheck) command-line tool, which is embedded in `gopls`. This provides a low-noise, reliable way to confirm whether your code actually invokes vulnerable functions. Because this analysis can be expensive to compute, it must be manually triggered by using the “Run govulncheck to verify” code action associated with the diagnostic reports from the Import-based analysis, or using the [`"codelenses.run_govulncheck"`](https://github.com/golang/tools/blob/master/gopls/doc/settings.md#run-govulncheck) code lens on `go.mod` files. ![Vulncheck](https://go.dev/doc/security/vuln/vscode.gif) _Go: Toggle Vulncheck_ [(vulncheck.mp4)](https://user-images.githubusercontent.com/4999471/206977512-a821107d-9ffb-4456-9b27-6a6a4f900ba6.mp4) These features are available in `gopls` v0.11.0 or newer. Please share your feedback at [go.dev/s/vsc-vulncheck-feedback](https://go.dev/s/vsc-vulncheck-feedback) . Editor-specific Instructions ---------------------------- ### VS Code The [Go extension](https://marketplace.visualstudio.com/items?itemName=golang.go) offers the integration with gopls. The following settings are required to enable the vulnerability scanning features: "go.diagnostic.vulncheck": "Imports", // enable the imports-based analysis by default. "gopls": { "ui.codelenses": { "run_govulncheck": true // "Run govulncheck" code lens on go.mod file. } } The [“Go Toggle Vulncheck”](https://github.com/golang/vscode-go/wiki/Commands#go-toggle-vulncheck) command can be used to toggle the imports-based analysis on and off for the current workspace. ### Vim/NeoVim When using [coc.nvim](https://www.vim.org/scripts/script.php?script_id=5779) , the following setting will enable the import-based analysis. { "codeLens.enable": true, "languageserver": { "go": { "command": "gopls", ... "initializationOptions": { "vulncheck": "Imports", } } } } Notes and Caveats ----------------- * The extension does not scan private packages nor send any information on private modules. All the analysis is done by pulling a list of known vulnerable modules from the Go vulnerability database and then computing the intersection locally. * The import-based analysis uses the list of packages in the workspace modules, which may be different from what you see from `go.mod` files if `go.work` or module `replace`/`exclude` is used. * The govulncheck analysis result can become stale as you modify code or the Go vulnerability database is updated. In order to invalidate the analysis results manually, use the `"Reset go.mod diagnostics"` codelens shown on the top of the `go.mod` file. Otherwise, the result will be automatically invalidated after an hour. * These features currently don’t report vulnerabilities in the standard libraries or tool chains. We are still investigating UX on where to surface the findings and how to help users handle the issues. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # FIPS 140-3 Compliance - The Go Programming Language FIPS 140-3 Compliance ===================== Starting with Go 1.24, Go binaries can natively operate in a mode that facilitates FIPS 140-3 compliance. Moreover, the toolchain can build against frozen versions of the cryptography packages that constitute the Go Cryptographic Module. FIPS 140-3 ---------- NIST FIPS 140-3 is a U.S. Government compliance regime for cryptography applications that amongst other things requires the use of a set of approved algorithms, and the use of [CMVP](https://csrc.nist.gov/projects/cryptographic-module-validation-program) \-validated cryptographic modules tested in the target operating environments. The mechanisms described in this page facilitate compliance for Go applications. Applications that have no need for FIPS 140-3 compliance can safely ignore them, and should not enable FIPS 140-3 mode. **NOTE:** Simply using a FIPS 140-3 compliant and validated cryptographic module may not—on its own—satisfy all relevant regulatory requirements. The Go team cannot provide any guarantees or support around how usage of the provided FIPS 140-3 mode may, or may not, satisfy specific regulatory requirements for individual users. Care should be taken in determining if usage of this module satisfies your specific requirements. The Go Cryptographic Module --------------------------- The Go Cryptographic Module is a collection of standard library Go packages under `crypto/internal/fips140/...` that implement FIPS 140-3 approved algorithms. Public API packages such as `crypto/ecdsa` and `crypto/rand` transparently use the Go Cryptographic Module to implement FIPS 140-3 algorithms. FIPS 140-3 mode --------------- The run-time `fips140` [GODEBUG](https://go.dev/doc/godebug) option controls whether the Go Cryptographic Module operates in FIPS 140-3 mode. It defaults to `off`. It can’t be changed after the program has started. When operating in FIPS 140-3 mode (the `fips140` GODEBUG setting is `on`): * The Go Cryptographic Module automatically performs an integrity self-check at `init` time, comparing the checksum of the module’s object file computed at build time with the symbols loaded in memory. * All algorithms perform known-answer self-tests according to the relevant FIPS 140-3 Implementation Guidance, either at `init` time, or on first use. * Pairwise consistency tests are performed on generated cryptographic keys. Note that this can cause a slowdown of up to 2x for certain key types, which is especially relevant for ephemeral keys. * [`crypto/rand.Reader`](https://go.dev/pkg/crypto/rand/#Reader) is implemented in terms of a NIST SP 800-90A DRBG. To guarantee the same level of security as `GODEBUG=fips140=off`, random bytes are also sourced from the platform’s CSPRNG at every `Read` and mixed into the output as uncredited additional data. * The [`crypto/tls`](https://go.dev/pkg/crypto/tls/) package will ignore and not negotiate any protocol version, cipher suite, signature algorithm, or key exchange mechanism that is not FIPS 140-3 approved. * [`crypto/rsa.SignPSS`](https://go.dev/pkg/crypto/rsa/#SignPSS) with [`PSSSaltLengthAuto`](https://go.dev/pkg/crypto/rsa/#PSSSaltLengthAuto) will cap the length of the salt at the length of the hash. When `GODEBUG=fips140=only` is used, in addition to the above, cryptographic algorithms that are not FIPS 140-3 compliant will return an error or panic. Note that this mode is a best effort and can’t guarantee compliance with all FIPS 140-3 requirements. `GODEBUG=fips140=on` and `only` are not supported on OpenBSD, Wasm, AIX, and 32-bit Windows platforms. The `crypto/fips140` package ---------------------------- The [`crypto/fips140.Enabled`](https://go.dev/pkg/crypto/fips140/#Enabled) function reports whether FIPS 140-3 mode is active. The `GOFIPS140` environment variable ------------------------------------ The `GOFIPS140` environment variable can be used with `go build`, `go install`, and `go test` to select the version of the Go Cryptographic Module to be linked into the executable program. * `off` is the default, and uses the `crypto/internal/fips140/...` packages in the standard library tree in use. * `latest` is like `off`, but enables FIPS 140-3 mode by default. * `v1.0.0` uses Go Cryptographic Module version v1.0.0, frozen in early 2025 and first shipped with Go 1.24. It enables FIPS 140-3 mode by default. Module Validations ------------------ Google currently has a contractual relationship with [Geomys](https://geomys.org/) to facilitate at least yearly CMVP validations of the Go Cryptographic Module. At the time of validation we will freeze the Go Cryptographic Module and create a new module version for submission. These validations are tested on a comprehensive set of Operating Environments, supporting many popular operating system and hardware platform combinations. Off-cycle validations may be performed if security issues are discovered in the module. ### Validated Module Versions List of module versions which have completed [CMVP validation](https://csrc.nist.gov/projects/cryptographic-module-validation-program/validated-modules/search?SearchMode=Basic&ModuleName=Go+Cryptographic+Module&CertificateStatus=Active&ValidationYear=0) : _There are currently no module versions which have completed validation._ ### In Process Module Versions List of module versions which are currently in the [CMVP Modules In Process List](https://csrc.nist.gov/Projects/cryptographic-module-validation-program/modules-in-process/modules-in-process-list) : * v1.0.0 ([CAVP Certificate A6650](https://csrc.nist.gov/projects/cryptographic-algorithm-validation-program/details?validation=39260) ), Review Pending, available in Go 1.24+ ### Implementation Under Test Module Versions List of module versions which are currently in the [CMVP Implementation Under Test List](https://csrc.nist.gov/Projects/cryptographic-module-validation-program/modules-in-process/iut-list) : _There are currently no module versions under test._ Go+BoringCrypto --------------- The previous, unsupported mechanism to use the BoringCrypto module for certain FIPS 140-3 approved algorithms is currently still available, but it is meant to be removed and replaced with the mechanism described in this page in a future release. Go+BoringCrypto is incompatible with the native FIPS 140-3 mode. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Tutorial: Find and fix vulnerable dependencies with VS Code Go - The Go Programming Language 1. [Documentation](https://go.dev/doc/) 2. [Tutorials](https://go.dev/doc/tutorial/) 3. [Tutorial: Find and fix vulnerable dependencies with VS Code Go](https://go.dev/doc/tutorial/govulncheck-ide) Tutorial: Find and fix vulnerable dependencies with VS Code Go ============================================================== [Back to Go Security](https://go.dev/security) You can scan your code for vulnerabilities directly out of your editor with the Go extension for Visual Studio Code. Note: for an explanation of the vulnerability fix included in the images below, see the [govulncheck tutorial](https://go.dev/doc/tutorial/govulncheck) . Prerequisites: -------------- * **Go.** We recommend using the latest version of Go to follow this tutorial. For installation instructions, see [Installing Go](https://go.dev/doc/install) . * **VS Code**, updated to the latest version. [Download here](https://code.visualstudio.com/) . You can also use Vim (see [here](https://go.dev/security/vuln/editor#editor-specific-instructions) for details), but this tutorial focuses on VS Code Go. * **VS Code Go extension**, which can be [downloaded here](https://marketplace.visualstudio.com/items?itemName=golang.go) . * **Editor-specific settings changes.** You will need to modify your IDE settings according to [these specifications](https://go.dev/security/vuln/editor#editor-specific-instructions) before being able to replicate the results below. How to scan for vulnerabilities using VS Code Go ------------------------------------------------ **Step 1.** Run “Go: Toggle Vulncheck” The [Toggle Vulncheck](https://github.com/golang/vscode-go/wiki/Commands#go-toggle-vulncheck) command displays vulnerability analysis for all the dependencies listed in your modules. To use this command, open the [command palette](https://code.visualstudio.com/docs/getstarted/userinterface#_command-palette) in your IDE (Ctrl+Shift+P on Linux/Windows or Cmd+Shift+P on Mac OS) and run “Go: Toggle Vulncheck.” In your go.mod file, you will see the diagnostics for vulnerable dependencies that are used both directly and indirectly in your code. ![Run Toggle Vulncheck](https://go.dev/doc/tutorial/editor_tutorial_1.png) Note: To reproduce this tutorial on your own editor, copy the code below into your main.go file. // This program takes language tags as command-line // arguments and parses them. package main import ( "fmt" "os" "golang.org/x/text/language" ) func main() { for _, arg := range os.Args[1:] { tag, err := language.Parse(arg) if err != nil { fmt.Printf("%s: error: %v\n", arg, err) } else if tag == language.Und { fmt.Printf("%s: undefined\n", arg) } else { fmt.Printf("%s: tag %s\n", arg, tag) } } } Then, make sure the corresponding go.mod file for the program looks like this: module module1 go 1.18 require golang.org/x/text v0.3.5 Now, run `go mod tidy` to ensure that your go.sum file is updated. **Step 2.** Run govulncheck via a code action. Running govulncheck using a code action allows you to focus on the dependencies that are actually called in your code. Code actions in VS Code are marked by lightbulb icons; hover over the relevant dependency to see information about the vulnerability, then select “Quick Fix” to be shown a menu of options. Of these, choose “run govulncheck to verify.” This will return the relevant govulncheck output in your terminal. ![govulncheck code action](https://go.dev/doc/tutorial/editor_tutorial_2.png) ![VS Code Go govulncheck output](https://go.dev/doc/tutorial/editor_tutorial_3.png) **Step 3**. Hover over a dependency listed in your go.mod file. The relevant govulncheck output about a specific dependency can also be found by hovering over the dependency in the go.mod file. For a quick look at dependency information, this option is even more efficient than using a code action. ![Hover over dependency for vulnerability information](https://go.dev/doc/tutorial/editor_tutorial_4.png) **Step 4.** Upgrade to a “fixed in” version of your dependency. Code actions can also be used to quickly upgrade to a version of your dependency where the vulnerability is fixed. Do this by selecting the “Upgrade” option in the code action drop-down menu. ![Upgrade to Latest via code action menu](https://go.dev/doc/tutorial/editor_tutorial_5.png) Additional resources -------------------- * See [this page](https://go.dev/security/vuln/editor) for more information about vulnerability scanning in your IDE. The [Notes and Caveats section](https://go.dev/security/vuln/editor#notes-and-caveats) , in particular, discusses special cases for which vulnerability scanning may be more complex than in the example above. * The [Go Vulnerability Database](https://pkg.go.dev/vuln/) contains information from many existing sources in addition to direct reports by Go package maintainers to the Go security team. * See [Go Vulnerability Management](https://go.dev/security/vuln/) page provides a high-level view of Go’s architecture for detecting, reporting and managing vulnerabilities. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Go 1.25 is released - The Go Programming Language [The Go Blog](https://go.dev/blog/) ==================================== Go 1.25 is released =================== Dmitri Shuralyov, on behalf of the Go team 12 August 2025 Today the Go team is pleased to release Go 1.25. You can find its binary archives and installers on the [download page](https://go.dev/dl/) . Go 1.25 comes with improvements over Go 1.24 across its [tools](https://go.dev/doc/go1.25#tools) , the [runtime](https://go.dev/doc/go1.25#runtime) , [compiler](https://go.dev/doc/go1.25#compiler) , [linker](https://go.dev/doc/go1.25#linker) , and the [standard library](https://go.dev/doc/go1.25#library) , including the addition of one [new package](https://go.dev/doc/go1.25#new-testingsynctest-package) . There are [port-specific](https://go.dev/doc/go1.25#ports) changes and [`GODEBUG` settings](https://go.dev/doc/godebug#go-125) updates. Some of the additions in Go 1.25 are in an experimental stage and become exposed only when you explicitly opt in. Notably, a [new experimental garbage collector](https://go.dev/doc/go1.25#new-experimental-garbage-collector) , and a [new experimental `encoding/json/v2` package](https://go.dev/doc/go1.25#json_v2) are available for you to try ahead of time and provide your feedback. It really helps if you’re able to do that! Please refer to the [Go 1.25 Release Notes](https://go.dev/doc/go1.25) for the complete list of additions, changes and improvements in Go 1.25. Over the next few weeks, follow-up blog posts will cover some of the topics relevant to Go 1.25 in more detail. Check back in later to read those posts. Thanks to everyone who contributed to this release by writing code, filing bugs, trying out experimental additions, sharing feedback, and testing the release candidates. Your efforts helped make Go 1.25 as stable as possible. As always, if you notice any problems, please [file an issue](https://go.dev/issue/new) . We hope you enjoy using the new release! **Next article:** [Container-aware GOMAXPROCS](https://go.dev/blog/container-aware-gomaxprocs) **Previous article:** [The FIPS 140-3 Go Cryptographic Module](https://go.dev/blog/fips140) **[Blog Index](https://go.dev/blog/all) ** go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # talks/static - The Go Programming Language Go talks ======== talks/static ------------ Opens in new window. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # talks/2019 - The Go Programming Language Go talks ======== talks/2019 ---------- #### Sub-directories: [playground-v3](https://go.dev/talks/2019/playground-v3) Opens in new window. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # talks/2011 - The Go Programming Language Go talks ======== talks/2011 ---------- #### Slide decks: [lex.slide](https://go.dev/talks/2011/lex.slide) : Lexical Scanning in Go #### Files: [Real\_World\_Go.pdf](https://go.dev/talks/2011/Real_World_Go.pdf) [Writing\_Web\_Apps\_in\_Go.pdf](https://go.dev/talks/2011/Writing_Web_Apps_in_Go.pdf) #### Sub-directories: [lex](https://go.dev/talks/2011/lex) Opens in new window. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Using prepared statements - The Go Programming Language Using prepared statements ========================= You can define a prepared statement for repeated use. This can help your code run a bit faster by avoiding the overhead of re-creating the statement each time your code performs the database operation. **Note:** Parameter placeholders in prepared statements vary depending on the DBMS and driver you’re using. For example, the [pq driver](https://pkg.go.dev/github.com/lib/pq) for Postgres requires a placeholder like `$1` instead of `?`. ### What is a prepared statement? A prepared statement is SQL that is parsed and saved by the DBMS, typically containing placeholders but with no actual parameter values. Later, the statement can be executed with a set of parameter values. ### How you use prepared statements When you expect to execute the same SQL repeatedly, you can use an `sql.Stmt` to prepare the SQL statement in advance, then execute it as needed. The following example creates a prepared statement that selects a specific album from the database. [`DB.Prepare`](https://pkg.go.dev/database/sql#DB.Prepare) returns an [`sql.Stmt`](https://pkg.go.dev/database/sql#Stmt) representing a prepared statement for a given SQL text. You can pass the parameters for the SQL statement to `Stmt.Exec`, `Stmt.QueryRow`, or `Stmt.Query` to run the statement. // AlbumByID retrieves the specified album. func AlbumByID(id int) (Album, error) { // Define a prepared statement. You'd typically define the statement // elsewhere and save it for use in functions such as this one. stmt, err := db.Prepare("SELECT * FROM album WHERE id = ?") if err != nil { log.Fatal(err) } defer stmt.Close() var album Album // Execute the prepared statement, passing in an id value for the // parameter whose placeholder is ? err := stmt.QueryRow(id).Scan(&album.ID, &album.Title, &album.Artist, &album.Price, &album.Quantity) if err != nil { if err == sql.ErrNoRows { // Handle the case of no rows returned. } return album, err } return album, nil } ### Prepared statement behavior A prepared [`sql.Stmt`](https://pkg.go.dev/database/sql#Stmt) provides the usual `Exec`, `QueryRow`, and `Query` methods for invoking the statement. For more on using these methods, see [Querying for data](https://go.dev/doc/database/querying) and [Executing SQL statements that don’t return data](https://go.dev/doc/database/change-data) . However, because an `sql.Stmt` already represents a preset SQL statement, its `Exec`, `QueryRow`, and `Query` methods take only the SQL parameter values corresponding to placeholders, omitting the SQL text. You can define a new `sql.Stmt` in different ways, depending on how you will use it. * `DB.Prepare` and `DB.PrepareContext` create a prepared statement that can be executed in isolation, by itself outside a transaction, just like `DB.Exec` and `DB.Query` are. * `Tx.Prepare`, `Tx.PrepareContext`, `Tx.Stmt`, and `Tx.StmtContext` create a prepared statement for use in a specific transaction. `Prepare` and `PrepareContext` use SQL text to define the statement. `Stmt` and `StmtContext` use the result of `DB.Prepare` or `DB.PrepareContext`. That is, they convert a not-for-transactions `sql.Stmt` into a for-this-transaction `sql.Stmt`. * `Conn.PrepareContext` creates a prepared statement from an `sql.Conn`, which represents a reserved connection. Be sure that `stmt.Close` is called when your code is finished with a statement. This will release any database resources (such as underlying connections) that may be associated with it. For statements that are only local variables in a function, it’s enough to `defer stmt.Close()`. #### Functions for creating a prepared statement | Function | Description | | --- | --- | | `[DB.Prepare](https://pkg.go.dev/database/sql#DB.Prepare) `
`[DB.PrepareContext](https://pkg.go.dev/database/sql#DB.PrepareContext) ` | Prepare a statement for execution in isolation or that will be converted to an in-transaction' prepared statement using Tx.Stmt. | | `[Tx.Prepare](https://pkg.go.dev/database/sql#Tx.Prepare) `
`[Tx.PrepareContext](https://pkg.go.dev/database/sql#Tx.PrepareContext) `
`[Tx.Stmt](https://pkg.go.dev/database/sql#Tx.Stmt) `
`[Tx.StmtContext](https://pkg.go.dev/database/sql#Tx.StmtContext) ` | Prepare a statement for use in a specific transaction. For more, see [Executing transactions](https://go.dev/doc/database/execute-transactions)
. | | `[Conn.PrepareContext](https://pkg.go.dev/database/sql#Conn.PrepareContext) ` | For use with reserved connections. For more, see [Managing connections](https://go.dev/doc/database/manage-connections)
. | go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Managing connections - The Go Programming Language Managing connections ==================== For the vast majority of programs, you needn’t adjust the `sql.DB` connection pool defaults. But for some advanced programs, you might need to tune the connection pool parameters or work with connections explicitly. This topic explains how. The [`sql.DB`](https://pkg.go.dev/database/sql#DB) database handle is safe for concurrent use by multiple goroutines (meaning the handle is what other languages might call “thread-safe”). Some other database access libraries are based on connections that can only be used for one operation at a time. To bridge that gap, each `sql.DB` manages a pool of active connections to the underlying database, creating new ones as needed for parallelism in your Go program. The connection pool is suitable for most data access needs. When you call an `sql.DB` `Query` or `Exec` method, the `sql.DB` implementation retrieves an available connection from the pool or, if needed, creates one. The package returns the connection to the pool when it’s no longer needed. This supports a high level of parallelism for database access. ### Setting connection pool properties You can set properties that guide how the `sql` package manages a connection pool. To get statistics about the effects of these properties, use [`DB.Stats`](https://pkg.go.dev/database/sql#DB.Stats) . #### Setting the maximum number of open connections [`DB.SetMaxOpenConns`](https://pkg.go.dev/database/sql#DB.SetMaxOpenConns) imposes a limit on the number of open connections. Past this limit, new database operations will wait for an existing operation to finish, at which time `sql.DB` will create another connection. By default, `sql.DB` creates a new connection any time all the existing connections are in use when a connection is needed. Keep in mind that setting a limit makes database usage similar to acquiring a lock or semaphore, with the result that your application can deadlock waiting for a new database connection. #### Setting the maximum number of idle connections [`DB.SetMaxIdleConns`](https://pkg.go.dev/database/sql#DB.SetMaxIdleConns) changes the limit on the maximum number of idle connections `sql.DB` maintains. When an SQL operation finishes on a given database connection, it is not typically shut down immediately: the application may need one again soon, and keeping the open connection around avoids having to reconnect to the database for the next operation. By default an `sql.DB` keeps two idle connections at any given moment. Raising the limit can avoid frequent reconnects in programs with significant parallelism. #### Setting the maximum amount a time a connection can be idle [`DB.SetConnMaxIdleTime`](https://pkg.go.dev/database/sql#DB.SetConnMaxIdleTime) sets the maximum length of time a connection can be idle before it is closed. This causes the `sql.DB` to close connections that have been idle for longer than the given duration. By default, when an idle connection is added to the connection pool, it remains there until it is needed again. When using `DB.SetMaxIdleConns` to increase the number of allowed idle connections during bursts of parallel activity, also using `DB.SetConnMaxIdleTime` can arrange to release those connections later when the system is quiet. #### Setting the maximum lifetime of connections Using [`DB.SetConnMaxLifetime`](https://pkg.go.dev/database/sql#DB.SetConnMaxLifetime) sets the maximum length of time a connection can be held open before it is closed. By default, a connection can be used and reused for an arbitrarily long amount of time, subject to the limits described above. In some systems, such as those using a load-balanced database server, it can be helpful to ensure that the application never uses a particular connection for too long without reconnecting. ### Using dedicated connections The `database/sql` package includes functions you can use when a database may assign implicit meaning to a sequence of operations executed on a particular connection. The most common example is transactions, which typically start with a `BEGIN` command, end with a `COMMIT` or `ROLLBACK` command, and include all the commands issued on the connection between those commands in the overall transaction. For this use case, use the `sql` package’s transaction support. See [Executing transactions](https://go.dev/doc/database/execute-transactions) . For other use cases where a sequence of individual operations must all execute on the same connection, the `sql` package provides dedicated connections. [`DB.Conn`](https://pkg.go.dev/database/sql#DB.Conn) obtains a dedicated connection, an [`sql.Conn`](https://pkg.go.dev/database/sql#Conn) . The `sql.Conn` has methods `BeginTx`, `ExecContext`, `PingContext`, `PrepareContext`, `QueryContext`, and `QueryRowContext` that behave like the equivalent methods on DB but only use the dedicated connection. When finished with the dedicated connection, your code must release it using `Conn.Close`. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # The Green Tea Garbage Collector - The Go Programming Language [The Go Blog](https://go.dev/blog/) ==================================== The Green Tea Garbage Collector =============================== Michael Knyszek and Austin Clements 29 October 2025 Go 1.25 includes a new experimental garbage collector called Green Tea, available by setting `GOEXPERIMENT=greenteagc` at build time. Many workloads spend around 10% less time in the garbage collector, but some workloads see a reduction of up to 40%! It’s production-ready and already in use at Google, so we encourage you to try it out. We know some workloads don’t benefit as much, or even at all, so your feedback is crucial to helping us move forward. Based on the data we have now, we plan to make it the default in Go 1.26. To report back with any problems, [file a new issue](https://go.dev/issue/new) . To report back with any successes, reply to [the existing Green Tea issue](https://go.dev/issue/73581) . What follows is a blog post based on Michael Knyszek’s GopherCon 2025 talk. Tracing garbage collection -------------------------- Before we discuss Green Tea let’s get us all on the same page about garbage collection. ### Objects and pointers The purpose of garbage collection is to automatically reclaim and reuse memory no longer used by the program. To this end, the Go garbage collector concerns itself with _objects_ and _pointers_. In the context of the Go runtime, _objects_ are Go values whose underlying memory is allocated from the heap. Heap objects are created when the Go compiler can’t figure out how else to allocate memory for a value. For example, the following code snippet allocates a single heap object: the backing store for a slice of pointers. var x = make([]*int, 10) // global The Go compiler can’t allocate the slice backing store anywhere except the heap, since it’s very hard, and maybe even impossible, for it to know how long `x` will refer to the object for. _Pointers_ are just numbers that indicate the location of a Go value in memory, and they’re how a Go program references objects. For example, to get the pointer to the beginning of the object allocated in the last code snippet, we can write: &x[0] // 0xc000104000 ### The mark-sweep algorithm Go’s garbage collector follows a strategy broadly referred to as _tracing garbage collection_, which just means that the garbage collector follows, or traces, the pointers in the program to identify which objects the program is still using. More specifically, the Go garbage collector implements the mark-sweep algorithm. This is much simpler than it sounds. Imagine objects and pointers as a sort of graph, in the computer science sense. Objects are nodes, pointers are edges. The mark-sweep algorithm operates on this graph, and as the name might suggest, proceeds in two phases. In the first phase, the mark phase, it walks the object graph from well-defined source edges called _roots_. Think global and local variables. Then, it _marks_ everything it finds along the way as _visited_, to avoid going in circles. This is analogous to your typical graph flood algorithm, like a depth-first or breadth-first search. Next is the sweep phase. Whatever objects were not visited in our graph walk are unused, or _unreachable_, by the program. We call this state unreachable because it is impossible with normal safe Go code to access that memory anymore, simply through the semantics of the language. To complete the sweep phase, the algorithm simply iterates through all the unvisited nodes and marks their memory as free, so the memory allocator can reuse it. ### That’s it? You may think I’m oversimplifying a bit here. Garbage collectors are frequently referred to as _magic_, and _black boxes_. And you’d be partially right, there are more complexities. For example, this algorithm is, in practice, executed concurrently with your regular Go code. Walking a graph that’s mutating underneath you brings challenges. We also parallelize this algorithm, which is a detail that’ll come up again later. But trust me when I tell you that these details are mostly separate from the core algorithm. It really is just a simple graph flood at the center. ### Graph flood example Let’s walk through an example. Navigate through the slideshow below to follow along. ← Prev Next → ![](https://go.dev/blog/greenteagc/marksweep-007.png) Here we have a diagram of some global variables and Go heap. Let's break it down, piece by piece. ![](https://go.dev/blog/greenteagc/marksweep-008.png) On the left here we have our roots. These are global variables x and y. They will be the starting point of our graph walk. Since they're marked blue, according to our handy legend in the bottom left, they're currently on our work list. ![](https://go.dev/blog/greenteagc/marksweep-009.png) On the right side, we have our heap. Currently, everything in our heap is grayed out because we haven't visited any of it yet. ![](https://go.dev/blog/greenteagc/marksweep-010.png) Each one of these rectangles represents an object. Each object is labeled with its type. This object in particular is an object of type T, whose type definition is on the top left. It's got a pointer to an array of children, and some value. We can surmise that this is some kind of recursive tree data structure. ![](https://go.dev/blog/greenteagc/marksweep-011.png) In addition to the objects of type T, you'll also notice that we have array objects containing \*Ts. These are pointed to by the "children" field of objects of type T. ![](https://go.dev/blog/greenteagc/marksweep-012.png) Each square inside of the rectangle represents 8 bytes of memory. A square with a dot is a pointer. If it has an arrow, it is a non-nil pointer pointing to some other object. ![](https://go.dev/blog/greenteagc/marksweep-013.png) And if it doesn't have a corresponding arrow, then it's a nil pointer. ![](https://go.dev/blog/greenteagc/marksweep-014.png) Next, these dotted rectangles represents free space, what I'll call a free "slot." We could put an object there, but there currently isn't one. ![](https://go.dev/blog/greenteagc/marksweep-015.png) You'll also notice that objects are grouped together by these labeled, dotted rounded rectangles. Each of these represents a _page_, which is a contiguous block of fixed-size, aligned memory. In Go, pages are 8 KiB (regardless of the hardware virtual memory page size). These pages are labeled A, B, C, and D, and I'll refer to them that way. ![](https://go.dev/blog/greenteagc/marksweep-015.png) In this diagram, each object is allocated as part of some page. Like in the real implementation, each page here only contains objects of a certain size. This is just how the Go heap is organized. ![](https://go.dev/blog/greenteagc/marksweep-016.png) Pages are also how we organize per-object metadata. Here you can see seven boxes, each corresponding to one of the seven object slots in page A. ![](https://go.dev/blog/greenteagc/marksweep-016.png) Each box represents one bit of information: whether or not we have seen the object before. This is actually how the real runtime manages whether an object has been visited, and it'll be an important detail later. ![](https://go.dev/blog/greenteagc/marksweep-017.png) That was a lot of detail, so thanks for reading along. This will all come into play later. For now, let's just see how our graph flood applies to this picture. ![](https://go.dev/blog/greenteagc/marksweep-018.png) We start by taking a root off of the work list. We mark it red to indicate that it's now active. ![](https://go.dev/blog/greenteagc/marksweep-019.png) Following that root's pointer, we find an object of type T, which we add to our work list. Following our legend, we draw the object in blue to indicate that it's on our work list. Note also that we set the seen bit corresponding to this object in our metadata. ![](https://go.dev/blog/greenteagc/marksweep-020.png) Same goes for the next root. ![](https://go.dev/blog/greenteagc/marksweep-021.png) Now that we've taken care of all the roots, we're left with two objects on our work list. Let's take an object off the work list. ![](https://go.dev/blog/greenteagc/marksweep-022.png) What we're going to do now is walk the pointers of the objects, to find more objects. By the way, we call walking the pointers of an object "scanning" the object. ![](https://go.dev/blog/greenteagc/marksweep-023.png) We find this valid array object… ![](https://go.dev/blog/greenteagc/marksweep-024.png) … and add it to our work list. ![](https://go.dev/blog/greenteagc/marksweep-025.png) From here, we proceed recursively. ![](https://go.dev/blog/greenteagc/marksweep-026.png) We walk the array's pointers. ![](https://go.dev/blog/greenteagc/marksweep-027.png) ![](https://go.dev/blog/greenteagc/marksweep-028.png) ![](https://go.dev/blog/greenteagc/marksweep-029.png) Find some more objects… ![](https://go.dev/blog/greenteagc/marksweep-030.png) ![](https://go.dev/blog/greenteagc/marksweep-031.png) ![](https://go.dev/blog/greenteagc/marksweep-032.png) Then we walk the objects that the array object referred to! ![](https://go.dev/blog/greenteagc/marksweep-033.png) And note that we still have to walk over all pointers, even if they're nil. We don't know ahead of time if they will be. ![](https://go.dev/blog/greenteagc/marksweep-034.png) One more object down this branch… ![](https://go.dev/blog/greenteagc/marksweep-035.png) ![](https://go.dev/blog/greenteagc/marksweep-036.png) And now we've reached the other branch, starting from that object in page A we found much earlier from one of the roots. ![](https://go.dev/blog/greenteagc/marksweep-036.png) You may be noticing a last-in-first-out discipline for our work list here, indicating that our work list is a stack, and hence our graph flood is approximately depth-first. This is intentional, and reflects the actual graph flood algorithm in the Go runtime. ![](https://go.dev/blog/greenteagc/marksweep-037.png) Let's keep going… ![](https://go.dev/blog/greenteagc/marksweep-038.png) Next we find another array object… ![](https://go.dev/blog/greenteagc/marksweep-039.png) And walk it… ![](https://go.dev/blog/greenteagc/marksweep-040.png) ![](https://go.dev/blog/greenteagc/marksweep-041.png) ![](https://go.dev/blog/greenteagc/marksweep-042.png) ![](https://go.dev/blog/greenteagc/marksweep-043.png) ![](https://go.dev/blog/greenteagc/marksweep-044.png) Just one more object left on our work list… ![](https://go.dev/blog/greenteagc/marksweep-045.png) Let's scan it… ![](https://go.dev/blog/greenteagc/marksweep-046.png) ![](https://go.dev/blog/greenteagc/marksweep-047.png) And we're done with the mark phase! There's nothing we're actively working on and there's nothing left on our work list. Every object drawn in black is reachable, and every object drawn in gray is unreachable. Let's sweep the unreachable objects, all in one go. ![](https://go.dev/blog/greenteagc/marksweep-048.png) We've converted those objects into free slots, ready to hold new objects. The problem ----------- After all that, I think we have a handle on what the Go garbage collector is actually doing. This process seems to work well enough today, so what’s the problem? Well, it turns out we can spend _a lot_ of time executing this particular algorithm in some programs, and it adds substantial overhead to nearly every Go program. It’s not that uncommon to see Go programs spending 20% or more of their CPU time in the garbage collector. Let’s break down where that time is being spent. ### Garbage collection costs At a high level, there are two parts to the cost of the garbage collector. The first is how often it runs, and the second is how much work it does each time it runs. Multiply those two together, and you get the total cost of the garbage collector. Total GC cost = Number of GC cycles × Average cost per GC cycle Over the years we’ve tackled both terms in this equation, and for more on _how often_ the garbage collector runs, see [Michael’s GopherCon EU talk from 2022](https://www.youtube.com/watch?v=07wduWyWx8M) about memory limits. [The guide to the Go garbage collector](https://go.dev/doc/gc-guide) also has a lot to say about this topic, and is worth a look if you want to dive deeper. But for now let’s focus only on the second part, the cost per cycle. From years of poring over CPU profiles to try to improve performance, we know two big things about Go’s garbage collector. The first is that about 90% of the cost of the garbage collector is spent marking, and only about 10% is sweeping. Sweeping turns out to be much easier to optimize than marking, and Go has had a very efficient sweeper for many years. The second is that, of that time spent marking, a substantial portion, usually at least 35%, is simply spent _stalled_ on accessing heap memory. This is bad enough on its own, but it completely gums up the works on what makes modern CPUs actually fast. ### “A microarchitectural disaster” What does “gum up the works” mean in this context? The specifics of modern CPUs can get pretty complicated, so let’s use an analogy. Imagine the CPU driving down a road, where that road is your program. The CPU wants to ramp up to a high speed, and to do that it needs to be able to see far ahead of it, and the way needs to be clear. But the graph flood algorithm is like driving through city streets for the CPU. The CPU can’t see around corners and it can’t predict what’s going to happen next. To make progress, it constantly has to slow down to make turns, stop at traffic lights, and avoid pedestrians. It hardly matters how fast your engine is because you never get a chance to get going. Let’s make that more concrete by looking at our example again. I’ve overlaid the heap here with the path that we took. Each left-to-right arrow represents a piece of scanning work that we did and the dashed arrows show how we jumped around between bits of scanning work. ![](https://go.dev/blog/greenteagc/graphflood-path.png) The path through the heap the garbage collector took in our graph flood example. Notice that we were jumping all over memory doing tiny bits of work in each place. In particular, we’re frequently jumping between pages, and between different parts of pages. Modern CPUs do a lot of caching. Going to main memory can be up to 100x slower than accessing memory that’s in our cache. CPU caches are populated with memory that’s been recently accessed, and memory that’s nearby to recently accessed memory. But there’s no guarantee that any two objects that point to each other will _also_ be close to each other in memory. The graph flood doesn’t take this into account. Quick side note: if we were just stalling fetches to main memory, it might not be so bad. CPUs issue memory requests asynchronously, so even slow ones could overlap if the CPU could see far enough ahead. But in the graph flood, every bit of work is small, unpredictable, and highly dependent on the last, so the CPU is forced to wait on nearly every individual memory fetch. And unfortunately for us, this problem is only getting worse. There’s an adage in the industry of “wait two years and your code will get faster.” But Go, as a garbage collected language that relies on the mark-sweep algorithm, risks the opposite. “Wait two years and your code will get slower.” The trends in modern CPU hardware are creating new challenges for garbage collector performance: **Non-uniform memory access.** For one, memory now tends to be associated with subsets of CPU cores. Accesses by _other_ CPU cores to that memory are slower than before. In other words, the cost of a main memory access [depends on which CPU core is accessing it](https://jprahman.substack.com/p/sapphire-rapids-core-to-core-latency) . It’s non-uniform, so we call this non-uniform memory access, or NUMA for short. **Reduced memory bandwidth.** Available memory bandwidth per CPU is trending downward over time. This just means that while we have more CPU cores, each core can submit relatively fewer requests to main memory, forcing non-cached requests to wait longer than before. **Ever more CPU cores.** Above, we looked at a sequential marking algorithm, but the real garbage collector performs this algorithm in parallel. This scales well to a limited number of CPU cores, but the shared queue of objects to scan becomes a bottleneck, even with careful design. **Modern hardware features.** New hardware has fancy features like vector instructions, which let us operate on a lot of data at once. While this has the potential for big speedups, it’s not immediately clear how to make that work for marking because marking does so much irregular and often small pieces of work. Green Tea --------- Finally, this brings us to Green Tea, our new approach to the mark-sweep algorithm. The key idea behind Green Tea is astonishingly simple: _Work with pages, not objects._ Sounds trivial, right? And yet, it took a lot of work to figure out how to order the object graph walk and what we needed to track to make this work well in practice. More concretely, this means: * Instead of scanning objects we scan whole pages. * Instead of tracking objects on our work list, we track whole pages. * We still need to mark objects at the end of the day, but we’ll track marked objects locally to each page, rather than across the whole heap. ### Green Tea example Let’s see what this means in practice by looking at our example heap again, but this time running Green Tea instead of the straightforward graph flood. As above, navigate through the annotated slideshow to follow along. ← Prev Next → ![](https://go.dev/blog/greenteagc/greentea-060.png) This is the same heap as before, but now with two bits of metadata per object rather than one. Again, each bit, or box, corresponds to one of the object slots in the page. In total, we now have fourteen bits that correspond to the seven slots in page A. ![](https://go.dev/blog/greenteagc/greentea-060.png) The top bits represent the same thing as before: whether or not we've seen a pointer to the object. I'll call these the "seen" bits. The bottom set of bits are new. These "scanned" bits track whether or not we've _scanned_ the object. ![](https://go.dev/blog/greenteagc/greentea-060.png) This new piece of metadata is necessary because, in Green tea, **the work list tracks pages, not objects**. We still need to track objects at some level, and that's the purpose of these bits. ![](https://go.dev/blog/greenteagc/greentea-062.png) We start off the same as before, walking objects from the roots. ![](https://go.dev/blog/greenteagc/greentea-063.png) ![](https://go.dev/blog/greenteagc/greentea-064.png) But this time, instead of putting an object on the work list, we put a whole page–in this case page A–on the work list, indicated by shading the whole page blue. ![](https://go.dev/blog/greenteagc/greentea-066.png) The object we found is also blue to indicate that when we do take this page off of the work list, we will need to look at that object. Note that the object's blue hue directly reflects the metadata in page A. Its corresponding seen bit is set, but its scanned bit is not. ![](https://go.dev/blog/greenteagc/greentea-069.png) We follow the next root, find another object, and again put the whole page–page C–on the work list and set the object's seen bit. ![](https://go.dev/blog/greenteagc/greentea-071.png) We're done following roots, so we turn to the work list and take page A off the work list. ![](https://go.dev/blog/greenteagc/greentea-072.png) Using the seen and scanned bits, we can tell there's one object to scan on page A. ![](https://go.dev/blog/greenteagc/greentea-074.png) We scan that object, following its pointers. And as a result, we add page B to the work list, since the first object in page A points to an object in page B. ![](https://go.dev/blog/greenteagc/greentea-075.png) We're done with page A. Next we take page C off the work list. ![](https://go.dev/blog/greenteagc/greentea-076.png) Similar to page A, there's a single object on page C to scan. ![](https://go.dev/blog/greenteagc/greentea-078.png) We found a pointer to another object in page B. Page B is already on the work list, so we don't need to add anything to the work list. We simply have to set the seen bit for the target object. ![](https://go.dev/blog/greenteagc/greentea-079.png) Now it's page B's turn. We've accumulated two objects to scan on page B, and we can process both of these objects in a row, in memory order! ![](https://go.dev/blog/greenteagc/greentea-081.png) We walk the pointers of the first object… ![](https://go.dev/blog/greenteagc/greentea-082.png) ![](https://go.dev/blog/greenteagc/greentea-083.png) ![](https://go.dev/blog/greenteagc/greentea-084.png) We find a pointer to an object in page A. Page A was previously on the work list, but isn't at this point, so we put it back on the work list. Unlike the original mark-sweep algorithm, where any given object is only added to the work list at most once per whole mark phase, in Green Tea, a given page can reappear on the work list several times during a mark phase. ![](https://go.dev/blog/greenteagc/greentea-085.png) ![](https://go.dev/blog/greenteagc/greentea-086.png) We scan the second seen object in the page immediately after the first. ![](https://go.dev/blog/greenteagc/greentea-087.png) ![](https://go.dev/blog/greenteagc/greentea-088.png) ![](https://go.dev/blog/greenteagc/greentea-089.png) We find a few more objects in page A… ![](https://go.dev/blog/greenteagc/greentea-090.png) ![](https://go.dev/blog/greenteagc/greentea-091.png) ![](https://go.dev/blog/greenteagc/greentea-092.png) ![](https://go.dev/blog/greenteagc/greentea-093.png) We're done scanning page B, so we pull page A off the work list. ![](https://go.dev/blog/greenteagc/greentea-094.png) This time we only need to scan three objects, not four, since we already scanned the first object. We know which objects to scan by looking at the difference between the "seen" and "scanned" bits. ![](https://go.dev/blog/greenteagc/greentea-095.png) We'll scan these objects in sequence. ![](https://go.dev/blog/greenteagc/greentea-096.png) ![](https://go.dev/blog/greenteagc/greentea-097.png) ![](https://go.dev/blog/greenteagc/greentea-098.png) ![](https://go.dev/blog/greenteagc/greentea-099.png) ![](https://go.dev/blog/greenteagc/greentea-100.png) ![](https://go.dev/blog/greenteagc/greentea-101.png) We're done! There are no more pages on the work list and there's nothing we're actively looking at. Notice that the metadata now all lines up nicely, since all reachable objects were both seen and scanned. ![](https://go.dev/blog/greenteagc/greentea-101.png) You may have also noticed during our traversal that the work list order is a little different from the graph flood. Where the graph flood had a last-in-first-out, or stack-like, order, here we're using a first-in-first-out, or queue-like, order for the pages on our work list. ![](https://go.dev/blog/greenteagc/greentea-101.png) This is intentional. We let seen objects accumulate on each page while the page sits on the queue, so we can process as many as we can at once. That's how we were able to hit so many objects on page A at once. Sometimes laziness is a virtue. ![](https://go.dev/blog/greenteagc/greentea-102.png) And finally we can sweep away the unvisited objects, as before. ### Getting on the highway Let’s come back around to our driving analogy. Are we finally getting on the highway? Let’s recall our graph flood picture before. ![](https://go.dev/blog/greenteagc/graphflood-path2.png) The path the original graph flood took through the heap required 7 separate scans. We jumped around a whole lot, doing little bits of work in different places. The path taken by Green Tea looks very different. ![](https://go.dev/blog/greenteagc/greentea-path.png) The path taken by Green Tea requires only 4 scans. Green Tea, in contrast, makes fewer, longer left-to-right passes over pages A and B. The longer these arrows, the better, and with bigger heaps, this effect can be much stronger. _That’s_ the magic of Green Tea. It’s also our opportunity to ride the highway. This all adds up to a better fit with the microarchitecture. We can now scan objects closer together with much higher probability, so there’s a better chance we can make use of our caches and avoid main memory. Likewise, per-page metadata is more likely to be in cache. Tracking pages instead of objects means work lists are smaller, and less pressure on work lists means less contention and fewer CPU stalls. And speaking of the highway, we can take our metaphorical engine into gears we’ve never been able to before, since now we can use vector hardware! ### Vector acceleration If you’re only vaguely familiar with vector hardware, you might be confused as to how we can use it here. But besides the usual arithmetic and trigonometric operations, recent vector hardware supports two things that are valuable for Green Tea: very wide registers, and sophisticated bit-wise operations. Most modern x86 CPUs support AVX-512, which has 512-bit wide vector registers. This is wide enough to hold all of the metadata for an entire page in just two registers, right on the CPU, enabling Green Tea to work on an entire page in just a few straight-line instructions. Vector hardware has long supported basic bit-wise operations on whole vector registers, but starting with AMD Zen 4 and Intel Ice Lake, it also supports a new bit vector “Swiss army knife” instruction that enables a key step of the Green Tea scanning process to be done in just a few CPU cycles. Together, these allow us to turbo-charge the Green Tea scan loop. This wasn’t even an option for the graph flood, where we’d be jumping between scanning objects that are all sorts of different sizes. Sometimes you needed two bits of metadata and sometimes you needed ten thousand. There simply wasn’t enough predictability or regularity to use vector hardware. If you want to nerd out on some of the details, read along! Otherwise, feel free to skip ahead to the [evaluation](https://go.dev/blog/greenteagc#evaluation) . #### AVX-512 scanning kernel To get a sense of what AVX-512 GC scanning looks like, take a look at the diagram below. ![](https://go.dev/blog/greenteagc/avx512.svg) The AVX-512 vector kernel for scanning. There’s a lot going on here and we could probably fill an entire blog post just on how this works. For now, let’s just break it down at a high level: 1. First we fetch the “seen” and “scanned” bits for a page. Recall, these are one bit per object in the page, and all objects in a page have the same size. 2. Next, we compare the two bit sets. Their union becomes the new “scanned” bits, while their difference is the “active objects” bitmap, which tells us which objects we need to scan in this pass over the page (versus previous passes). 3. We take the difference of the bitmaps and “expand” it, so that instead of one bit per object, we have one bit per word (8 bytes) of the page. We call this the “active words” bitmap. For example, if the page stores 6-word (48-byte) objects, each bit in the active objects bitmap will be copied to 6 bits in the active words bitmap. Like so: 0 0 1 1 ... → 000000 000000 111111 111111 ... 4. Next we fetch the pointer/scalar bitmap for the page. Here, too, each bit corresponds to a word (8 bytes) of the page, and it tells us whether that word stores a pointer. This data is managed by the memory allocator. 5. Now, we take the intersection of the pointer/scalar bitmap and the active words bitmap. The result is the “active pointer bitmap”: a bitmap that tells us the location of every pointer in the entire page contained in any live object we haven’t scanned yet. 6. Finally, we can iterate over the memory of the page and collect all the pointers. Logically, we iterate over each set bit in the active pointer bitmap, load the pointer value at that word, and write it back to a buffer that will later be used to mark objects seen and add pages to the work list. Using vector instructions, we’re able to do this 64 bytes at a time, in just a couple instructions. Part of what makes this fast is the `VGF2P8AFFINEQB` instruction, part of the “Galois Field New Instructions” x86 extension, and the bit manipulation Swiss army knife we referred to above. It’s the real star of the show, since it lets us do step (3) in the scanning kernel very, very efficiently. It performs a bit-wise [affine transformations](https://en.wikipedia.org/wiki/Affine_transformation) , treating each byte in a vector as itself a mathematical vector of 8 bits and multiplying it by an 8x8 bit matrix. This is all done over the [Galois field](https://en.wikipedia.org/wiki/Finite_field) `GF(2)`, which just means multiplication is AND and addition is XOR. The upshot of this is that we can define a few 8x8 bit matrices for each object size that perform exactly the 1:n bit expansion we need. For the full assembly code, see [this file](https://cs.opensource.google/go/go/+/master:src/internal/runtime/gc/scan/scan_amd64.s;l=23;drc=041f564b3e6fa3f4af13a01b94db14c1ee8a42e0) . The “expanders” use different matrices and different permutations for each size class, so they’re in a [separate file](https://cs.opensource.google/go/go/+/master:src/internal/runtime/gc/scan/expand_amd64.s;drc=041f564b3e6fa3f4af13a01b94db14c1ee8a42e0) that’s written by a [code generator](https://cs.opensource.google/go/go/+/master:src/internal/runtime/gc/scan/mkasm.go;drc=041f564b3e6fa3f4af13a01b94db14c1ee8a42e0) . Aside from the expansion functions, it’s really not a lot of code. Most of it is dramatically simplified by the fact that we can perform most of the above operations on data that sits purely in registers. And, hopefully soon this assembly code [will be replaced with Go code](https://go.dev/issue/73787) ! Credit to Austin Clements for devising this process. It’s incredibly cool, and incredibly fast! ### Evaluation So that’s it for how it works. How much does it actually help? It can be quite a lot. Even without the vector enhancements, we see reductions in garbage collection CPU costs between 10% and 40% in our benchmark suite. For example, if an application spends 10% of its time in the garbage collector, then that would translate to between a 1% and 4% overall CPU reduction, depending on the specifics of the workload. A 10% reduction in garbage collection CPU time is roughly the modal improvement. (See the [GitHub issue](https://go.dev/issue/73581) for some of these details.) We’ve rolled Green Tea out inside Google, and we see similar results at scale. We’re still rolling out the vector enhancements, but benchmarks and early results suggest this will net an additional 10% GC CPU reduction. While most workloads benefit to some degree, there are some that don’t. Green Tea is based on the hypothesis that we can accumulate enough objects to scan on a single page in one pass to counteract the costs of the accumulation process. This is clearly the case if the heap has a very regular structure: objects of the same size at a similar depth in the object graph. But there are some workloads that often require us to scan only a single object per page at a time. This is potentially worse than the graph flood because we might be doing more work than before while trying to accumulate objects on pages and failing. The implementation of Green Tea has a special case for pages that have only a single object to scan. This helps reduce regressions, but doesn’t completely eliminate them. However, it takes a lot less per-page accumulation to outperform the graph flood than you might expect. One surprise result of this work was that scanning a mere 2% of a page at a time can yield improvements over the graph flood. ### Availability Green Tea is already available as an experiment in the recent Go 1.25 release and can be enabled by setting the environment variable `GOEXPERIMENT` to `greenteagc` at build time. This doesn’t include the aforementioned vector acceleration. We expect to make it the default garbage collector in Go 1.26, but you’ll still be able to opt-out with `GOEXPERIMENT=nogreenteagc` at build time. Go 1.26 will also add vector acceleration on newer x86 hardware, and include a whole bunch of tweaks and improvements based on feedback we’ve collected so far. If you can, we encourage you to try at Go tip-of-tree! If you prefer to use Go 1.25, we’d still love your feedback. See [this GitHub comment](https://go.dev/issue/73581#issuecomment-2847696497) with some details on what diagnostics we’d be interested in seeing, if you can share, and the preferred channels for reporting feedback. The journey ----------- Before we wrap up this blog post, let’s take a moment to talk about the journey that got us here. The human element of the technology. The core of Green Tea may seem like a single, simple idea. Like the spark of inspiration that just one single person had. But that’s not true at all. Green Tea is the result of work and ideas from many people over several years. Several people on the Go team contributed to the ideas, including Michael Pratt, Cherry Mui, David Chase, and Keith Randall. Microarchitectural insights from Yves Vandriessche, who was at Intel at the time, also really helped direct the design exploration. There were a lot of ideas that didn’t work, and there were a lot of details that needed figuring out. Just to make this single, simple idea viable. ![](https://go.dev/blog/greenteagc/timeline.png) A timeline depicting a subset of the ideas we tried in this vein before getting to where we are today. The seeds of this idea go all the way back to 2018. What’s funny is that everyone on the team thinks someone else thought of this initial idea. Green Tea got its name in 2024 when Austin worked out a prototype of an earlier version while cafe crawling in Japan and drinking LOTS of matcha! This prototype showed that the core idea of Green Tea was viable. And from there we were off to the races. Throughout 2025, as Michael implemented and productionized Green Tea, the ideas evolved and changed even further. This took so much collaborative exploration because Green Tea is not just an algorithm, but an entire design space. One that we don’t think any of us could’ve navigated alone. It’s not enough to just have the idea, but you need to figure out the details and prove it. And now that we’ve done it, we can finally iterate. The future of Green Tea is bright. Once again, please try it out by setting `GOEXPERIMENT=greenteagc` and let us know how it goes! We’re really excited about this work and want to hear from you! **Next article:** [Go’s Sweet 16](https://go.dev/blog/16years) **Previous article:** [Flight Recorder in Go 1.25](https://go.dev/blog/flight-recorder) **[Blog Index](https://go.dev/blog/all) ** go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Unknown http://golang.org Monday, October 18, 2010 http://golang.org The Expressiveness of Go Rob Pike JAOO Oct 5, 2010 Monday, October 18, 2010 Who 2 Monday, October 18, 2010 Te a m 3 Russ Cox Robert Griesemer Rob Pike Ian Taylor Ken Thompson plus David Symonds, Nigel Tao, Andrew Gerrand, Stephen Ma, and others, plus many contributions from the open source community. Monday, October 18, 2010 Why 4 Why a new language? Monday, October 18, 2010 Why Go? 5 A response to Google’s internal needs: - efficient large scale programming - speed of compilation - distributed systems - multicore, networked hardware And a reaction to: “speed and safety or ease of use; pick one.” - complexity, weight, noise (C++, Java) vs. - no static checking (JavaScript, Python) Go is statically typed and compiled, like C++ or Java (with no VM), but in many ways feels as lightweight and dynamic as JavaScript or Python. Monday, October 18, 2010 The surprise 6 It turned out to be a nice general purpose language. That was unexpected. The most productive language I’ve ever used. And some people agree. Monday, October 18, 2010 Acceptance 7 Go was the 2009 TIOBE "Language of the year" two months after it was released and it won an InfoWorld BOSSIE award. Monday, October 18, 2010 Why the success? 8 This acceptance was a pleasant surprise. But in retrospect, the way we approached the design was important to the expressiveness and productivity of Go. Monday, October 18, 2010 Principles 9 The axioms of Go’s design Monday, October 18, 2010 Principles 10 Go is: Simple - concepts are easy to understand - (the implementation might still be sophisticated) Orthogonal - concepts mix cleanly - easy to understand and predict what happens Succinct - no need to predeclare every intention Safe - misbehavior should be detected These combine to give expressiveness. Monday, October 18, 2010 Respect 11 Go trusts the programmer to write down what is meant. In turn, Go tries to respect the programmer's intentions. It's possible to be safe and fun. There's a difference between seat belts and training wheels. Monday, October 18, 2010 Simplicity 12 Number of keywords is an approximate measure of complexity. C (K&R)K&R32 C++199148 Java3rd edition50 C#201077 C++0x201072+11\* JavaScriptECMA-26226+16\* Python2.731 PascalISO35 Modula-2198040 Oberon199032 Go201025 \* extra count is for reserved words and alternate spellings Monday, October 18, 2010 An example 13 A complete (if simple) web server Monday, October 18, 2010 Hello, world 2.0 14 Serving http://localhost:8080/world: package main import ( "fmt" "http" ) func handler(c \*http.Conn, r \*http.Request) { fmt.Fprintf(c, "Hello, %s.", r.URL.Path\[1:\]) } func main() { http.ListenAndServe(":8080", http.HandlerFunc(handler)) } Monday, October 18, 2010 Stories 15 A few design tales that illustrate how the principles play out. Not close to a complete tour of Go. Monday, October 18, 2010 Basics 16 Some fundamentals Monday, October 18, 2010 Starting points 17 Go started with a few important simplifications relative to C/C++, informed by our experience with those languages: There are pointers but no pointer arithmetic - pointers are important to performance, pointer arithmetic not. - although it's OK to point inside a struct. - important to control layout of memory, avoid allocation Increment/decrement (p++) is a statement, not expression. - no confusion about order of evaluation Addresses last as long as they are needed. - take the address of a local variable, the implementation guarantees the memory survives while it's referenced. No implicit numerical conversions (float to int, etc.). - C's "usual arithmetic conversions" are a minefield. Monday, October 18, 2010 Constants are ideal 18 Implicit conversions often involve constants (sin(Pi/4)), but Go mitigates the issue by having nicer constants. Constants are "ideal numbers": no size or sign, hence no L or U or UL endings. 077 // octal 0xFEEDBEEEEEEEEEEEEEEEEEEEEF // hexadecimal 1 << 100 They are just numbers that can be assigned to variables; no conversions necessary. A typed element in the expression sets the true type of the constant. Here 1e9 becomes int64. seconds := time.Nanoseconds()/1e9 Monday, October 18, 2010 High precision constants 19 Arithmetic with constants is high precision. Only when assigned to a variable are they rounded or truncated to fit. const Ln2= 0.6931471805599453094172321214581\\ 76568075500134360255254120680009 const Log2E= 1/Ln2 // accurate reciprocal var x float64 = Log2E The value assigned to x will be as precise as possible in a 64-bit floating point number. Simple, clear model. Simple constant syntax. Constants orthogonal (nearly) to type system. Monday, October 18, 2010 Types and data 20 Structs, methods, and interfaces Monday, October 18, 2010 Structs 21 Structs describe (and control) the layout of data. Some early proposals included methods in the struct, but that idea was dropped. Instead methods are declared like ordinary functions, outside the type, and with a "receiver". type Point struct { x, y float } func (p Point) Abs() float { return math.Sqrt(p.x\*p.x + p.y\*p.y) } The (p Point) declares the receiver (no automatic "this" variable; also notice p is not a pointer, although it could be.) Methods are not mixed with the data definition. They are orthogonal to types. Monday, October 18, 2010 Methods are orthogonal to types 22 Orthogonality of methods allows any type to have them. type Vector \[\]float func (v Vector) Abs() float { sumOfSquares := 0.0 for i := range v { sumOfSquares += v\[i\]\*v\[i\] } return math.Sqrt(sumOfSquares) } It also allows receivers to be values or pointers: func (p \*Point) Scale(ratio float) { p.x, p.y = ratio\*p.x, ratio\*p.y } Orthogonality leads to generality. Monday, October 18, 2010 Interfaces 23 Interfaces are just sets of methods; work for any type. type Abser interface { Abs() float } var a Abser a = Point{3, 4} print(a.Abs()) a = Vector{1, 2, 3, 4} print(a.Abs()) Interfaces are satisfied implicitly. Point and Vector do not declare that they implement Abser, they just do. @mjmoriarity: The way Go handles interfaces is the way I wish every language handled them. Monday, October 18, 2010 Interfaces are abstract, other types are concrete 24 In some of our early proposals, interfaces could contain data, but this conflated representation and behavior. Made them distinct: - concrete type such as structs define data - interfaces define behavior As with methods, now anything can satisfy an interface. type Value float // basic type func (v Value) Abs() float { if v < 0 { v = -v } return float(v) } a = Value(-27) print(a.Abs()) Monday, October 18, 2010 Interfaces are satisfied implicitly 25 Point and Vector satisfied Abser implicitly; other types might too. A type satisfies an interface simply by implementing its methods. There is no "implements" declaration; interfaces are satisfied implicitly. It's a form of duck typing, but (usually) checkable at compile time. It's also another form of orthogonality. An object can (and usually does) satisfy many interfaces simultaneously. For instance, Point and Vector satisfy Abser and also the empty interface: interface{}, which is satisfied by any value (analogous to C++ void\* or Java Object) In Go, interfaces are usually one or two (or zero) methods. Monday, October 18, 2010 Reader 26 type Reader interface { Read(p \[\]byte) (n int, err os.Error) // two return vals } // And similarly for Writer Anything with a Read method implements Reader. - Sources: files, buffers, network connections, pipes - Filters: buffers, checksums, decompressors, decrypters JPEG decoder takes a Reader, so it can decode from disk, network, gzipped HTTP, .... Buffering just wraps a Reader: var bufferedInput Reader = bufio.NewReader(os.Stdin) Fprintf uses a Writer: func Fprintf(w Writer, fmt string, a ...interface{}) Monday, October 18, 2010 Interfaces enable retrofitting 27 Had an existing RPC implementation that used custom wire format. Changed to an interface: type Encoding interface { ReadRequestHeader(\*Request) os.Error ReadRequestBody(interface{}) os.Error WriteResponse(\*Response, interface{}) os.Error Close() os.Error } Two functions (send, receive) changed signature. Before: func sendResponse(sending \*sync.Mutex, req \*Request, reply interface{}, enc \*gob.Encoder, err string) After (and similarly for receiving): func sendResponse(sending \*sync.Mutex, req \*Request, reply interface{}, enc Encoding, err string) That is almost the whole change to the RPC implementation. Monday, October 18, 2010 Post facto abstraction 28 We saw an opportunity: RPC needed only Encode and Decode methods. Put those in an interface and you've abstracted the codec. Total time: 20 minutes, including writing and testing the JSON implementation of the interface. (We also wrote a trivial wrapper to adapt the existing codec for the new rpc.Encoding interface.) In Java, RPC would be refactored into a half-abstract class, subclassed to create JsonRPC and StandardRPC. You'd have to plan ahead. In Go it's simpler and the design adapts through experience. Monday, October 18, 2010 Principles redux 29 Types and data examples: Simple - interfaces are just method sets Orthogonal - representation (data) and behavior (methods) are independent - empty interface can represent any value Succinct - no implements declarations; interfaces just satisfy Safe - static type checking Expressiveness: implicit satisfaction lets pieces fit together seamlessly. Monday, October 18, 2010 Names 30 Visibility Monday, October 18, 2010 Visibility 31 With methods not declared as part of the type definition, how to say private/public? Long design debate with several suggestions: - placement in the file - export keyword - name marker (e.g. Point{ +x,+y int }) Resolution (anguished decision): Don't make it part of the declaration, make it part of the name. After all, that's what you see whenever you use it! Case of first character determines visibility outside package: ThisNameIsPublic thisOneIsNot Monday, October 18, 2010 Visibility is orthogonal to type 32 One of the best decisions in the language, yet really hard to make! Lose control over how to use case, e.g. can't use it to distinguish Types from variables. In return, though: - extremely simple, easy rule to understand - consequences clear - see a variable, can see whether it's public without going to the declaration - can make any type, variable, or constant public or not with the same mechanism Orthogonality again! Monday, October 18, 2010 Concurrency and closures 33 Goroutines, channels, stacks and closures Monday, October 18, 2010 The model 34 Go has native support for concurrent operations in the CSP tradition. Two styles of concurrency exist: deterministic (well-defined ordering) and non-deterministic (mutual exclusion but order undefined). Go's goroutines and channels promote deterministic concurrency (e.g. channels with one sender, one receiver), which is easier to reason about. Simpler for the programmer. Monday, October 18, 2010 Go concurrency basics 35 Start a goroutine: go f() Channel send (arrow points in direction of flow): ch <- value Channel receive: value = <-ch Channels are unbuffered by default, which combines synchronization with communication. Monday, October 18, 2010 A simple worker pool 36 A unit of work: type Work struct { x, y, z int } A worker task: func worker(in <-chan \*Work, out chan <- \*Work) { for w := range in { w.z = w.x \* w.y out <- w } } Driver: func Run() { in, out := make(chan \*Work), make(chan \*Work) for i := 0; i < 10; i++ { go worker(in, out) } go sendLotsOfWork(in) receiveLotsOfResults(out) } No low-level synchronization needed. Monday, October 18, 2010 Secondary support 37 To make concurrency feasible, need several things: - language support (axiomatic) - garbage collection (near axiomatic) To make it good, you need: - stack management - closures Monday, October 18, 2010 Stacks 38 Goroutines have "segmented stacks": go f() starts f() executing concurrently on a new stack. Stack grows and shrinks as needed. No programmer concern about stack size. No possibility for stack overflow. A couple of instructions of overhead on each function call, a huge improvement in simplicity and expressiveness. Concurrent execution is orthogonal to everything else. Monday, October 18, 2010 Launching a goroutine 39 Start a service, return a channel to communicate with it: func (s \*Service) Start() chan request { ch := make(chan request) go s.serve(ch) // s.serve runs concurrently return ch // returns immediately } A common pattern, given channels as first-class values. Monday, October 18, 2010 Closures 40 Closures are just local functions func Compose(f, g func(x float) float) func(x float) float { return func(x float) float { return f(g(x)) } } func main() { print(Compose(sin, cos)(0.5)) } Fit easily into implementation since local variables already move to heap when necessary. Monday, October 18, 2010 Closures and concurrency 41 Query servers in replicated database, return first response. func Query(conns \[\]Conn, query string) Result { ch := make(chan Result, 1) // buffer of 1 item for \_, conn := range conns { go func(c Conn) { \_ = ch <- c.DoQuery(query) }(conn) } return <-ch } Monday, October 18, 2010 Principles redux 42 Concurrency and closure examples: Simple - stacks just work; goroutines too cheap to meter Orthogonal - concurrency orthogonal to rest of language - orthogonality of functions make closures easy Succinct - go f() - closure syntax clear Safe - no stack overflows - garbage collection avoids many concurrency problems Expressiveness: complex behaviors easily expressed. Monday, October 18, 2010 Conclusion 43 Expressiveness comes from orthogonal composition of simple concepts. Monday, October 18, 2010 Conclusion 44 Go is not a small language (goroutines, channels, garbage collection, methods, interfaces, closures, ...) but it is an expressive and comprehensible one. Expressiveness comes from orthogonal composition of constructs. Comprehensibility comes from simple constructs that interact in easily understood ways. Build a language from simple orthogonal constructs and you have a language that will be easy and productive to use. The surprises you discover will be pleasant ones. Monday, October 18, 2010 Implementation 45 The language is designed and usable. Two compiler suites: Gc, written in C, generates OK code very quickly. - unusual design based on the Plan 9 compiler suite Gccgo, written in C++, generates good code more slowly - uses GCC's code generator and tools Libraries good and growing, but some pieces are still preliminary. Garbage collector works fine (simple mark and sweep) but is being rewritten for more concurrency, less latency. Available for Linux etc., Mac OS X. Windows port working. All available as open source; see http://golang.org. Monday, October 18, 2010 Try it out 46 This is a true open source project. Much more information at http://golang.org including full source, documentation, and a playground that lets you try Go code right from the browser. Monday, October 18, 2010 http://golang.org Monday, October 18, 2010 http://golang.org The Expressiveness of Go Rob Pike JAOO Oct 5, 2010 Monday, October 18, 2010 --- # Avoiding SQL injection risk - The Go Programming Language Avoiding SQL injection risk =========================== You can avoid an SQL injection risk by providing SQL parameter values as `sql` package function arguments. Many functions in the `sql` package provide parameters for the SQL statement and for values to be used in that statement’s parameters (others provide a parameter for a prepared statement and parameters). Code in the following example uses the `?` symbol as a placeholder for the `id` parameter, which is provided as a function argument: // Correct format for executing an SQL statement with parameters. rows, err := db.Query("SELECT * FROM user WHERE id = ?", id) `sql` package functions that perform database operations create prepared statements from the arguments you supply. At run time, the `sql` package turns the SQL statement into a prepared statement and sends it along with the parameter, which is separate. **Note:** Parameter placeholders vary depending on the DBMS and driver you’re using. For example, [pq driver](https://pkg.go.dev/github.com/lib/pq) for Postgres accepts a placeholder form such as `$1` instead of `?`. You might be tempted to use a function from the `fmt` package to assemble the SQL statement as a string with parameters included – like this: // SECURITY RISK! rows, err := db.Query(fmt.Sprintf("SELECT * FROM user WHERE id = %s", id)) This is not secure! When you do this, Go assembles the entire SQL statement, replacing the `%s` format verb with the parameter value, before sending the full statement to the DBMS. This poses an [SQL injection](https://en.wikipedia.org/wiki/SQL_injection) risk because the code’s caller could send an unexpected SQL snippet as the `id` argument. That snippet could complete the SQL statement in unpredictable ways that are dangerous to your application. For example, by passing a certain `%s` value, you might end up with something like the following, which could return all user records in your database: SELECT * FROM user WHERE id = 1 OR 1=1; go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Editor plugins and IDEs - The Go Programming Language 1. [Documentation](https://go.dev/doc/) 2. [Editor plugins and IDEs](https://go.dev/doc/editors) Editor plugins and IDEs ======================= Introduction ------------ This document lists commonly used editor plugins and IDEs from the Go ecosystem that make Go development more productive and seamless. A comprehensive list of editor support and IDEs for Go development is available at [the wiki](https://go.dev/wiki/IDEsAndTextEditorPlugins) . Options ------- The Go ecosystem provides a variety of editor plugins and IDEs to enhance your day-to-day editing, navigation, testing, and debugging experience. * [Visual Studio Code](https://marketplace.visualstudio.com/items?itemName=golang.go) : Go extension provides support for the Go programming language * [GoLand](https://www.jetbrains.com/go) : GoLand is distributed either as a standalone IDE or as a plugin for IntelliJ IDEA Ultimate * [vim](https://github.com/fatih/vim-go) : vim-go plugin provides Go programming language support Note that these are only a few top solutions; a more comprehensive community-maintained list of [IDEs and text editor plugins](https://go.dev/wiki/IDEsAndTextEditorPlugins) is available at the Wiki. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Canceling in-progress operations - The Go Programming Language Canceling in-progress operations ================================ You can manage in-progress operations by using Go [`context.Context`](https://pkg.go.dev/context#Context) . A `Context` is a standard Go data value that can report whether the overall operation it represents has been canceled and is no longer needed. By passing a `context.Context` across function calls and services in your application, those can stop working early and return an error when their processing is no longer needed. For more about `Context`, see [Go Concurrency Patterns: Context](https://go.dev/blog/context) . For example, you might want to: * End long-running operations, including database operations that are taking too long to complete. * Propagate cancellation requests from elsewhere, such as when a client closes a connection. Many APIs for Go developers include methods that take a `Context` argument, making it easier for you to use `Context` throughout your application. ### Canceling database operations after a timeout You can use a `Context` to set a timeout or deadline after which an operation will be canceled. To derive a `Context` with a timeout or deadline, call [`context.WithTimeout`](https://pkg.go.dev/context#WithTimeout) or [`context.WithDeadline`](https://pkg.go.dev/context#WithDeadline) . Code in the following timeout example derives a `Context` and passes it into the `sql.DB` [`QueryContext`](https://pkg.go.dev/database/sql#DB.QueryContext) method. func QueryWithTimeout(ctx context.Context) { // Create a Context with a timeout. queryCtx, cancel := context.WithTimeout(ctx, 5*time.Second) defer cancel() // Pass the timeout Context with a query. rows, err := db.QueryContext(queryCtx, "SELECT * FROM album") if err != nil { log.Fatal(err) } defer rows.Close() // Handle returned rows. } When one context is derived from an outer context, as `queryCtx` is derived from `ctx` in this example, if the outer context is canceled, then the derived context is automatically canceled as well. For example, in HTTP servers, the `http.Request.Context` method returns a context associated with the request. That context is canceled if the HTTP client disconnects or cancels the HTTP request (possible in HTTP/2). Passing an HTTP request’s context to `QueryWithTimeout` above would cause the database query to stop early _either_ if the overall HTTP request was canceled or if the query took more than five seconds. **Note:** Always defer a call to the `cancel` function that’s returned when you create a new `Context` with a timeout or deadline. This releases resources held by the new `Context` when the containing function exits. It also cancels `queryCtx`, but by the time the function returns, nothing should be using `queryCtx` anymore. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Executing SQL statements that don't return data - The Go Programming Language Executing SQL statements that don't return data =============================================== When you perform database actions that don’t return data, use an `Exec` or `ExecContext` method from the `database/sql` package. SQL statements you’d execute this way include `INSERT`, `DELETE`, and `UPDATE`. When your query might return rows, use a `Query` or `QueryContext` method instead. For more, see [Querying a database](https://go.dev/doc/database/querying) . An `ExecContext` method works as an `Exec` method does, but with an additional `context.Context` argument, as described in [Canceling in-progress operations](https://go.dev/doc/database/cancel-operations) . Code in the following example uses [`DB.Exec`](https://pkg.go.dev/database/sql#DB.Exec) to execute a statement to add a new record album to an `album` table. func AddAlbum(alb Album) (int64, error) { result, err := db.Exec("INSERT INTO album (title, artist) VALUES (?, ?)", alb.Title, alb.Artist) if err != nil { return 0, fmt.Errorf("AddAlbum: %v", err) } // Get the new album's generated ID for the client. id, err := result.LastInsertId() if err != nil { return 0, fmt.Errorf("AddAlbum: %v", err) } // Return the new album's ID. return id, nil } `DB.Exec` returns values: an [`sql.Result`](https://pkg.go.dev/database/sql#Result) and an error. When the error is `nil`, you can use the `Result` to get the ID of the last inserted item (as in the example) or to retrieve the number of rows affected by the operation. **Note:** Parameter placeholders in prepared statements vary depending on the DBMS and driver you’re using. For example, the [pq driver](https://pkg.go.dev/github.com/lib/pq) for Postgres requires a placeholder like `$1` instead of `?`. If your code will be executing the same SQL statement repeatedly, consider using an `sql.Stmt` to create a reusable prepared statement from the SQL statement. For more, see [Using prepared statements](https://go.dev/doc/database/prepared-statements) . **Caution:** Don’t use string formatting functions such as `fmt.Sprintf` to assemble an SQL statement! You could introduce an SQL injection risk. For more, see [Avoiding SQL injection risk](https://go.dev/doc/database/sql-injection) . #### Functions for executing SQL statements that don’t return rows | Function | Description | | --- | --- | | `[DB.Exec](https://pkg.go.dev/database/sql#DB.Exec) `
`[DB.ExecContext](https://pkg.go.dev/database/sql#DB.ExecContext) ` | Execute a single SQL statement in isolation. | | `[Tx.Exec](https://pkg.go.dev/database/sql#Tx.Exec) `
`[Tx.ExecContext](https://pkg.go.dev/database/sql#Tx.ExecContext) ` | Execute a SQL statement within a larger transaction. For more, see [Executing transactions](https://go.dev/doc/database/execute-transactions)
. | | `[Stmt.Exec](https://pkg.go.dev/database/sql#Stmt.Exec) `
`[Stmt.ExecContext](https://pkg.go.dev/database/sql#Stmt.ExecContext) ` | Execute an already-prepared SQL statement. For more, see [Using prepared statements](https://go.dev/doc/database/prepared-statements)
. | | `[Conn.ExecContext](https://pkg.go.dev/database/sql#Conn.ExecContext) ` | For use with reserved connections. For more, see [Managing connections](https://go.dev/doc/database/manage-connections)
. | go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Managing module source - The Go Programming Language Managing module source ====================== When you’re developing modules to publish for others to use, you can help ensure that your modules are easier for other developers to use by following the repository conventions described in this topic. This topic describes actions you might take when managing your module repository. For information about the sequence of workflow steps you’d take when revising from version to version, see [Module release and versioning workflow](https://go.dev/doc/modules/release-workflow) . Some of the conventions described here are required in modules, while others are best practices. This content assumes you’re familiar with the basic module use practices described in [Managing dependencies](https://go.dev/doc/modules/managing-dependencies) . Go supports the following repositories for publishing modules: Git, Subversion, Mercurial, Bazaar, and Fossil. For an overview of module development, see [Developing and publishing modules](https://go.dev/doc/modules/developing) . How Go tools find your published module --------------------------------------- In Go’s decentralized system for publishing modules and retrieving their code, you can publish your module while leaving the code in your repository. Go tools rely on naming rules that have repository paths and repository tags indicating a module’s name and version number. When your repository follows these requirements, your module code is downloadable from your repository by Go tools such as the [`go get` command](https://go.dev/ref/mod#go-get) . When a developer uses the `go get` command to get source code for packages their code imports, the command does the following: 1. From `import` statements in Go source code, `go get` identifies the module path within the package path. 2. Using a URL derived from the module path, the command locates the module source on a module proxy server or at its repository directly. 3. Locates source for the module version to download by matching the module’s version number to a repository tag to discover the code in the repository. When a version number to use is not yet known, `go get` locates the latest release version. 4. Retrieves module source and downloads it to the developer’s local module cache. Organizing code in the repository --------------------------------- You can keep maintenance simple and improve developers’ experience with your module by following the conventions described here. Getting your module code into a repository is generally as simple as with other code. The following diagram illustrates a source hierarchy for a simple module with two packages. ![Diagram illustrating a module source code hierarchy](https://go.dev/doc/modules/images/source-hierarchy.png) Your initial commit should include files listed in the following table: | File | Description | | --- | --- | | LICENSE | The module's license. | | go.mod | Describes the module, including its module path (in effect, its name) and its dependencies. For more, see the [go.mod reference](https://go.dev/doc/modules/gomod-ref)
.

The module path will be given in a module directive, such as:

module example.com/mymodule

For more about choosing a module path, see [Managing dependencies](https://go.dev/doc/modules/managing-dependencies#naming_module)
.

Though you can edit the go.mod file, you'll find it more reliable to make changes through `go` commands. | | go.sum | Contains cryptographic hashes that represent the module's dependencies. Go tools use these hashes to authenticate downloaded modules, attempting to confirm that the downloaded module is authentic. Where this confirmation fails, Go will display a security error.

The file will be empty or not present when there are no dependencies. You shouldn't edit this file except by using the `go mod tidy` command, which removes unneeded entries. | | Package directories and .go sources. | Directories and .go files that comprise the Go packages and sources in the module. | From the command-line, you can create an empty repository, add the files that will be part of your initial commit, and commit with a message. Here’s an example using git: $ git init $ git add --all $ git commit -m "mycode: initial commit" $ git push Choosing repository scope ------------------------- You publish code in a module when the code should be versioned independently from code in other modules. Designing your repository so that it hosts a single module at its root directory will help keep maintenance simpler, particularly over time as you publish new minor and patch versions, branch into new major versions, and so on. However, if your needs require it, you can instead maintain a collection of modules in a single repository. ### Sourcing one module per repository You can maintain a repository that has a single module’s source in it. In this model, you place your go.mod file at the repository root, with package subdirectories containing Go source beneath. This is the simplest approach, making your module likely easier to manage over time. It helps you avoid the need to prefix a module version number with a directory path. ![Diagram illustrating a single module's source in its repository](https://go.dev/doc/modules/images/single-module.png) ### Sourcing multiple modules in a single repository You can publish multiple modules from a single repository. For example, you might have code in a single repository that constitutes multiple modules, but want to version those modules separately. Each subdirectory that is a module root directory must have its own go.mod file. Sourcing module code in subdirectories changes the form of the version tag you must use when publishing a module. You must prefix the version number part of the tag with the name of the subdirectory that is the module root. For more about version numbers, see [Module version numbering](https://go.dev/doc/modules/version-numbers) . For example, for module `example.com/mymodules/module1` below, you would have the following for version v1.2.3: * Module path: `example.com/mymodules/module1` * Version tag: `module1/v1.2.3` * Package path imported by a user: `example.com/mymodules/module1/package1` * Module path and version as specified in a user’s require directive: `example.com/mymodules/module1 v1.2.3` ![Diagram illustrating two modules in a single repository](https://go.dev/doc/modules/images/multiple-modules.png) go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Managing dependencies - The Go Programming Language Managing dependencies ===================== When your code uses external packages, those packages (distributed as modules) become dependencies. Over time, you may need to upgrade them or replace them. Go provides dependency management tools that help you keep your Go applications secure as you incorporate external dependencies. This topic describes how to perform tasks to manage dependencies you take on in your code. You can perform most of these with Go tools. This topic also describes how to perform a few other dependency-related tasks you might find useful. **See also** * If you’re new to working with dependencies as modules, take a look at the [Getting started tutorial](https://go.dev/doc/tutorial/getting-started) for a brief introduction. * Using the `go` command to manage dependencies helps ensure that your requirements remain consistent and the content of your go.mod file is valid. For reference on the commands, see [Command go](https://go.dev/cmd/go/) . You can also get help from the command line by typing `go help` _command-name_, as with `go help mod tidy`. * Go commands you use to make dependency changes edit your go.mod file. For more about the contents of the file, see [go.mod file reference](https://go.dev/doc/modules/gomod-ref) . * Making your editor or IDE aware of Go modules can make the work of managing them easier. For more on editors that support Go, see [Editor plugins and IDEs](https://go.dev/doc/editors.html) . * This topic doesn’t describe how to develop, publish, and version modules for others to use. For more on that, see [Developing and publishing modules](https://go.dev/doc/modules/developing) . Workflow for using and managing dependencies -------------------------------------------- You can get and use useful packages with Go tools. On [pkg.go.dev](https://pkg.go.dev/) , you can search for packages you might find useful, then use the `go` command to import those packages into your own code to call their functions. The following lists the most common dependency management steps. For more about each, see the sections in this topic. 1. [Locate useful packages](https://go.dev/doc/modules/managing-dependencies#locating_packages) on [pkg.go.dev](https://pkg.go.dev/) . 2. [Import the packages](https://go.dev/doc/modules/managing-dependencies#locating_packages) you want in your code. 3. Add your code to a module for dependency tracking (if it isn’t in a module already). See [Enabling dependency tracking](https://go.dev/doc/modules/managing-dependencies#enable_tracking) 4. [Add external packages as dependencies](https://go.dev/doc/modules/managing-dependencies#adding_dependency) so you can manage them. 5. [Upgrade or downgrade dependency versions](https://go.dev/doc/modules/managing-dependencies#upgrading) as needed over time. Managing dependencies as modules -------------------------------- In Go, you manage dependencies as modules that contain the packages you import. This process is supported by: * A **decentralized system for publishing** modules and retrieving their code. Developers make their modules available for other developers to use from their own repository and publish with a version number. * A **package search engine** and documentation browser (pkg.go.dev) at which you can find modules. See [Locating and importing useful packages](https://go.dev/doc/modules/managing-dependencies#locating_packages) . * A module **version numbering convention** to help you understand a module’s stability and backward compatibility guarantees. See [Module version numbering](https://go.dev/doc/modules/version-numbers) . * **Go tools** that make it easier for you to manage dependencies, including getting a module’s source, upgrading, and so on. See sections of this topic for more. Locating and importing useful packages -------------------------------------- You can search [pkg.go.dev](https://pkg.go.dev/) to find packages with functions you might find useful. When you’ve found a package you want to use in your code, locate the package path at the top of the page and click the Copy path button to copy the path to your clipboard. In your own code, paste the path into an import statement, as in the following example: import "rsc.io/quote" After your code imports the package, enable dependency tracking and get the package’s code to compile with. For more, see [Enabling dependency tracking in your code](https://go.dev/doc/modules/managing-dependencies#enable_tracking) and [Adding a dependency](https://go.dev/doc/modules/managing-dependencies#adding_dependency) . Enabling dependency tracking in your code ----------------------------------------- To track and manage the dependencies you add, you begin by putting your code in its own module. This creates a go.mod file at the root of your source tree. Dependencies you add will be listed in that file. To add your code to its own module, use the [`go mod init` command](https://go.dev/ref/mod#go-mod-init) . For example, from the command line, change to your code’s root directory, then run the command as in the following example: $ go mod init example/mymodule The `go mod init` command’s argument is your module’s module path. If possible, the module path should be the repository location of your source code. If at first you don’t know the module’s eventual repository location, use a safe substitute. This might be the name of a domain you own or another name you control (such as your company name), along with a path following from the module’s name or source directory. For more, see [Naming a module](https://go.dev/doc/modules/managing-dependencies#naming_module) . As you use Go tools to manage dependencies, the tools update the go.mod file so that it maintains a current list of your dependencies. When you add dependencies, Go tools also create a go.sum file that contains checksums of modules you depend on. Go uses this to verify the integrity of downloaded module files, especially for other developers working on your project. Include the go.mod and go.sum files in your repository with your code. See the [go.mod reference](https://go.dev/doc/modules/gomod-ref) for more. Naming a module --------------- When you run `go mod init` to create a module for tracking dependencies, you specify a module path that serves as the module’s name. The module path becomes the import path prefix for packages in the module. Be sure to specify a module path that won’t conflict with the module path of other modules. At a minimum, a module path need only indicate something about its origin, such as a company or author or owner name. But the path might also be more descriptive about what the module is or does. The module path is typically of the following form: / * The _prefix_ is typically a string that partially describes the module, such as a string that describes its origin. This might be: * The location of the repository where Go tools can find the module’s source code (required if you’re publishing the module). For example, it might be `github.com//`. Use this best practice if you think you might publish the module for others to use. For more about publishing, see [Developing and publishing modules](https://go.dev/doc/modules/developing) . * A name you control. If you’re not using a repository name, be sure to choose a prefix that you’re confident won’t be used by others. A good choice is your company’s name. Avoid common terms such as `widgets`, `utilities`, or `app`. * For the _descriptive text_, a good choice would be a project name. Remember that package names carry most of the weight of describing functionality. The module path creates a namespace for those package names. **Reserved module path prefixes** Go guarantees that the following strings won’t be used in package names. * `test` – You can use `test` as a module path prefix for a module whose code is designed to locally test functions in another module. Use the `test` path prefix for modules that are created as part of a test. For example, your test itself might run `go mod init test` and then set up that module in some particular way in order to test with a Go source code analysis tool. * `example` – Used as a module path prefix in some Go documentation, such as in tutorials where you’re creating a module just to track dependencies. Note that Go documentation also uses `example.com` to illustrate when the example might be a published module. Adding a dependency ------------------- Once you’re importing packages from a published module, you can add that module to manage as a dependency by using the [`go get` command](https://go.dev/cmd/go/#hdr-Add_dependencies_to_current_module_and_install_them) . The command does the following: * If needed, it adds `require` directives to your go.mod file for modules needed to build packages named on the command line. A `require` directive tracks the minimum version of a module that your module depends on. See the [go.mod reference](https://go.dev/doc/modules/gomod-ref) for more. * If needed, it downloads module source code so you can compile packages that depend on them. It can download modules from a module proxy like proxy.golang.org or directly from version control repositories. The source is cached locally. You can set the location from which Go tools download modules. For more, see [Specifying a module proxy server](https://go.dev/doc/modules/managing-dependencies#proxy_server) . The following describes a few examples. * To add all dependencies for a package in your module, run a command like the one below ("." refers to the package in the current directory): $ go get . * To add a specific dependency, specify its module path as an argument to the command. $ go get example.com/theirmodule The command also authenticates each module it downloads. This ensures that it’s unchanged from when the module was published. If the module has changed since it was published – for example, the developer changed the contents of the commit – Go tools will present a security error. This authentication check protects you from modules that might have been tampered with. Getting a specific dependency version ------------------------------------- You can get a specific version of a dependency module by specifying its version in the `go get` command. The command updates the `require` directive in your go.mod file (though you can also update that manually). You might want to do this if: * You want to get a specific pre-release version of a module to try out. * You’ve discovered that the version you’re currently requiring isn’t working for you, so you want to get a version you know you can rely on. * You want to upgrade or downgrade a module you’re already requiring. Here are examples for using the [`go get` command](https://go.dev/ref/mod#go-get) : * To get a specific numbered version, append the module path with an @ sign followed by the version you want: $ go get example.com/theirmodule@v1.3.4 * To get the latest version, append the module path with `@latest`: $ go get example.com/theirmodule@latest The following go.mod file `require` directive example (see the [go.mod reference](https://go.dev/doc/modules/gomod-ref) for more) illustrates how to require a specific version number: require example.com/theirmodule v1.3.4 Discovering available updates ----------------------------- You can check to see if there are newer versions of dependencies you’re already using in your current module. Use the `go list` command to display a list of your module’s dependencies, along with the latest version available for that module. Once you’ve discovered available upgrades, you can try them out with your code to decide whether or not to upgrade to new versions. For more about the `go list` command, see [`go list -m`](https://go.dev/ref/mod#go-list-m) . Here are a couple of examples. * List all of the modules that are dependencies of your current module, along with the latest version available for each: $ go list -m -u all * Display the latest version available for a specific module: $ go list -m -u example.com/theirmodule Upgrading or downgrading a dependency ------------------------------------- You can upgrade or downgrade a dependency module by using Go tools to discover available versions, then add a different version as a dependency. 1. To discover new versions use the `go list` command as described in [Discovering available updates](https://go.dev/doc/modules/managing-dependencies#discovering_updates) . 2. To add a particular version as a dependency, use the `go get` command as described in [Getting a specific dependency version](https://go.dev/doc/modules/managing-dependencies#getting_version) . Synchronizing your code’s dependencies -------------------------------------- You can ensure that you’re managing dependencies for all of your code’s imported packages while also removing dependencies for packages you’re no longer importing. This can be useful when you’ve been making changes to your code and dependencies, possibly creating a collection of managed dependencies and downloaded modules that no longer match the collection specifically required by the packages imported in your code. To keep your managed dependency set tidy, use the `go mod tidy` command. Using the set of packages imported in your code, this command edits your go.mod file to add modules that are necessary but missing. It also removes unused modules that don’t provide any relevant packages. The command has no arguments except for one flag, -v, that prints information about removed modules. $ go mod tidy Developing and testing against unpublished module code ------------------------------------------------------ You can specify that your code should use dependency modules that may not be published. The code for these modules might be in their respective repositories, in a fork of those repositories, or on a drive with the current module that consumes them. You might want to do this when: * You want to make your own changes to an external module’s code, such as after forking and/or cloning it. For example, you might want to prepare a fix to the module, then send it as a pull request to the module’s developer. * You’re building a new module and haven’t yet published it, so it’s unavailable on a repository where the `go get` command can reach it. ### Requiring module code in a local directory You can specify that the code for a required module is on the same local drive as the code that requires it. You might find this useful when you are: * Developing your own separate module and want to test from the current module. * Fixing issues in or adding features to an external module and want to test from the current module. (Note that you can also require the external module from your own fork of its repository. For more, see [Requiring external module code from your own repository fork](https://go.dev/doc/modules/managing-dependencies#external_fork) .) To tell Go commands to use the local copy of the module’s code, use the `replace` directive in your go.mod file to replace the module path given in a `require` directive. See the [go.mod reference](https://go.dev/doc/modules/gomod-ref) for more about directives. In the following go.mod file example, the current module requires the external module `example.com/theirmodule`, with a nonexistent version number (`v0.0.0-unpublished`) used to ensure the replacement works correctly. The `replace` directive then replaces the original module path with `../theirmodule`, a directory that is at the same level as the current module’s directory. module example.com/mymodule go 1.23.0 require example.com/theirmodule v0.0.0-unpublished replace example.com/theirmodule v0.0.0-unpublished => ../theirmodule When setting up a `require`/`replace` pair, use the [`go mod edit`](https://go.dev/ref/mod#go-mod-edit) and [`go get`](https://go.dev/ref/mod#go-get) commands to ensure that requirements described by the file remain consistent: $ go mod edit -replace=example.com/theirmodule@v0.0.0-unpublished=../theirmodule $ go get example.com/theirmodule@v0.0.0-unpublished **Note:** When you use the replace directive, Go tools don’t authenticate external modules as described in [Adding a dependency](https://go.dev/doc/modules/managing-dependencies#adding_dependency) . For more about version numbers, see [Module version numbering](https://go.dev/doc/modules/version-numbers) . ### Requiring external module code from your own repository fork When you have forked an external module’s repository (such as to fix an issue in the module’s code or to add a feature), you can have Go tools use your fork for the module’s source. This can be useful for testing changes from your own code. (Note that you can also require the module code in a directory that’s on the local drive with the module that requires it. For more, see [Requiring module code in a local directory](https://go.dev/doc/modules/managing-dependencies#local_directory) .) You do this by using a `replace` directive in your go.mod file to replace the external module’s original module path with a path to the fork in your repository. This directs Go tools to use the replacement path (the fork’s location) when compiling, for example, while allowing you to leave `import` statements unchanged from the original module path. For more about the `replace` directive, see the [go.mod file reference](https://go.dev/doc/modules/gomod-ref) . In the following go.mod file example, the current module requires the external module `example.com/theirmodule`. The `replace` directive then replaces the original module path with `example.com/myfork/theirmodule`, a fork of the module’s own repository. module example.com/mymodule go 1.23.0 require example.com/theirmodule v1.2.3 replace example.com/theirmodule v1.2.3 => example.com/myfork/theirmodule v1.2.3-fixed When setting up a `require`/`replace` pair, use Go tool commands to ensure that requirements described by the file remain consistent. Use the [`go list`](https://go.dev/ref/mod#go-list-m) command to get the version in use by the current module. Then use the [`go mod edit`](https://go.dev/ref/mod#go-mod-edit) command to replace the required module with the fork: $ go list -m example.com/theirmodule example.com/theirmodule v1.2.3 $ go mod edit -replace=example.com/theirmodule@v1.2.3=example.com/myfork/theirmodule@v1.2.3-fixed **Note:** When you use the `replace` directive, Go tools don’t authenticate external modules as described in [Adding a dependency](https://go.dev/doc/modules/managing-dependencies#adding_dependency) . For more about version numbers, see [Module version numbering](https://go.dev/doc/modules/version-numbers) . Getting a specific commit using a repository identifier ------------------------------------------------------- You can use the `go get` command to add unpublished code for a module from a specific commit in its repository. To do this, you use the `go get` command, specifying the code you want with an `@` sign. When you use `go get`, the command will add to your go.mod file a `require` directive that requires the external module, using a pseudo-version number based on details about the commit. The following examples provide a few illustrations. These are based on a module whose source is in a git repository. * To get the module at a specific commit, append the form @_commithash_: $ go get example.com/theirmodule@4cf76c2 * To get the module at a specific branch, append the form @_branchname_: $ go get example.com/theirmodule@bugfixes Removing a dependency --------------------- When your code no longer uses any packages in a module, you can stop tracking the module as a dependency. To stop tracking all unused modules, run the [`go mod tidy` command](https://go.dev/ref/mod#go-mod-tidy) . This command may also add missing dependencies needed to build packages in your module. $ go mod tidy To remove a specific dependency, use the [`go get` command](https://go.dev/ref/mod#go-get) , specifying the module’s module path and appending `@none`, as in the following example: $ go get example.com/theirmodule@none The `go get` command will also downgrade or remove other dependencies that depend on the removed module. Tool dependencies ----------------- Tool dependencies let you manage developer tools that are written in Go and used when working on your module. For example, you might use [`stringer`](https://pkg.go.dev/golang.org/x/tools/cmd/stringer) with [`go generate`](https://go.dev/blog/generate) , or a specific linter or formatter as part of preparing your change for submission. In Go 1.24 and above, you can add a tool dependency with: $ go get -tool golang.org/x/tools/cmd/stringer This will add a [`tool` directive](https://go.dev/ref/mod/#go-mod-file-tool) to your `go.mod` file, and ensure the necessary require directives are present. Once this directive is added you can run the tool by passing the last [non-major-version](https://go.dev/ref/mod#major-version-suffixes) component of the tool’s import path to `go tool`: $ go tool stringer In the case that multiple tools share the last path fragment, or the path fragment matches one of the tools shipped with the Go distribution, you must pass the full package path instead: $ go tool golang.org/x/tools/cmd/stringer To see a list of all tools currently available, run `go tool` with no arguments: $ go tool You can manually add a `tool` directive to your `go.mod`, but you must ensure that there is a `require` directive for the module that defines the tool. The easiest way to add any missing `require` directives is to run: $ go mod tidy Requirements needed to satisfy tool dependencies behave like any other requirements in your [module graph](https://go.dev/ref/mod#glos-module-graph) . They participate in [minimal version selection](https://go.dev/ref/mod#minimal-version-selection) and respect `require`, `replace` and `exclude` directives. Due to module pruning, when you depend on a module that itself has a tool dependency, requirements that exist just to satisfy that tool dependency do not usually become requirements of your module. The `tool` [meta-pattern](https://go.dev/cmd/go#hdr-Package_lists_and_patterns) provides a way to perform operations on all tools simultaneously. For example you can upgrade all tools with `go get -u tool`, or install them all to $GOBIN with `go install tool`. In Go versions before 1.24, you can acheive something similar to a `tool` directive by adding a blank import to a go file within the module that is excluded from the build using [build constraints](https://go.dev/pkg/go/build/#hdr-Build_Constraints) . If you do this, you can then use `go run` with the full package path to run the tool. Specifying a module proxy server -------------------------------- When you use Go tools to work with modules, the tools by default download modules from proxy.golang.org (a public Google-run module mirror) or directly from the module’s repository. You can specify that Go tools should instead use another proxy server for downloading and authenticating modules. You might want to do this if you (or your team) have set up or chosen a different module proxy server that you want to use. For example, some set up a module proxy server in order to have greater control over how dependencies are used. To specify another module proxy server for Go tools use, set the `GOPROXY` environment variable to the URL of one or more servers. Go tools will try each URL in the order you specify. By default, `GOPROXY` specifies a public Google-run module proxy first, then direct download from the module’s repository (as specified in its module path): GOPROXY="https://proxy.golang.org,direct" For more about the `GOPROXY` environment variable, including values to support other behavior, see the [`go` command reference](https://go.dev/cmd/go/#hdr-Module_downloading_and_verification) . You can set the variable to URLs for other module proxy servers, separating URLs with either a comma or a pipe. * When you use a comma, Go tools will try the next URL in the list only if the current URL returns an HTTP 404 or 410. GOPROXY="https://proxy.example.com,https://proxy2.example.com" * When you use a pipe, Go tools will try the next URL in the list regardless of the HTTP error code. GOPROXY="https://proxy.example.com|https://proxy2.example.com" Go modules are frequently developed and distributed on version control servers and module proxies that aren’t available on the public internet. You can set the `GOPRIVATE` environment variable to configure the `go` command to download and build modules from private sources. Then the go command can download and build modules from private sources. The `GOPRIVATE` or `GONOPROXY` environment variables may be set to lists of glob patterns matching module prefixes that are private and should not be requested from any proxy. For example: GOPRIVATE=*.corp.example.com,*.research.example.com go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Organizing a Go module - The Go Programming Language Organizing a Go module ====================== A common question developers new to Go have is “How do I organize my Go project?”, in terms of the layout of files and folders. The goal of this document is to provide some guidelines that will help answer this question. To make the most of this document, make sure you’re familiar with the basics of Go modules by reading [the tutorial](https://go.dev/doc/tutorial/create-module) and [managing module source](https://go.dev/doc/modules/managing-source) . Go projects can include packages, command-line programs or a combination of the two. This guide is organized by project type. ### Basic package A basic Go package has all its code in the project’s root directory. The project consists of a single module, which consists of a single package. The package name matches the last path component of the module name. For a very simple package requiring a single Go file, the project structure is: project-root-directory/ go.mod modname.go modname_test.go _\[throughout this document, file/package names are entirely arbitrary\]_ Assuming this directory is uploaded to a GitHub repository at `github.com/someuser/modname`, the `module` line in the `go.mod` file should say `module github.com/someuser/modname`. The code in `modname.go` declares the package with: package modname // ... package code here Users can then rely on this package by `import`\-ing it in their Go code with: import "github.com/someuser/modname" A Go package can be split into multiple files, all residing within the same directory, e.g.: project-root-directory/ go.mod modname.go modname_test.go auth.go auth_test.go hash.go hash_test.go All the files in the directory declare `package modname`. ### Basic command A basic executable program (or command-line tool) is structured according to its complexity and code size. The simplest program can consist of a single Go file where `func main` is defined. Larger programs can have their code split across multiple files, all declaring `package main`: project-root-directory/ go.mod auth.go auth_test.go client.go main.go Here the `main.go` file contains `func main`, but this is just a convention. The “main” file can also be called `modname.go` (for an appropriate value of `modname`) or anything else. Assuming this directory is uploaded to a GitHub repository at `github.com/someuser/modname`, the `module` line in the `go.mod` file should say: module github.com/someuser/modname And a user should be able to install it on their machine with: $ go install github.com/someuser/modname@latest ### Package or command with supporting packages Larger packages or commands may benefit from splitting off some functionality into supporting packages. Initially, it’s recommended placing such packages into a directory named `internal`; [this prevents](https://pkg.go.dev/cmd/go#hdr-Internal_Directories) other modules from depending on packages we don’t necessarily want to expose and support for external uses. Since other projects cannot import code from our `internal` directory, we’re free to refactor its API and generally move things around without breaking external users. The project structure for a package is thus: project-root-directory/ internal/ auth/ auth.go auth_test.go hash/ hash.go hash_test.go go.mod modname.go modname_test.go The `modname.go` file declares `package modname`, `auth.go` declares `package auth` and so on. `modname.go` can import the `auth` package as follows: import "github.com/someuser/modname/internal/auth" The layout for a command with supporting packages in an `internal` directory is very similar, except that the file(s) in the root directory declare `package main`. ### Multiple packages A module can consist of multiple importable packages; each package has its own directory, and can be structured hierarchically. Here’s a sample project structure: project-root-directory/ go.mod modname.go modname_test.go auth/ auth.go auth_test.go token/ token.go token_test.go hash/ hash.go internal/ trace/ trace.go As a reminder, we assume that the `module` line in `go.mod` says: module github.com/someuser/modname The `modname` package resides in the root directory, declares `package modname` and can be imported by users with: import "github.com/someuser/modname" Sub-packages can be imported by users as follows: import "github.com/someuser/modname/auth" import "github.com/someuser/modname/auth/token" import "github.com/someuser/modname/hash" Package `trace` that resides in `internal/trace` cannot be imported outside this module. It’s recommended to keep packages in `internal` as much as possible. ### Multiple commands Multiple programs in the same repository will typically have separate directories: project-root-directory/ go.mod internal/ ... shared internal packages prog1/ main.go prog2/ main.go In each directory, the program’s Go files declare `package main`. A top-level `internal` directory can contain shared packages used by all commands in the repository. Users can install these programs as follows: $ go install github.com/someuser/modname/prog1@latest $ go install github.com/someuser/modname/prog2@latest A common convention is placing all commands in a repository into a `cmd` directory; while this isn’t strictly necessary in a repository that consists only of commands, it’s very useful in a mixed repository that has both commands and importable packages, as we will discuss next. ### Packages and commands in the same repository Sometimes a repository will provide both importable packages and installable commands with related functionality. Here’s a sample project structure for such a repository: project-root-directory/ go.mod modname.go modname_test.go auth/ auth.go auth_test.go internal/ ... internal packages cmd/ prog1/ main.go prog2/ main.go Assuming this module is called `github.com/someuser/modname`, users can now both import packages from it: import "github.com/someuser/modname" import "github.com/someuser/modname/auth" And install programs from it: $ go install github.com/someuser/modname/cmd/prog1@latest $ go install github.com/someuser/modname/cmd/prog2@latest ### Server project Go is a common language choice for implementing _servers_. There is a very large variance in the structure of such projects, given the many aspects of server development: protocols (REST? gRPC?), deployments, front-end files, containerization, scripts and so on. We will focus our guidance here on the parts of the project written in Go. Server projects typically won’t have packages for export, since a server is usually a self-contained binary (or a group of binaries). Therefore, it’s recommended to keep the Go packages implementing the server’s logic in the `internal` directory. Moreover, since the project is likely to have many other directories with non-Go files, it’s a good idea to keep all Go commands together in a `cmd` directory: project-root-directory/ go.mod internal/ auth/ ... metrics/ ... model/ ... cmd/ api-server/ main.go metrics-analyzer/ main.go ... ... the project's other directories with non-Go code In case the server repository grows packages that become useful for sharing with other projects, it’s best to split these off to separate modules. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Profile-guided optimization - The Go Programming Language Profile-guided optimization =========================== Starting in Go 1.20, the Go compiler supports profile-guided optimization (PGO) to further optimize builds. Table of Contents: [Overview](https://go.dev/doc/pgo#overview) [Collecting profiles](https://go.dev/doc/pgo#collecting-profiles) [Building with PGO](https://go.dev/doc/pgo#building) [Notes](https://go.dev/doc/pgo#notes) [Frequently Asked Questions](https://go.dev/doc/pgo#faq) [Appendix: alternative profile sources](https://go.dev/doc/pgo#alternative-sources) Overview ======== Profile-guided optimization (PGO), also known as feedback-directed optimization (FDO), is a compiler optimization technique that feeds information (a profile) from representative runs of the application back into to the compiler for the next build of the application, which uses that information to make more informed optimization decisions. For example, the compiler may decide to more aggressively inline functions which the profile indicates are called frequently. In Go, the compiler uses CPU pprof profiles as the input profile, such as from [runtime/pprof](https://pkg.go.dev/runtime/pprof) or [net/http/pprof](https://pkg.go.dev/net/http/pprof) . As of Go 1.22, benchmarks for a representative set of Go programs show that building with PGO improves performance by around 2-14%. We expect performance gains to generally increase over time as additional optimizations take advantage of PGO in future versions of Go. Collecting profiles =================== The Go compiler expects a CPU pprof profile as the input to PGO. Profiles generated by the Go runtime (such as from [runtime/pprof](https://pkg.go.dev/runtime/pprof) and [net/http/pprof](https://pkg.go.dev/net/http/pprof) ) can be used directly as the compiler input. It may also be possible to use/convert profiles from other profiling systems. See [the appendix](https://go.dev/doc/pgo#alternative-sources) for additional information. For best results, it is important that profiles are _representative_ of actual behavior in the application’s production environment. Using an unrepresentative profile is likely to result in a binary with little to no improvement in production. Thus, collecting profiles directly from the production environment is recommended, and is the primary method that Go’s PGO is designed for. The typical workflow is as follows: 1. Build and release an initial binary (without PGO). 2. Collect profiles from production. 3. When it’s time to release an updated binary, build from the latest source and provide the production profile. 4. GOTO 2 Go PGO is generally robust to skew between the profiled version of an application and the version building with the profile, as well as to building with profiles collected from already-optimized binaries. This is what makes this iterative lifecycle possible. See the [AutoFDO](https://go.dev/doc/pgo#autofdo) section for additional details about this workflow. If it is difficult or impossible to collect from the production environment (e.g., a command-line tool distributed to end users), it is also possible to collect from a representative benchmark. Note that constructing representative benchmarks is often quite difficult (as is keeping them representative as the application evolves). In particular, _microbenchmarks are usually bad candidates for PGO profiling_, as they only exercise a small part of the application, which yields small gains when applied to the whole program. Building with PGO ================= The standard approach to building is to store a pprof CPU profile with filename `default.pgo` in the main package directory of the profiled binary. By default, `go build` will detect `default.pgo` files automatically and enable PGO. Committing profiles directly in the source repository is recommended as profiles are an input to the build important for reproducible (and performant!) builds. Storing alongside the source simplifies the build experience as there are no additional steps to get the profile beyond fetching the source. For more complex scenarios, the `go build -pgo` flag controls PGO profile selection. This flag defaults to `-pgo=auto` for the `default.pgo` behavior described above. Setting the flag to `-pgo=off` disables PGO optimizations entirely. If you cannot use `default.pgo` (e.g., different profiles for different scenarios of one binary, unable to store profile with source, etc), you may directly pass a path to the profile to use (e.g., `go build -pgo=/tmp/foo.pprof`). _Note: A path passed to `-pgo` applies to all main packages. e.g., `go build -pgo=/tmp/foo.pprof ./cmd/foo ./cmd/bar` applies `foo.pprof` to both binaries `foo` and `bar`, which is often not what you want. Usually different binaries should have different profiles, passed via separate `go build` invocations._ _Note: Before Go 1.21, the default is `-pgo=off`. PGO must be explicitly enabled._ Notes ===== Collecting representative profiles from production -------------------------------------------------- Your production environment is the best source of representative profiles for your application, as described in [Collecting profiles](https://go.dev/doc/pgo#collecting-profiles) . The simplest way to start with this is to add [net/http/pprof](https://pkg.go.dev/net/http/pprof) to your application and then fetch `/debug/pprof/profile?seconds=30` from an arbitrary instance of your service. This is a great way to get started, but there are ways that this may be unrepresentative: * This instance may not be doing anything at the moment it gets profiled, even though it is usually busy. * Traffic patterns may change throughout the day, making behavior change throughout the day. * Instances may perform long-running operations (e.g., 5 minutes doing operation A, then 5 minutes doing operation B, etc). A 30s profile will likely only cover a single operation type. * Instances may not receive fair distributions of requests (some instances receive more of one type of request than others). A more robust strategy is collecting multiple profiles at different times from different instances to limit the impact of differences between individual instance profiles. Multiple profiles may then be [merged](https://go.dev/doc/pgo#merging-profiles) into a single profile for use with PGO. Many organizations run “continuous profiling” services that perform this kind of fleet-wide sampling profiling automatically, which could then be used as a source of profiles for PGO. Merging profiles ---------------- The pprof tool can merge multiple profiles like this: $ go tool pprof -proto a.pprof b.pprof > merged.pprof This merge is effectively a straightforward sum of samples in the input, regardless of wall duration of the profile. As a result, when profiling a small time slice of an application (e.g., a server that runs indefinitely), you likely want to ensure that all profiles have the same wall duration (i.e., all profiles are collected for 30s). Otherwise, profiles with longer wall duration will be overrepresented in the merged profile. AutoFDO ------- Go PGO is designed to support an “[AutoFDO](https://research.google/pubs/pub45290/) ” style workflow. Let’s take a closer look at the workflow described in [Collecting profiles](https://go.dev/doc/pgo#collecting-profiles) : 1. Build and release an initial binary (without PGO). 2. Collect profiles from production. 3. When it’s time to release an updated binary, build from the latest source and provide the production profile. 4. GOTO 2 This sounds deceptively simple, but there are a few important properties to note here: * Development is always ongoing, so the source code of the profiled version of the binary (step 2) is likely slightly different from the latest source code getting built (step 3). Go PGO is designed to be robust to this, which we refer to as _source stability_. * This is a closed loop. That is, after the first iteration the profiled version of the binary is already PGO-optimized with a profile from a previous iteration. Go PGO is also designed to be robust to this, which we refer to as _iterative stability_. _Source stability_ is achieved using heuristics to match samples from the profile to the compiling source. As a result, many changes to source code, such as adding new functions, have no impact on matching existing code. When the compiler is not able to match changed code, some optimizations are lost, but note that this is a _graceful degradation_. A single function failing to match may lose out on optimization opportunities, but overall PGO benefit is usually spread across many functions. See the [source stability](https://go.dev/doc/pgo#source-stability) section for more details about matching and degradation. _Iterative stability_ is the prevention of cycles of variable performance in successive PGO builds (e.g., build #1 is fast, build #2 is slow, build #3 is fast, etc). We use CPU profiles to identify hot functions to target with optimizations. In theory, a hot function could be sped up so much by PGO that it no longer appears hot in the next profile and does not get optimized, making it slow again. The Go compiler takes a conservative approach to PGO optimizations, which we believe prevents significant variance. If you do observe this kind of instability, please file an issue at [go.dev/issue/new](https://go.dev/issue/new) . Together, source and iterative stability eliminate the requirement for two-stage builds where a first, unoptimized build is profiled as a canary, and then rebuilt with PGO for production (unless absolutely peak performance is required). Source stability and refactoring -------------------------------- As described in above, Go’s PGO makes a best-effort attempt to continue matching samples from older profiles to the current source code. Specifically, Go uses line offsets within functions (e.g., call on 5th line of function foo). Many common changes will not break matching, including: * Changes in a file outside of a hot function (adding/changing code above or below the function). * Moving a function to another file in the same package (the compiler ignores source filenames altogether). Some changes that may break matching: * Changes within a hot function (may affect line offsets). * Renaming the function (and/or type for methods) (changes symbol name). * Moving the function to another package (changes symbol name). If the profile is relatively recent, then differences likely only affect a small number of hot functions, limiting the impact of missed optimizations in functions that fail to match. Still, degradation will slowly accumulate over time since code is rarely refactored _back_ to its old form, so it is important to collect new profiles regularly to limit source skew from production. One situation where profile matching may significantly degrade is a large-scale refactor that renames many functions or moves them between packages. In this case, you may take a short-term performance hit until a new profile shows the new structure. For rote renames, an existing profile could theoretically be rewritten to change the old symbol names to the new names. [github.com/google/pprof/profile](https://pkg.go.dev/github.com/google/pprof/profile) contains the primitives required to rewrite a pprof profile in this way, but as of writing no off-the-shelf tool exists for this. Performance of new code ----------------------- When adding new code or enabling new code paths with a flag flip, that code will not be present in the profile on the first build, and thus won’t receive PGO optimizations until a new profile reflecting the new code is collected. Keep in mind when evaluating the rollout of new code that the initial release will not represent its steady state performance. Frequently Asked Questions ========================== Is it possible to optimize Go standard library packages with PGO? ----------------------------------------------------------------- Yes. PGO in Go applies to the entire program. All packages are rebuilt to consider potential profile-guided optimizations, including standard library packages. Is it possible to optimize packages in dependent modules with PGO? ------------------------------------------------------------------ Yes. PGO in Go applies to the entire program. All packages are rebuilt to consider potential profile-guided optimizations, including packages in dependencies. This means that the unique way your application uses a dependency impacts the optimizations applied to that dependency. Will PGO with an unrepresentative profile make my program slower than no PGO? ----------------------------------------------------------------------------- It should not. While a profile that is not representative of production behavior will result in optimizations in cold parts of the application, it should not make hot parts of the application slower. If you encounter a program where PGO results in worse performance than disabling PGO, please file an issue at [go.dev/issue/new](https://go.dev/issue/new) . Can I use the same profile for different GOOS/GOARCH builds? ------------------------------------------------------------ Yes. The format of the profiles is equivalent across OS and architecture configurations, so they may be used across different configurations. For example, a profile collected from a linux/arm64 binary may be used in a windows/amd64 build. That said, the source stability caveats discussed [above](https://go.dev/doc/pgo#autofdo) apply here as well. Any source code that differs across these configurations will not be optimized. For most applications, the vast majority of code is platform-independent, so degradation of this form is limited. As a specific example, the internals of file handling in package `os` differ between Linux and Windows. If these functions are hot in the Linux profile, the Windows equivalents will not get PGO optimizations because they do not match the profiles. You may merge profiles of different GOOS/GOARCH builds. See the next question for the tradeoffs of doing so. How should I handle a single binary used for different workload types? ---------------------------------------------------------------------- There is no obvious choice here. A single binary used for different types of workloads (e.g., a database used in a read-heavy way in one service, and write-heavy in another service) may have different hot components, which benefit from different optimizations. There are three options: 1. Build different versions of the binary for each workload: use profiles from each workload to build multiple workload-specific builds of the binary. This will provide the best performance for each workload, but may add operational complexity with regard to handling multiple binaries and profile sources. 2. Build a single binary using only profiles from the “most important” workload: select the “most important” workload (largest footprint, most performance sensitive), and build using profiles only from that workload. This provides the best performance for the selected workload, and likely still modest performance improvements for other workloads from optimizations to common code shared across workloads. 3. Merge profiles across workloads: take profiles from each workload (weighted by total footprint) and merge them into a single “fleet-wide” profile used to build a single common profile used to build. This likely provides modest performance improvements for all workloads. How does PGO affect build time? ------------------------------- Enabling PGO builds will likely cause measurable increases in package build times. The most noticeable component of this is that PGO profiles apply to all packages in a binary, meaning that the first use of a profile requires a rebuild of every package in the dependency graph. These builds are cached like any other, so subsequent incremental builds using the same profile do not require complete rebuilds. If you experience extreme increases in build time, please file an issue at [go.dev/issue/new](https://go.dev/issue/new) . How does PGO affect binary size? -------------------------------- PGO can result in slightly larger binaries due to additional function inlining. Appendix: alternative profile sources ===================================== CPU profiles generated by the Go runtime (via [runtime/pprof](https://pkg.go.dev/runtime/pprof) , etc) are already in the correct format for direct use as PGO inputs. However, organizations may have alternative preferred tooling (e.g., Linux perf), or existing fleet-wide continuous profiling systems which they wish to use with Go PGO. Profiles from alternative source may be used with Go PGO if converted to the [pprof format](https://github.com/google/pprof/tree/main/proto) , provided they follow these general requirements: * One of the sample indices should have type/unit “samples”/“count” or “cpu”/“nanoseconds”. * Samples should represent samples of CPU time at the sample location. * The profile must be symbolized ([Function.name](https://github.com/google/pprof/blob/76d1ae5aea2b3f738f2058d17533b747a1a5cd01/proto/profile.proto#L208) must be set). * Samples must contain stack frames for inlined functions. If inlined functions are omitted, Go will not be able to maintain iterative stability. * [Function.start\_line](https://github.com/google/pprof/blob/76d1ae5aea2b3f738f2058d17533b747a1a5cd01/proto/profile.proto#L215) must be set. This is the line number of the start of the function. i.e., the line containing the `func` keyword. The Go compiler uses this field to compute line offsets of samples (`Location.Line.line - Function.start_line`). **Note that many existing pprof converters omit this field.** _Note: Before Go 1.21, DWARF metadata omits function start lines (`DW_AT_decl_line`), which may make it difficult for tools to determine the start line._ See the [PGO Tools](https://go.dev/wiki/PGO-Tools) page on the Go Wiki for additional information about PGO compatibility of specific third-party tools. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Developing a major version update - The Go Programming Language Developing a major version update ================================= You must update to a major version when changes you’re making in a potential new version can’t guarantee backward compatibility for the module’s users. For example, you’ll make this change if you change your module’s public API such that it breaks client code using previous versions of the module. > **Note:** Each release type – major, minor, patch, or pre-release – has a different meaning for a module’s users. Those users rely on these differences to understand the level of risk a release represents to their own code. In other words, when preparing a release, be sure that its version number accurately reflects the nature of the changes since the preceding release. For more on version numbers, see [Module version numbering](https://go.dev/doc/modules/version-numbers) > . **See also** * For an overview of module development, see [Developing and publishing modules](https://go.dev/doc/modules/developing) . * For an end-to-end view, see [Module release and versioning workflow](https://go.dev/doc/modules/release-workflow) . Considerations for a major version update ----------------------------------------- You should only update to a new major version when it’s absolutely necessary. A major version update represents significant churn for both you and your module’s users. When you’re considering a major version update, think about the following: * Be clear with your users about what releasing the new major version means for your support of previous major versions. Are previous versions deprecated? Supported as they were before? Will you be maintaining previous versions, including with bug fixes? * Be ready to take on the maintenance of two versions: the old and the new. For example, if you fix bugs in one, you’ll often be porting those fixes into the other. * Remember that a new major version is a new module from a dependency management perspective. Your users will need to update to use a new module after you release, rather than simply upgrading. That’s because a new major version has a different module path from the preceding major version. For example, for a module whose module path is example.com/mymodule, a v2 version would have the module path example.com/mymodule/v2. * When you’re developing a new major version, you must also update import paths wherever code imports packages from the new module. Your module’s users must also update their import paths if they want to upgrade to the new major version. Branching for a major release ----------------------------- The most straightforward approach to handling source when preparing to develop a new major version is to branch the repository at the latest version of the previous major version. For example, in a command prompt you might change to your module’s root directory, then create a new v2 branch there. $ cd mymodule $ git checkout -b v2 Switched to a new branch "v2" ![Diagram illustrating a repository branched from master to v2](https://go.dev/doc/modules/images/v2-branch-module.png) Once you have the source branched, you’ll need to make the following changes to the source for your new version: * In the new version’s go.mod file, append new major version number to the module path, as in the following example: * Existing version: `example.com/mymodule` * New version: `example.com/mymodule/v2` * In your Go code, update every imported package path where you import a package from the module, appending the major version number to the module path portion. * Old import statement: `import "example.com/mymodule/package1"` * New import statement: `import "example.com/mymodule/v2/package1"` For publishing steps, see [Publishing a module](https://go.dev/doc/modules/publishing) . go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Managing Go installations - The Go Programming Language Managing Go installations ========================= This topic describes how to install multiple versions of Go on the same machine, as well as how to uninstall Go. For other content on installing, you might be interested in: * [Download and install](https://go.dev/doc/install) -- The simplest way to get installed and running. * [Installing Go from source](https://go.dev/doc/install/source) -- How to check out the sources, build them on your own machine, and run them. Installing multiple Go versions ------------------------------- You can install multiple Go versions on the same machine. For example, you might want to test your code on multiple Go versions. For a list of versions you can install this way, see the [download page](https://go.dev/dl/) . **Note:** To install using the method described here, you'll need to have [git](https://git-scm.com/) installed. To install additional Go versions, run the [`go install` command](https://go.dev/cmd/go/#hdr-Compile_and_install_packages_and_dependencies) , specifying the download location of the version you want to install. The following example illustrates with version 1.10.7: $ go install golang.org/dl/go1.10.7@latest $ go1.10.7 download To run `go` commands with the newly-downloaded version, append the version number to the `go` command, as follows: $ go1.10.7 version go version go1.10.7 linux/amd64 When you have multiple versions installed, you can discover where each is installed, look at the version's `GOROOT` value. For example, run a command such as the following: $ go1.10.7 env GOROOT To uninstall a downloaded version, just remove the directory specified by its `GOROOT` environment variable and the goX.Y.Z binary. Uninstalling Go --------------- You can remove Go from your system using the steps described in this topic. ### Removing user config and data Go stores user configuration in the `go` directory within the user configuration directory, as returned by [`os.UserConfigDir`](https://go.dev/pkg/os#UserConfigDir) . This can also be found as the directory containing the config file returned by `go env GOENV`. Go stores intermediate build artifacts in the directory returned by `go env GOCACHE`. These can be removed with `go clean -cache`. Go stores downloaded dependencies in the directory returned by `go env GOMODCACHE`. These can be removed with `go clean -modcache`. ### Linux / macOS / FreeBSD 1. Delete the go directory. This is usually /usr/local/go. 2. Remove the Go bin directory from your `PATH` environment variable. Under Linux and FreeBSD, edit /etc/profile or $HOME/.profile. If you installed Go with the macOS package, remove the /etc/paths.d/go file. ### Windows The simplest way to remove Go is via Add/Remove Programs in the Windows control panel: 1. In Control Panel, double-click **Add/Remove Programs**. 2. In **Add/Remove Programs**, select **Go Programming Language,** click Uninstall, then follow the prompts. For removing Go with tools, you can also use the command line: * Uninstall using the command line by running the following command: msiexec /x go{{version}}.windows-{{cpu-arch}}.msi /q **Note:** Using this uninstall process for Windows will automatically remove Windows environment variables created by the original installation. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # The FIPS 140-3 Go Cryptographic Module - The Go Programming Language [The Go Blog](https://go.dev/blog/) ==================================== The FIPS 140-3 Go Cryptographic Module ====================================== Filippo Valsorda (Geomys), Daniel McCarney (Geomys), and Roland Shoemaker (Google) 15 July 2025 FIPS 140 is a standard for cryptography implementations and, although it doesn’t necessarily improve security, FIPS 140 compliance is a requirement in certain regulated environments that are increasingly adopting Go. Until now, FIPS 140 compliance has been a significant source of friction for Go users, requiring unsupported solutions with safety, developer experience, functionality, release velocity, and compliance issues. Go is addressing this growing need with native FIPS 140 support built right into the standard library and the `go` command, making Go the easiest, most secure way to comply with FIPS 140. The FIPS 140-3 validated Go Cryptographic Module now underlies Go’s built-in crypto libraries, starting with the Go Cryptographic Module v1.0.0 that is included in Go 1.24, released last February. The v1.0.0 module has been awarded [Cryptographic Algorithm Validation Program (CAVP) certificate A6650](https://csrc.nist.gov/projects/cryptographic-algorithm-validation-program/details?validation=39260) , was submitted to the Cryptographic Module Validation Program (CMVP), and reached the [Modules In Process List](https://csrc.nist.gov/Projects/cryptographic-module-validation-program/modules-in-process/modules-in-process-list) in May. Modules on the MIP list are awaiting NIST review and can already be deployed in certain regulated environments. [Geomys](https://geomys.org/) led the implementation effort in collaboration with the Go Security Team, and is pursuing a broadly applicable FIPS 140-3 validation for the benefit of the Go community. Google and other industry stakeholders have a contractual relationship with Geomys to include specific Operating Environments in the certificate. Further details on the module are available in the [documentation](https://go.dev/doc/security/fips140) . Some Go users currently rely on the [Go+BoringCrypto](https://go.dev/doc/security/fips140#goboringcrypto) GOEXPERIMENT, or on one of its forks, as part of their FIPS 140 compliance strategy. Unlike the FIPS 140-3 Go Cryptographic Module, Go+BoringCrypto was never officially supported and had significant developer experience issues, since it was produced exclusively for the internal needs of Google. It will be removed in a future release once Google migrates to the native module. A native developer experience ----------------------------- The module integrates completely transparently into Go applications. In fact, every Go program built with Go 1.24 already uses it for all FIPS 140-3 approved algorithms! The module is just another name for the `crypto/internal/fips140/...` packages of the standard library, which provide the implementation of operations exposed by packages such as `crypto/ecdsa` and `crypto/rand`. These packages involve no cgo, meaning they cross-compile like any other Go program, they pay no FFI performance overhead, and they don’t suffer from [memory management security issues](https://go.dev/blog/tob-crypto-audit#cgo-memory-management) , unlike Go+BoringCrypto and its forks. When starting a Go binary, the module can be put into FIPS 140-3 mode with the `fips140=on` [GODEBUG option](https://go.dev/doc/godebug) , which can be set as an environment variable or through the `go.mod` file. If FIPS 140-3 mode is enabled, the module will use the NIST DRBG for randomness, `crypto/tls` will automatically only negotiate FIPS 140-3 approved TLS versions and algorithms, and it will perform the mandatory self-tests on initialization and during key generation. That’s it; there are no other behavior differences. There is also an experimental stricter mode, `fips140=only`, which causes all non-approved algorithms to return errors or panic. We understand this might be too inflexible for most deployments and are [looking for feedback](https://go.dev/issue/74630) on what a policy enforcement framework might look like. Finally, applications can use the [`GOFIPS140` environment variable](https://go.dev/doc/security/fips140#the-gofips140-environment-variable) to build against older, validated versions of the `crypto/internal/fips140/...` packages. `GOFIPS140` works like `GOOS` and `GOARCH`, and if set to `GOFIPS140=v1.0.0` the program will be built against the v1.0.0 snapshot of the packages as they were submitted for validation to CMVP. This snapshot ships with the rest of the Go standard library, as `lib/fips140/v1.0.0.zip`. When using `GOFIPS140`, the `fips140` GODEBUG defaults to `on`, so putting it all together, all that’s needed to build against the FIPS 140-3 module and run in FIPS 140-3 mode is `GOFIPS140=v1.0.0 go build`. That’s it. If a toolchain is built with `GOFIPS140` set, all builds it produces will default to that value. The `GOFIPS140` version used to build a binary can be verified with `go version -m`. Future versions of Go will continue shipping and working with v1.0.0 of the Go Cryptographic Module until the next version is fully certified by Geomys, but some new cryptography features might not be available when building against old modules. Starting with Go 1.24.3, you can use `GOFIPS140=inprocess` to dynamically select the latest module for which a Geomys validation has reached the In Process stage. Geomys plans to validate new module versions at least every year—to avoid leaving FIPS 140 builds too far behind—and every time a vulnerability in the module can’t be mitigated in the calling standard library code. Uncompromising security ----------------------- Our first priority in developing the module has been matching or exceeding the security of the existing Go standard library cryptography packages. It might be surprising, but sometimes the easiest way to achieve and demonstrate compliance with the FIPS 140 security requirements is not to exceed them. We declined to accept that. For example, `crypto/ecdsa` [always produced hedged signatures](https://cs.opensource.google/go/go/+/refs/tags/go1.23.0:src/crypto/ecdsa/ecdsa.go;l=417) . Hedged signatures generate nonces by combining the private key, the message, and random bytes. Like [deterministic ECDSA](https://www.rfc-editor.org/rfc/rfc6979) , they protect against failure of the random number generator, which would otherwise leak the private key(!). Unlike deterministic ECDSA, they are also resistant to [API issues](https://github.com/MystenLabs/ed25519-unsafe-libs) and [fault attacks](https://en.wikipedia.org/wiki/Differential_fault_analysis) , and they don’t leak message equality. FIPS 186-5 introduced support for [RFC 6979](https://www.rfc-editor.org/rfc/rfc6979) deterministic ECDSA, but not for hedged ECDSA. Instead of downgrading to regular randomized or deterministic ECDSA signatures in FIPS 140-3 mode (or worse, across modes), we [switched the hedging algorithm](https://github.com/golang/go/commit/9776d028f4b99b9a935dae9f63f32871b77c49af) and connected dots across half a dozen documents to [prove the new one is a compliant composition of a DRBG and traditional ECDSA](https://github.com/cfrg/draft-irtf-cfrg-det-sigs-with-noise/issues/6#issuecomment-2067819904) . While at it, we also [added opt-in support for deterministic signatures](https://go.dev/doc/go1.24#cryptoecdsapkgcryptoecdsa) . Another example is random number generation. FIPS 140-3 has strict rules on how cryptographic randomness is generated, which essentially enforce the use of a userspace [CSPRNG](https://en.wikipedia.org/wiki/Cryptographically_secure_pseudorandom_number_generator) . Conversely, we believe the kernel is best suited to produce secure random bytes, because it’s best positioned to collect entropy from the system, and to detect when processes or even virtual machines are cloned (which could lead to reuse of supposedly random bytes). Hence, [crypto/rand](https://pkg.go.dev/crypto/rand) routes every read operation to the kernel. To square this circle, in FIPS 140-3 mode we maintain a compliant userspace NIST DRBG based on AES-256-CTR, and then inject into it 128 bits sourced from the kernel at every read operation. This extra entropy is considered “uncredited” additional data for FIPS 140-3 purposes, but in practice makes it as strong as reading directly from the kernel—even if slower. Finally, all of the Go Cryptographic Module v1.0.0 was in scope for the [recent security audit by Trail of Bits](https://go.dev/blog/tob-crypto-audit) , and was not affected by the only non-informational finding. Combined with the memory safety guarantees provided by the Go compiler and runtime, we believe this delivers on our goal of making Go one of the easiest, most secure solutions for FIPS 140 compliance. Broad platform support ---------------------- A FIPS 140-3 module is only compliant if operated on a tested or “Vendor Affirmed” Operating Environment, essentially a combination of operating system and hardware platform. To enable as many Go use cases as possible, the Geomys validation is tested on [one of the most comprehensive sets of Operating Environments](https://csrc.nist.gov/projects/cryptographic-algorithm-validation-program/details?product=19371&displayMode=Aggregated) in the industry. Geomys’s laboratory tested various Linux flavors (Alpine Linux on Podman, Amazon Linux, Google Prodimage, Oracle Linux, Red Hat Enterprise Linux, and SUSE Linux Enterprise Server), macOS, Windows, and FreeBSD on a mix of x86-64 (AMD and Intel), ARMv8/9 (Ampere Altra, Apple M, AWS Graviton, and Qualcomm Snapdragon), ARMv7, MIPS, z/ Architecture, and POWER, for a total of 23 tested environments. Some of these were paid for by stakeholders, others were funded by Geomys for the benefit of the Go community. Moreover, the Geomys validation lists a broad set of generic platforms as Vendor Affirmed Operating Environments: * Linux 3.10+ on x86-64 and ARMv7/8/9, * macOS 11–15 on Apple M processors, * FreeBSD 12–14 on x86-64, * Windows 10 and Windows Server 2016–2022 on x86-64, and * Windows 11 and Windows Server 2025 on x86-64 and ARMv8/9. Comprehensive algorithm coverage -------------------------------- It may be surprising, but even using a FIPS 140-3 approved algorithm implemented by a FIPS 140-3 module on a supported Operating Environment is not necessarily enough for compliance; the algorithm must have been specifically covered by testing as part of validation. Hence, to make it as easy as possible to build FIPS 140 compliant applications in Go, all FIPS 140-3 approved algorithms in the standard library are implemented by the Go Cryptographic Module and were tested as part of the validation, from digital signatures to the TLS key schedule. The post-quantum ML-KEM key exchange (FIPS 203), [introduced in Go 1.24](https://go.dev/doc/go1.24#crypto-mlkem) , is also validated, meaning `crypto/tls` can establish FIPS 140-3 compliant post-quantum secure connections with X25519MLKEM768. In some cases, we validated the same algorithms under multiple different NIST designations, to make it possible to use them in full compliance for different purposes. For example, [HKDF is tested and validated under _four_ names](https://words.filippo.io/dispatches/fips-hkdf/) : SP 800-108 Feedback KDF, SP 800-56C two-step KDF, Implementation Guidance D.P OneStepNoCounter KDF, and SP 800-133 Section 6.3 KDF. Finally, we validated some internal algorithms such as CMAC Counter KDF, to make it possible to expose future functionality such as [XAES-256-GCM](https://c2sp.org/XAES-256-GCM) . Overall, the native FIPS 140-3 module delivers a better compliance profile than Go+BoringCrypto, while making more algorithms available to FIPS 140-3 restricted applications. We look forward to the new native Go Cryptographic Module making it easier and safer for Go developers to run FIPS 140 compliant workloads. **Next article:** [Go 1.25 is released](https://go.dev/blog/go1.25) **Previous article:** [Generic interfaces](https://go.dev/blog/generic-interfaces) **[Blog Index](https://go.dev/blog/all) ** go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Codewalk: First-Class Functions in Go - The Go Programming Language Codewalk: First-Class Functions in Go ===================================== [![Pop Out Code](https://go.dev/doc/codewalk/popout.png "View code in new window")](https://go.dev/doc/codewalk/functions/) doc/codewalk/pig.go code on [left](https://go.dev/doc/codewalk/functions/#) • [right](https://go.dev/doc/codewalk/functions/#) code width 70% filepaths [shown](https://go.dev/doc/codewalk/functions/#) • [hidden](https://go.dev/doc/codewalk/functions/#) [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fpig.go&lo=0&hi=0#mark) Introduction Go supports first class functions, higher-order functions, user-defined function types, function literals, closures, and multiple return values. This rich feature set supports a functional programming style in a strongly typed language. In this codewalk we will look at a simple program that simulates a dice game called [Pig](http://en.wikipedia.org/wiki/Pig_(dice)) and evaluates basic strategies. doc/codewalk/pig.go [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fpig.go&lo=17&hi=21#mark) Game overview Pig is a two-player game played with a 6-sided die. Each turn, you may roll or stay. * If you roll a 1, you lose all points for your turn and play passes to your opponent. Any other roll adds its value to your turn score. * If you stay, your turn score is added to your total score, and play passes to your opponent. The first person to reach 100 total points wins. The `score` type stores the scores of the current and opposing players, in addition to the points accumulated during the current turn. doc/codewalk/pig.go:17,21 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fpig.go&lo=23&hi=24#mark) User-defined function types In Go, functions can be passed around just like any other value. A function's type signature describes the types of its arguments and return values. The `action` type is a function that takes a `score` and returns the resulting `score` and whether the current turn is over. If the turn is over, the `player` and `opponent` fields in the resulting `score` should be swapped, as it is now the other player's turn. doc/codewalk/pig.go:23,24 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fpig.go&lo=26&hi=41#mark) Multiple return values Go functions can return multiple values. The functions `roll` and `stay` each return a pair of values. They also match the `action` type signature. These `action` functions define the rules of Pig. doc/codewalk/pig.go:26,41 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fpig.go&lo=43&hi=44#mark) Higher-order functions A function can use other functions as arguments and return values. A `strategy` is a function that takes a `score` as input and returns an `action` to perform. (Remember, an `action` is itself a function.) doc/codewalk/pig.go:43,44 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fpig.go&lo=48&hi=53#mark) Function literals and closures Anonymous functions can be declared in Go, as in this example. Function literals are closures: they inherit the scope of the function in which they are declared. One basic strategy in Pig is to continue rolling until you have accumulated at least k points in a turn, and then stay. The argument `k` is enclosed by this function literal, which matches the `strategy` type signature. doc/codewalk/pig.go:48,53 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fpig.go&lo=56&hi=70#mark) Simulating games We simulate a game of Pig by calling an `action` to update the `score` until one player reaches 100 points. Each `action` is selected by calling the `strategy` function associated with the current player. doc/codewalk/pig.go:56,70 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fpig.go&lo=72&hi=89#mark) Simulating a tournament The `roundRobin` function simulates a tournament and tallies wins. Each strategy plays each other strategy `gamesPerSeries` times. doc/codewalk/pig.go:72,89 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fpig.go&lo=91&hi=94#mark) Variadic function declarations Variadic functions like `ratioString` take a variable number of arguments. These arguments are available as a slice inside the function. doc/codewalk/pig.go:91,94 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fpig.go&lo=110&hi=121#mark) Simulation results The `main` function defines 100 basic strategies, simulates a round robin tournament, and then prints the win/loss record of each strategy. Among these strategies, staying at 25 is best, but the [optimal strategy for Pig](http://www.google.com/search?q=optimal+play+pig) is much more complex. doc/codewalk/pig.go:110,121 [previous step](https://go.dev/doc/codewalk/functions/#) • [next step](https://go.dev/doc/codewalk/functions/#) go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Executing transactions - The Go Programming Language Executing transactions ====================== You can execute database transactions using an [`sql.Tx,`](https://pkg.go.dev/database/sql#Tx) which represents a transaction. In addition to `Commit` and `Rollback` methods representing transaction-specific semantics, `sql.Tx` has all of the methods you use to perform common database operations. To get the `sql.Tx`, you call `DB.Begin` or `DB.BeginTx`. A [database transaction](https://en.wikipedia.org/wiki/Database_transaction) groups multiple operations as part of a larger goal. All of the operations must succeed or none can, with the data’s integrity preserved in either case. Typically, a transaction workflow includes: 1. Beginning the transaction. 2. Performing a set of database operations. 3. If no error occurs, committing the transaction to make database changes. 4. If an error occurs, rolling back the transaction to leave the database unchanged. The `sql` package provides methods for beginning and concluding a transaction, as well as methods for performing the intervening database operations. These methods correspond to the four steps in the workflow above. * Begin a transaction. [`DB.Begin`](https://pkg.go.dev/database/sql#DB.Begin) or [`DB.BeginTx`](https://pkg.go.dev/database/sql#DB.BeginTx) begin a new database transaction, returning an `sql.Tx` that represents it. * Perform database operations. Using an `sql.Tx`, you can query or update the database in a series of operations that use a single connection. To support this, `Tx` exports the following methods: * [`Exec`](https://pkg.go.dev/database/sql#Tx.Exec) and [`ExecContext`](https://pkg.go.dev/database/sql#Tx.ExecContext) for making database changes through SQL statements such as `INSERT`, `UPDATE`, and `DELETE`. For more, see [Executing SQL statements that don’t return data](https://go.dev/doc/database/change-data) . * [`Query`](https://pkg.go.dev/database/sql#Tx.Query) , [`QueryContext`](https://pkg.go.dev/database/sql#Tx.QueryContext) , [`QueryRow`](https://pkg.go.dev/database/sql#Tx.QueryRow) , and [`QueryRowContext`](https://pkg.go.dev/database/sql#Tx.QueryRowContext) for operations that return rows. For more, see [Querying for data](https://go.dev/doc/database/querying) . * [`Prepare`](https://pkg.go.dev/database/sql#Tx.Prepare) , [`PrepareContext`](https://pkg.go.dev/database/sql#Tx.PrepareContext) , [`Stmt`](https://pkg.go.dev/database/sql#Tx.Stmt) , and [`StmtContext`](https://pkg.go.dev/database/sql#Tx.StmtContext) for pre-defining prepared statements. For more, see [Using prepared statements](https://go.dev/doc/database/prepared-statements) . * End the transaction with _one_ of the following: * Commit the transaction using [`Tx.Commit`](https://pkg.go.dev/database/sql#Tx.Commit) . If `Commit` succeeds (returns a `nil` error), then all the query results are confirmed as valid and all the executed updates are applied to the database as a single atomic change. If `Commit` fails, then all the results from `Query` and `Exec` on the `Tx` should be discarded as invalid. * Roll back the transaction using [`Tx.Rollback`](https://pkg.go.dev/database/sql#Tx.Rollback) . Even if `Tx.Rollback` fails, the transaction will no longer be valid, nor will it have been committed to the database. ### Best practices Follow the best practices below to better navigate the complicated semantics and connection management that transactions sometimes require. * Use the APIs described in this section to manage transactions. Do _not_ use transaction-related SQL statements such as `BEGIN` and `COMMIT` directly—doing so can leave your database in an unpredictable state, especially in concurrent programs. * When using a transaction, take care not to call the non-transaction `sql.DB` methods directly, too, as those will execute outside the transaction, giving your code an inconsistent view of the state of the database or even causing deadlocks. ### Example Code in the following example uses a transaction to create a new customer order for an album. Along the way, the code will: 1. Begin a transaction. 2. Defer the transaction’s rollback. If the transaction succeeds, it will be committed before the function exits, making the deferred rollback call a no-op. If the transaction fails it won’t be committed, meaning that the rollback will be called as the function exits. 3. Confirm that there’s sufficient inventory for the album the customer is ordering. 4. If there’s enough, update the inventory count, reducing it by the number of albums ordered. 5. Create a new order and retrieve the new order’s generated ID for the client. 6. Commit the transaction and return the ID. This example uses `Tx` methods that take a `context.Context` argument. This makes it possible for the function’s execution – including database operations – to be canceled if it runs too long or the client connection closes. For more, see [Canceling in-progress operations](https://go.dev/doc/database/cancel-operations) . // CreateOrder creates an order for an album and returns the new order ID. func CreateOrder(ctx context.Context, albumID, quantity, custID int) (orderID int64, err error) { // Create a helper function for preparing failure results. fail := func(err error) (int64, error) { return 0, fmt.Errorf("CreateOrder: %v", err) } // Get a Tx for making transaction requests. tx, err := db.BeginTx(ctx, nil) if err != nil { return fail(err) } // Defer a rollback in case anything fails. defer tx.Rollback() // Confirm that album inventory is enough for the order. var enough bool if err = tx.QueryRowContext(ctx, "SELECT (quantity >= ?) from album where id = ?", quantity, albumID).Scan(&enough); err != nil { if err == sql.ErrNoRows { return fail(fmt.Errorf("no such album")) } return fail(err) } if !enough { return fail(fmt.Errorf("not enough inventory")) } // Update the album inventory to remove the quantity in the order. _, err = tx.ExecContext(ctx, "UPDATE album SET quantity = quantity - ? WHERE id = ?", quantity, albumID) if err != nil { return fail(err) } // Create a new row in the album_order table. result, err := tx.ExecContext(ctx, "INSERT INTO album_order (album_id, cust_id, quantity, date) VALUES (?, ?, ?, ?)", albumID, custID, quantity, time.Now()) if err != nil { return fail(err) } // Get the ID of the order item just created. orderID, err = result.LastInsertId() if err != nil { return fail(err) } // Commit the transaction. if err = tx.Commit(); err != nil { return fail(err) } // Return the order ID. return orderID, nil } go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Opening a database handle - The Go Programming Language 1. [Documentation](https://go.dev/doc/) 2. [Accessing relational databases](https://go.dev/doc/database/) 3. [Opening a database handle](https://go.dev/doc/database/open-handle) Opening a database handle ========================= The [`database/sql`](https://pkg.go.dev/database/sql) package simplifies database access by reducing the need for you to manage connections. Unlike many data access APIs, with `database/sql` you don’t explicitly open a connection, do work, then close the connection. Instead, your code opens a database handle that represents a connection pool, then executes data access operations with the handle, calling a `Close` method only when needed to free resources, such as those held by retrieved rows or a prepared statement. In other words, it’s the database handle, represented by an [`sql.DB`](https://pkg.go.dev/database/sql#DB) , that handles connections, opening and closing them on your code’s behalf. As your code uses the handle to execute database operations, those operations have concurrent access to the database. For more, see [Managing connections](https://go.dev/doc/database/manage-connections) . **Note:** You can also reserve a database connection. For more information, see [Using dedicated connections](https://go.dev/doc/database/manage-connections#dedicated_connections) . In addition to the APIs available in the `database/sql` package, the Go community has developed drivers for all of the most common (and many uncommon) database management systems (DBMSes). When opening a database handle, you follow these high-level steps: 1. Locate a driver. A driver translates requests and responses between your Go code and the database. For more, see [Locating and importing a database driver](https://go.dev/doc/database/open-handle#database_driver) . 2. Open a database handle. After you’ve imported the driver, you can open a handle for a specific database. For more, see [Opening a database handle](https://go.dev/doc/database/open-handle#opening_handle) . 3. Confirm a connection. Once you’ve opened a database handle, your code can check that a connection is available. For more, see [Confirming a connection](https://go.dev/doc/database/open-handle#confirm_connection) . Your code typically won’t explicitly open or close database connections – that’s done by the database handle. However, your code should free resources it obtains along the way, such as an `sql.Rows` containing query results. For more, see [Freeing resources](https://go.dev/doc/database/open-handle#free_resources) . ### Locating and importing a database driver You’ll need a database driver that supports the DBMS you’re using. To locate a driver for your database, see [SQLDrivers](https://go.dev/wiki/SQLDrivers) . To make the driver available to your code, you import it as you would another Go package. Here’s an example: import "github.com/go-sql-driver/mysql" Note that if you’re not calling any functions directly from the driver package –- such as when it’s being used implicitly by the `sql` package – you’ll need to use a blank import, which prefixes the import path with an underscore: import _ "github.com/go-sql-driver/mysql" **Note:** As a best practice, avoid using the database driver’s own API for database operations. Instead, use functions in the `database/sql` package. This will help keep your code loosely coupled with the DBMS, making it easier to switch to a different DBMS if you need to. ### Opening a database handle An `sql.DB` database handle provides the ability to read from and write to a database, either individually or in a transaction. You can get a database handle by calling either `sql.Open` (which takes a connection string) or `sql.OpenDB` (which takes a `driver.Connector`). Both return a pointer to an [`sql.DB`](https://pkg.go.dev/database/sql#DB) . **Note:** Be sure to keep your database credentials out of your Go source. For more, see [Storing database credentials](https://go.dev/doc/database/open-handle#store_credentials) . #### Opening with a connection string Use the [`sql.Open` function](https://pkg.go.dev/database/sql#Open) when you want to connect using a connection string. The format for the string will vary depending on the driver you’re using. Here’s an example for MySQL: db, err = sql.Open("mysql", "username:password@tcp(127.0.0.1:3306)/jazzrecords") if err != nil { log.Fatal(err) } However, you’ll likely find that capturing connection properties in a more structured way gives you code that’s more readable. The details will vary by driver. For example, you could replace the preceding example with the following, which uses the MySQL driver’s [`Config`](https://pkg.go.dev/github.com/go-sql-driver/mysql#Config) to specify properties and its [`FormatDSN method`](https://pkg.go.dev/github.com/go-sql-driver/mysql#Config.FormatDSN) to build a connection string. // Specify connection properties. cfg := mysql.NewConfig() cfg.User = username cfg.Passwd = password cfg.Net = "tcp" cfg.Addr = "127.0.0.1:3306" cfg.DBName = "jazzrecords" // Get a database handle. db, err = sql.Open("mysql", cfg.FormatDSN()) if err != nil { log.Fatal(err) } #### Opening with a Connector Use the [`sql.OpenDB function`](https://pkg.go.dev/database/sql#OpenDB) when you want to take advantage of driver-specific connection features that aren’t available in a connection string. Each driver supports its own set of connection properties, often providing ways to customize the connection request specific to the DBMS. Adapting the preceding `sql.Open` example to use `sql.OpenDB`, you could create a handle with code such as the following: // Specify connection properties. cfg := mysql.NewConfig() cfg.User = username cfg.Passwd = password cfg.Net = "tcp" cfg.Addr = "127.0.0.1:3306" cfg.DBName = "jazzrecords" // Get a driver-specific connector. connector, err := mysql.NewConnector(&cfg) if err != nil { log.Fatal(err) } // Get a database handle. db = sql.OpenDB(connector) #### Handling errors Your code should check for an error from attempting to create a handle, such as with `sql.Open`. This won’t be a connection error. Instead, you’ll get an error if `sql.Open` was unable to initialize the handle. This could happen, for example, if it’s unable to parse the DSN you specified. ### Confirming a connection When you open a database handle, the `sql` package may not create a new database connection itself right away. Instead, it may create the connection when your code needs it. If you won’t be using the database right away and want to confirm that a connection could be established, call [`Ping`](https://pkg.go.dev/database/sql#DB.Ping) or [`PingContext`](https://pkg.go.dev/database/sql#DB.PingContext) . Code in the following example pings the database to confirm a connection. db, err = sql.Open("mysql", connString) // Confirm a successful connection. if err := db.Ping(); err != nil { log.Fatal(err) } ### Storing database credentials Avoid storing database credentials in your Go source, which could expose the contents of your database to others. Instead, find a way to store them in a location outside your code but available to it. For example, consider a secret keeper app that stores credentials and provides an API your code can use to retrieve credentials for authenticating with your DBMS. One popular approach is to store the secrets in the environment before the program starts, perhaps loaded from a secret manager, and then your Go program can read them using [`os.Getenv`](https://pkg.go.dev/os#Getenv) : username := os.Getenv("DB_USER") password := os.Getenv("DB_PASS") This approach also lets you set the environment variables yourself for local testing. ### Freeing resources Although you don’t manage or close connections explicitly with the `database/sql` package, your code should free resources it has obtained when they’re no longer needed. Those can include resources held by an `sql.Rows` representing data returned from a query or an `sql.Stmt` representing a prepared statement. Typically, you close resources by deferring a call to a `Close` function so that resources are released before the enclosing function exits. Code in the following example defers `Close` to free the resource held by [`sql.Rows`](https://pkg.go.dev/database/sql#Rows) . rows, err := db.Query("SELECT * FROM album WHERE artist = ?", artist) if err != nil { log.Fatal(err) } defer rows.Close() // Loop through returned rows. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # go.mod file reference - The Go Programming Language go.mod file reference ===================== Each Go module is defined by a go.mod file that describes the module’s properties, including its dependencies on other modules and on versions of Go. These properties include: * The current module’s **module path**. This should be a location from which the module can be downloaded by Go tools, such as the module code’s repository location. This serves as a unique identifier, when combined with the module’s version number. It is also the prefix of the package path for all packages in the module. For more about how Go locates the module, see the [Go Modules Reference](https://go.dev/ref/mod#vcs-find) . * The minimum **version of Go** required by the current module. * A list of minimum versions of other **modules required** by the current module. * Instructions, optionally, to **replace** a required module with another module version or a local directory, or to **exclude** a specific version of a required module. Go generates a go.mod file when you run the [`go mod init` command](https://go.dev/ref/mod#go-mod-init) . The following example creates a go.mod file, setting the module’s module path to example/mymodule: $ go mod init example/mymodule Use `go` commands to manage dependencies. The commands ensure that the requirements described in your go.mod file remain consistent and the content of your go.mod file is valid. These commands include the [`go get`](https://go.dev/ref/mod#go-get) and [`go mod tidy`](https://go.dev/ref/mod#go-mod-tidy) and [`go mod edit`](https://go.dev/ref/mod#go-mod-edit) commands. For reference on `go` commands, see [Command go](https://go.dev/cmd/go/) . You can get help from the command line by typing `go help` _command-name_, as with `go help mod tidy`. **See also** * Go tools make changes to your go.mod file as you use them to manage dependencies. For more, see [Managing dependencies](https://go.dev/doc/modules/managing-dependencies) . * For more details and constraints related to go.mod files, see the [Go modules reference](https://go.dev/ref/mod#go-mod-file) . Example ------- A go.mod file includes directives as shown in the following example. These are described elsewhere in this topic. module example.com/mymodule go 1.14 require ( example.com/othermodule v1.2.3 example.com/thismodule v1.2.3 example.com/thatmodule v1.2.3 ) replace example.com/thatmodule => ../thatmodule exclude example.com/thismodule v1.3.0 module ------ Declares the module’s module path, which is the module’s unique identifier (when combined with the module version number). The module path becomes the import prefix for all packages the module contains. For more, see [`module` directive](https://go.dev/ref/mod#go-mod-file-module) in the Go Modules Reference. ### Syntax module module-path module-path The module's module path, usually the repository location from which the module can be downloaded by Go tools. For module versions v2 and later, this value must end with the major version number, such as `/v2`. ### Examples The following examples substitute `example.com` for a repository domain from which the module could be downloaded. * Module declaration for a v0 or v1 module: module example.com/mymodule * Module path for a v2 module: module example.com/mymodule/v2 ### Notes The module path must uniquely identify your module. For most modules, the path is a URL where the `go` command can find the code (or a redirect to the code). For modules that won’t ever be downloaded directly, the module path can be just some name you control that will ensure uniqueness. The prefix `example/` is also reserved for use in examples like these. For more details, see [Managing dependencies](https://go.dev/doc/modules/managing-dependencies#naming_module) . In practice, the module path is typically the module source’s repository domain and path to the module code within the repository. The `go` command relies on this form when downloading module versions to resolve dependencies on the module user’s behalf. Even if you’re not at first intending to make your module available for use from other code, using its repository path is a best practice that will help you avoid having to rename the module if you publish it later. If at first you don’t know the module’s eventual repository location, consider temporarily using a safe substitute, such as the name of a domain you own or a name you control (such as your company name), along with a path following from the module’s name or source directory. For more, see [Managing dependencies](https://go.dev/doc/modules/managing-dependencies#naming_module) . For example, if you’re developing in a `stringtools` directory, your temporary module path might be `/stringtools`, as in the following example, where _company-name_ is your company’s name: go mod init /stringtools go -- Indicates that the module was written assuming the semantics of the Go version specified by the directive. For more, see [`go` directive](https://go.dev/ref/mod#go-mod-file-go) in the Go Modules Reference. ### Syntax go minimum-go-version minimum-go-version The minimum version of Go required to compile packages in this module. ### Examples * Module must run on Go version 1.14 or later: go 1.14 ### Notes The `go` directive sets the minimum version of Go required to use this module. Before Go 1.21, the directive was advisory only; now it is a mandatory requirement: Go toolchains refuse to use modules declaring newer Go versions. The `go` directive is an input into selecting which Go toolchain to run. See “[Go toolchains](https://go.dev/doc/toolchain) ” for details. The `go` directive affects use of new language features: * For packages within the module, the compiler rejects use of language features introduced after the version specified by the `go` directive. For example, if a module has the directive `go 1.12`, its packages may not use numeric literals like `1_000_000`, which were introduced in Go 1.13. * If an older Go version builds one of the module’s packages and encounters a compile error, the error notes that the module was written for a newer Go version. For example, suppose a module has `go 1.13` and a package uses the numeric literal `1_000_000`. If that package is built with Go 1.12, the compiler notes that the code is written for Go 1.13. The `go` directive also affects the behavior of the `go` command: * At `go 1.14` or higher, automatic [vendoring](https://go.dev/ref/mod#vendoring) may be enabled. If the file `vendor/modules.txt` is present and consistent with `go.mod`, there is no need to explicitly use the `-mod=vendor` flag. * At `go 1.16` or higher, the `all` package pattern matches only packages transitively imported by packages and tests in the [main module](https://go.dev/ref/mod#glos-main-module) . This is the same set of packages retained by [`go mod vendor`](https://go.dev/ref/mod#go-mod-vendor) since modules were introduced. In lower versions, `all` also includes tests of packages imported by packages in the main module, tests of those packages, and so on. * At `go 1.17` or higher: * The `go.mod` file includes an explicit [`require` directive](https://go.dev/ref/mod#go-mod-file-require) for each module that provides any package transitively imported by a package or test in the main module. (At `go 1.16` and lower, an indirect dependency is included only if [minimal version selection](https://go.dev/ref/mod#minimal-version-selection) would otherwise select a different version.) This extra information enables [module graph pruning](https://go.dev/ref/mod#graph-pruning) and [lazy module loading](https://go.dev/ref/mod#lazy-loading) . * Because there may be many more `// indirect` dependencies than in previous `go` versions, indirect dependencies are recorded in a separate block within the `go.mod` file. * `go mod vendor` omits `go.mod` and `go.sum` files for vendored dependencies. (That allows invocations of the `go` command within subdirectories of `vendor` to identify the correct main module.) * `go mod vendor` records the `go` version from each dependency’s `go.mod` file in `vendor/modules.txt`. * At `go 1.21` or higher: * The `go` line declares a required minimum version of Go to use with this module. * The `go` line must be greater than or equal to the `go` line of all dependencies. * The `go` command no longer attempts to maintain compatibility with the previous older version of Go. * The `go` command is more careful about keeping checksums of `go.mod` files in the `go.sum` file. A `go.mod` file may contain at most one `go` directive. Most commands will add a `go` directive with the current Go version if one is not present. toolchain --------- Declares a suggested Go toolchain to use with this module. Only takes effect when the module is the main module and the default toolchain is older than the suggested toolchain. For more see “[Go toolchains](https://go.dev/doc/toolchain) ” and [`toolchain` directive](https://go.dev/ref/mod/#go-mod-file-toolchain) in the Go Modules Reference. ### Syntax toolchain toolchain-name toolchain-name The suggested Go toolchain's name. Standard toolchain names take the form `go_V_` for a Go version _V_, as in `go1.21.0` and `go1.18rc1`. The special value `default` disables automatic toolchain switching. ### Examples * Suggest using Go 1.21.0 or newer: toolchain go1.21.0 ### Notes See “[Go toolchains](https://go.dev/doc/toolchain) ” for details about how the `toolchain` line affects Go toolchain selection. godebug ------- Indicates the default [GODEBUG](https://go.dev/doc/godebug) settings to be applied to the main packages of this module. These override any toolchain defaults, and are overridden by explicit `//go:debug` lines in main packages. ### Syntax godebug debug-key\=debug-value debug-key The name of the setting to be applied. A list of settings and the versions they were introduced in can be found at [GODEBUG History](https://go.dev/doc/godebug#history) . debug-value The value provided to the setting. If not otherwise specified, `0` to disable and `1` to enable the named behavior. ### Examples * Use the new 1.23 `asynctimerchan=0` behavior: godebug asynctimerchan=0 * Use the default GODEBUGs from Go 1.21, but the old `panicnil=1` behavior: godebug ( default=go1.21 panicnil=1 ) ### Notes GODEBUG settings only apply for builds of main packages and test binaries in the current module. They have no effect when a module is used as a dependency. See “[Go, Backwards Compatibility, and GODEBUG](https://go.dev/doc/godebug) ” for details on backwards compatibility. require ------- Declares a module as a dependency of the current module, specifying the minimum version of the module required. For more, see [`require` directive](https://go.dev/ref/mod#go-mod-file-require) in the Go Modules Reference. ### Syntax require module-path module-version module-path The module's module path, usually a concatenation of the module source's repository domain and the module name. For module versions v2 and later, this value must end with the major version number, such as `/v2`. module-version The module's version. This can be either a release version number, such as v1.2.3, or a Go-generated pseudo-version number, such as v0.0.0-20200921210052-fa0125251cc4. ### Examples * Requiring a released version v1.2.3: require example.com/othermodule v1.2.3 * Requiring a version not yet tagged in its repository by using a pseudo-version number generated by Go tools: require example.com/othermodule v0.0.0-20200921210052-fa0125251cc4 ### Notes When you run a `go` command such as `go get`, Go inserts `require` directives for each module containing imported packages. When a module isn’t yet tagged in its repository, Go assigns a pseudo-version number it generates when you run the command. You can have Go require a module from a location other than its repository by using the [`replace` directive](https://go.dev/doc/modules/gomod-ref#replace) . For more about version numbers, see [Module version numbering](https://go.dev/doc/modules/version-numbers) . For more about managing dependencies, see the following: * [Adding a dependency](https://go.dev/doc/modules/managing-dependencies#adding_dependency) * [Getting a specific dependency version](https://go.dev/doc/modules/managing-dependencies#getting_version) * [Discovering available updates](https://go.dev/doc/modules/managing-dependencies#discovering_updates) * [Upgrading or downgrading a dependency](https://go.dev/doc/modules/managing-dependencies#upgrading) * [Synchronizing your code’s dependencies](https://go.dev/doc/modules/managing-dependencies#synchronizing) tool ---- Adds a package as a dependency of the current module, and makes it available to run with `go tool` when the current working directory is within this module. ### Syntax tool package-path package-path The tool's package path, a concatenation of the module containing the tool and the (possibly empty) path to the package implementing the tool within the module. ### Examples * Declaring a tool implemented in the current module: module example.com/mymodule tool example.com/mymodule/cmd/mytool * Declaring a tool implemented in a separate module: module example.com/mymodule tool example.com/atool/cmd/atool require example.com/atool v1.2.3 ### Notes You can use `go tool` to run tools declared in your module by fully qualified package path or, if there is no ambiguity, by the last path segment. In the first example above you could run `go tool mytool` or `go tool example.com/mymodule/cmd/mytool`. In workspace mode, you can use `go tool` to run a tool declared in any workspace module. Tools are built using the same module graph as the module itself. A [`require` directive](https://go.dev/doc/modules/gomod-ref#require) is needed to select the version of the module that implements the tool. Any [`replace` directives](https://go.dev/doc/modules/gomod-ref#replace) , or [`exclude` directives](https://go.dev/doc/modules/gomod-ref#exclude) also apply to the tool and its dependencies. For more information see [Tool dependencies](https://go.dev/doc/modules/managing-dependencies#tools) . replace ------- Replaces the content of a module at a specific version (or all versions) with another module version or with a local directory. Go tools will use the replacement path when resolving the dependency. For more, see [`replace` directive](https://go.dev/ref/mod#go-mod-file-replace) in the Go Modules Reference. ### Syntax replace module-path \[module-version\] => replacement-path \[replacement-version\] module-path The module path of the module to replace. module-version Optional. A specific version to replace. If this version number is omitted, all versions of the module are replaced with the content on the right side of the arrow. replacement-path The path at which Go should look for the required module. This can be a module path or a path to a directory on the file system local to the replacement module. If this is a module path, you must specify a _replacement-version_ value. If this is a local path, you may not use a _replacement-version_ value. replacement-version The version of the replacement module. The replacement version may only be specified if _replacement-path_ is a module path (not a local directory). ### Examples * Replacing with a fork of the module repository In the following example, any version of example.com/othermodule is replaced with the specified fork of its code. require example.com/othermodule v1.2.3 replace example.com/othermodule => example.com/myfork/othermodule v1.2.3-fixed When you replace one module path with another, do not change import statements for packages in the module you’re replacing. For more on using a forked copy of module code, see [Requiring external module code from your own repository fork](https://go.dev/doc/modules/managing-dependencies#external_fork) . * Replacing with a different version number The following example specifies that version v1.2.3 should be used instead of any other version of the module. require example.com/othermodule v1.2.2 replace example.com/othermodule => example.com/othermodule v1.2.3 The following example replaces module version v1.2.5 with version v1.2.3 of the same module. replace example.com/othermodule v1.2.5 => example.com/othermodule v1.2.3 * Replacing with local code The following example specifies that a local directory should be used as a replacement for all versions of the module. require example.com/othermodule v1.2.3 replace example.com/othermodule => ../othermodule The following example specifies that a local directory should be used as a replacement for v1.2.5 only. require example.com/othermodule v1.2.5 replace example.com/othermodule v1.2.5 => ../othermodule For more on using a local copy of module code, see [Requiring module code in a local directory](https://go.dev/doc/modules/managing-dependencies#local_directory) . ### Notes Use the `replace` directive to temporarily substitute a module path value with another value when you want Go to use the other path to find the module’s source. This has the effect of redirecting Go’s search for the module to the replacement’s location. You needn’t change package import paths to use the replacement path. Use the `exclude` and `replace` directives to control build-time dependency resolution when building the current module. These directives are ignored in modules that depend on the current module. The `replace` directive can be useful in situations such as the following: * You’re developing a new module whose code is not yet in the repository. You want to test with clients using a local version. * You’ve identified an issue with a dependency, have cloned the dependency’s repository, and you’re testing a fix with the local repository. Note that a `replace` directive alone does not add a module to the [module graph](https://go.dev/ref/mod#glos-module-graph) . A [`require` directive](https://go.dev/doc/modules/gomod-ref#require) that refers to a replaced module version is also needed, either in the main module’s `go.mod` file or a dependency’s `go.mod` file. If you don’t have a specific version to replace, you can use a fake version, as in the example below. Note that this will break modules that depend on your module, since `replace` directives are only applied in the main module. require example.com/mod v0.0.0-replace replace example.com/mod v0.0.0-replace => ./mod For more on replacing a required module, including using Go tools to make the change, see: * [Requiring external module code from your own repository fork](https://go.dev/doc/modules/managing-dependencies#external_fork) * [Requiring module code in a local directory](https://go.dev/doc/modules/managing-dependencies#local_directory) For more about version numbers, see [Module version numbering](https://go.dev/doc/modules/version-numbers) . exclude ------- Specifies a module or module version to exclude from the current module’s dependency graph. For more, see [`exclude` directive](https://go.dev/ref/mod#go-mod-file-exclude) in the Go Modules Reference. ### Syntax exclude module-path module-version module-path The module path of the module to exclude. module-version The specific version to exclude. ### Example * Exclude example.com/theirmodule version v1.3.0 exclude example.com/theirmodule v1.3.0 ### Notes Use the `exclude` directive to exclude a specific version of a module that is indirectly required but can’t be loaded for some reason. For example, you might use it to exclude a version of a module that has an invalid checksum. Use the `exclude` and `replace` directives to control build-time dependency resolution when building the current module (the main module you’re building). These directives are ignored in modules that depend on the current module. You can use the [`go mod edit`](https://go.dev/ref/mod#go-mod-edit) command to exclude a module, as in the following example. go mod edit -exclude=example.com/theirmodule@v1.3.0 For more about version numbers, see [Module version numbering](https://go.dev/doc/modules/version-numbers) . retract ------- Indicates that a version or range of versions of the module defined by `go.mod` should not be depended upon. A `retract` directive is useful when a version was published prematurely or a severe problem was discovered after the version was published. For more, see [`retract` directive](https://go.dev/ref/mod#go-mod-file-retract) in the Go Modules Reference. ### Syntax retract version // rationale retract \[version-low,version-high\] // rationale version A single version to retract. version-low Lower bound of a range of versions to retract. version-high Upper bound of a range of versions to retract. Both version-low and version-high are included in the range. rationale Optional comment explaining the retraction. May be shown in messages to the user. ### Example * Retracting a single version retract v1.1.0 // Published accidentally. * Retracting a range of versions retract [v1.0.0,v1.0.5] // Build broken on some platforms. ### Notes Use the `retract` directive to indicate that a previous version of your module should not be used. Users will not automatically upgrade to a retracted version with `go get`, `go mod tidy`, or other commands. Users will not see a retracted version as an available update with `go list -m -u`. Retracted versions should remain available so users that already depend on them are able to build their packages. Even if a retracted version is deleted from the source repository, it may remain available on mirrors such as [proxy.golang.org](https://proxy.golang.org/) . Users that depend on retracted versions may be notified when they run `go get` or `go list -m -u` on related modules. The `go` command discovers retracted versions by reading `retract` directives in the `go.mod` file in the latest version of a module. The latest version is, in order of precedence: 1. Its highest release version, if any 2. Its highest pre-release version, if any 3. A pseudo-version for the tip of the repository’s default branch. When you add a retraction, you almost always need to tag a new, higher version so the command will see it in the latest version of the module. You can publish a version whose sole purpose is to signal retractions. In this case, the new version may also retract itself. For example, if you accidentally tag `v1.0.0`, you can tag `v1.0.1` with the following directives: retract v1.0.0 // Published accidentally. retract v1.0.1 // Contains retraction only. Unfortunately, once a version is published, it cannot be changed. If you later tag `v1.0.0` at a different commit, the `go` command may detect a mismatched sum in `go.sum` or in the [checksum database](https://go.dev/ref/mod#checksum-database) . Retracted versions of a module do not normally appear in the output of `go list -m -versions`, but you can use the `-retracted` to show them. For more, see [`go list -m`](https://go.dev/ref/mod#go-list-m) in the Go Modules Reference. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Developing and publishing modules - The Go Programming Language Developing and publishing modules ================================= You can collect related packages into modules, then publish the modules for other developers to use. This topic gives an overview of developing and publishing modules. To support developing, publishing, and using modules, you use: * A **workflow** through which you develop and publish modules, revising them with new versions over time. See [Workflow for developing and publishing modules](https://go.dev/doc/modules/developing#workflow) . * **Design practices** that help a module’s users understand it and upgrade to new versions in a stable way. See [Design and development](https://go.dev/doc/modules/developing#design) . * A **decentralized system for publishing** modules and retrieving their code. You make your module available for other developers to use from your own repository and publish with a version number. See [Decentralized publishing](https://go.dev/doc/modules/developing#decentralized) . * A **package search engine** and documentation browser (pkg.go.dev) at which developers can find your module. See [Package discovery](https://go.dev/doc/modules/developing#discovery) . * A module **version numbering convention** to communicate expectations of stability and backward compatibility to developers using your module. See [Versioning](https://go.dev/doc/modules/developing#versioning) . * **Go tools** that make it easier for other developers to manage dependencies, including getting your module’s source, upgrading, and so on. See [Managing dependencies](https://go.dev/doc/modules/managing-dependencies) . **See also** * If you’re interested simply in using packages developed by others, this isn’t the topic for you. Instead, see [Managing dependencies](https://go.dev/doc/modules/managing-dependencies) . * For a tutorial that includes a few module development basics, see [Tutorial: Create a Go module](https://go.dev/doc/tutorial/create-module) . Workflow for developing and publishing modules ---------------------------------------------- When you want to publish your modules for others, you adopt a few conventions to make using those modules easier. The following high-level steps are described in more detail in [Module release and versioning workflow](https://go.dev/doc/modules/release-workflow) . 1. Design and code the packages that the module will include. 2. Commit code to your repository using conventions that ensure it’s available to others via Go tools. 3. Publish the module to make it discoverable by developers. 4. Over time, revise the module with versions that use a version numbering convention that signals each version’s stability and backward compatibility. Design and development ---------------------- Your module will be easier for developers to find and use if the functions and packages in it form a coherent whole. When you’re designing a module’s public API, try to keep its functionality focused and discrete. Also, designing and developing your module with backward compatibility in mind helps its users upgrade while minimizing churn to their own code. You can use certain techniques in code to avoid releasing a version that breaks backward compatibility. For more about those techniques, see [Keeping your modules compatible](https://go.dev/blog/module-compatibility) on the Go blog. Before you publish a module, you can reference it on the local file system using the replace directive. This makes it easier to write client code that calls functions in the module while the module is still in development. For more information, see “Coding against an unpublished module” in [Module release and versioning workflow](https://go.dev/doc/modules/release-workflow#unpublished) . Decentralized publishing ------------------------ In Go, you publish your module by tagging its code in your repository to make it available for other developers to use. You don’t need to push your module to a centralized service because Go tools can download your module directly from your repository (located using the module’s path, which is a URL with the scheme omitted) or from a proxy server. After importing your package in their code, developers use Go tools (including the `go get` command) to download your module’s code to compile with. To support this model, you follow conventions and best practices that make it possible for Go tools (on behalf of another developer) to retrieve your module’s source from your repository. For example, Go tools use the module’s module path you specify, along with the module version number you use to tag the module for release, to locate and download the module for its users. For more about source and publishing conventions and best practices, see [Managing module source](https://go.dev/doc/modules/managing-source) . For step-by-step instructions on publishing a module, see [Publishing a module](https://go.dev/doc/modules/publishing) . Package discovery ----------------- After you’ve published your module and someone has fetched it with Go tools, it will become visible on the Go package discovery site at [pkg.go.dev](https://pkg.go.dev/) . There, developers can search the site to find it and read its documentation. To begin using the module, a developer imports packages from the module, then runs the `go get` command to download its source code to compile with. For more about how developers find and use modules, see [Managing dependencies](https://go.dev/doc/modules/managing-dependencies) . Versioning ---------- As you revise and improve your module over time, you assign version numbers (based on the semantic versioning model) designed to signal each version’s stability and backward compatibility. This helps developers using your module determine when the module is stable and whether an upgrade may include significant changes in behavior. You indicate a module’s version number by tagging the module’s source in the repository with the number. For more on developing major version updates, see [Developing a major version update](https://go.dev/doc/modules/major-version) . For more about how you use the semantic versioning model for Go modules, see [Module version numbering](https://go.dev/doc/modules/version-numbers) . go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Accessing relational databases - The Go Programming Language 1. [Documentation](https://go.dev/doc/) 2. [Accessing relational databases](https://go.dev/doc/database/) Accessing relational databases ============================== Using Go, you can incorporate a wide variety of databases and data access approaches into your applications. Topics in this section describe how to use the standard library’s [`database/sql`](https://pkg.go.dev/database/sql) package to access relational databases. For an introductory tutorial to data access with Go, please see [Tutorial: Accessing a relational database](https://go.dev/doc/tutorial/database-access) . Go supports other data access technologies as well, including ORM libraries for higher-level access to relational databases, and also non-relational NoSQL data stores. * **Object-relational mapping (ORM) libraries.** While the `database/sql` package includes functions for lower-level data access logic, you can also use Go to access data stores at a higher abstraction level. For more about two popular object-relational mapping (ORM) libraries for Go, see [GORM](https://gorm.io/index.html) ([package reference](https://pkg.go.dev/gorm.io/gorm) ) and [ent](https://entgo.io/) ([package reference](https://pkg.go.dev/entgo.io/ent) ). * **NoSQL data stores.** The Go community has developed drivers for the majority of NoSQL data stores, including [MongoDB](https://docs.mongodb.com/drivers/go/) and [Couchbase](https://docs.couchbase.com/go-sdk/current/hello-world/overview.html) . You can search [pkg.go.dev](https://pkg.go.dev/) for more. ### Supported database management systems Go supports all of the most common relational database management systems, including MySQL, Oracle, Postgres, SQL Server, SQLite, and more. You’ll find a complete list of drivers at the [SQLDrivers](https://go.dev/wiki/SQLDrivers) page. ### Functions to execute queries or make database changes The `database/sql` package includes functions specifically designed for the kind of database operation you’re executing. For example, while you can use `Query` or `QueryRow` to execute queries, `QueryRow` is designed for the case when you’re expecting only a single row, omitting the overhead of returning an `sql.Rows` that includes only one row. You can use the `Exec` function to make database changes with SQL statements such as `INSERT`, `UPDATE`, or `DELETE`. For more, see the following: * [Executing SQL statements that don’t return data](https://go.dev/doc/database/change-data) * [Querying for data](https://go.dev/doc/database/querying) ### Transactions Through `sql.Tx`, you can write code to execute database operations in a transaction. In a transaction, multiple operations can be performed together and conclude with a final commit, to apply all the changes in one atomic step, or a rollback, to discard them. For more about transactions, see [Executing transactions](https://go.dev/doc/database/execute-transactions) . ### Query cancellation You can use `context.Context` when you want the ability to cancel a database operation, such as when the client’s connection closes or the operation runs longer than you want it to. For any database operation, you can use a `database/sql` package function that takes `Context` as an argument. Using the `Context`, you can specify a timeout or deadline for the operation. You can also use the `Context` to propagate a cancellation request through your application to the function executing an SQL statement, ensuring that resources are freed up if they’re no longer needed. For more, see [Canceling in-progress operations](https://go.dev/doc/database/cancel-operations) . ### Managed connection pool When you use the `sql.DB` database handle, you’re connecting with a built-in connection pool that creates and disposes of connections according to your code’s needs. A handle through `sql.DB` is the most common way to do database access with Go. For more, see [Opening a database handle](https://go.dev/doc/database/open-handle) . The `database/sql` package manages the connection pool for you. However, for more advanced needs, you can set connection pool properties as described in [Setting connection pool properties](https://go.dev/doc/database/manage-connections#connection_pool_properties) . For those operations in which you need a single reserved connection, the `database/sql` package provides [`sql.Conn`](https://pkg.go.dev/database/sql#Conn) . `Conn` is especially useful when a transaction with `sql.Tx` would be a poor choice. For example, your code might need to: * Make schema changes through a DDL, including logic that contains its own transaction semantics. Mixing `sql` package transaction functions with SQL transaction statements is a poor practice, as described in [Executing transactions](https://go.dev/doc/database/execute-transactions) . * Perform query locking operations that create temporary tables. For more, see [Using dedicated connections](https://go.dev/doc/database/manage-connections#dedicated_connections) . go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Module version numbering - The Go Programming Language Module version numbering ======================== A module’s developer uses each part of a module’s version number to signal the version’s stability and backward compatibility. For each new release, a module’s release version number specifically reflects the nature of the module’s changes since the preceding release. When you’re developing code that uses external modules, you can use the version numbers to understand an external module’s stability when you’re considering an upgrade. When you’re developing your own modules, your version numbers will signal your modules’ stability and backward compatibility to other developers. This topic describes what module version numbers mean. **See also** * When you’re using external packages in your code, you can manage those dependencies with Go tools. For more, see [Managing dependencies](https://go.dev/doc/modules/managing-dependencies) . * If you’re developing modules for others to use, you apply a version number when you publish the module, tagging the module in its repository. For more, see [Publishing a module](https://go.dev/doc/modules/publishing) . A released module is published with a version number in the semantic versioning model, as in the following illustration: ![Diagram illustrating a semantic version number showing major version 1, minor version 4, patch version 0, and pre-release version beta 2](https://go.dev/doc/modules/images/version-number.png) The following table describes how the parts of a version number signify a module’s stability and backward compatibility. | Version stage | Example | Message to developers | | --- | --- | --- | | [In development](https://go.dev/doc/modules/version-numbers#in-development) | Automatic pseudo-version number

v**0**.x.x | Signals that the module is still **in development and unstable**. This release carries no backward compatibility or stability guarantees. | | [Major version](https://go.dev/doc/modules/version-numbers#major) | v**1**.x.x | Signals **backward-incompatible public API changes**. This release carries no guarantee that it will be backward compatible with preceding major versions. | | [Minor version](https://go.dev/doc/modules/version-numbers#minor) | vx.**4**.x | Signals **backward-compatible public API changes**. This release guarantees backward compatibility and stability. | | [Patch version](https://go.dev/doc/modules/version-numbers#patch) | vx.x.**1** | Signals **changes that don't affect the module's public API** or its dependencies. This release guarantees backward compatibility and stability. | | [Pre-release version](https://go.dev/doc/modules/version-numbers#pre-release) | vx.x.x-**beta.2** | Signals that this is a **pre-release milestone, such as an alpha or beta**. This release carries no stability guarantees. | In development -------------- Signals that the module is still in development and **unstable**. This release carries no backward compatibility or stability guarantees. The version number can take one of the following forms: **Pseudo-version number** > v0.0.0-20170915032832-14c0d48ead0c **v0 number** > v0.x.x ### Pseudo-version number When a module has not been tagged in its repository, Go tools will generate a pseudo-version number for use in the go.mod file of code that calls functions in the module. **Note:** As a best practice, always allow Go tools to generate the pseudo-version number rather than creating your own. Pseudo-versions are useful when a developer of code consuming the module’s functions needs to develop against a commit that hasn’t been tagged with a semantic version tag yet. A pseudo-version number has three parts separated by dashes, as shown in the following form: #### Syntax _baseVersionPrefix_\-_timestamp_\-_revisionIdentifier_ #### Parts * **baseVersionPrefix** (vX.0.0 or vX.Y.Z-0) is a value derived either from a semantic version tag that precedes the revision or from vX.0.0 if there is no such tag. * **timestamp** (yymmddhhmmss) is the UTC time the revision was created. In Git, this is the commit time, not the author time. * **revisionIdentifier** (abcdefabcdef) is a 12-character prefix of the commit hash, or in Subversion, a zero-padded revision number. ### v0 number A module published with a v0 number will have a formal semantic version number with a major, minor, and patch part, as well as an optional pre-release identifier. Though a v0 version can be used in production, it makes no stability or backward compatibility guarantees. In addition, versions v1 and later are allowed to break backward compatibility for code using the v0 versions. For this reason, a developer with code consuming functions in a v0 module is responsible for adapting to incompatible changes until v1 is released. Pre-release version ------------------- Signals that this is a pre-release milestone, such as an alpha or beta. This release carries no stability guarantees. #### Example vx.x.x-beta.2 A module’s developer can use a pre-release identifier with any major.minor.patch combination by appending a hyphen and the pre-release identifier. Minor version ------------- Signals backward-compatible changes to the module’s public API. This release guarantees backward compatibility and stability. #### Example vx.4.x This version changes the module’s public API, but not in a way that breaks calling code. This might include changes to a module’s own dependencies or the addition of new functions, methods, struct fields, or types. In other words, this version might include enhancements through new functions that another developer might want to use. However, a developer using previous minor versions needn’t change their code otherwise. Patch version ------------- Signals changes that don’t affect the module’s public API or its dependencies. This release guarantees backward compatibility and stability. #### Example vx.x.1 An update that increments this number is only for minor changes such as bug fixes. Developers of consuming code can upgrade to this version safely without needing to change their code. Major version ------------- Signals backward-incompatible changes in a module’s public API. This release carries no guarantee that it will be backward compatible with preceding major versions. #### Example v1.x.x A v1 or above version number signals that the module is stable for use (with exceptions for its pre-release versions). Note that because a version 0 makes no stability or backward compatibility guarantees, a developer upgrading a module from v0 to v1 is responsible for adapting to changes that break backward compatibility. A module developer should increment this number past v1 only when necessary because the version upgrade represents significant disruption for developers whose code uses function in the upgraded module. This disruption includes backward-incompatible changes to the public API, as well as the need for developers using the module to update the package path wherever they import packages from the module. A major version update to a number higher than v1 will also have a new module path. That’s because the module path will have the major version number appended, as in the following example: module example.com/mymodule/v2 v2.0.0 A major version update makes this a new module with a separate history from the module’s previous version. If you’re developing modules to publish for others, see “Publishing breaking API changes” in [Module release and versioning workflow](https://go.dev/doc/modules/release-workflow) . For more on the module directive, see [go.mod reference](https://go.dev/doc/modules/gomod-ref) . go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Go's Declaration Syntax - The Go Programming Language [The Go Blog](https://go.dev/blog/) ==================================== Go's Declaration Syntax ======================= Rob Pike 7 July 2010 Introduction ------------ Newcomers to Go wonder why the declaration syntax is different from the tradition established in the C family. In this post we’ll compare the two approaches and explain why Go’s declarations look as they do. C syntax -------- First, let’s talk about C syntax. C took an unusual and clever approach to declaration syntax. Instead of describing the types with special syntax, one writes an expression involving the item being declared, and states what type that expression will have. Thus int x; declares x to be an int: the expression ‘x’ will have type int. In general, to figure out how to write the type of a new variable, write an expression involving that variable that evaluates to a basic type, then put the basic type on the left and the expression on the right. Thus, the declarations int *p; int a[3]; state that p is a pointer to int because ‘\*p’ has type int, and that a is an array of ints because a\[3\] (ignoring the particular index value, which is punned to be the size of the array) has type int. What about functions? Originally, C’s function declarations wrote the types of the arguments outside the parens, like this: int main(argc, argv) int argc; char *argv[]; { /* ... */ } Again, we see that main is a function because the expression main(argc, argv) returns an int. In modern notation we’d write int main(int argc, char *argv[]) { /* ... */ } but the basic structure is the same. This is a clever syntactic idea that works well for simple types but can get confusing fast. The famous example is declaring a function pointer. Follow the rules and you get this: int (*fp)(int a, int b); Here, fp is a pointer to a function because if you write the expression (\*fp)(a, b) you’ll call a function that returns int. What if one of fp’s arguments is itself a function? int (*fp)(int (*ff)(int x, int y), int b) That’s starting to get hard to read. Of course, we can leave out the name of the parameters when we declare a function, so main can be declared int main(int, char *[]) Recall that argv is declared like this, char *argv[] so you drop the name from the middle of its declaration to construct its type. It’s not obvious, though, that you declare something of type char \*\[\] by putting its name in the middle. And look what happens to fp’s declaration if you don’t name the parameters: int (*fp)(int (*)(int, int), int) Not only is it not obvious where to put the name inside int (*)(int, int) it’s not exactly clear that it’s a function pointer declaration at all. And what if the return type is a function pointer? int (*(*fp)(int (*)(int, int), int))(int, int) It’s hard even to see that this declaration is about fp. You can construct more elaborate examples but these should illustrate some of the difficulties that C’s declaration syntax can introduce. There’s one more point that needs to be made, though. Because type and declaration syntax are the same, it can be difficult to parse expressions with types in the middle. This is why, for instance, C casts always parenthesize the type, as in (int)M_PI Go syntax --------- Languages outside the C family usually use a distinct type syntax in declarations. Although it’s a separate point, the name usually comes first, often followed by a colon. Thus our examples above become something like (in a fictional but illustrative language) x: int p: pointer to int a: array[3] of int These declarations are clear, if verbose - you just read them left to right. Go takes its cue from here, but in the interests of brevity it drops the colon and removes some of the keywords: x int p *int a [3]int There is no direct correspondence between the look of \[3\]int and how to use a in an expression. (We’ll come back to pointers in the next section.) You gain clarity at the cost of a separate syntax. Now consider functions. Let’s transcribe the declaration for main as it would read in Go, although the real main function in Go takes no arguments: func main(argc int, argv []string) int Superficially that’s not much different from C, other than the change from `char` arrays to strings, but it reads well from left to right: function main takes an int and a slice of strings and returns an int. Drop the parameter names and it’s just as clear - they’re always first so there’s no confusion. func main(int, []string) int One merit of this left-to-right style is how well it works as the types become more complex. Here’s a declaration of a function variable (analogous to a function pointer in C): f func(func(int,int) int, int) int Or if f returns a function: f func(func(int,int) int, int) func(int, int) int It still reads clearly, from left to right, and it’s always obvious which name is being declared - the name comes first. The distinction between type and expression syntax makes it easy to write and invoke closures in Go: sum := func(a, b int) int { return a+b } (3, 4) Pointers -------- Pointers are the exception that proves the rule. Notice that in arrays and slices, for instance, Go’s type syntax puts the brackets on the left of the type but the expression syntax puts them on the right of the expression: var a []int x = a[1] For familiarity, Go’s pointers use the \* notation from C, but we could not bring ourselves to make a similar reversal for pointer types. Thus pointers work like this var p *int x = *p We couldn’t say var p *int x = p* because that postfix \* would conflate with multiplication. We could have used the Pascal ^, for example: var p ^int x = p^ and perhaps we should have (and chosen another operator for xor), because the prefix asterisk on both types and expressions complicates things in a number of ways. For instance, although one can write []int("hi") as a conversion, one must parenthesize the type if it starts with a \*: (*int)(nil) Had we been willing to give up \* as pointer syntax, those parentheses would be unnecessary. So Go’s pointer syntax is tied to the familiar C form, but those ties mean that we cannot break completely from using parentheses to disambiguate types and expressions in the grammar. Overall, though, we believe Go’s type syntax is easier to understand than C’s, especially when things get complicated. Notes ----- Go’s declarations read left to right. It’s been pointed out that C’s read in a spiral! See [The “Clockwise/Spiral Rule”](http://c-faq.com/decl/spiral.anderson.html) by David Anderson. **Next article:** [Share Memory By Communicating](https://go.dev/blog/codelab-share) **Previous article:** [Go Programming session video from Google I/O](https://go.dev/blog/io2010) **[Blog Index](https://go.dev/blog/all) ** go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Module release and versioning workflow - The Go Programming Language Module release and versioning workflow ====================================== When you develop modules for use by other developers, you can follow a workflow that helps ensure a reliable, consistent experience for developers using the module. This topic describes the high-level steps in that workflow. For an overview of module development, see [Developing and publishing modules](https://go.dev/doc/modules/developing) . **See also** * If you’re merely wanting to use external packages in your code, be sure to see [Managing dependencies](https://go.dev/doc/modules/managing-dependencies) . * With each new version, you signal the changes to your module with its version number. For more, see [Module version numbering](https://go.dev/doc/modules/version-numbers) . Common workflow steps --------------------- The following sequence illustrates release and versioning workflow steps for an example new module. For more about each step, see the sections in this topic. 1. **Begin a module** and organize its sources to make it easier for developers to use and for you to maintain. If you’re brand new to developing modules, check out [Tutorial: Create a Go module](https://go.dev/doc/tutorial/create-module) . In Go’s decentralized module publishing system, how you organize your code matters. For more, see [Managing module source](https://go.dev/doc/modules/managing-source) . 2. Set up to **write local client code** that calls functions in the unpublished module. Before you publish a module, it’s unavailable for the typical dependency management workflow using commands such as `go get`. A good way to test your module code at this stage is to try it while it is in a directory local to your calling code. See [Coding against an unpublished module](https://go.dev/doc/modules/release-workflow#unpublished) for more about local development. 3. When the module’s code is ready for other developers to try it out, **begin publishing v0 pre-releases** such as alphas and betas. See [Publishing pre-release versions](https://go.dev/doc/modules/release-workflow#pre-release) for more. 4. **Release a v0** that’s not guaranteed to be stable, but which users can try out. For more, see [Publishing the first (unstable) version](https://go.dev/doc/modules/release-workflow#first-unstable) . 5. After your v0 version is published, you can (and should!) continue to **release new versions** of it. These new versions might include bug fixes (patch releases), additions to the module’s public API (minor releases), and even breaking changes. Because a v0 release makes no guarantees of stability or backward compatibility, you can make breaking changes in its versions. For more, see [Publishing bug fixes](https://go.dev/doc/modules/release-workflow#bug-fixes) and [Publishing non-breaking API changes](https://go.dev/doc/modules/release-workflow#non-breaking) . 6. When you’re getting a stable version ready for release, you **publish pre-releases as alphas and betas**. For more, see [Publishing pre-release versions](https://go.dev/doc/modules/release-workflow#pre-release) . 7. Release a v1 as the **first stable release**. This is the first release that makes commitments about the module’s stability. For more, see [Publishing the first stable version](https://go.dev/doc/modules/release-workflow#first-stable) . 8. In the v1 version, **continue to fix bugs** and, where necessary, make additions to the module’s public API. For more, see [Publishing bug fixes](https://go.dev/doc/modules/release-workflow#bug-fixes) and [Publishing non-breaking API changes](https://go.dev/doc/modules/release-workflow#non-breaking) . 9. When it can’t be avoided, publish breaking changes in a **new major version**. A major version update – such as from v1.x.x to v2.x.x – can be a very disruptive upgrade for your module’s users. It should be a last resort. For more, see [Publishing breaking API changes](https://go.dev/doc/modules/release-workflow#breaking) . Coding against an unpublished module ------------------------------------ When you begin developing a module or a new version of a module, you won’t yet have published it. Before you publish a module, you won’t be able to use Go commands to add the module as a dependency. Instead, at first, when writing client code in a different module that calls functions in the unpublished module, you’ll need to reference a copy of the module on the local file system. You can reference a module locally from the client module’s go.mod file by using the `replace` directive in the client module’s go.mod file. For more information, see in [Requiring module code in a local directory](https://go.dev/doc/modules/managing-dependencies#local_directory) . Publishing pre-release versions ------------------------------- You can publish pre-release versions to make a module available for others to try it out and give you feedback. A pre-release version includes no guarantee of stability. Pre-release version numbers are appended with a pre-release identifier. For more on version numbers, see [Module version numbering](https://go.dev/doc/modules/version-numbers) . Here are two examples: v0.2.1-beta.1 v1.2.3-alpha When making a pre-release available, keep in mind that developers using the pre-release will need to explicitly specify it by version with the `go get` command. That’s because, by default, the `go` command prefers release versions over pre-release versions when locating the module you’re asking for. So developers must get the pre-release by specifying it explicitly, as in the following example: go get example.com/theirmodule@v1.2.3-alpha You publish a pre-release by tagging the module code in your repository, specifying the pre-release identifier in the tag. For more, see [Publishing a module](https://go.dev/doc/modules/publishing) . Publishing the first (unstable) version --------------------------------------- As when you publish a pre-release version, you can publish release versions that don’t guarantee stability or backward compatibility, but give your users an opportunity to try out the module and give you feedback. Unstable releases are those whose version numbers are in the v0.x.x range. A v0 version makes no stability or backward compatibility guarantees. But it gives you a way to get feedback and refine your API before making stability commitments with v1 and later. For more see, [Module version numbering](https://go.dev/doc/modules/version-numbers) . As with other published versions, you can increment the minor and patch parts of the v0 version number as you make changes toward releasing a stable v1 version. For example, after releasing a v.0.0.0, you might release a v0.0.1 with the first set of bug fixes. Here’s an example version number: v0.1.3 You publish an unstable release by tagging the module code in your repository, specifying a v0 version number in the tag. For more, see [Publishing a module](https://go.dev/doc/modules/publishing) . Publishing the first stable version ----------------------------------- Your first stable release will have a v1.x.x version number. The first stable release follows pre-release and v0 releases through which you got feedback, fixed bugs, and stabilized the module for users. With a v1 release, you’re making the following commitments to developers using your module: * They can upgrade to the major version’s subsequent minor and patch releases without breaking their own code. * You won’t be making further changes to the module’s public API – including its function and method signatures – that break backward compatibility. * You won’t be removing any exported types, which would break backward compatibility. * Future changes to your API (such as adding a new field to a struct) will be backward compatible and will be included in a new minor release. * Bug fixes (such as a security fix) will be included in a patch release or as part of a minor release. **Note:** While your first major version might be a v0 release, a v0 version does not signal stability or backward compatibility guarantees. As a result, when you increment from v0 to v1, you needn’t be mindful of breaking backward compatibility because the v0 release was not considered stable. For more about version numbers, see [Module version numbering](https://go.dev/doc/modules/version-numbers) . Here’s an example of a stable version number: v1.0.0 You publish a first stable release by tagging the module code in your repository, specifying a v1 version number in the tag. For more, see [Publishing a module](https://go.dev/doc/modules/publishing) . Publishing bug fixes -------------------- You can publish a release in which the changes are limited to bug fixes. This is known as a patch release. A _patch release_ includes only minor changes. In particular, it includes no changes to the module’s public API. Developers of consuming code can upgrade to this version safely and without needing to change their code. **Note:** Your patch release should try not to upgrade any of that module’s own transitive dependencies by more than a patch release. Otherwise, someone upgrading to the patch of your module could wind up accidentally pulling in a more invasive change to a transitive dependency that they use. A patch release increments the patch part of the module’s version number. For more see, [Module version numbering](https://go.dev/doc/modules/version-numbers) . In the following example, v1.0.1 is a patch release. Old version: `v1.0.0` New version: `v1.0.1` You publish a patch release by tagging the module code in your repository, incrementing the patch version number in the tag. For more, see [Publishing a module](https://go.dev/doc/modules/publishing) . Publishing non-breaking API changes ----------------------------------- You can make non-breaking changes to your module’s public API and publish those changes in a _minor_ version release. This version changes the API, but not in a way that breaks calling code. This might include changes to a module’s own dependencies or the addition of new functions, methods, struct fields, or types. Even with the changes it includes, this kind of release guarantees backward compatibility and stability for existing code that calls the module’s functions. A minor release increments the minor part of the module’s version number. For more, see [Module version numbering](https://go.dev/doc/modules/version-numbers) . In the following example, v1.1.0 is a minor release. Old version: `v1.0.1` New version: `v1.1.0` You publish a minor release by tagging the module code in your repository, incrementing the minor version number in the tag. For more, see [Publishing a module](https://go.dev/doc/modules/publishing) . Publishing breaking API changes ------------------------------- You can publish a version that breaks backward compatibility by publishing a _major_ version release. A major version release doesn’t guarantee backward compatibility, typically because it includes changes to the module’s public API that would break code using the module’s previous versions. Given the disruptive effect a major version upgrade can have on code relying on the module, you should avoid a major version update if you can. For more about major version updates, see [Developing a major version update](https://go.dev/doc/modules/major-version) . For strategies to avoid making breaking changes, see the blog post [Keeping your modules compatible](https://go.dev/blog/module-compatibility) . Where publishing other kinds of versions requires essentially tagging the module code with the version number, publishing a major version update requires more steps. 1. Before beginning development of the new major version, in your repository create a place for the new version’s source. One way to do this is to create a new branch in your repository that is specifically for the new major version and its subsequent minor and patch versions. For more, see [Managing module source](https://go.dev/doc/modules/managing-source) . 2. In the module’s go.mod file, revise the module path to append the new major version number, as in the following example: example.com/mymodule/v2 Given that the module path is the module’s identifier, this change effectively creates a new module. It also changes the package path, ensuring that developers won’t unintentionally import a version that breaks their code. Instead, those wanting to upgrade will explicitly replace occurrences of the old path with the new one. 3. In your code, change any package paths where you’re importing packages in the module you’re updating, including packages in the module you’re updating. You need to do this because you changed your module path. 4. As with any new release, you should publish pre-release versions to get feedback and bug reports before publishing an official release. 5. Publish the new major version by tagging the module code in your repository, incrementing the major version number in the tag – such as from v1.5.2 to v2.0.0. For more, see [Publishing a module](https://go.dev/doc/modules/publishing) . go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Chrome Content Optimization Service Runs on Go - The Go Programming Language 1. [Why Go](https://go.dev/solutions/) 2. [Using Go at Google](https://go.dev/solutions/google/) 3. [Chrome Content Optimization Service Runs on Go](https://go.dev/solutions/google/chrome) Chrome Content Optimization Service Runs on Go ============================================== ![Chrome](https://go.dev/images/go_chrome_case_study.png) When the product Chrome comes to mind, you probably think solely of the user-installed browser. But behind the scenes, Chrome has an extensive fleet of backends. Among these is the Chrome Optimization Guide service. This service forms an important basis for Chrome’s user experience strategy, operating in the critical path for users, and is implemented in Go. The Chrome Optimization Guide service is designed to bring the power of Google to Chrome by providing hints to the installed browser about what optimizations may be performed on a page load, as well as when they can be applied most effectively. It comprises a conjunction of real-time servers and batch logs analysis. All Lite mode users of Chrome receive data via the service through the following mechanisms: a data blob push that provides hints for well-known sites in their geography, a check-in to Google servers to retrieve hints for hosts that the specific user visits often, and on demand for page loads for which a hint is not already on the device. Were the Chrome Optimization Guide service to suddenly disappear, users might notice a dramatic change in the speed of their page loads and the amount of data consumed while browsing the web. “ Given that Go was a success for us, we plan to continue to use it where appropriate ” — Sophie Chang ,  Software Engineer When the Chrome engineering team started building the service, only a few members had comfort with Go. Most of the team was more familiar with C++, but they found the complex boilerplate required to stand up a C++ server to be too much. The team shared that “\[they\] were pretty motivated to learn Go due to its simplicity, fast ramp-up, and ecosystem.” and that “\[their\] sense of adventure was rewarded.” Millions of users rely on this service to make their Chrome experience better, and choosing Go was no small decision. After their experience so far, the team also shared that “given that Go was a success for us, we plan to continue to use it where appropriate.” In addition to the Chrome Optimization Guide team, engineering teams across Google have adopted Go in their development process. Read about how the [Core Data Solutions](https://go.dev/solutions/google/coredata/) and [Firebase Hosting](https://go.dev/solutions/google/firebase/) teams use Go to build fast, reliable, and efficient software at scale. _Editorial note: The Go team would like to thank Sophie Chang for her contributions to this story._ ![Chrome](https://go.dev/images/logos/chrome.svg) ![Chrome](https://go.dev/images/logos/chrome.svg) ### About Chrome Google Chrome is a more simple, secure, and faster web browser than ever, with Google’s smarts built-in. In this case study, the Chrome Optimization Guide team shared how they experimented with Go, ramped up quickly, and their plans to use Go going forward. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Codewalk: Share Memory By Communicating - The Go Programming Language Codewalk: Share Memory By Communicating ======================================= [![Pop Out Code](https://go.dev/doc/codewalk/popout.png "View code in new window")](https://go.dev/doc/codewalk/sharemem/) doc/codewalk/urlpoll.go code on [left](https://go.dev/doc/codewalk/sharemem/#) • [right](https://go.dev/doc/codewalk/sharemem/#) code width 70% filepaths [shown](https://go.dev/doc/codewalk/sharemem/#) • [hidden](https://go.dev/doc/codewalk/sharemem/#) [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2furlpoll.go&lo=0&hi=0#mark) Introduction Go's approach to concurrency differs from the traditional use of threads and shared memory. Philosophically, it can be summarized: _Don't communicate by sharing memory; share memory by communicating._ Channels allow you to pass references to data structures between goroutines. If you consider this as passing around ownership of the data (the ability to read and write it), they become a powerful and expressive synchronization mechanism. In this codewalk we will look at a simple program that polls a list of URLs, checking their HTTP response codes and periodically printing their state. doc/codewalk/urlpoll.go [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2furlpoll.go&lo=26&hi=30#mark) State type The State type represents the state of a URL. The Pollers send State values to the StateMonitor, which maintains a map of the current state of each URL. doc/codewalk/urlpoll.go:26,30 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2furlpoll.go&lo=60&hi=64#mark) Resource type A Resource represents the state of a URL to be polled: the URL itself and the number of errors encountered since the last successful poll. When the program starts, it allocates one Resource for each URL. The main goroutine and the Poller goroutines send the Resources to each other on channels. doc/codewalk/urlpoll.go:60,64 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2furlpoll.go&lo=86&hi=92#mark) Poller function Each Poller receives Resource pointers from an input channel. In this program, the convention is that sending a Resource pointer on a channel passes ownership of the underlying data from the sender to the receiver. Because of this convention, we know that no two goroutines will access this Resource at the same time. This means we don't have to worry about locking to prevent concurrent access to these data structures. The Poller processes the Resource by calling its Poll method. It sends a State value to the status channel, to inform the StateMonitor of the result of the Poll. Finally, it sends the Resource pointer to the out channel. This can be interpreted as the Poller saying "I'm done with this Resource" and returning ownership of it to the main goroutine. Several goroutines run Pollers, processing Resources in parallel. doc/codewalk/urlpoll.go:86,92 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2furlpoll.go&lo=66&hi=77#mark) The Poll method The Poll method (of the Resource type) performs an HTTP HEAD request for the Resource's URL and returns the HTTP response's status code. If an error occurs, Poll logs the message to standard error and returns the error string instead. doc/codewalk/urlpoll.go:66,77 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2furlpoll.go&lo=94&hi=116#mark) main function The main function starts the Poller and StateMonitor goroutines and then loops passing completed Resources back to the pending channel after appropriate delays. doc/codewalk/urlpoll.go:94,116 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2furlpoll.go&lo=95&hi=96#mark) Creating channels First, main makes two channels of \*Resource, pending and complete. Inside main, a new goroutine sends one Resource per URL to pending and the main goroutine receives completed Resources from complete. The pending and complete channels are passed to each of the Poller goroutines, within which they are known as in and out. doc/codewalk/urlpoll.go:95,96 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2furlpoll.go&lo=98&hi=99#mark) Initializing StateMonitor StateMonitor will initialize and launch a goroutine that stores the state of each Resource. We will look at this function in detail later. For now, the important thing to note is that it returns a channel of State, which is saved as status and passed to the Poller goroutines. doc/codewalk/urlpoll.go:98,99 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2furlpoll.go&lo=101&hi=104#mark) Launching Poller goroutines Now that it has the necessary channels, main launches a number of Poller goroutines, passing the channels as arguments. The channels provide the means of communication between the main, Poller, and StateMonitor goroutines. doc/codewalk/urlpoll.go:101,104 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2furlpoll.go&lo=106&hi=111#mark) Send Resources to pending To add the initial work to the system, main starts a new goroutine that allocates and sends one Resource per URL to pending. The new goroutine is necessary because unbuffered channel sends and receives are synchronous. That means these channel sends will block until the Pollers are ready to read from pending. Were these sends performed in the main goroutine with fewer Pollers than channel sends, the program would reach a deadlock situation, because main would not yet be receiving from complete. Exercise for the reader: modify this part of the program to read a list of URLs from a file. (You may want to move this goroutine into its own named function.) doc/codewalk/urlpoll.go:106,111 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2furlpoll.go&lo=113&hi=115#mark) Main Event Loop When a Poller is done with a Resource, it sends it on the complete channel. This loop receives those Resource pointers from complete. For each received Resource, it starts a new goroutine calling the Resource's Sleep method. Using a new goroutine for each ensures that the sleeps can happen in parallel. Note that any single Resource pointer may only be sent on either pending or complete at any one time. This ensures that a Resource is either being handled by a Poller goroutine or sleeping, but never both simultaneously. In this way, we share our Resource data by communicating. doc/codewalk/urlpoll.go:113,115 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2furlpoll.go&lo=79&hi=84#mark) The Sleep method Sleep calls time.Sleep to pause before sending the Resource to done. The pause will either be of a fixed length (pollInterval) plus an additional delay proportional to the number of sequential errors (r.errCount). This is an example of a typical Go idiom: a function intended to run inside a goroutine takes a channel, upon which it sends its return value (or other indication of completed state). doc/codewalk/urlpoll.go:79,84 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2furlpoll.go&lo=32&hi=50#mark) StateMonitor The StateMonitor receives State values on a channel and periodically outputs the state of all Resources being polled by the program. doc/codewalk/urlpoll.go:32,50 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2furlpoll.go&lo=36&hi=36#mark) The updates channel The variable updates is a channel of State, on which the Poller goroutines send State values. This channel is returned by the function. doc/codewalk/urlpoll.go:36 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2furlpoll.go&lo=37&hi=37#mark) The urlStatus map The variable urlStatus is a map of URLs to their most recent status. doc/codewalk/urlpoll.go:37 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2furlpoll.go&lo=38&hi=38#mark) The Ticker object A time.Ticker is an object that repeatedly sends a value on a channel at a specified interval. In this case, ticker triggers the printing of the current state to standard output every updateInterval nanoseconds. doc/codewalk/urlpoll.go:38 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2furlpoll.go&lo=39&hi=48#mark) The StateMonitor goroutine StateMonitor will loop forever, selecting on two channels: ticker.C and update. The select statement blocks until one of its communications is ready to proceed. When StateMonitor receives a tick from ticker.C, it calls logState to print the current state. When it receives a State update from updates, it records the new status in the urlStatus map. Notice that this goroutine owns the urlStatus data structure, ensuring that it can only be accessed sequentially. This prevents memory corruption issues that might arise from parallel reads and/or writes to a shared map. doc/codewalk/urlpoll.go:39,48 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2furlpoll.go&lo=0&hi=0#mark) Conclusion In this codewalk we have explored a simple example of using Go's concurrency primitives to share memory through communication. This should provide a starting point from which to explore the ways in which goroutines and channels can be used to write expressive and concise concurrent programs. doc/codewalk/urlpoll.go [previous step](https://go.dev/doc/codewalk/sharemem/#) • [next step](https://go.dev/doc/codewalk/sharemem/#) go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Tutorial: Getting started with multi-module workspaces - The Go Programming Language 1. [Documentation](https://go.dev/doc/) 2. [Tutorials](https://go.dev/doc/tutorial/) 3. [Tutorial: Getting started with multi-module workspaces](https://go.dev/doc/tutorial/workspaces) Tutorial: Getting started with multi-module workspaces ====================================================== This tutorial introduces the basics of multi-module workspaces in Go. With multi-module workspaces, you can tell the Go command that you’re writing code in multiple modules at the same time and easily build and run code in those modules. In this tutorial, you’ll create two modules in a shared multi-module workspace, make changes across those modules, and see the results of those changes in a build. **Note:** For other tutorials, see [Tutorials](https://go.dev/doc/tutorial/index.html) . Prerequisites ------------- * **An installation of Go 1.18 or later.** * **A tool to edit your code.** Any text editor you have will work fine. * **A command terminal.** Go works well using any terminal on Linux and Mac, and on PowerShell or cmd in Windows. This tutorial requires go1.18 or later. Make sure you’ve installed Go at Go 1.18 or later using the links at [go.dev/dl](https://go.dev/dl) . Create a module for your code ----------------------------- To begin, create a module for the code you’ll write. 1. Open a command prompt and change to your home directory. On Linux or Mac: $ cd On Windows: C:\> cd %HOMEPATH% The rest of the tutorial will show a $ as the prompt. The commands you use will work on Windows too. 2. From the command prompt, create a directory for your code called workspace. $ mkdir workspace $ cd workspace 3. Initialize the module Our example will create a new module `hello` that will depend on the golang.org/x/example module. Create the hello module: $ mkdir hello $ cd hello $ go mod init example.com/hello go: creating new go.mod: module example.com/hello Add a dependency on the golang.org/x/example/hello/reverse package by using `go get`. $ go get golang.org/x/example/hello/reverse Create hello.go in the hello directory with the following contents: package main import ( "fmt" "golang.org/x/example/hello/reverse" ) func main() { fmt.Println(reverse.String("Hello")) } Now, run the hello program: $ go run . olleH Create the workspace -------------------- In this step, we’ll create a `go.work` file to specify a workspace with the module. #### Initialize the workspace In the `workspace` directory, run: $ go work init ./hello The `go work init` command tells `go` to create a `go.work` file for a workspace containing the modules in the `./hello` directory. The `go` command produces a `go.work` file that looks like this: go 1.18 use ./hello The `go.work` file has similar syntax to `go.mod`. The `go` directive tells Go which version of Go the file should be interpreted with. It’s similar to the `go` directive in the `go.mod` file. The `use` directive tells Go that the module in the `hello` directory should be main modules when doing a build. So in any subdirectory of `workspace` the module will be active. #### Run the program in the workspace directory In the `workspace` directory, run: $ go run ./hello olleH The Go command includes all the modules in the workspace as main modules. This allows us to refer to a package in the module, even outside the module. Running the `go run` command outside the module or the workspace would result in an error because the `go` command wouldn’t know which modules to use. Next, we’ll add a local copy of the `golang.org/x/example/hello` module to the workspace. That module is stored in a subdirectory of the `go.googlesource.com/example` Git repository. We’ll then add a new function to the `reverse` package that we can use instead of `String`. Download and modify the `golang.org/x/example/hello` module ----------------------------------------------------------- In this step, we’ll download a copy of the Git repo containing the `golang.org/x/example/hello` module, add it to the workspace, and then add a new function to it that we will use from the hello program. 1. Clone the repository From the workspace directory, run the `git` command to clone the repository: $ git clone https://go.googlesource.com/example Cloning into 'example'... remote: Total 165 (delta 27), reused 165 (delta 27) Receiving objects: 100% (165/165), 434.18 KiB | 1022.00 KiB/s, done. Resolving deltas: 100% (27/27), done. 2. Add the module to the workspace The Git repo was just checked out into `./example`. The source code for the `golang.org/x/example/hello` module is in `./example/hello`. Add it to the workspace: $ go work use ./example/hello The `go work use` command adds a new module to the go.work file. It will now look like this: go 1.18 use ( ./hello ./example/hello ) The workspace now includes both the `example.com/hello` module and the `golang.org/x/example/hello` module, which provides the `golang.org/x/example/hello/reverse` package. This will allow us to use the new code we will write in our copy of the `reverse` package instead of the version of the package in the module cache that we downloaded with the `go get` command. 3. Add the new function. We’ll add a new function to reverse a number to the `golang.org/x/example/hello/reverse` package. Create a new file named `int.go` in the `workspace/example/hello/reverse` directory containing the following contents: package reverse import "strconv" // Int returns the decimal reversal of the integer i. func Int(i int) int { i, _ = strconv.Atoi(String(strconv.Itoa(i))) return i } 4. Modify the hello program to use the function. Modify the contents of `workspace/hello/hello.go` to contain the following contents: package main import ( "fmt" "golang.org/x/example/hello/reverse" ) func main() { fmt.Println(reverse.String("Hello"), reverse.Int(24601)) } #### Run the code in the workspace From the workspace directory, run $ go run ./hello olleH 10642 The Go command finds the `example.com/hello` module specified in the command line in the `hello` directory specified by the `go.work` file, and similarly resolves the `golang.org/x/example/hello/reverse` import using the `go.work` file. `go.work` can be used instead of adding [`replace`](https://go.dev/ref/mod#go-mod-file-replace) directives to work across multiple modules. Since the two modules are in the same workspace it’s easy to make a change in one module and use it in another. #### Future step Now, to properly release these modules we’d need to make a release of the `golang.org/x/example/hello` module, for example at `v0.1.0`. This is usually done by tagging a commit on the module’s version control repository. See the [module release workflow documentation](https://go.dev/doc/modules/release-workflow) for more details. Once the release is done, we can increase the requirement on the `golang.org/x/example/hello` module in `hello/go.mod`: cd hello go get golang.org/x/example/hello@v0.1.0 That way, the `go` command can properly resolve the modules outside the workspace. Learn more about workspaces --------------------------- The `go` command has a couple of subcommands for working with workspaces in addition to `go work init` which we saw earlier in the tutorial: * `go work use [-r] [dir]` adds a `use` directive to the `go.work` file for `dir`, if it exists, and removes the `use` directory if the argument directory doesn’t exist. The `-r` flag examines subdirectories of `dir` recursively. * `go work edit` edits the `go.work` file similarly to `go mod edit` * `go work sync` syncs dependencies from the workspace’s build list into each of the workspace modules. See [Workspaces](https://go.dev/ref/mod#workspaces) in the Go Modules Reference for more detail on workspaces and `go.work` files. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Querying for data - The Go Programming Language Querying for data ================= When executing an SQL statement that returns data, use one of the `Query` methods provided in the `database/sql` package. Each of these returns a `Row` or `Rows` whose data you can copy to variables using the `Scan` method. You’d use these methods to, for example, execute `SELECT` statements. When executing a statement that doesn’t return data, you can use an `Exec` or `ExecContext` method instead. For more, see [Executing statements that don’t return data](https://go.dev/doc/database/change-data) . The `database/sql` package provides two ways to execute a query for results. * **Querying for a single row** – `QueryRow` returns at most a single `Row` from the database. For more, see [Querying for a single row](https://go.dev/doc/database/querying#single_row) . * **Querying for multiple rows** – `Query` returns all matching rows as a `Rows` struct your code can loop over. For more, see [Querying for multiple rows](https://go.dev/doc/database/querying#multiple_rows) . If your code will be executing the same SQL statement repeatedly, consider using a prepared statement. For more, see [Using prepared statements](https://go.dev/doc/database/prepared-statements) . **Caution:** Don’t use string formatting functions such as `fmt.Sprintf` to assemble an SQL statement! You could introduce an SQL injection risk. For more, see [Avoiding SQL injection risk](https://go.dev/doc/database/sql-injection) . ### Querying for a single row `QueryRow` retrieves at most a single database row, such as when you want to look up data by a unique ID. If multiple rows are returned by the query, the `Scan` method discards all but the first. `QueryRowContext` works like `QueryRow` but with a `context.Context` argument. For more, see [Canceling in-progress operations](https://go.dev/doc/database/cancel-operations) . The following example uses a query to find out if there’s enough inventory to support a purchase. The SQL statement returns `true` if there’s enough, `false` if not. [`Row.Scan`](https://pkg.go.dev/database/sql#Row.Scan) copies the boolean return value into the `enough` variable through a pointer. func canPurchase(id int, quantity int) (bool, error) { var enough bool // Query for a value based on a single row. if err := db.QueryRow("SELECT (quantity >= ?) from album where id = ?", quantity, id).Scan(&enough); err != nil { if err == sql.ErrNoRows { return false, fmt.Errorf("canPurchase %d: unknown album", id) } return false, fmt.Errorf("canPurchase %d: %v", id, err) } return enough, nil } **Note:** Parameter placeholders in prepared statements vary depending on the DBMS and driver you’re using. For example, the [pq driver](https://pkg.go.dev/github.com/lib/pq) for Postgres requires a placeholder like `$1` instead of `?`. #### Handling errors `QueryRow` itself returns no error. Instead, `Scan` reports any error from the combined lookup and scan. It returns [`sql.ErrNoRows`](https://pkg.go.dev/database/sql#ErrNoRows) when the query finds no rows. #### Functions for returning a single row | Function | Description | | --- | --- | | `[DB.QueryRow](https://pkg.go.dev/database/sql#DB.QueryRow) `
`[DB.QueryRowContext](https://pkg.go.dev/database/sql#DB.QueryRowContext) ` | Run a single-row query in isolation. | | `[Tx.QueryRow](https://pkg.go.dev/database/sql#Tx.QueryRow) `
`[Tx.QueryRowContext](https://pkg.go.dev/database/sql#Tx.QueryRowContext) ` | Run a single-row query inside a larger transaction. For more, see [Executing transactions](https://go.dev/doc/database/execute-transactions)
. | | `[Stmt.QueryRow](https://pkg.go.dev/database/sql#Stmt.QueryRow) `
`[Stmt.QueryRowContext](https://pkg.go.dev/database/sql#Stmt.QueryRowContext) ` | Run a single-row query using an already-prepared statement. For more, see [Using prepared statements](https://go.dev/doc/database/prepared-statements)
. | | `[Conn.QueryRowContext](https://pkg.go.dev/database/sql#Conn.QueryRowContext) ` | For use with reserved connections. For more, see [Managing connections](https://go.dev/doc/database/manage-connections)
. | ### Querying for multiple rows You can query for multiple rows using `Query` or `QueryContext`, which return a `Rows` representing the query results. Your code iterates over the returned rows using [`Rows.Next`](https://pkg.go.dev/database/sql#Rows.Next) . Each iteration calls `Scan` to copy column values into variables. `QueryContext` works like `Query` but with a `context.Context` argument. For more, see [Canceling in-progress operations](https://go.dev/doc/database/cancel-operations) . The following example executes a query to return the albums by a specified artist. The albums are returned in an `sql.Rows`. The code uses [`Rows.Scan`](https://pkg.go.dev/database/sql#Rows.Scan) to copy column values into variables represented by pointers. func albumsByArtist(artist string) ([]Album, error) { rows, err := db.Query("SELECT * FROM album WHERE artist = ?", artist) if err != nil { return nil, err } defer rows.Close() // An album slice to hold data from returned rows. var albums []Album // Loop through rows, using Scan to assign column data to struct fields. for rows.Next() { var alb Album if err := rows.Scan(&alb.ID, &alb.Title, &alb.Artist, &alb.Price, &alb.Quantity); err != nil { return albums, err } albums = append(albums, alb) } if err = rows.Err(); err != nil { return albums, err } return albums, nil } Note the deferred call to [`rows.Close`](https://pkg.go.dev/database/sql#Rows.Close) . This releases any resources held by the rows no matter how the function returns. Looping all the way through the rows also closes it implicitly, but it is better to use `defer` to make sure `rows` is closed no matter what. **Note:** Parameter placeholders in prepared statements vary depending on the DBMS and driver you’re using. For example, the [pq driver](https://pkg.go.dev/github.com/lib/pq) for Postgres requires a placeholder like `$1` instead of `?`. #### Handling errors Be sure to check for an error from `sql.Rows` after looping over query results. If the query failed, this is how your code finds out. #### Functions for returning multiple rows | Function | Description | | --- | --- | | `[DB.Query](https://pkg.go.dev/database/sql#DB.Query) `
`[DB.QueryContext](https://pkg.go.dev/database/sql#DB.QueryContext) ` | Run a query in isolation. | | `[Tx.Query](https://pkg.go.dev/database/sql#Tx.Query) `
`[Tx.QueryContext](https://pkg.go.dev/database/sql#Tx.QueryContext) ` | Run a query inside a larger transaction. For more, see [Executing transactions](https://go.dev/doc/database/execute-transactions)
. | | `[Stmt.Query](https://pkg.go.dev/database/sql#Stmt.Query) `
`[Stmt.QueryContext](https://pkg.go.dev/database/sql#Stmt.QueryContext) ` | Run a query using an already-prepared statement. For more, see [Using prepared statements](https://go.dev/doc/database/prepared-statements)
. | | `[Conn.QueryContext](https://pkg.go.dev/database/sql#Conn.QueryContext) ` | For use with reserved connections. For more, see [Managing connections](https://go.dev/doc/database/manage-connections)
. | ### Handling nullable column values The `database/sql` package provides several special types you can use as arguments for the `Scan` function when a column’s value might be null. Each includes a `Valid` field that reports whether the value is non-null, and a field holding the value if so. Code in the following example queries for a customer name. If the name value is null, the code substitutes another value for use in the application. var s sql.NullString err := db.QueryRow("SELECT name FROM customer WHERE id = ?", id).Scan(&s) if err != nil { log.Fatal(err) } // Find customer name, using placeholder if not present. name := "Valued Customer" if s.Valid { name = s.String } See more about each type in the `sql` package reference: * [`NullBool`](https://pkg.go.dev/database/sql#NullBool) * [`NullFloat64`](https://pkg.go.dev/database/sql#NullFloat64) * [`NullInt32`](https://pkg.go.dev/database/sql#NullInt32) * [`NullInt64`](https://pkg.go.dev/database/sql#NullInt64) * [`NullString`](https://pkg.go.dev/database/sql#NullString) * [`NullTime`](https://pkg.go.dev/database/sql#NullTime) ### Getting data from columns When looping over the rows returned by a query, you use `Scan` to copy a row’s column values into Go values, as described in the [`Rows.Scan`](https://pkg.go.dev/database/sql#Rows.Scan) reference. There is a base set of data conversions supported by all drivers, such as converting SQL `INT` to Go `int`. Some drivers extend this set of conversions; see each individual driver’s documentation for details. As you might expect, `Scan` will convert from column types to Go types that are similar. For example, `Scan` will convert from SQL `CHAR`, `VARCHAR`, and `TEXT` to Go `string`. However, `Scan` will also perform a conversion to another Go type that is a good fit for the column value. For example, if the column is a `VARCHAR` that will always contain a number, you can specify a numeric Go type, such as `int`, to receive the value, and `Scan` will convert it using `strconv.Atoi` for you. For more detail about conversions made by the `Scan` function, see the [`Rows.Scan`](https://pkg.go.dev/database/sql#Rows.Scan) reference. ### Handling multiple result sets When your database operation might return multiple result sets, you can retrieve those by using [`Rows.NextResultSet`](https://pkg.go.dev/database/sql#Rows.NextResultSet) . This can be useful, for example, when you’re sending SQL that separately queries multiple tables, returning a result set for each. `Rows.NextResultSet` prepares the next result set so that a call to `Rows.Next` retrieves the first row from that next set. It returns a boolean indicating whether there is a next result set at all. Code in the following example uses `DB.Query` to execute two SQL statements. The first result set is from the first query in the procedure, retrieving all of the rows in the `album` table. The next result set is from the second query, retrieving rows from the `song` table. rows, err := db.Query("SELECT * from album; SELECT * from song;") if err != nil { log.Fatal(err) } defer rows.Close() // Loop through the first result set. for rows.Next() { // Handle result set. } // Advance to next result set. rows.NextResultSet() // Loop through the second result set. for rows.Next() { // Handle second set. } // Check for any error in either result set. if err := rows.Err(); err != nil { log.Fatal(err) } go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Diagnostics - The Go Programming Language 1. [Documentation](https://go.dev/doc/) 2. [Diagnostics](https://go.dev/doc/diagnostics) Diagnostics =========== Introduction ------------ The Go ecosystem provides a large suite of APIs and tools to diagnose logic and performance problems in Go programs. This page summarizes the available tools and helps Go users pick the right one for their specific problem. Diagnostics solutions can be categorized into the following groups: * **Profiling**: Profiling tools analyze the complexity and costs of a Go program such as its memory usage and frequently called functions to identify the expensive sections of a Go program. * **Tracing**: Tracing is a way to instrument code to analyze latency throughout the lifecycle of a call or user request. Traces provide an overview of how much latency each component contributes to the overall latency in a system. Traces can span multiple Go processes. * **Debugging**: Debugging allows us to pause a Go program and examine its execution. Program state and flow can be verified with debugging. * **Runtime statistics and events**: Collection and analysis of runtime stats and events provides a high-level overview of the health of Go programs. Spikes/dips of metrics helps us to identify changes in throughput, utilization, and performance. Note: Some diagnostics tools may interfere with each other. For example, precise memory profiling skews CPU profiles and goroutine blocking profiling affects scheduler trace. Use tools in isolation to get more precise info. Profiling --------- Profiling is useful for identifying expensive or frequently called sections of code. The Go runtime provides [profiling data](https://go.dev/pkg/runtime/pprof/) in the format expected by the [pprof visualization tool](https://github.com/google/pprof/blob/master/doc/README.md) . The profiling data can be collected during testing via `go` `test` or endpoints made available from the [net/http/pprof](https://go.dev/pkg/net/http/pprof/) package. Users need to collect the profiling data and use pprof tools to filter and visualize the top code paths. Predefined profiles provided by the [runtime/pprof](https://go.dev/pkg/runtime/pprof) package: * **cpu**: CPU profile determines where a program spends its time while actively consuming CPU cycles (as opposed to while sleeping or waiting for I/O). * **heap**: Heap profile reports memory allocation samples; used to monitor current and historical memory usage, and to check for memory leaks. * **threadcreate**: Thread creation profile reports the sections of the program that lead the creation of new OS threads. * **goroutine**: Goroutine profile reports the stack traces of all current goroutines. * **block**: Block profile shows where goroutines block waiting on synchronization primitives (including timer channels). Block profile is not enabled by default; use `runtime.SetBlockProfileRate` to enable it. * **mutex**: Mutex profile reports the lock contentions. When you think your CPU is not fully utilized due to a mutex contention, use this profile. Mutex profile is not enabled by default, see `runtime.SetMutexProfileFraction` to enable it. **What other profilers can I use to profile Go programs?** On Linux, [perf tools](https://perf.wiki.kernel.org/index.php/Tutorial) can be used for profiling Go programs. Perf can profile and unwind cgo/SWIG code and kernel, so it can be useful to get insights into native/kernel performance bottlenecks. On macOS, [Instruments](https://developer.apple.com/library/content/documentation/DeveloperTools/Conceptual/InstrumentsUserGuide/) suite can be used profile Go programs. **Can I profile my production services?** Yes. It is safe to profile programs in production, but enabling some profiles (e.g. the CPU profile) adds cost. You should expect to see performance downgrade. The performance penalty can be estimated by measuring the overhead of the profiler before turning it on in production. You may want to periodically profile your production services. Especially in a system with many replicas of a single process, selecting a random replica periodically is a safe option. Select a production process, profile it for X seconds for every Y seconds and save the results for visualization and analysis; then repeat periodically. Results may be manually and/or automatically reviewed to find problems. Collection of profiles can interfere with each other, so it is recommended to collect only a single profile at a time. **What are the best ways to visualize the profiling data?** The Go tools provide text, graph, and [callgrind](http://valgrind.org/docs/manual/cl-manual.html) visualization of the profile data using `[go tool pprof](https://github.com/google/pprof/blob/master/doc/README.md) `. Read [Profiling Go programs](https://go.dev/blog/profiling-go-programs) to see them in action. ![](https://go.dev/images/diagnostics/pprof-text.png) Listing of the most expensive calls as text. ![](https://go.dev/images/diagnostics/pprof-dot.png) Visualization of the most expensive calls as a graph. Weblist view displays the expensive parts of the source line by line in an HTML page. In the following example, 530ms is spent in the `runtime.concatstrings` and cost of each line is presented in the listing. ![](https://go.dev/images/diagnostics/pprof-weblist.png) Visualization of the most expensive calls as weblist. Another way to visualize profile data is a [flame graph](http://www.brendangregg.com/flamegraphs.html) . Flame graphs allow you to move in a specific ancestry path, so you can zoom in/out of specific sections of code. The [upstream pprof](https://github.com/google/pprof) has support for flame graphs. ![](https://go.dev/images/diagnostics/flame.png) Flame graphs offers visualization to spot the most expensive code-paths. **Am I restricted to the built-in profiles?** Additionally to what is provided by the runtime, Go users can create their custom profiles via [pprof.Profile](https://go.dev/pkg/runtime/pprof/#Profile) and use the existing tools to examine them. **Can I serve the profiler handlers (/debug/pprof/...) on a different path and port?** Yes. The `net/http/pprof` package registers its handlers to the default mux by default, but you can also register them yourself by using the handlers exported from the package. For example, the following example will serve the pprof.Profile handler on :7777 at /custom\_debug\_path/profile: package main import ( "log" "net/http" "net/http/pprof" ) func main() { mux := http.NewServeMux() mux.HandleFunc("/custom\_debug\_path/profile", pprof.Profile) log.Fatal(http.ListenAndServe(":7777", mux)) } Tracing ------- Tracing is a way to instrument code to analyze latency throughout the lifecycle of a chain of calls. Go provides [golang.org/x/net/trace](https://godoc.org/golang.org/x/net/trace) package as a minimal tracing backend per Go node and provides a minimal instrumentation library with a simple dashboard. Go also provides an execution tracer to trace the runtime events within an interval. Tracing enables us to: * Instrument and analyze application latency in a Go process. * Measure the cost of specific calls in a long chain of calls. * Figure out the utilization and performance improvements. Bottlenecks are not always obvious without tracing data. In monolithic systems, it's relatively easy to collect diagnostic data from the building blocks of a program. All modules live within one process and share common resources to report logs, errors, and other diagnostic information. Once your system grows beyond a single process and starts to become distributed, it becomes harder to follow a call starting from the front-end web server to all of its back-ends until a response is returned back to the user. This is where distributed tracing plays a big role to instrument and analyze your production systems. Distributed tracing is a way to instrument code to analyze latency throughout the lifecycle of a user request. When a system is distributed and when conventional profiling and debugging tools don’t scale, you might want to use distributed tracing tools to analyze the performance of your user requests and RPCs. Distributed tracing enables us to: * Instrument and profile application latency in a large system. * Track all RPCs within the lifecycle of a user request and see integration issues that are only visible in production. * Figure out performance improvements that can be applied to our systems. Many bottlenecks are not obvious before the collection of tracing data. The Go ecosystem provides various distributed tracing libraries per tracing system and backend-agnostic ones. **Is there a way to automatically intercept each function call and create traces?** Go doesn’t provide a way to automatically intercept every function call and create trace spans. You need to manually instrument your code to create, end, and annotate spans. **How should I propagate trace headers in Go libraries?** You can propagate trace identifiers and tags in the [`context.Context`](https://go.dev/pkg/context#Context) . There is no canonical trace key or common representation of trace headers in the industry yet. Each tracing provider is responsible for providing propagation utilities in their Go libraries. **What other low-level events from the standard library or runtime can be included in a trace?** The standard library and runtime are trying to expose several additional APIs to notify on low level internal events. For example, [`httptrace.ClientTrace`](https://go.dev/pkg/net/http/httptrace#ClientTrace) provides APIs to follow low-level events in the life cycle of an outgoing request. There is an ongoing effort to retrieve low-level runtime events from the runtime execution tracer and allow users to define and record their user events. Debugging --------- Debugging is the process of identifying why a program misbehaves. Debuggers allow us to understand a program’s execution flow and current state. There are several styles of debugging; this section will only focus on attaching a debugger to a program and core dump debugging. Go users mostly use the following debuggers: * [Delve](https://github.com/derekparker/delve) : Delve is a debugger for the Go programming language. It has support for Go’s runtime concepts and built-in types. Delve is trying to be a fully featured reliable debugger for Go programs. * [GDB](https://go.dev/doc/gdb) : Go provides GDB support via the standard Go compiler and Gccgo. The stack management, threading, and runtime contain aspects that differ enough from the execution model GDB expects that they can confuse the debugger, even when the program is compiled with gccgo. Even though GDB can be used to debug Go programs, it is not ideal and may create confusion. **How well do debuggers work with Go programs?** The `gc` compiler performs optimizations such as function inlining and variable registerization. These optimizations sometimes make debugging with debuggers harder. There is an ongoing effort to improve the quality of the DWARF information generated for optimized binaries. Until those improvements are available, we recommend disabling optimizations when building the code being debugged. The following command builds a package with no compiler optimizations: $ go build -gcflags=all="-N -l" As part of the improvement effort, Go 1.10 introduced a new compiler flag `-dwarflocationlists`. The flag causes the compiler to add location lists that helps debuggers work with optimized binaries. The following command builds a package with optimizations but with the DWARF location lists: $ go build -gcflags="-dwarflocationlists=true" **What’s the recommended debugger user interface?** Even though both delve and gdb provides CLIs, most editor integrations and IDEs provides debugging-specific user interfaces. **Is it possible to do postmortem debugging with Go programs?** A core dump file is a file that contains the memory dump of a running process and its process status. It is primarily used for post-mortem debugging of a program and to understand its state while it is still running. These two cases make debugging of core dumps a good diagnostic aid to postmortem and analyze production services. It is possible to obtain core files from Go programs and use delve or gdb to debug, see the [core dump debugging](https://go.dev/wiki/CoreDumpDebugging) page for a step-by-step guide. Runtime statistics and events ----------------------------- The runtime provides stats and reporting of internal events for users to diagnose performance and utilization problems at the runtime level. Users can monitor these stats to better understand the overall health and performance of Go programs. Some frequently monitored stats and states: * `[runtime.ReadMemStats](https://go.dev/pkg/runtime/#ReadMemStats) ` reports the metrics related to heap allocation and garbage collection. Memory stats are useful for monitoring how much memory resources a process is consuming, whether the process can utilize memory well, and to catch memory leaks. * `[debug.ReadGCStats](https://go.dev/pkg/runtime/debug/#ReadGCStats) ` reads statistics about garbage collection. It is useful to see how much of the resources are spent on GC pauses. It also reports a timeline of garbage collector pauses and pause time percentiles. * `[debug.Stack](https://go.dev/pkg/runtime/debug/#Stack) ` returns the current stack trace. Stack trace is useful to see how many goroutines are currently running, what they are doing, and whether they are blocked or not. * `[debug.WriteHeapDump](https://go.dev/pkg/runtime/debug/#WriteHeapDump) ` suspends the execution of all goroutines and allows you to dump the heap to a file. A heap dump is a snapshot of a Go process' memory at a given time. It contains all allocated objects as well as goroutines, finalizers, and more. * `[runtime.NumGoroutine](https://go.dev/pkg/runtime#NumGoroutine) ` returns the number of current goroutines. The value can be monitored to see whether enough goroutines are utilized, or to detect goroutine leaks. ### Execution tracer Go comes with a runtime execution tracer to capture a wide range of runtime events. Scheduling, syscall, garbage collections, heap size, and other events are collected by runtime and available for visualization by the go tool trace. Execution tracer is a tool to detect latency and utilization problems. You can examine how well the CPU is utilized, and when networking or syscalls are a cause of preemption for the goroutines. Tracer is useful to: * Understand how your goroutines execute. * Understand some of the core runtime events such as GC runs. * Identify poorly parallelized execution. However, it is not great for identifying hot spots such as analyzing the cause of excessive memory or CPU usage. Use profiling tools instead first to address them. ![](https://go.dev/images/diagnostics/tracer-lock.png) Above, the go tool trace visualization shows the execution started fine, and then it became serialized. It suggests that there might be lock contention for a shared resource that creates a bottleneck. See [`go` `tool` `trace`](https://go.dev/cmd/trace/) to collect and analyze runtime traces. ### GODEBUG Runtime also emits events and information if [GODEBUG](https://go.dev/pkg/runtime/#hdr-Environment_Variables) environmental variable is set accordingly. * GODEBUG=gctrace=1 prints garbage collector events at each collection, summarizing the amount of memory collected and the length of the pause. * GODEBUG=inittrace=1 prints a summary of execution time and memory allocation information for completed package initialization work. * GODEBUG=schedtrace=X prints scheduling events every X milliseconds. The GODEBUG environmental variable can be used to disable use of instruction set extensions in the standard library and runtime. * GODEBUG=cpu.all=off disables the use of all optional instruction set extensions. * GODEBUG=cpu._extension_\=off disables use of instructions from the specified instruction set extension. _extension_ is the lower case name for the instruction set extension such as _sse41_ or _avx_. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # The Laws of Reflection - The Go Programming Language [The Go Blog](https://go.dev/blog/) ==================================== The Laws of Reflection ====================== Rob Pike 6 September 2011 Introduction ------------ Reflection in computing is the ability of a program to examine its own structure, particularly through types; it’s a form of metaprogramming. It’s also a great source of confusion. In this article we attempt to clarify things by explaining how reflection works in Go. Each language’s reflection model is different (and many languages don’t support it at all), but this article is about Go, so for the rest of this article the word “reflection” should be taken to mean “reflection in Go”. Note added January 2022: This blog post was written in 2011 and predates parametric polymorphism (a.k.a. generics) in Go. Although nothing important in the article has become incorrect as a result of that development in the language, it has been tweaked in a few places to avoid confusing someone familiar with modern Go. Types and interfaces -------------------- Because reflection builds on the type system, let’s start with a refresher about types in Go. Go is statically typed. Every variable has a static type, that is, exactly one type known and fixed at compile time: `int`, `float32`, `*MyType`, `[]byte`, and so on. If we declare type MyInt int var i int var j MyInt then `i` has type `int` and `j` has type `MyInt`. The variables `i` and `j` have distinct static types and, although they have the same underlying type, they cannot be assigned to one another without a conversion. One important category of type is interface types, which represent fixed sets of methods. (When discussing reflection, we can ignore the use of interface definitions as constraints within polymorphic code.) An interface variable can store any concrete (non-interface) value as long as that value implements the interface’s methods. A well-known pair of examples is `io.Reader` and `io.Writer`, the types `Reader` and `Writer` from the [io package](https://go.dev/pkg/io/) : // Reader is the interface that wraps the basic Read method. type Reader interface { Read(p []byte) (n int, err error) } // Writer is the interface that wraps the basic Write method. type Writer interface { Write(p []byte) (n int, err error) } Any type that implements a `Read` (or `Write`) method with this signature is said to implement `io.Reader` (or `io.Writer`). For the purposes of this discussion, that means that a variable of type `io.Reader` can hold any value whose type has a `Read` method: var r io.Reader r = os.Stdin r = bufio.NewReader(r) r = new(bytes.Buffer) // and so on It’s important to be clear that whatever concrete value `r` may hold, `r`’s type is always `io.Reader`: Go is statically typed and the static type of `r` is `io.Reader`. An extremely important example of an interface type is the empty interface: interface{} or its equivalent alias, any It represents the empty set of methods and is satisfied by any value at all, since every value has zero or more methods. Some people say that Go’s interfaces are dynamically typed, but that is misleading. They are statically typed: a variable of interface type always has the same static type, and even though at run time the value stored in the interface variable may change type, that value will always satisfy the interface. We need to be precise about all this because reflection and interfaces are closely related. The representation of an interface ---------------------------------- Russ Cox has written a [detailed blog post](https://research.swtch.com/2009/12/go-data-structures-interfaces.html) about the representation of interface values in Go. It’s not necessary to repeat the full story here, but a simplified summary is in order. A variable of interface type stores a pair: the concrete value assigned to the variable, and that value’s type descriptor. To be more precise, the value is the underlying concrete data item that implements the interface and the type describes the full type of that item. For instance, after var r io.Reader tty, err := os.OpenFile("/dev/tty", os.O_RDWR, 0) if err != nil { return nil, err } r = tty `r` contains, schematically, the (value, type) pair, (`tty`, `*os.File`). Notice that the type `*os.File` implements methods other than `Read`; even though the interface value provides access only to the `Read` method, the value inside carries all the type information about that value. That’s why we can do things like this: var w io.Writer w = r.(io.Writer) The expression in this assignment is a type assertion; what it asserts is that the item inside `r` also implements `io.Writer`, and so we can assign it to `w`. After the assignment, `w` will contain the pair (`tty`, `*os.File`). That’s the same pair as was held in `r`. The static type of the interface determines what methods may be invoked with an interface variable, even though the concrete value inside may have a larger set of methods. Continuing, we can do this: var empty interface{} empty = w and our empty interface value `empty` will again contain that same pair, (`tty`, `*os.File`). That’s handy: an empty interface can hold any value and contains all the information we could ever need about that value. (We don’t need a type assertion here because it’s known statically that `w` satisfies the empty interface. In the example where we moved a value from a `Reader` to a `Writer`, we needed to be explicit and use a type assertion because `Writer`’s methods are not a subset of `Reader`’s.) One important detail is that the pair inside an interface variable always has the form (value, concrete type) and cannot have the form (value, interface type). Interfaces do not hold interface values. Now we’re ready to reflect. The first law of reflection --------------------------- 1\. Reflection goes from interface value to reflection object. -------------------------------------------------------------- At the basic level, reflection is just a mechanism to examine the type and value pair stored inside an interface variable. To get started, there are two types we need to know about in [package reflect](https://go.dev/pkg/reflect/) : [Type](https://go.dev/pkg/reflect/#Type) and [Value](https://go.dev/pkg/reflect/#Value) . Those two types give access to the contents of an interface variable, and two simple functions, called `reflect.TypeOf` and `reflect.ValueOf`, retrieve `reflect.Type` and `reflect.Value` pieces out of an interface value. (Also, from a `reflect.Value` it’s easy to get to the corresponding `reflect.Type`, but let’s keep the `Value` and `Type` concepts separate for now.) Let’s start with `TypeOf`: package main import ( "fmt" "reflect" ) func main() { var x float64 = 3.4 fmt.Println("type:", reflect.TypeOf(x)) } This program prints type: float64 You might be wondering where the interface is here, since the program looks like it’s passing the `float64` variable `x`, not an interface value, to `reflect.TypeOf`. But it’s there; as [godoc reports](https://go.dev/pkg/reflect/#TypeOf) , the signature of `reflect.TypeOf` includes an empty interface: // TypeOf returns the reflection Type of the value in the interface{}. func TypeOf(i interface{}) Type When we call `reflect.TypeOf(x)`, `x` is first stored in an empty interface, which is then passed as the argument; `reflect.TypeOf` unpacks that empty interface to recover the type information. The `reflect.ValueOf` function, of course, recovers the value (from here on we’ll elide the boilerplate and focus just on the executable code): var x float64 = 3.4 fmt.Println("value:", reflect.ValueOf(x).String()) prints value: (We call the `String` method explicitly because by default the `fmt` package digs into a `reflect.Value` to show the concrete value inside. The `String` method does not.) Both `reflect.Type` and `reflect.Value` have lots of methods to let us examine and manipulate them. One important example is that `Value` has a `Type` method that returns the `Type` of a `reflect.Value`. Another is that both `Type` and `Value` have a `Kind` method that returns a constant indicating what sort of item is stored: `Uint`, `Float64`, `Slice`, and so on. Also methods on `Value` with names like `Int` and `Float` let us grab values (as `int64` and `float64`) stored inside: var x float64 = 3.4 v := reflect.ValueOf(x) fmt.Println("type:", v.Type()) fmt.Println("kind is float64:", v.Kind() == reflect.Float64) fmt.Println("value:", v.Float()) prints type: float64 kind is float64: true value: 3.4 There are also methods like `SetInt` and `SetFloat` but to use them we need to understand settability, the subject of the third law of reflection, discussed below. The reflection library has a couple of properties worth singling out. First, to keep the API simple, the “getter” and “setter” methods of `Value` operate on the largest type that can hold the value: `int64` for all the signed integers, for instance. That is, the `Int` method of `Value` returns an `int64` and the `SetInt` value takes an `int64`; it may be necessary to convert to the actual type involved: var x uint8 = 'x' v := reflect.ValueOf(x) fmt.Println("type:", v.Type()) // uint8. fmt.Println("kind is uint8: ", v.Kind() == reflect.Uint8) // true. x = uint8(v.Uint()) // v.Uint returns a uint64. The second property is that the `Kind` of a reflection object describes the underlying type, not the static type. If a reflection object contains a value of a user-defined integer type, as in type MyInt int var x MyInt = 7 v := reflect.ValueOf(x) the `Kind` of `v` is still `reflect.Int`, even though the static type of `x` is `MyInt`, not `int`. In other words, the `Kind` cannot discriminate an `int` from a `MyInt` even though the `Type` can. The second law of reflection ---------------------------- 2\. Reflection goes from reflection object to interface value. -------------------------------------------------------------- Like physical reflection, reflection in Go generates its own inverse. Given a `reflect.Value` we can recover an interface value using the `Interface` method; in effect the method packs the type and value information back into an interface representation and returns the result: // Interface returns v's value as an interface{}. func (v Value) Interface() interface{} As a consequence we can say y := v.Interface().(float64) // y will have type float64. fmt.Println(y) to print the `float64` value represented by the reflection object `v`. We can do even better, though. The arguments to `fmt.Println`, `fmt.Printf` and so on are all passed as empty interface values, which are then unpacked by the `fmt` package internally just as we have been doing in the previous examples. Therefore all it takes to print the contents of a `reflect.Value` correctly is to pass the result of the `Interface` method to the formatted print routine: fmt.Println(v.Interface()) (Since this article was first written, a change was made to the `fmt` package so that it automatically unpacks a `reflect.Value` like this, so we could just say fmt.Println(v) for the same result, but for clarity we’ll keep the `.Interface()` calls here.) Since our value is a `float64`, we can even use a floating-point format if we want: fmt.Printf("value is %7.1e\n", v.Interface()) and get in this case 3.4e+00 Again, there’s no need to type-assert the result of `v.Interface()` to `float64`; the empty interface value has the concrete value’s type information inside and `Printf` will recover it. In short, the `Interface` method is the inverse of the `ValueOf` function, except that its result is always of static type `interface{}`. Reiterating: Reflection goes from interface values to reflection objects and back again. The third law of reflection --------------------------- 3\. To modify a reflection object, the value must be settable. -------------------------------------------------------------- The third law is the most subtle and confusing, but it’s easy enough to understand if we start from first principles. Here is some code that does not work, but is worth studying. var x float64 = 3.4 v := reflect.ValueOf(x) v.SetFloat(7.1) // Error: will panic. If you run this code, it will panic with the cryptic message panic: reflect.Value.SetFloat using unaddressable value The problem is not that the value `7.1` is not addressable; it’s that `v` is not settable. Settability is a property of a reflection `Value`, and not all reflection `Values` have it. The `CanSet` method of `Value` reports the settability of a `Value`; in our case, var x float64 = 3.4 v := reflect.ValueOf(x) fmt.Println("settability of v:", v.CanSet()) prints settability of v: false It is an error to call a `Set` method on a non-settable `Value`. But what is settability? Settability is a bit like addressability, but stricter. It’s the property that a reflection object can modify the actual storage that was used to create the reflection object. Settability is determined by whether the reflection object holds the original item. When we say var x float64 = 3.4 v := reflect.ValueOf(x) we pass a copy of `x` to `reflect.ValueOf`, so the interface value created as the argument to `reflect.ValueOf` is a copy of `x`, not `x` itself. Thus, if the statement v.SetFloat(7.1) were allowed to succeed, it would not update `x`, even though `v` looks like it was created from `x`. Instead, it would update the copy of `x` stored inside the reflection value and `x` itself would be unaffected. That would be confusing and useless, so it is illegal, and settability is the property used to avoid this issue. If this seems bizarre, it’s not. It’s actually a familiar situation in unusual garb. Think of passing `x` to a function: f(x) We would not expect `f` to be able to modify `x` because we passed a copy of `x`’s value, not `x` itself. If we want `f` to modify `x` directly we must pass our function the address of `x` (that is, a pointer to `x`): f(&x) This is straightforward and familiar, and reflection works the same way. If we want to modify `x` by reflection, we must give the reflection library a pointer to the value we want to modify. Let’s do that. First we initialize `x` as usual and then create a reflection value that points to it, called `p`. var x float64 = 3.4 p := reflect.ValueOf(&x) // Note: take the address of x. fmt.Println("type of p:", p.Type()) fmt.Println("settability of p:", p.CanSet()) The output so far is type of p: *float64 settability of p: false The reflection object `p` isn’t settable, but it’s not `p` we want to set, it’s (in effect) `*p`. To get to what `p` points to, we call the `Elem` method of `Value`, which indirects through the pointer, and save the result in a reflection `Value` called `v`: v := p.Elem() fmt.Println("settability of v:", v.CanSet()) Now `v` is a settable reflection object, as the output demonstrates, settability of v: true and since it represents `x`, we are finally able to use `v.SetFloat` to modify the value of `x`: v.SetFloat(7.1) fmt.Println(v.Interface()) fmt.Println(x) The output, as expected, is 7.1 7.1 Reflection can be hard to understand but it’s doing exactly what the language does, albeit through reflection `Types` and `Values` that can disguise what’s going on. Just keep in mind that reflection Values need the address of something in order to modify what they represent. Structs ------- In our previous example `v` wasn’t a pointer itself, it was just derived from one. A common way for this situation to arise is when using reflection to modify the fields of a structure. As long as we have the address of the structure, we can modify its fields. Here’s a simple example that analyzes a struct value, `t`. We create the reflection object with the address of the struct because we’ll want to modify it later. Then we set `typeOfT` to its type and iterate over the fields using straightforward method calls (see [package reflect](https://go.dev/pkg/reflect/) for details). Note that we extract the names of the fields from the struct type, but the fields themselves are regular `reflect.Value` objects. type T struct { A int B string } t := T{23, "skidoo"} s := reflect.ValueOf(&t).Elem() typeOfT := s.Type() for i := 0; i < s.NumField(); i++ { f := s.Field(i) fmt.Printf("%d: %s %s = %v\n", i, typeOfT.Field(i).Name, f.Type(), f.Interface()) } The output of this program is 0: A int = 23 1: B string = skidoo There’s one more point about settability introduced in passing here: the field names of `T` are upper case (exported) because only exported fields of a struct are settable. Because `s` contains a settable reflection object, we can modify the fields of the structure. s.Field(0).SetInt(77) s.Field(1).SetString("Sunset Strip") fmt.Println("t is now", t) And here’s the result: t is now {77 Sunset Strip} If we modified the program so that `s` was created from `t`, not `&t`, the calls to `SetInt` and `SetString` would fail as the fields of `t` would not be settable. Conclusion ---------- Here again are the laws of reflection: * Reflection goes from interface value to reflection object. * Reflection goes from reflection object to interface value. * To modify a reflection object, the value must be settable. Once you understand these laws reflection in Go becomes much easier to use, although it remains subtle. It’s a powerful tool that should be used with care and avoided unless strictly necessary. There’s plenty more to reflection that we haven’t covered — sending and receiving on channels, allocating memory, using slices and maps, calling methods and functions — but this post is long enough. We’ll cover some of those topics in a later article. **Next article:** [The Go image package](https://go.dev/blog/image) **Previous article:** [Two Go Talks: "Lexical Scanning in Go" and "Cuddle: an App Engine Demo"](https://go.dev/blog/sydney-gtug) **[Blog Index](https://go.dev/blog/all) ** go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Actuating Google Production: How Google’s Site Reliability Engineering Team Uses Go - The Go Programming Language 1. [Why Go](https://go.dev/solutions/) 2. [Using Go at Google](https://go.dev/solutions/google/) 3. [Actuating Google Production: How Google’s Site Reliability Engineering Team Uses Go](https://go.dev/solutions/google/sitereliability) Actuating Google Production: How Google’s Site Reliability Engineering Team Uses Go =================================================================================== Pierre Palatin, Site Reliability Engineer ![Google Site Reliability Engineering (SRE)](https://go.dev/images/go_sitereliability_case_study.png) Google runs a small number of very large services. Those services are powered by a global infrastructure covering everything a developer needs: storage systems, load balancers, network, logging, monitoring, and much more. Nevertheless, it is not a static system—it cannot be. Architecture evolves, new products and ideas are created, new versions must be rolled out, configs pushed, database schema updated, and more. We end up deploying changes to our systems dozens of times per second. Because of this scale and critical need for reliability, Google pioneered Site Reliability Engineering (SRE), a role that many other companies have since adopted. “SRE is what you get when you treat operations as if it’s a software problem. Our mission is to protect, provide for, and progress the software and systems behind all of Google’s public services with an ever-watchful eye on their availability, latency, performance, and capacity.” — [Site Reliability Engineering (SRE)](https://sre.google/) . “ Go promised a sweet spot between performance and readability that neither of the other languages \[Python and C++\] were able to offer. ” In 2013-2014, Google’s SRE team realized that our approach to production management was not cutting it anymore in many ways. We had advanced far beyond shell scripts, but our scale had so many moving pieces and complexities that a new approach was needed. We determined that we needed to move toward a declarative model of our production, called “Prodspec”, driving a dedicated control plane, called “Annealing”. When we started those projects, Go was just becoming a viable option for critical services at Google. Most engineers were more familiar with Python and C++, either of which would have been valid choices. Nevertheless, Go captured our interest. The appeal of novelty was certainly a factor of course. But, more importantly, Go promised a sweet spot between performance and readability that neither of the other languages were able to offer. We started a small experiment with Go for some initial parts of Annealing and Prodspec. As the projects progressed, those initial parts written in Go found themselves at the core. We were happy with Go—its simplicity grew on us, the performance was there, and concurrency primitives would have been hard to replace. “ Now the majority of Google production is managed and maintained by our systems written in Go. ” At no point was there ever a mandate or requirement to use Go, but we had no desire to return to Python or C++. Go grew organically in Annealing and Prodspec. It was the right choice, and thus is now our language of choice. Now the majority of Google production is managed and maintained by our systems written in Go. The power of having a simple language in those projects is hard to overstate. There have been cases where some feature was indeed missing, such as the ability to enforce in the code that some complex structure should not be mutated. But for each one of those cases, there have undoubtedly been tens or hundred of cases where the simplicity helped. “ Go’s simplicity means that the code is easy to follow, whether it is to spot bugs during review or when trying to determine exactly what happened during a service disruption. ” For example, Annealing impacts a wide variety of teams and services meaning that we relied heavily on contributions across the company. The simplicity of Go made it possible for people outside our team to see why some part or another was not working for them, and often provide fixes or features themselves. This allowed us to quickly grow. Prodspec and Annealing are in charge of some quite critical components. Go’s simplicity means that the code is easy to follow, whether it is to spot bugs during review or when trying to determine exactly what happened during a service disruption. Go performance and concurrency support have also been key for our work. As our model of production is declarative, we tend to manipulate a lot of structured data, which describes what production is and what it should be. We have large services so the data can grow large, often making purely sequential processing not efficient enough. We are manipulating this data in many ways and many places. It is not a matter of having a smart person come up with a parallel version of our algorithm. It is a matter of casual parallelism, finding the next bottleneck and parallelising that code section. And Go enables exactly that. As a result of our success with Go, we now use Go for every new development for Prodspec and Annealing. In addition to the Site Reliability Engineering team, engineering teams across Google have adopted Go in their development process. Read about how the [Core Data Solutions](https://go.dev/solutions/google/coredata/) , [Firebase Hosting](https://go.dev/solutions/google/firebase/) , and [Chrome](https://go.dev/solutions/google/chrome/) teams use Go to build fast, reliable, and efficient software at scale. ![Google Site Reliability Engineering (SRE)](https://go.dev/images/logos/sitereliability.svg) ![Google Site Reliability Engineering (SRE)](https://go.dev/images/logos/sitereliability.svg) ### About Google Site Reliability Engineering (SRE) Google’s Site Reliability Engineering team has a mission to protect, provide for, and progress the software and systems behind all of Google’s public services — Google Search, Ads, Gmail, Android, YouTube, and App Engine, to name just a few — with an ever-watchful eye on their availability, latency, performance, and capacity. They shared their experience building core production management systems with Go, coming from experience with Python and C++. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Publishing a module - The Go Programming Language Publishing a module =================== When you want to make a module available for other developers, you publish it so that it’s visible to Go tools. Once you’ve published the module, developers importing its packages will be able to resolve a dependency on the module by running commands such as `go get`. > **Note:** Don’t change a tagged version of a module after publishing it. For developers using the module, Go tools authenticate a downloaded module against the first downloaded copy. If the two differ, Go tools will return a security error. Instead of changing the code for a previously published version, publish a new version. **See also** * For an overview of module development, see [Developing and publishing modules](https://go.dev/doc/modules/developing) * For a high-level module development workflow – which includes publishing – see [Module release and versioning workflow](https://go.dev/doc/modules/release-workflow) . Publishing steps ---------------- Use the following steps to publish a module. 1. Open a command prompt and change to your module’s root directory in the local repository. 2. Run `go mod tidy`, which removes any dependencies the module might have accumulated that are no longer necessary. $ go mod tidy 3. Run `go test ./...` a final time to make sure everything is working. This runs the unit tests you’ve written to use the Go testing framework. $ go test ./... ok example.com/mymodule 0.015s 4. Tag the project with a new version number using the `git tag` command. For the version number, use a number that signals to users the nature of changes in this release. For more, see [Module version numbering](https://go.dev/doc/modules/version-numbers) . $ git commit -m "mymodule: changes for v0.1.0" $ git tag v0.1.0 5. Push the new tag to the origin repository. $ git push origin v0.1.0 6. Make the module available by running the [`go list` command](https://go.dev/cmd/go/#hdr-List_packages_or_modules) to prompt Go to update its index of modules with information about the module you’re publishing. Precede the command with a statement to set the `GOPROXY` environment variable to a Go proxy. This will ensure that your request reaches the proxy. $ GOPROXY=proxy.golang.org go list -m example.com/mymodule@v0.1.0 Developers interested in your module import a package from it and run the [`go get` command](https://go.dev/doc/modules/publishing) just as they would with any other module. They can run the [`go get` command](https://go.dev/doc/modules/publishing) for latest versions or they can specify a particular version, as in the following example: $ go get example.com/mymodule@v0.1.0 go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # JSON and Go - The Go Programming Language [The Go Blog](https://go.dev/blog/) ==================================== JSON and Go =========== Andrew Gerrand 25 January 2011 Introduction ------------ JSON (JavaScript Object Notation) is a simple data interchange format. Syntactically it resembles the objects and lists of JavaScript. It is most commonly used for communication between web back-ends and JavaScript programs running in the browser, but it is used in many other places, too. Its home page, [json.org](http://json.org/) , provides a wonderfully clear and concise definition of the standard. With the [json package](https://go.dev/pkg/encoding/json/) it’s a snap to read and write JSON data from your Go programs. Encoding -------- To encode JSON data we use the [`Marshal`](https://go.dev/pkg/encoding/json/#Marshal) function. func Marshal(v interface{}) ([]byte, error) Given the Go data structure, `Message`, type Message struct { Name string Body string Time int64 } and an instance of `Message` m := Message{"Alice", "Hello", 1294706395881547000} we can marshal a JSON-encoded version of m using `json.Marshal`: b, err := json.Marshal(m) If all is well, `err` will be `nil` and `b` will be a `[]byte` containing this JSON data: b == []byte(`{"Name":"Alice","Body":"Hello","Time":1294706395881547000}`) Only data structures that can be represented as valid JSON will be encoded: * JSON objects only support strings as keys; to encode a Go map type it must be of the form `map[string]T` (where `T` is any Go type supported by the json package). * Channel, complex, and function types cannot be encoded. * Cyclic data structures are not supported; they will cause `Marshal` to go into an infinite loop. * Pointers will be encoded as the values they point to (or ’null’ if the pointer is `nil`). The json package only accesses the exported fields of struct types (those that begin with an uppercase letter). Therefore only the exported fields of a struct will be present in the JSON output. Decoding -------- To decode JSON data we use the [`Unmarshal`](https://go.dev/pkg/encoding/json/#Unmarshal) function. func Unmarshal(data []byte, v interface{}) error We must first create a place where the decoded data will be stored var m Message and call `json.Unmarshal`, passing it a `[]byte` of JSON data and a pointer to `m` err := json.Unmarshal(b, &m) If `b` contains valid JSON that fits in `m`, after the call `err` will be `nil` and the data from `b` will have been stored in the struct `m`, as if by an assignment like: m = Message{ Name: "Alice", Body: "Hello", Time: 1294706395881547000, } How does `Unmarshal` identify the fields in which to store the decoded data? For a given JSON key `"Foo"`, `Unmarshal` will look through the destination struct’s fields to find (in order of preference): * An exported field with a tag of `"Foo"` (see the [Go spec](https://go.dev/ref/spec#Struct_types) for more on struct tags), * An exported field named `"Foo"`, or * An exported field named `"FOO"` or `"FoO"` or some other case-insensitive match of `"Foo"`. What happens when the structure of the JSON data doesn’t exactly match the Go type? b := []byte(`{"Name":"Bob","Food":"Pickle"}`) var m Message err := json.Unmarshal(b, &m) `Unmarshal` will decode only the fields that it can find in the destination type. In this case, only the Name field of m will be populated, and the Food field will be ignored. This behavior is particularly useful when you wish to pick only a few specific fields out of a large JSON blob. It also means that any unexported fields in the destination struct will be unaffected by `Unmarshal`. But what if you don’t know the structure of your JSON data beforehand? Generic JSON with interface --------------------------- The `interface{}` (empty interface) type describes an interface with zero methods. Every Go type implements at least zero methods and therefore satisfies the empty interface. The empty interface serves as a general container type: var i interface{} i = "a string" i = 2011 i = 2.777 A type assertion accesses the underlying concrete type: r := i.(float64) fmt.Println("the circle's area", math.Pi*r*r) Or, if the underlying type is unknown, a type switch determines the type: switch v := i.(type) { case int: fmt.Println("twice i is", v*2) case float64: fmt.Println("the reciprocal of i is", 1/v) case string: h := len(v) / 2 fmt.Println("i swapped by halves is", v[h:]+v[:h]) default: // i isn't one of the types above } The json package uses `map[string]interface{}` and `[]interface{}` values to store arbitrary JSON objects and arrays; it will happily unmarshal any valid JSON blob into a plain `interface{}` value. The default concrete Go types are: * `bool` for JSON booleans, * `float64` for JSON numbers, * `string` for JSON strings, and * `nil` for JSON null. Decoding arbitrary data ----------------------- Consider this JSON data, stored in the variable `b`: b := []byte(`{"Name":"Wednesday","Age":6,"Parents":["Gomez","Morticia"]}`) Without knowing this data’s structure, we can decode it into an `interface{}` value with `Unmarshal`: var f interface{} err := json.Unmarshal(b, &f) At this point the Go value in `f` would be a map whose keys are strings and whose values are themselves stored as empty interface values: f = map[string]interface{}{ "Name": "Wednesday", "Age": 6, "Parents": []interface{}{ "Gomez", "Morticia", }, } To access this data we can use a type assertion to access `f`’s underlying `map[string]interface{}`: m := f.(map[string]interface{}) We can then iterate through the map with a range statement and use a type switch to access its values as their concrete types: for k, v := range m { switch vv := v.(type) { case string: fmt.Println(k, "is string", vv) case float64: fmt.Println(k, "is float64", vv) case []interface{}: fmt.Println(k, "is an array:") for i, u := range vv { fmt.Println(i, u) } default: fmt.Println(k, "is of a type I don't know how to handle") } } In this way you can work with unknown JSON data while still enjoying the benefits of type safety. Reference Types --------------- Let’s define a Go type to contain the data from the previous example: type FamilyMember struct { Name string Age int Parents []string } var m FamilyMember err := json.Unmarshal(b, &m) Unmarshaling that data into a `FamilyMember` value works as expected, but if we look closely we can see a remarkable thing has happened. With the var statement we allocated a `FamilyMember` struct, and then provided a pointer to that value to `Unmarshal`, but at that time the `Parents` field was a `nil` slice value. To populate the `Parents` field, `Unmarshal` allocated a new slice behind the scenes. This is typical of how `Unmarshal` works with the supported reference types (pointers, slices, and maps). Consider unmarshaling into this data structure: type Foo struct { Bar *Bar } If there were a `Bar` field in the JSON object, `Unmarshal` would allocate a new `Bar` and populate it. If not, `Bar` would be left as a `nil` pointer. From this a useful pattern arises: if you have an application that receives a few distinct message types, you might define “receiver” structure like type IncomingMessage struct { Cmd *Command Msg *Message } and the sending party can populate the `Cmd` field and/or the `Msg` field of the top-level JSON object, depending on the type of message they want to communicate. `Unmarshal`, when decoding the JSON into an `IncomingMessage` struct, will only allocate the data structures present in the JSON data. To know which messages to process, the programmer need simply test that either `Cmd` or `Msg` is not `nil`. Streaming Encoders and Decoders ------------------------------- The json package provides `Decoder` and `Encoder` types to support the common operation of reading and writing streams of JSON data. The `NewDecoder` and `NewEncoder` functions wrap the [`io.Reader`](https://go.dev/pkg/io/#Reader) and [`io.Writer`](https://go.dev/pkg/io/#Writer) interface types. func NewDecoder(r io.Reader) *Decoder func NewEncoder(w io.Writer) *Encoder Here’s an example program that reads a series of JSON objects from standard input, removes all but the `Name` field from each object, and then writes the objects to standard output: package main import ( "encoding/json" "log" "os" ) func main() { dec := json.NewDecoder(os.Stdin) enc := json.NewEncoder(os.Stdout) for { var v map[string]interface{} if err := dec.Decode(&v); err != nil { log.Println(err) return } for k := range v { if k != "Name" { delete(v, k) } } if err := enc.Encode(&v); err != nil { log.Println(err) } } } Due to the ubiquity of Readers and Writers, these `Encoder` and `Decoder` types can be used in a broad range of scenarios, such as reading and writing to HTTP connections, WebSockets, or files. References ---------- For more information see the [json package documentation](https://go.dev/pkg/encoding/json/) . For an example usage of json see the source files of the [jsonrpc package](https://go.dev/pkg/net/rpc/jsonrpc/) . **Next article:** [Go becomes more stable](https://go.dev/blog/stable-releases) **Previous article:** [Go Slices: usage and internals](https://go.dev/blog/slices-intro) **[Blog Index](https://go.dev/blog/all) ** go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # How the Firebase Hosting Team Scaled With Go - The Go Programming Language 1. [Why Go](https://go.dev/solutions/) 2. [Using Go at Google](https://go.dev/solutions/google/) 3. [How the Firebase Hosting Team Scaled With Go](https://go.dev/solutions/google/firebase) How the Firebase Hosting Team Scaled With Go ============================================ ![Firebase](https://go.dev/images/go_firebase_case_study.png) The Firebase Hosting team provides static web hosting services for Google Cloud customers. They provide a static web host that sits behind a global content delivery network, and offer users tools that are easy to use. The team also develops features that range from uploading site files to registering domains to tracking usage. Before joining Google, Firebase Hosting’s tech stack was written in Node.js. The team started to use Go when they needed to interoperate with several other Google services. They decided to use Go to help them scale easily and efficiently, knowing that “concurrency would continue to be a big need.” They “were confident Go would be more performant,” and “liked that Go is more terse” than other languages they were considering, said Michael Bleigh, a software engineer on the team. Starting with one small service written in Go, the team migrated their entire backend in a series of moves. The team progressively identified large features they wanted to implement and, in the process, rewrote them in Go and moved to Google Cloud and Google’s internal cluster management system. **Now the Firebase Hosting team has replaced 100% of backend Node.js code with Go.** The team’s experience writing in Go began with one engineer. “Through peer-to-peer learning and Go being generally easy to get started with, everyone on the team now has Go dev experience,” said Bleigh. They’ve found that while a majority of people who are new to the team haven’t had any experience with Go, “most of them are productive within a couple weeks.” “Using Go, it’s easy to see how the code is organized and what the code does,” said Bleigh, speaking for the team. “Go is generally very readable and understandable. The language’s error handling, receivers, and interfaces are all easy to understand due to the idioms in the language.” Concurrency continues to be a focus for the team as they scale. Robert Rossney, a software engineer, shared that “Go makes it very easy to put all of the hard concurrency stuff in one place, and everywhere else it’s abstracted.” Rossney also spoke to the benefits of using a language built with concurrency in mind, saying that “there are also a lot of ways to do concurrency in Go. We’ve had to learn when each route is best, how to determine when a problem is a concurrency problem, how to debug–but that comes out of the fact that you actually can write these patterns in Go code.” “ Generally speaking, there’s not a time on the team where we’re feeling frustrated with Go, it just kind of gets out of the way and lets you do work. ” — Robert Rossney ,  Software Engineer Hundreds of thousands of customers host their websites with Firebase Hosting, which means Go code is used to serve billions of requests per day. “Our customer base and traffic have doubled multiple times since migrating to Go without ever requiring fine-tuned optimizations” shared Bleigh. With Go, the team has seen performance improvements both in the software and on the team, with excellent productivity gains. “Generally speaking,” Rossney mentioned, “…there’s not a time on the team where we’re feeling frustrated with Go, it just kind of gets out of the way and lets you do work.” In addition to the Firebase Hosting team, engineering teams across Google have adopted Go in their development process. Read about how the [Core Data Solutions](https://go.dev/solutions/google/coredata/) and [Chrome](https://go.dev/solutions/google/chrome/) teams use Go to build fast, reliable, and efficient software at scale. ![Firebase](https://go.dev/images/logos/firebase.svg) ![Firebase](https://go.dev/images/logos/firebase.svg) ### About Firebase Firebase is Google’s mobile platform that helps you quickly develop high-quality apps and grow your business. The Firebase Hosting team shared their journey with Go, including their backend migration from Node.js, the ease of onboarding new Go developers, and how Go has helped them scale. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # A Quick Guide to Go's Assembler - The Go Programming Language A Quick Guide to Go's Assembler =============================== A Quick Guide to Go's Assembler ------------------------------- This document is a quick outline of the unusual form of assembly language used by the `gc` Go compiler. The document is not comprehensive. The assembler is based on the input style of the Plan 9 assemblers, which is documented in detail [elsewhere](https://9p.io/sys/doc/asm.html) . If you plan to write assembly language, you should read that document although much of it is Plan 9-specific. The current document provides a summary of the syntax and the differences with what is explained in that document, and describes the peculiarities that apply when writing assembly code to interact with Go. The most important thing to know about Go's assembler is that it is not a direct representation of the underlying machine. Some of the details map precisely to the machine, but some do not. This is because the compiler suite (see [this description](https://9p.io/sys/doc/compiler.html) ) needs no assembler pass in the usual pipeline. Instead, the compiler operates on a kind of semi-abstract instruction set, and instruction selection occurs partly after code generation. The assembler works on the semi-abstract form, so when you see an instruction like `MOV` what the toolchain actually generates for that operation might not be a move instruction at all, perhaps a clear or load. Or it might correspond exactly to the machine instruction with that name. In general, machine-specific operations tend to appear as themselves, while more general concepts like memory move and subroutine call and return are more abstract. The details vary with architecture, and we apologize for the imprecision; the situation is not well-defined. The assembler program is a way to parse a description of that semi-abstract instruction set and turn it into instructions to be input to the linker. If you want to see what the instructions look like in assembly for a given architecture, say amd64, there are many examples in the sources of the standard library, in packages such as [`runtime`](https://go.dev/pkg/runtime/) and [`math/big`](https://go.dev/pkg/math/big/) . You can also examine what the compiler emits as assembly code (the actual output may differ from what you see here): $ cat x.go package main func main() { println(3) } $ GOOS=linux GOARCH=amd64 go tool compile -S x.go # or: go build -gcflags -S x.go "".main STEXT size=74 args=0x0 locals=0x10 0x0000 00000 (x.go:3) TEXT "".main(SB), $16-0 0x0000 00000 (x.go:3) MOVQ (TLS), CX 0x0009 00009 (x.go:3) CMPQ SP, 16(CX) 0x000d 00013 (x.go:3) JLS 67 0x000f 00015 (x.go:3) SUBQ $16, SP 0x0013 00019 (x.go:3) MOVQ BP, 8(SP) 0x0018 00024 (x.go:3) LEAQ 8(SP), BP 0x001d 00029 (x.go:3) FUNCDATA $0, gclocals·33cdeccccebe80329f1fdbee7f5874cb(SB) 0x001d 00029 (x.go:3) FUNCDATA $1, gclocals·33cdeccccebe80329f1fdbee7f5874cb(SB) 0x001d 00029 (x.go:3) FUNCDATA $2, gclocals·33cdeccccebe80329f1fdbee7f5874cb(SB) 0x001d 00029 (x.go:4) PCDATA $0, $0 0x001d 00029 (x.go:4) PCDATA $1, $0 0x001d 00029 (x.go:4) CALL runtime.printlock(SB) 0x0022 00034 (x.go:4) MOVQ $3, (SP) 0x002a 00042 (x.go:4) CALL runtime.printint(SB) 0x002f 00047 (x.go:4) CALL runtime.printnl(SB) 0x0034 00052 (x.go:4) CALL runtime.printunlock(SB) 0x0039 00057 (x.go:5) MOVQ 8(SP), BP 0x003e 00062 (x.go:5) ADDQ $16, SP 0x0042 00066 (x.go:5) RET 0x0043 00067 (x.go:5) NOP 0x0043 00067 (x.go:3) PCDATA $1, $-1 0x0043 00067 (x.go:3) PCDATA $0, $-1 0x0043 00067 (x.go:3) CALL runtime.morestack\_noctxt(SB) 0x0048 00072 (x.go:3) JMP 0 ... The `FUNCDATA` and `PCDATA` directives contain information for use by the garbage collector; they are introduced by the compiler. To see what gets put in the binary after linking, use `go tool objdump`: $ go build -o x.exe x.go $ go tool objdump -s main.main x.exe TEXT main.main(SB) /tmp/x.go x.go:3 0x10501c0 65488b0c2530000000 MOVQ GS:0x30, CX x.go:3 0x10501c9 483b6110 CMPQ 0x10(CX), SP x.go:3 0x10501cd 7634 JBE 0x1050203 x.go:3 0x10501cf 4883ec10 SUBQ $0x10, SP x.go:3 0x10501d3 48896c2408 MOVQ BP, 0x8(SP) x.go:3 0x10501d8 488d6c2408 LEAQ 0x8(SP), BP x.go:4 0x10501dd e86e45fdff CALL runtime.printlock(SB) x.go:4 0x10501e2 48c7042403000000 MOVQ $0x3, 0(SP) x.go:4 0x10501ea e8e14cfdff CALL runtime.printint(SB) x.go:4 0x10501ef e8ec47fdff CALL runtime.printnl(SB) x.go:4 0x10501f4 e8d745fdff CALL runtime.printunlock(SB) x.go:5 0x10501f9 488b6c2408 MOVQ 0x8(SP), BP x.go:5 0x10501fe 4883c410 ADDQ $0x10, SP x.go:5 0x1050202 c3 RET x.go:3 0x1050203 e83882ffff CALL runtime.morestack\_noctxt(SB) x.go:3 0x1050208 ebb6 JMP main.main(SB) ### Constants Although the assembler takes its guidance from the Plan 9 assemblers, it is a distinct program, so there are some differences. One is in constant evaluation. Constant expressions in the assembler are parsed using Go's operator precedence, not the C-like precedence of the original. Thus `3&1<<2` is 4, not 0—it parses as `(3&1)<<2` not `3&(1<<2)`. Also, constants are always evaluated as 64-bit unsigned integers. Thus `-2` is not the integer value minus two, but the unsigned 64-bit integer with the same bit pattern. The distinction rarely matters but to avoid ambiguity, division or right shift where the right operand's high bit is set is rejected. ### Symbols Some symbols, such as `R1` or `LR`, are predefined and refer to registers. The exact set depends on the architecture. There are four predeclared symbols that refer to pseudo-registers. These are not real registers, but rather virtual registers maintained by the toolchain, such as a frame pointer. The set of pseudo-registers is the same for all architectures: * `FP`: Frame pointer: arguments and locals. * `PC`: Program counter: jumps and branches. * `SB`: Static base pointer: global symbols. * `SP`: Stack pointer: the highest address within the local stack frame. All user-defined symbols are written as offsets to the pseudo-registers `FP` (arguments and locals) and `SB` (globals). The `SB` pseudo-register can be thought of as the origin of memory, so the symbol `foo(SB)` is the name `foo` as an address in memory. This form is used to name global functions and data. Adding `<>` to the name, as in `foo<>(SB)`, makes the name visible only in the current source file, like a top-level `static` declaration in a C file. Adding an offset to the name refers to that offset from the symbol's address, so `foo+4(SB)` is four bytes past the start of `foo`. The `FP` pseudo-register is a virtual frame pointer used to refer to function arguments. The compilers maintain a virtual frame pointer and refer to the arguments on the stack as offsets from that pseudo-register. Thus `0(FP)` is the first argument to the function, `8(FP)` is the second (on a 64-bit machine), and so on. However, when referring to a function argument this way, it is necessary to place a name at the beginning, as in `first_arg+0(FP)` and `second_arg+8(FP)`. (The meaning of the offset—offset from the frame pointer—distinct from its use with `SB`, where it is an offset from the symbol.) The assembler enforces this convention, rejecting plain `0(FP)` and `8(FP)`. The actual name is semantically irrelevant but should be used to document the argument's name. It is worth stressing that `FP` is always a pseudo-register, not a hardware register, even on architectures with a hardware frame pointer. For assembly functions with Go prototypes, `go` `vet` will check that the argument names and offsets match. On 32-bit systems, the low and high 32 bits of a 64-bit value are distinguished by adding a `_lo` or `_hi` suffix to the name, as in `arg_lo+0(FP)` or `arg_hi+4(FP)`. If a Go prototype does not name its result, the expected assembly name is `ret`. The `SP` pseudo-register is a virtual stack pointer used to refer to frame-local variables and the arguments being prepared for function calls. It points to the highest address within the local stack frame, so references should use negative offsets in the range \[−framesize, 0): `x-8(SP)`, `y-4(SP)`, and so on.\ \ On architectures with a hardware register named `SP`, the name prefix distinguishes references to the virtual stack pointer from references to the architectural `SP` register. That is, `x-8(SP)` and `-8(SP)` are different memory locations: the first refers to the virtual stack pointer pseudo-register, while the second refers to the hardware's `SP` register.\ \ On machines where `SP` and `PC` are traditionally aliases for a physical, numbered register, in the Go assembler the names `SP` and `PC` are still treated specially; for instance, references to `SP` require a symbol, much like `FP`. To access the actual hardware register use the true `R` name. For example, on the ARM architecture the hardware `SP` and `PC` are accessible as `R13` and `R15`.\ \ Branches and direct jumps are always written as offsets to the PC, or as jumps to labels:\ \ label:\ MOVW $0, R1\ JMP label\ \ Each label is visible only within the function in which it is defined. It is therefore permitted for multiple functions in a file to define and use the same label names. Direct jumps and call instructions can target text symbols, such as `name(SB)`, but not offsets from symbols, such as `name+4(SB)`.\ \ Instructions, registers, and assembler directives are always in UPPER CASE to remind you that assembly programming is a fraught endeavor. (Exception: the `g` register renaming on ARM.)\ \ In Go object files and binaries, the full name of a symbol is the package path followed by a period and the symbol name: `fmt.Printf` or `math/rand.Int`. Because the assembler's parser treats period and slash as punctuation, those strings cannot be used directly as identifier names. Instead, the assembler allows the middle dot character U+00B7 and the division slash U+2215 in identifiers and rewrites them to plain period and slash. Within an assembler source file, the symbols above are written as `fmt·Printf` and `math∕rand·Int`. The assembly listings generated by the compilers when using the `-S` flag show the period and slash directly instead of the Unicode replacements required by the assemblers.\ \ Most hand-written assembly files do not include the full package path in symbol names, because the linker inserts the package path of the current object file at the beginning of any name starting with a period: in an assembly source file within the math/rand package implementation, the package's Int function can be referred to as `·Int`. This convention avoids the need to hard-code a package's import path in its own source code, making it easier to move the code from one location to another.\ \ ### Directives\ \ The assembler uses various directives to bind text and data to symbol names. For example, here is a simple complete function definition. The `TEXT` directive declares the symbol `runtime·profileloop` and the instructions that follow form the body of the function. The last instruction in a `TEXT` block must be some sort of jump, usually a `RET` (pseudo-)instruction. (If it's not, the linker will append a jump-to-itself instruction; there is no fallthrough in `TEXTs`.) After the symbol, the arguments are flags (see below) and the frame size, a constant (but see below):\ \ TEXT runtime·profileloop(SB),NOSPLIT,$8\ MOVQ $runtime·profileloop1(SB), CX\ MOVQ CX, 0(SP)\ CALL runtime·externalthreadhandler(SB)\ RET\ \ In the general case, the frame size is followed by an argument size, separated by a minus sign. (It's not a subtraction, just idiosyncratic syntax.) The frame size `$24-8` states that the function has a 24-byte frame and is called with 8 bytes of argument, which live on the caller's frame. If `NOSPLIT` is not specified for the `TEXT`, the argument size must be provided. For assembly functions with Go prototypes, `go` `vet` will check that the argument size is correct.\ \ Note that the symbol name uses a middle dot to separate the components and is specified as an offset from the static base pseudo-register `SB`. This function would be called from Go source for package `runtime` using the simple name `profileloop`.\ \ Global data symbols are defined by a sequence of initializing `DATA` directives followed by a `GLOBL` directive. Each `DATA` directive initializes a section of the corresponding memory. The memory not explicitly initialized is zeroed. The general form of the `DATA` directive is\ \ DATA symbol+offset(SB)/width, value\ \ which initializes the symbol memory at the given offset and width with the given value. The `DATA` directives for a given symbol must be written with increasing offsets.\ \ The `GLOBL` directive declares a symbol to be global. The arguments are optional flags and the size of the data being declared as a global, which will have initial value all zeros unless a `DATA` directive has initialized it. The `GLOBL` directive must follow any corresponding `DATA` directives.\ \ For example,\ \ DATA divtab<>+0x00(SB)/4, $0xf4f8fcff\ DATA divtab<>+0x04(SB)/4, $0xe6eaedf0\ ...\ DATA divtab<>+0x3c(SB)/4, $0x81828384\ GLOBL divtab<>(SB), RODATA, $64\ \ GLOBL runtime·tlsoffset(SB), NOPTR, $4\ \ declares and initializes `divtab<>`, a read-only 64-byte table of 4-byte integer values, and declares `runtime·tlsoffset`, a 4-byte, implicitly zeroed variable that contains no pointers.\ \ There may be one or two arguments to the directives. If there are two, the first is a bit mask of flags, which can be written as numeric expressions, added or or-ed together, or can be set symbolically for easier absorption by a human. Their values, defined in the standard `#include` file `textflag.h`, are:\ \ * `NOPROF` = 1 \ (For `TEXT` items.) Don't profile the marked function. This flag is deprecated.\ * `DUPOK` = 2 \ It is legal to have multiple instances of this symbol in a single binary. The linker will choose one of the duplicates to use.\ * `NOSPLIT` = 4 \ (For `TEXT` items.) Don't insert the preamble to check if the stack must be split. The frame for the routine, plus anything it calls, must fit in the spare space remaining in the current stack segment. Used to protect routines such as the stack splitting code itself.\ * `RODATA` = 8 \ (For `DATA` and `GLOBL` items.) Put this data in a read-only section.\ * `NOPTR` = 16 \ (For `DATA` and `GLOBL` items.) This data contains no pointers and therefore does not need to be scanned by the garbage collector.\ * `WRAPPER` = 32 \ (For `TEXT` items.) This is a wrapper function and should not count as disabling `recover`.\ * `NEEDCTXT` = 64 \ (For `TEXT` items.) This function is a closure so it uses its incoming context register.\ * `LOCAL` = 128 \ This symbol is local to the dynamic shared object.\ * `TLSBSS` = 256 \ (For `DATA` and `GLOBL` items.) Put this data in thread local storage.\ * `NOFRAME` = 512 \ (For `TEXT` items.) Do not insert instructions to allocate a stack frame and save/restore the return address, even if this is not a leaf function. Only valid on functions that declare a frame size of 0.\ * `TOPFRAME` = 2048 \ (For `TEXT` items.) Function is the outermost frame of the call stack. Traceback should stop at this function.\ \ ### Special instructions\ \ The `PCALIGN` pseudo-instruction is used to indicate that the next instruction should be aligned to a specified boundary by padding with no-op instructions.\ \ It is currently supported on arm64, amd64, ppc64, loong64 and riscv64. For example, the start of the `MOVD` instruction below is aligned to 32 bytes:\ \ PCALIGN $32\ MOVD $2, R0\ \ ### Interacting with Go types and constants\ \ If a package has any .s files, then `go build` will direct the compiler to emit a special header called `go_asm.h`, which the .s files can then `#include`. The file contains symbolic `#define` constants for the offsets of Go struct fields, the sizes of Go struct types, and most Go `const` declarations defined in the current package. Go assembly should avoid making assumptions about the layout of Go types and instead use these constants. This improves the readability of assembly code, and keeps it robust to changes in data layout either in the Go type definitions or in the layout rules used by the Go compiler.\ \ Constants are of the form `const__name_`. For example, given the Go declaration `const bufSize = 1024`, assembly code can refer to the value of this constant as `const_bufSize`.\ \ Field offsets are of the form `_type___field_`. Struct sizes are of the form `_type___size`. For example, consider the following Go definition:\ \ type reader struct {\ buf \[bufSize\]byte\ r int\ }\ \ Assembly can refer to the size of this struct as `reader__size` and the offsets of the two fields as `reader_buf` and `reader_r`. Hence, if register `R1` contains a pointer to a `reader`, assembly can reference the `r` field as `reader_r(R1)`.\ \ If any of these `#define` names are ambiguous (for example, a struct with a `_size` field), `#include "go_asm.h"` will fail with a "redefinition of macro" error.\ \ ### Runtime Coordination\ \ For garbage collection to run correctly, the runtime must know the location of pointers in all global data and in most stack frames. The Go compiler emits this information when compiling Go source files, but assembly programs must define it explicitly.\ \ A data symbol marked with the `NOPTR` flag (see above) is treated as containing no pointers to runtime-allocated data. A data symbol with the `RODATA` flag is allocated in read-only memory and is therefore treated as implicitly marked `NOPTR`. A data symbol with a total size smaller than a pointer is also treated as implicitly marked `NOPTR`. It is not possible to define a symbol containing pointers in an assembly source file; such a symbol must be defined in a Go source file instead. Assembly source can still refer to the symbol by name even without `DATA` and `GLOBL` directives. A good general rule of thumb is to define all non-`RODATA` symbols in Go instead of in assembly.\ \ Each function also needs annotations giving the location of live pointers in its arguments, results, and local stack frame. For an assembly function with no pointer results and either no local stack frame or no function calls, the only requirement is to define a Go prototype for the function in a Go source file in the same package. The name of the assembly function must not contain the package name component (for example, function `Syscall` in package `syscall` should use the name `·Syscall` instead of the equivalent name `syscall·Syscall` in its `TEXT` directive). For more complex situations, explicit annotation is needed. These annotations use pseudo-instructions defined in the standard `#include` file `funcdata.h`.\ \ If a function has no arguments and no results, the pointer information can be omitted. This is indicated by an argument size annotation of `$_n_-0` on the `TEXT` instruction. Otherwise, pointer information must be provided by a Go prototype for the function in a Go source file, even for assembly functions not called directly from Go. (The prototype will also let `go` `vet` check the argument references.) At the start of the function, the arguments are assumed to be initialized but the results are assumed uninitialized. If the results will hold live pointers during a call instruction, the function should start by zeroing the results and then executing the pseudo-instruction `GO_RESULTS_INITIALIZED`. This instruction records that the results are now initialized and should be scanned during stack movement and garbage collection. It is typically easier to arrange that assembly functions do not return pointers or do not contain call instructions; no assembly functions in the standard library use `GO_RESULTS_INITIALIZED`.\ \ If a function has no local stack frame, the pointer information can be omitted. This is indicated by a local frame size annotation of `$0-_n_` on the `TEXT` instruction. The pointer information can also be omitted if the function contains no call instructions. Otherwise, the local stack frame must not contain pointers, and the assembly must confirm this fact by executing the pseudo-instruction `NO_LOCAL_POINTERS`. Because stack resizing is implemented by moving the stack, the stack pointer may change during any function call: even pointers to stack data must not be kept in local variables.\ \ Assembly functions should always be given Go prototypes, both to provide pointer information for the arguments and results and to let `go` `vet` check that the offsets being used to access them are correct.\ \ Architecture-specific details\ -----------------------------\ \ It is impractical to list all the instructions and other details for each machine. To see what instructions are defined for a given machine, say ARM, look in the source for the `obj` support library for that architecture, located in the directory `src/cmd/internal/obj/arm`. In that directory is a file `a.out.go`; it contains a long list of constants starting with `A`, like this:\ \ const (\ AAND = obj.ABaseARM + obj.A\_ARCHSPECIFIC + iota\ AEOR\ ASUB\ ARSB\ AADD\ ...\ \ This is the list of instructions and their spellings as known to the assembler and linker for that architecture. Each instruction begins with an initial capital `A` in this list, so `AAND` represents the bitwise and instruction, `AND` (without the leading `A`), and is written in assembly source as `AND`. The enumeration is mostly in alphabetical order. (The architecture-independent `AXXX`, defined in the `cmd/internal/obj` package, represents an invalid instruction). The sequence of the `A` names has nothing to do with the actual encoding of the machine instructions. The `cmd/internal/obj` package takes care of that detail.\ \ The instructions for both the 386 and AMD64 architectures are listed in `cmd/internal/obj/x86/a.out.go`.\ \ The architectures share syntax for common addressing modes such as `(R1)` (register indirect), `4(R1)` (register indirect with offset), and `$foo(SB)` (absolute address). The assembler also supports some (not necessarily all) addressing modes specific to each architecture. The sections below list these.\ \ One detail evident in the examples from the previous sections is that data in the instructions flows from left to right: `MOVQ` `$0,` `CX` clears `CX`. This rule applies even on architectures where the conventional notation uses the opposite direction.\ \ Here follow some descriptions of key Go-specific details for the supported architectures.\ \ ### 32-bit Intel 386\ \ The runtime pointer to the `g` structure is maintained through the value of an otherwise unused (as far as Go is concerned) register in the MMU. In the runtime package, assembly code can include `go_tls.h`, which defines an OS- and architecture-dependent macro `get_tls` for accessing this register. The `get_tls` macro takes one argument, which is the register to load the `g` pointer into.\ \ For example, the sequence to load `g` and `m` using `CX` looks like this:\ \ #include "go\_tls.h"\ #include "go\_asm.h"\ ...\ get\_tls(CX)\ MOVL g(CX), AX // Move g into AX.\ MOVL g\_m(AX), BX // Move g.m into BX.\ \ The `get_tls` macro is also defined on [amd64](https://go.dev/doc/asm#amd64)\ .\ \ Addressing modes:\ \ * `(DI)(BX*2)`: The location at address `DI` plus `BX*2`.\ * `64(DI)(BX*2)`: The location at address `DI` plus `BX*2` plus 64. These modes accept only 1, 2, 4, and 8 as scale factors.\ \ When using the compiler and assembler's `-dynlink` or `-shared` modes, any load or store of a fixed memory location such as a global variable must be assumed to overwrite `CX`. Therefore, to be safe for use with these modes, assembly sources should typically avoid CX except between memory references.\ \ ### 64-bit Intel 386 (a.k.a. amd64)\ \ The two architectures behave largely the same at the assembler level. Assembly code to access the `m` and `g` pointers on the 64-bit version is the same as on the 32-bit 386, except it uses `MOVQ` rather than `MOVL`:\ \ get\_tls(CX)\ MOVQ g(CX), AX // Move g into AX.\ MOVQ g\_m(AX), BX // Move g.m into BX.\ \ Register `BP` is callee-save. The assembler automatically inserts `BP` save/restore when frame size is larger than zero. Using `BP` as a general purpose register is allowed, however it can interfere with sampling-based profiling.\ \ ### ARM\ \ The registers `R10` and `R11` are reserved by the compiler and linker.\ \ `R10` points to the `g` (goroutine) structure. Within assembler source code, this pointer must be referred to as `g`; the name `R10` is not recognized.\ \ To make it easier for people and compilers to write assembly, the ARM linker allows general addressing forms and pseudo-operations like `DIV` or `MOD` that may not be expressible using a single hardware instruction. It implements these forms as multiple instructions, often using the `R11` register to hold temporary values. Hand-written assembly can use `R11`, but doing so requires being sure that the linker is not also using it to implement any of the other instructions in the function.\ \ When defining a `TEXT`, specifying frame size `$-4` tells the linker that this is a leaf function that does not need to save `LR` on entry.\ \ The name `SP` always refers to the virtual stack pointer described earlier. For the hardware register, use `R13`.\ \ Condition code syntax is to append a period and the one- or two-letter code to the instruction, as in `MOVW.EQ`. Multiple codes may be appended: `MOVM.IA.W`. The order of the code modifiers is irrelevant.\ \ Addressing modes:\ \ * `R0->16` \ `R0>>16` \ `R0<<16` \ `R0@>16`: For `<<`, left shift `R0` by 16 bits. The other codes are `->` (arithmetic right shift), `>>` (logical right shift), and `@>` (rotate right).\ * `R0->R1` \ `R0>>R1` \ `R0<R1`: For `<<`, left shift `R0` by the count in `R1`. The other codes are `->` (arithmetic right shift), `>>` (logical right shift), and `@>` (rotate right).\ * `[R0,g,R12-R15]`: For multi-register instructions, the set comprising `R0`, `g`, and `R12` through `R15` inclusive.\ * `(R5, R6)`: Destination register pair.\ \ ### ARM64\ \ `R18` is the "platform register", reserved on the Apple platform. To prevent accidental misuse, the register is named `R18_PLATFORM`. `R27` and `R28` are reserved by the compiler and linker. `R29` is the frame pointer. `R30` is the link register.\ \ Instruction modifiers are appended to the instruction following a period. The only modifiers are `P` (postincrement) and `W` (preincrement): `MOVW.P`, `MOVW.W`\ \ Addressing modes:\ \ * `R0->16` \ `R0>>16` \ `R0<<16` \ `R0@>16`: These are the same as on the 32-bit ARM.\ * `$(8<<12)`: Left shift the immediate value `8` by `12` bits.\ * `8(R0)`: Add the value of `R0` and `8`.\ * `(R2)(R0)`: The location at `R0` plus `R2`.\ * `R0.UXTB` \ `R0.UXTB<= len(funcs) { panic("invalid function index") } ... complex code ... // compiler must NOT reload i = \*p here funcs\[i\]() If the complex code needs many registers, a compiler for single-threaded programs could discard `i` without saving a copy and then reload `i = *p` just before `funcs[i]()`. A Go compiler must not, because the value of `*p` may have changed. (Instead, the compiler could spill `i` to the stack.) Not allowing a single write to write multiple values also means not using the memory where a local variable will be written as temporary storage before the write. For example, a compiler must not use `*p` as temporary storage in this program: \*p = i + \*p/2 That is, it must not rewrite the program into this one: \*p /= 2 \*p += i If `i` and `*p` start equal to 2, the original code does `*p = 3`, so a racing thread can read only 2 or 3 from `*p`. The rewritten code does `*p = 1` and then `*p = 3`, allowing a racing thread to read 1 as well. Note that all these optimizations are permitted in C/C++ compilers: a Go compiler sharing a back end with a C/C++ compiler must take care to disable optimizations that are invalid for Go. Note that the prohibition on introducing data races does not apply if the compiler can prove that the races do not affect correct execution on the target platform. For example, on essentially all CPUs, it is valid to rewrite n := 0 for i := 0; i < m; i++ { n += \*shared } into: n := 0 local := \*shared for i := 0; i < m; i++ { n += local } provided it can be proved that `*shared` will not fault on access, because the potential added read will not affect any existing concurrent reads or writes. On the other hand, the rewrite would not be valid in a source-to-source translator. Conclusion ---------- Go programmers writing data-race-free programs can rely on sequentially consistent execution of those programs, just as in essentially all other modern programming languages. When it comes to programs with races, both programmers and compilers should remember the advice: don't be clever. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Coverage profiling support for integration tests - The Go Programming Language Coverage profiling support for integration tests ================================================ Table of Contents: [Overview](https://go.dev/doc/build-cover#overview) [Building a binary for coverage profiling](https://go.dev/doc/build-cover#building) [Running a coverage-instrumented binary](https://go.dev/doc/build-cover#running) [Working with coverage data files](https://go.dev/doc/build-cover#working) [Frequently Asked Questions](https://go.dev/doc/build-cover#FAQ) [Resources](https://go.dev/doc/build-cover#resources) [Glossary](https://go.dev/doc/build-cover#glossary) Beginning in Go 1.20, Go supports collection of coverage profiles from applications and from integration tests, larger and more complex tests for Go programs. Overview ======== Go provides easy-to-use support for collecting coverage profiles at the level of package unit tests via the “`go test -coverprofile=... `” command. Starting with Go 1.20, users can now collect coverage profiles for larger [integration tests](https://go.dev/doc/build-cover#glos-integration-test) : more heavy-weight, complex tests that perform multiple runs of a given application binary. For unit tests, collecting a coverage profile and generating a report requires two steps: a `go test -coverprofile=...` run, followed by an invocation of `go tool cover {-func,-html}` to generate a report. For integration tests, three steps are needed: a [build](https://go.dev/doc/build-cover#building) step, a [run](https://go.dev/doc/build-cover#running) step (which may involve multiple invocations of the binary from the build step), and finally a [reporting](https://go.dev/doc/build-cover#reporting) step, as described below. Building a binary for coverage profiling ======================================== To build an application for collecting coverage profiles, pass the `-cover` flag when invoking `go build` on your application binary target. See the section [below](https://go.dev/doc/build-cover#packageselection) for a sample `go build -cover` invocation. The resulting binary can then be run using an environment variable setting to capture coverage profiles (see the next section on [running](https://go.dev/doc/build-cover#running) ). How packages are selected for instrumentation --------------------------------------------- During a given “`go build -cover`” invocation, the Go command will select packages in the main module for coverage profiling; other packages that feed into the build (dependencies listed in go.mod, or packages that are part of the Go standard library) will not be included by default. For example, here is a toy program containing a main package, a local main-module package `greetings` and a set of packages imported from outside the module, including (among others) `rsc.io/quote` and `fmt` ([link to full program](https://go.dev/play/p/VSQJN8xkkf-?v=gotip) ). $ cat go.mod module mydomain.com go 1.20 require rsc.io/quote v1.5.2 require ( golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c // indirect rsc.io/sampler v1.3.0 // indirect ) $ cat myprogram.go package main import ( "fmt" "mydomain.com/greetings" "rsc.io/quote" ) func main() { fmt.Printf("I say %q and %q\n", quote.Hello(), greetings.Goodbye()) } $ cat greetings/greetings.go package greetings func Goodbye() string { return "see ya" } $ go build -cover -o myprogram.exe . $ If you build this program with the “`-cover`” command line flag and run it, exactly two packages will be included in the profile: `main` and `mydomain.com/greetings`; the other dependent packages will be excluded. Users who want to have more control over which packages are included for coverage can build with the “`-coverpkg`” flag. Example: $ go build -cover -o myprogramMorePkgs.exe -coverpkg=io,mydomain.com,rsc.io/quote . $ In the build above, the main package from `mydomain.com` as well as the `rsc.io/quote` and `io` packages are selected for profiling; since `mydomain.com/greetings` isn’t specifically listed, it will be excluded from the profile, even though it resides in the main module. Running a coverage-instrumented binary ====================================== Binaries built with “`-cover`” write out profile data files at the end of their execution to a directory specified via the environment variable `GOCOVERDIR`. Example: $ go build -cover -o myprogram.exe myprogram.go $ mkdir somedata $ GOCOVERDIR=somedata ./myprogram.exe I say "Hello, world." and "see ya" $ ls somedata covcounters.c6de772f99010ef5925877a7b05db4cc.2424989.1670252383678349347 covmeta.c6de772f99010ef5925877a7b05db4cc $ Note the two files that were written to the directory `somedata`: these (binary) files contain the coverage results. See the following section on [reporting](https://go.dev/doc/build-cover#reporting) for more on how to produce human-readable results from these data files. If the `GOCOVERDIR` environment variable is not set, a coverage-instrumented binary will still execute correctly, but will issue a warning. Example: $ ./myprogram.exe warning: GOCOVERDIR not set, no coverage data emitted I say "Hello, world." and "see ya" $ Tests involving multiple runs ----------------------------- Integration tests can in many cases involve multiple program runs; when the program is built with “`-cover`”, each run will produce a new data file. Example $ mkdir somedata2 $ GOCOVERDIR=somedata2 ./myprogram.exe // first run I say "Hello, world." and "see ya" $ GOCOVERDIR=somedata2 ./myprogram.exe -flag // second run I say "Hello, world." and "see ya" $ ls somedata2 covcounters.890814fca98ac3a4d41b9bd2a7ec9f7f.2456041.1670259309405583534 covcounters.890814fca98ac3a4d41b9bd2a7ec9f7f.2456047.1670259309410891043 covmeta.890814fca98ac3a4d41b9bd2a7ec9f7f $ Coverage data output files come in two flavors: meta-data files (containing the items that are invariant from run to run, such as source file names and function names), and counter data files (which record the parts of the program that executed). In the example above, the first run produced two files (counter and meta), whereas the second run generated only a counter data file: since meta-data doesn’t change from run to run, it only needs to be written once. Working with coverage data files ================================ Go 1.20 introduces a new tool, ‘`covdata`’, that can be used to read and manipulate coverage data files from a `GOCOVERDIR` directory. Go’s `covdata` tool runs in a variety of modes. The general form of a `covdata` tool invocation takes the form $ go tool covdata -i= ...flags... where the “`-i`” flag provides a list of directories to read, where each directories is derived from an execution of a coverage-instrumented binary (via `GOCOVERDIR`). Creating coverage profile reports --------------------------------- This section discusses how to use “`go tool covdata`” to produce human-readable reports from coverage data files. ### Reporting percent statements covered To report a “percent statements covered” metric for each instrumented package, use the command “`go tool covdata percent -i=`”. Using the example from the [running](https://go.dev/doc/build-cover#running) section above: $ ls somedata covcounters.c6de772f99010ef5925877a7b05db4cc.2424989.1670252383678349347 covmeta.c6de772f99010ef5925877a7b05db4cc $ go tool covdata percent -i=somedata main coverage: 100.0% of statements mydomain.com/greetings coverage: 100.0% of statements $ The “statements covered” percentages here correspond directly to those reported by `go test -cover`. Converting to legacy text format -------------------------------- You can convert binary coverage data files into the legacy textual format generated by “`go test -coverprofile=`” using the covdata `textfmt` selector. The resulting text file can then be used with “`go tool cover -func`” or “`go tool cover -html`” to create additional reports. Example: $ ls somedata covcounters.c6de772f99010ef5925877a7b05db4cc.2424989.1670252383678349347 covmeta.c6de772f99010ef5925877a7b05db4cc $ go tool covdata textfmt -i=somedata -o profile.txt $ cat profile.txt mode: set mydomain.com/myprogram.go:10.13,12.2 1 1 mydomain.com/greetings/greetings.go:3.23,5.2 1 1 $ go tool cover -func=profile.txt mydomain.com/greetings/greetings.go:3: Goodbye 100.0% mydomain.com/myprogram.go:10: main 100.0% total: (statements) 100.0% $ Merging ------- The `merge` subcommand of “`go tool covdata`” can be used to merge together profiles from multiple data directories. For example, consider a program that runs on both macOS and on Windows. The author of this program might want to combine coverage profiles from separate runs on each operating system into a single profile corpus, so as to produce a cross-platform coverage summary. For example: $ ls windows_datadir covcounters.f3833f80c91d8229544b25a855285890.1025623.1667481441036838252 covcounters.f3833f80c91d8229544b25a855285890.1025628.1667481441042785007 covmeta.f3833f80c91d8229544b25a855285890 $ ls macos_datadir covcounters.b245ad845b5068d116a4e25033b429fb.1025358.1667481440551734165 covcounters.b245ad845b5068d116a4e25033b429fb.1025364.1667481440557770197 covmeta.b245ad845b5068d116a4e25033b429fb $ ls macos_datadir $ mkdir merged $ go tool covdata merge -i=windows_datadir,macos_datadir -o merged $ The merge operation above will combine the data from the specified input directories and write a new set of merged data files to the directory “merged”. Package selection ----------------- Most “`go tool covdata`” commands support a “`-pkg`” flag to perform package selection as part of the operation; the argument to “`-pkg`” takes the same form as that used by the Go command’s “`-coverpkg`” flag. Example: $ ls somedata covcounters.c6de772f99010ef5925877a7b05db4cc.2424989.1670252383678349347 covmeta.c6de772f99010ef5925877a7b05db4cc $ go tool covdata percent -i=somedata -pkg=mydomain.com/greetings mydomain.com/greetings coverage: 100.0% of statements $ go tool covdata percent -i=somedata -pkg=nonexistentpackage $ The “`-pkg`” flag can be used to select the specific subset of packages of interest for a given report. Frequently Asked Questions -------------------------- 1. [How can I request coverage instrumentation for all imported packages mentioned in my `go.mod` file](https://go.dev/doc/build-cover#gomodselect) 2. [Can I use `go build -cover` in GOPATH/GO111MODULE=off mode?](https://go.dev/doc/build-cover#gopathmode) 3. [If my program panics, will coverage data be written?](https://go.dev/doc/build-cover#panicprof) 4. [Will `-coverpkg=main` select my main package for profiling?](https://go.dev/doc/build-cover#mainpkg) #### How can I request coverage instrumentation for all imported packages mentioned in my `go.mod` file By default, `go build -cover` will instrument all main module packages for coverage, but will not instrument imports outside the main module (e.g. standard library packages or imports listed in `go.mod`). One way to request instrumentation for all non-stdlib dependencies is to feed the output of `go list` into `-coverpkg`. Here is an example, again using the [example program](https://go.dev/play/p/VSQJN8xkkf-?v=gotip) cited above: $ go list -f '{{if not .Standard}}{{.ImportPath}}{{end}}' -deps . | paste -sd "," > pkgs.txt $ go build -o myprogram.exe -coverpkg=`cat pkgs.txt` . $ mkdir somedata $ GOCOVERDIR=somedata ./myprogram.exe $ go tool covdata percent -i=somedata golang.org/x/text/internal/tag coverage: 78.4% of statements golang.org/x/text/language coverage: 35.5% of statements mydomain.com coverage: 100.0% of statements mydomain.com/greetings coverage: 100.0% of statements rsc.io/quote coverage: 25.0% of statements rsc.io/sampler coverage: 86.7% of statements $ #### Can I use `go build -cover` in GO111MODULE=off mode? Yes, `go build -cover` does work with `GO111MODULE=off`. When building a program in GO111MODULE=off mode, only the package specifically named as the target on the command line will be instrumented for profiling. Use the `-coverpkg` flag to include additional packages in the profile. #### If my program panics, will coverage data be written? Programs built with `go build -cover` will only write out complete profile data at the end of execution if the program invokes `os.Exit()` or returns normally from `main.main`. If a program terminates in an unrecovered panic, or if the program hits a fatal exception (such as a segmentation violation, divide by zero, etc), profile data from statements executed during the run will be lost. #### Will `-coverpkg=main` select my main package for profiling? The `-coverpkg` flag accepts a list of import paths, not a list of package names. If you want to select your `main` package for coverage instrumention, please identify it by import path, not by name. Example (using [this example program](https://go.dev/play/p/VSQJN8xkkf-?v=gotip) ): $ go list -m mydomain.com $ go build -coverpkg=main -o oops.exe . warning: no packages being built depend on matches for pattern main $ go build -coverpkg=mydomain.com -o myprogram.exe . $ mkdir somedata $ GOCOVERDIR=somedata ./myprogram.exe I say "Hello, world." and "see ya" $ go tool covdata percent -i=somedata mydomain.com coverage: 100.0% of statements $ Resources --------- * **Blog post introducing unit test coverage in Go 1.2**: * Coverage profiling for unit tests was introduced as part of the Go 1.2 release; see [this blog post](https://go.dev/blog/cover) for details. * **Documentation**: * The [`cmd/go`](https://pkg.go.dev/cmd/go) package docs describe the build and test flags associated with coverage. * **Technical details**: * [Design draft](https://go.dev/design/51430-revamp-code-coverage) * [Proposal](https://go.dev/issue/51430) Glossary -------- **unit test:** Tests within a `*_test.go` file associated with a specific Go package, utilizing Go’s `testing` package. **integration test:** A more comprehensive, heavier weight test for a given application or binary. Integration tests typically involve building a program or set of programs, then performing a series of runs of the programs using multiple inputs and scenarios, under control of a test harness that may or may not be based on Go’s `testing` package. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # JSON-RPC: a tale of interfaces - The Go Programming Language [The Go Blog](https://go.dev/blog/) ==================================== JSON-RPC: a tale of interfaces ============================== Andrew Gerrand 27 April 2010 Here we present an example where Go’s [interfaces](https://go.dev/doc/effective_go.html#interfaces_and_types) made it easy to refactor some existing code to make it more flexible and extensible. Originally, the standard library’s [RPC package](https://go.dev/pkg/net/rpc/) used a custom wire format called [gob](https://go.dev/pkg/encoding/gob/) . For a particular application, we wanted to use [JSON](https://go.dev/pkg/encoding/json/) as an alternate wire format. We first defined a pair of interfaces to describe the functionality of the existing wire format, one for the client, and one for the server (depicted below). type ServerCodec interface { ReadRequestHeader(*Request) error ReadRequestBody(interface{}) error WriteResponse(*Response, interface{}) error Close() error } On the server side, we then changed two internal function signatures to accept the `ServerCodec` interface instead of our existing `gob.Encoder`. Here’s one of them: func sendResponse(sending *sync.Mutex, req *Request, reply interface{}, enc *gob.Encoder, errmsg string) became func sendResponse(sending *sync.Mutex, req *Request, reply interface{}, enc ServerCodec, errmsg string) We then wrote a trivial `gobServerCodec` wrapper to reproduce the original functionality. From there it is simple to build a `jsonServerCodec`. After some similar changes to the client side, this was the full extent of the work we needed to do on the RPC package. This whole exercise took about 20 minutes! After tidying up and testing the new code, the [final changeset](https://github.com/golang/go/commit/dcff89057bc0e0d7cb14cf414f2df6f5fb1a41ec) was submitted. In an inheritance-oriented language like Java or C++, the obvious path would be to generalize the RPC class, and create JsonRPC and GobRPC subclasses. However, this approach becomes tricky if you want to make a further generalization orthogonal to that hierarchy. (For example, if you were to implement an alternate RPC standard). In our Go package, we took a route that is both conceptually simpler and requires less code be written or changed. A vital quality for any codebase is maintainability. As needs change, it is essential to adapt your code easily and cleanly, lest it become unwieldy to work with. We believe Go’s lightweight, composition-oriented type system provides a means of structuring code that scales. **Next article:** [New Talk and Tutorials](https://go.dev/blog/new-talk-and-tutorials) **Previous article:** [Third-party libraries: goprotobuf and beyond](https://go.dev/blog/protobuf) **[Blog Index](https://go.dev/blog/all) ** go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Go Slices: usage and internals - The Go Programming Language [The Go Blog](https://go.dev/blog/) ==================================== Go Slices: usage and internals ============================== Andrew Gerrand 5 January 2011 Introduction ------------ Go’s slice type provides a convenient and efficient means of working with sequences of typed data. Slices are analogous to arrays in other languages, but have some unusual properties. This article will look at what slices are and how they are used. Arrays ------ The slice type is an abstraction built on top of Go’s array type, and so to understand slices we must first understand arrays. An array type definition specifies a length and an element type. For example, the type `[4]int` represents an array of four integers. An array’s size is fixed; its length is part of its type (`[4]int` and `[5]int` are distinct, incompatible types). Arrays can be indexed in the usual way, so the expression `s[n]` accesses the nth element, starting from zero. var a [4]int a[0] = 1 i := a[0] // i == 1 Arrays do not need to be initialized explicitly; the zero value of an array is a ready-to-use array whose elements are themselves zeroed: // a[2] == 0, the zero value of the int type The in-memory representation of `[4]int` is just four integer values laid out sequentially: ![](https://go.dev/blog/slices-intro/slice-array.png) Go’s arrays are values. An array variable denotes the entire array; it is not a pointer to the first array element (as would be the case in C). This means that when you assign or pass around an array value you will make a copy of its contents. (To avoid the copy you could pass a _pointer_ to the array, but then that’s a pointer to an array, not an array.) One way to think about arrays is as a sort of struct but with indexed rather than named fields: a fixed-size composite value. An array literal can be specified like so: b := [2]string{"Penn", "Teller"} Or, you can have the compiler count the array elements for you: b := [...]string{"Penn", "Teller"} In both cases, the type of `b` is `[2]string`. Slices ------ Arrays have their place, but they’re a bit inflexible, so you don’t see them too often in Go code. Slices, though, are everywhere. They build on arrays to provide great power and convenience. The type specification for a slice is `[]T`, where `T` is the type of the elements of the slice. Unlike an array type, a slice type has no specified length. A slice literal is declared just like an array literal, except you leave out the element count: letters := []string{"a", "b", "c", "d"} A slice can be created with the built-in function called `make`, which has the signature, func make([]T, len, cap) []T where T stands for the element type of the slice to be created. The `make` function takes a type, a length, and an optional capacity. When called, `make` allocates an array and returns a slice that refers to that array. var s []byte s = make([]byte, 5, 5) // s == []byte{0, 0, 0, 0, 0} When the capacity argument is omitted, it defaults to the specified length. Here’s a more succinct version of the same code: s := make([]byte, 5) The length and capacity of a slice can be inspected using the built-in `len` and `cap` functions. len(s) == 5 cap(s) == 5 The next two sections discuss the relationship between length and capacity. The zero value of a slice is `nil`. The `len` and `cap` functions will both return 0 for a nil slice. A slice can also be formed by “slicing” an existing slice or array. Slicing is done by specifying a half-open range with two indices separated by a colon. For example, the expression `b[1:4]` creates a slice including elements 1 through 3 of `b` (the indices of the resulting slice will be 0 through 2). b := []byte{'g', 'o', 'l', 'a', 'n', 'g'} // b[1:4] == []byte{'o', 'l', 'a'}, sharing the same storage as b The start and end indices of a slice expression are optional; they default to zero and the slice’s length respectively: // b[:2] == []byte{'g', 'o'} // b[2:] == []byte{'l', 'a', 'n', 'g'} // b[:] == b This is also the syntax to create a slice given an array: x := [3]string{"Лайка", "Белка", "Стрелка"} s := x[:] // a slice referencing the storage of x Slice internals --------------- A slice is a descriptor of an array segment. It consists of a pointer to the array, the length of the segment, and its capacity (the maximum length of the segment). ![](https://go.dev/blog/slices-intro/slice-struct.png) Our variable `s`, created earlier by `make([]byte, 5)`, is structured like this: ![](https://go.dev/blog/slices-intro/slice-1.png) The length is the number of elements referred to by the slice. The capacity is the number of elements in the underlying array (beginning at the element referred to by the slice pointer). The distinction between length and capacity will be made clear as we walk through the next few examples. As we slice `s`, observe the changes in the slice data structure and their relation to the underlying array: s = s[2:4] ![](https://go.dev/blog/slices-intro/slice-2.png) Slicing does not copy the slice’s data. It creates a new slice value that points to the original array. This makes slice operations as efficient as manipulating array indices. Therefore, modifying the _elements_ (not the slice itself) of a re-slice modifies the elements of the original slice: d := []byte{'r', 'o', 'a', 'd'} e := d[2:] // e == []byte{'a', 'd'} e[1] = 'm' // e == []byte{'a', 'm'} // d == []byte{'r', 'o', 'a', 'm'} Earlier we sliced `s` to a length shorter than its capacity. We can grow s to its capacity by slicing it again: s = s[:cap(s)] ![](https://go.dev/blog/slices-intro/slice-3.png) A slice cannot be grown beyond its capacity. Attempting to do so will cause a runtime panic, just as when indexing outside the bounds of a slice or array. Similarly, slices cannot be re-sliced below zero to access earlier elements in the array. Growing slices (the copy and append functions) ---------------------------------------------- To increase the capacity of a slice one must create a new, larger slice and copy the contents of the original slice into it. This technique is how dynamic array implementations from other languages work behind the scenes. The next example doubles the capacity of `s` by making a new slice, `t`, copying the contents of `s` into `t`, and then assigning the slice value `t` to `s`: t := make([]byte, len(s), (cap(s)+1)*2) // +1 in case cap(s) == 0 for i := range s { t[i] = s[i] } s = t The looping piece of this common operation is made easier by the built-in copy function. As the name suggests, copy copies data from a source slice to a destination slice. It returns the number of elements copied. func copy(dst, src []T) int The `copy` function supports copying between slices of different lengths (it will copy only up to the smaller number of elements). In addition, `copy` can handle source and destination slices that share the same underlying array, handling overlapping slices correctly. Using `copy`, we can simplify the code snippet above: t := make([]byte, len(s), (cap(s)+1)*2) copy(t, s) s = t A common operation is to append data to the end of a slice. This function appends byte elements to a slice of bytes, growing the slice if necessary, and returns the updated slice value: func AppendByte(slice []byte, data ...byte) []byte { m := len(slice) n := m + len(data) if n > cap(slice) { // if necessary, reallocate // allocate double what's needed, for future growth. newSlice := make([]byte, (n+1)*2) copy(newSlice, slice) slice = newSlice } slice = slice[0:n] copy(slice[m:n], data) return slice } One could use `AppendByte` like this: p := []byte{2, 3, 5} p = AppendByte(p, 7, 11, 13) // p == []byte{2, 3, 5, 7, 11, 13} Functions like `AppendByte` are useful because they offer complete control over the way the slice is grown. Depending on the characteristics of the program, it may be desirable to allocate in smaller or larger chunks, or to put a ceiling on the size of a reallocation. But most programs don’t need complete control, so Go provides a built-in `append` function that’s good for most purposes; it has the signature func append(s []T, x ...T) []T The `append` function appends the elements `x` to the end of the slice `s`, and grows the slice if a greater capacity is needed. a := make([]int, 1) // a == []int{0} a = append(a, 1, 2, 3) // a == []int{0, 1, 2, 3} To append one slice to another, use `...` to expand the second argument to a list of arguments. a := []string{"John", "Paul"} b := []string{"George", "Ringo", "Pete"} a = append(a, b...) // equivalent to "append(a, b[0], b[1], b[2])" // a == []string{"John", "Paul", "George", "Ringo", "Pete"} Since the zero value of a slice (`nil`) acts like a zero-length slice, you can declare a slice variable and then append to it in a loop: // Filter returns a new slice holding only // the elements of s that satisfy fn() func Filter(s []int, fn func(int) bool) []int { var p []int // == nil for _, v := range s { if fn(v) { p = append(p, v) } } return p } A possible “gotcha” ------------------- As mentioned earlier, re-slicing a slice doesn’t make a copy of the underlying array. The full array will be kept in memory until it is no longer referenced. Occasionally this can cause the program to hold all the data in memory when only a small piece of it is needed. For example, this `FindDigits` function loads a file into memory and searches it for the first group of consecutive numeric digits, returning them as a new slice. var digitRegexp = regexp.MustCompile("[0-9]+") func FindDigits(filename string) []byte { b, _ := ioutil.ReadFile(filename) return digitRegexp.Find(b) } This code behaves as advertised, but the returned `[]byte` points into an array containing the entire file. Since the slice references the original array, as long as the slice is kept around the garbage collector can’t release the array; the few useful bytes of the file keep the entire contents in memory. To fix this problem one can copy the interesting data to a new slice before returning it: func CopyDigits(filename string) []byte { b, _ := ioutil.ReadFile(filename) b = digitRegexp.Find(b) c := make([]byte, len(b)) copy(c, b) return c } A more concise version of this function could be constructed by using `append`. This is left as an exercise for the reader. Further Reading --------------- [Effective Go](https://go.dev/doc/effective_go.html) contains an in-depth treatment of [slices](https://go.dev/doc/effective_go.html#slices) and [arrays](https://go.dev/doc/effective_go.html#arrays) , and the Go [language specification](https://go.dev/doc/go_spec.html) defines [slices](https://go.dev/doc/go_spec.html#Slice_types) and their [associated](https://go.dev/doc/go_spec.html#Length_and_capacity) [helper](https://go.dev/doc/go_spec.html#Making_slices_maps_and_channels) [functions](https://go.dev/doc/go_spec.html#Appending_and_copying_slices) . **Next article:** [JSON and Go](https://go.dev/blog/json) **Previous article:** [Go: one year ago today](https://go.dev/blog/1year) **[Blog Index](https://go.dev/blog/all) ** go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Debugging Go Code with GDB - The Go Programming Language 1. [Documentation](https://go.dev/doc/) 2. [Debugging Go Code with GDB](https://go.dev/doc/gdb) Debugging Go Code with GDB ========================== _The following instructions apply to the standard toolchain (the `gc` Go compiler and tools). Gccgo has native gdb support._ _Note that [Delve](https://github.com/go-delve/delve) is a better alternative to GDB when debugging Go programs built with the standard toolchain. It understands the Go runtime, data structures, and expressions better than GDB. Delve currently supports Linux, OSX, and Windows on `amd64`. For the most up-to-date list of supported platforms, please see [the Delve documentation](https://github.com/go-delve/delve/tree/master/Documentation/installation) ._ GDB does not understand Go programs well. The stack management, threading, and runtime contain aspects that differ enough from the execution model GDB expects that they can confuse the debugger and cause incorrect results even when the program is compiled with gccgo. As a consequence, although GDB can be useful in some situations (e.g., debugging Cgo code, or debugging the runtime itself), it is not a reliable debugger for Go programs, particularly heavily concurrent ones. Moreover, it is not a priority for the Go project to address these issues, which are difficult. In short, the instructions below should be taken only as a guide to how to use GDB when it works, not as a guarantee of success. Besides this overview you might want to consult the [GDB manual](https://sourceware.org/gdb/current/onlinedocs/gdb/) . Introduction ------------ When you compile and link your Go programs with the `gc` toolchain on Linux, macOS, FreeBSD or NetBSD, the resulting binaries contain DWARFv4 debugging information that recent versions (≥7.5) of the GDB debugger can use to inspect a live process or a core dump. Pass the `'-w'` flag to the linker to omit the debug information (for example, `go` `build` `-ldflags=-w` `prog.go`). The code generated by the `gc` compiler includes inlining of function invocations and registerization of variables. These optimizations can sometimes make debugging with `gdb` harder. If you find that you need to disable these optimizations, build your program using `go` `build` `-gcflags=all="-N -l"`. If you want to use gdb to inspect a core dump, you can trigger a dump on a program crash, on systems that permit it, by setting `GOTRACEBACK=crash` in the environment (see the [runtime package documentation](https://go.dev/pkg/runtime/#hdr-Environment_Variables) for more info). ### Common Operations * Show file and line number for code, set breakpoints and disassemble: (gdb) **list** (gdb) **list _line_** (gdb) **list _file.go_:_line_** (gdb) **break _line_** (gdb) **break _file.go_:_line_** (gdb) **disas** * Show backtraces and unwind stack frames: (gdb) **bt** (gdb) **frame _n_** * Show the name, type and location on the stack frame of local variables, arguments and return values: (gdb) **info locals** (gdb) **info args** (gdb) **p variable** (gdb) **whatis variable** * Show the name, type and location of global variables: (gdb) **info variables _regexp_** ### Go Extensions A recent extension mechanism to GDB allows it to load extension scripts for a given binary. The toolchain uses this to extend GDB with a handful of commands to inspect internals of the runtime code (such as goroutines) and to pretty print the built-in map, slice and channel types. * Pretty printing a string, slice, map, channel or interface: (gdb) **p _var_** * A $len() and $cap() function for strings, slices and maps: (gdb) **p $len(_var_)** * A function to cast interfaces to their dynamic types: (gdb) **p $dtype(_var_)** (gdb) **iface _var_** **Known issue:** GDB can’t automatically find the dynamic type of an interface value if its long name differs from its short name (annoying when printing stacktraces, the pretty printer falls back to printing the short type name and a pointer). * Inspecting goroutines: (gdb) **info goroutines** (gdb) **goroutine _n_ _cmd_** (gdb) **help goroutine** For example: (gdb) **goroutine 12 bt** You can inspect all goroutines by passing `all` instead of a specific goroutine's ID. For example: (gdb) **goroutine all bt** If you'd like to see how this works, or want to extend it, take a look at [src/runtime/runtime-gdb.py](https://go.dev/src/runtime/runtime-gdb.py) in the Go source distribution. It depends on some special magic types (`hash`) and variables (`runtime.m` and `runtime.g`) that the linker ([src/cmd/link/internal/ld/dwarf.go](https://go.dev/src/cmd/link/internal/ld/dwarf.go) ) ensures are described in the DWARF code. If you're interested in what the debugging information looks like, run `objdump` `-W` `a.out` and browse through the `.debug_*` sections. ### Known Issues 1. String pretty printing only triggers for type string, not for types derived from it. 2. Type information is missing for the C parts of the runtime library. 3. GDB does not understand Go’s name qualifications and treats `"fmt.Print"` as an unstructured literal with a `"."` that needs to be quoted. It objects even more strongly to method names of the form `pkg.(*MyType).Meth`. 4. As of Go 1.11, debug information is compressed by default. Older versions of gdb, such as the one available by default on MacOS, do not understand the compression. You can generate uncompressed debug information by using `go build -ldflags=-compressdwarf=false`. (For convenience you can put the `-ldflags` option in the [`GOFLAGS` environment variable](https://go.dev/cmd/go/#hdr-Environment_variables) so that you don't have to specify it each time.) Tutorial -------- In this tutorial we will inspect the binary of the [regexp](https://go.dev/pkg/regexp/) package's unit tests. To build the binary, change to `$GOROOT/src/regexp` and run `go` `test` `-c`. This should produce an executable file named `regexp.test`. ### Getting Started Launch GDB, debugging `regexp.test`: $ **gdb regexp.test** GNU gdb (GDB) 7.2-gg8 Copyright (C) 2010 Free Software Foundation, Inc. License GPLv 3+: GNU GPL version 3 or later Type "show copying" and "show warranty" for licensing/warranty details. This GDB was configured as "x86\_64-linux". Reading symbols from /home/user/go/src/regexp/regexp.test... done. Loading Go Runtime support. (gdb) The message "Loading Go Runtime support" means that GDB loaded the extension from `$GOROOT/src/runtime/runtime-gdb.py`. To help GDB find the Go runtime sources and the accompanying support script, pass your `$GOROOT` with the `'-d'` flag: $ **gdb regexp.test -d $GOROOT** If for some reason GDB still can't find that directory or that script, you can load it by hand by telling gdb (assuming you have the go sources in `~/go/`): (gdb) **source ~/go/src/runtime/runtime-gdb.py** Loading Go Runtime support. ### Inspecting the source Use the `"l"` or `"list"` command to inspect source code. (gdb) **l** List a specific part of the source parameterizing `"list"` with a function name (it must be qualified with its package name). (gdb) **l main.main** List a specific file and line number: (gdb) **l regexp.go:1** (gdb) _\# Hit enter to repeat last command. Here, this lists next 10 lines._ ### Naming Variable and function names must be qualified with the name of the packages they belong to. The `Compile` function from the `regexp` package is known to GDB as `'regexp.Compile'`. Methods must be qualified with the name of their receiver types. For example, the `*Regexp` type’s `String` method is known as `'regexp.(*Regexp).String'`. Variables that shadow other variables are magically suffixed with a number in the debug info. Variables referenced by closures will appear as pointers magically prefixed with '&'. ### Setting breakpoints Set a breakpoint at the `TestFind` function: (gdb) **b 'regexp.TestFind'** Breakpoint 1 at 0x424908: file /home/user/go/src/regexp/find\_test.go, line 148. Run the program: (gdb) **run** Starting program: /home/user/go/src/regexp/regexp.test Breakpoint 1, regexp.TestFind (t=0xf8404a89c0) at /home/user/go/src/regexp/find\_test.go:148 148 func TestFind(t \*testing.T) { Execution has paused at the breakpoint. See which goroutines are running, and what they're doing: (gdb) **info goroutines** 1 waiting runtime.gosched \* 13 running runtime.goexit the one marked with the `*` is the current goroutine. ### Inspecting the stack Look at the stack trace for where we’ve paused the program: (gdb) **bt** _\# backtrace_ #0 regexp.TestFind (t=0xf8404a89c0) at /home/user/go/src/regexp/find\_test.go:148 #1 0x000000000042f60b in testing.tRunner (t=0xf8404a89c0, test=0x573720) at /home/user/go/src/testing/testing.go:156 #2 0x000000000040df64 in runtime.initdone () at /home/user/go/src/runtime/proc.c:242 #3 0x000000f8404a89c0 in ?? () #4 0x0000000000573720 in ?? () #5 0x0000000000000000 in ?? () The other goroutine, number 1, is stuck in `runtime.gosched`, blocked on a channel receive: (gdb) **goroutine 1 bt** #0 0x000000000040facb in runtime.gosched () at /home/user/go/src/runtime/proc.c:873 #1 0x00000000004031c9 in runtime.chanrecv (c=void, ep=void, selected=void, received=void) at /home/user/go/src/runtime/chan.c:342 #2 0x0000000000403299 in runtime.chanrecv1 (t=void, c=void) at/home/user/go/src/runtime/chan.c:423 #3 0x000000000043075b in testing.RunTests (matchString={void (struct string, struct string, bool \*, error \*)} 0x7ffff7f9ef60, tests= \[\]testing.InternalTest = {...}) at /home/user/go/src/testing/testing.go:201 #4 0x00000000004302b1 in testing.Main (matchString={void (struct string, struct string, bool \*, error \*)} 0x7ffff7f9ef80, tests= \[\]testing.InternalTest = {...}, benchmarks= \[\]testing.InternalBenchmark = {...}) at /home/user/go/src/testing/testing.go:168 #5 0x0000000000400dc1 in main.main () at /home/user/go/src/regexp/\_testmain.go:98 #6 0x00000000004022e7 in runtime.mainstart () at /home/user/go/src/runtime/amd64/asm.s:78 #7 0x000000000040ea6f in runtime.initdone () at /home/user/go/src/runtime/proc.c:243 #8 0x0000000000000000 in ?? () The stack frame shows we’re currently executing the `regexp.TestFind` function, as expected. (gdb) **info frame** Stack level 0, frame at 0x7ffff7f9ff88: rip = 0x425530 in regexp.TestFind (/home/user/go/src/regexp/find\_test.go:148); saved rip 0x430233 called by frame at 0x7ffff7f9ffa8 source language minimal. Arglist at 0x7ffff7f9ff78, args: t=0xf840688b60 Locals at 0x7ffff7f9ff78, Previous frame's sp is 0x7ffff7f9ff88 Saved registers: rip at 0x7ffff7f9ff80 The command `info` `locals` lists all variables local to the function and their values, but is a bit dangerous to use, since it will also try to print uninitialized variables. Uninitialized slices may cause gdb to try to print arbitrary large arrays. The function’s arguments: (gdb) **info args** t = 0xf840688b60 When printing the argument, notice that it’s a pointer to a `Regexp` value. Note that GDB has incorrectly put the `*` on the right-hand side of the type name and made up a 'struct' keyword, in traditional C style. (gdb) **p re** (gdb) p t $1 = (struct testing.T \*) 0xf840688b60 (gdb) p t $1 = (struct testing.T \*) 0xf840688b60 (gdb) p \*t $2 = {errors = "", failed = false, ch = 0xf8406f5690} (gdb) p \*t->ch $3 = struct hchan<\*testing.T> That `struct` `hchan<*testing.T>` is the runtime-internal representation of a channel. It is currently empty, or gdb would have pretty-printed its contents. Stepping forward: (gdb) **n** _\# execute next line_ 149 for \_, test := range findTests { (gdb) _\# enter is repeat_ 150 re := MustCompile(test.pat) (gdb) **p test.pat** $4 = "" (gdb) **p re** $5 = (struct regexp.Regexp \*) 0xf84068d070 (gdb) **p \*re** $6 = {expr = "", prog = 0xf840688b80, prefix = "", prefixBytes = \[\]uint8, prefixComplete = true, prefixRune = 0, cond = 0 '\\000', numSubexp = 0, longest = false, mu = {state = 0, sema = 0}, machine = \[\]\*regexp.machine} (gdb) **p \*re->prog** $7 = {Inst = \[\]regexp/syntax.Inst = {{Op = 5 '\\005', Out = 0, Arg = 0, Rune = \[\]int}, {Op = 6 '\\006', Out = 2, Arg = 0, Rune = \[\]int}, {Op = 4 '\\004', Out = 0, Arg = 0, Rune = \[\]int}}, Start = 1, NumCap = 2} We can step into the `String`function call with `"s"`: (gdb) **s** regexp.(\*Regexp).String (re=0xf84068d070, noname=void) at /home/user/go/src/regexp/regexp.go:97 97 func (re \*Regexp) String() string { Get a stack trace to see where we are: (gdb) **bt** #0 regexp.(\*Regexp).String (re=0xf84068d070, noname=void) at /home/user/go/src/regexp/regexp.go:97 #1 0x0000000000425615 in regexp.TestFind (t=0xf840688b60) at /home/user/go/src/regexp/find\_test.go:151 #2 0x0000000000430233 in testing.tRunner (t=0xf840688b60, test=0x5747b8) at /home/user/go/src/testing/testing.go:156 #3 0x000000000040ea6f in runtime.initdone () at /home/user/go/src/runtime/proc.c:243 .... Look at the source code: (gdb) **l** 92 mu sync.Mutex 93 machine \[\]\*machine 94 } 95 96 // String returns the source text used to compile the regular expression. 97 func (re \*Regexp) String() string { 98 return re.expr 99 } 100 101 // Compile parses a regular expression and returns, if successful, ### Pretty Printing GDB's pretty printing mechanism is triggered by regexp matches on type names. An example for slices: (gdb) **p utf** $22 = \[\]uint8 = {0 '\\000', 0 '\\000', 0 '\\000', 0 '\\000'} Since slices, arrays and strings are not C pointers, GDB can't interpret the subscripting operation for you, but you can look inside the runtime representation to do that (tab completion helps here): (gdb) **p slc** $11 = \[\]int = {0, 0} (gdb) **p slc->**__ array slc len (gdb) **p slc->array** $12 = (int \*) 0xf84057af00 (gdb) **p slc->array\[1\]** $13 = 0 The extension functions $len and $cap work on strings, arrays and slices: (gdb) **p $len(utf)** $23 = 4 (gdb) **p $cap(utf)** $24 = 4 Channels and maps are 'reference' types, which gdb shows as pointers to C++-like types `hash*`. Dereferencing will trigger prettyprinting Interfaces are represented in the runtime as a pointer to a type descriptor and a pointer to a value. The Go GDB runtime extension decodes this and automatically triggers pretty printing for the runtime type. The extension function `$dtype` decodes the dynamic type for you (examples are taken from a breakpoint at `regexp.go` line 293.) (gdb) **p i** $4 = {str = "cbb"} (gdb) **whatis i** type = regexp.input (gdb) **p $dtype(i)** $26 = (struct regexp.inputBytes \*) 0xf8400b4930 (gdb) **iface i** regexp.input: struct regexp.inputBytes \* go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # How Google's Core Data Solutions Team Uses Go - The Go Programming Language 1. [Why Go](https://go.dev/solutions/) 2. [Using Go at Google](https://go.dev/solutions/google/) 3. [How Google's Core Data Solutions Team Uses Go](https://go.dev/solutions/google/coredata) How Google's Core Data Solutions Team Uses Go ============================================= Prasanna Meda, Software Engineer, Core Data Solutions ![Core Data](https://go.dev/images/go_core_data_case_study.png) Google’s mission is “to organize the world’s information and make it universally accessible and useful.” One of the teams responsible for organizing that information is Google’s Core Data Solutions team. The team, among other things, maintains services to index web pages across the globe. These web indexing services help support products like Google Search by keeping search results updated and comprehensive, and they’re written in Go. In 2015, to keep up with Google’s scale, our team needed to rewrite our indexing stack from a single monolithic binary written in C++ to multiple components in a microservices architecture. We decided to rewrite many indexing services in Go, which we now use to power the majority of our architecture. “ Go’s built-in concurrency is a natural fit because engineers on the team are encouraged to use concurrency and parallel algorithms. ” — Minjae Hwang ,  Software Engineer When choosing a language, our team found that several of Go’s features made it particularly suitable. For instance, Go’s built-in concurrency is a natural fit because engineers on the team are encouraged to use concurrency and parallel algorithms. Engineers have also found that “Go code is more natural,” allowing them to spend their time focusing on business logic and analysis rather than on managing memory and optimizing performance. Writing code is much simpler when writing in Go, as it helps lessen cognitive burden during the development process. For example, when working with C++, sophisticated IDEs might, “show that the source code has no compile error when there actually is one” whereas “in Go, \[the code\] will always compile when \[the IDE\] says the code has no compile error,” said MinJae Hwang, a software engineer on the Core Data Solutions team. Reducing small friction points along the development process, such as shortening the cycle of fixing compile errors, helped our team ship faster during the original rewrite, and has helped keep our maintenance costs low. “When I’m in C++ and I want to use more packages, I have to write pieces such as headers. When I’m writing in Go, **built-in tools allow me to use packages more easily. My development velocity is much faster,**” Hwang also shared. With simple language syntax and support of Go tools, several members of our team find it much easier to write in Go code. We’ve also found that Go does a really good job of static type checking and that certain Go fundamentals, such as the godoc command, have helped the team build a more disciplined culture around writing documentation. “ …Google’s web indexing was re-architected within a year. More impressively, most developers on the team were rewriting in Go while also learning it. ” — Prasanna Meda ,  Software Engineer Working on a product used so heavily around the world is no small task and our team’s decision to use Go wasn’t a simple one, but doing so helped us move faster. As a result, Google’s web indexing was re-architected within a year. More impressively, most developers on the team were rewriting in Go while also learning it. In addition to the Core Data Solutions team, engineering teams across Google have adopted Go in their development process. Read about how the [Chrome](https://go.dev/solutions/google/chrome/) and [Firebase Hosting](https://go.dev/solutions/google/firebase/) teams use Go to build fast, reliable, and efficient software at scale. ![Core Data](https://go.dev/images/logos/google.svg) ![Core Data](https://go.dev/images/logos/google.svg) ### About Core Data Google is a technology company whose mission is to organize the world’s information and make it universally accessible and useful. In this case study, Google’s Core Data Solutions team shares their journey with Go, including their decision to rewrite web indexing services in Go, taking advantage of Go’s built-in concurrency, and observing how Go helps to improve the development process. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Go Concurrency Patterns: Timing out, moving on - The Go Programming Language [The Go Blog](https://go.dev/blog/) ==================================== Go Concurrency Patterns: Timing out, moving on ============================================== Andrew Gerrand 23 September 2010 Concurrent programming has its own idioms. A good example is timeouts. Although Go’s channels do not support them directly, they are easy to implement. Say we want to receive from the channel `ch`, but want to wait at most one second for the value to arrive. We would start by creating a signalling channel and launching a goroutine that sleeps before sending on the channel: timeout := make(chan bool, 1) go func() { time.Sleep(1 * time.Second) timeout <- true }() We can then use a `select` statement to receive from either `ch` or `timeout`. If nothing arrives on `ch` after one second, the timeout case is selected and the attempt to read from ch is abandoned. select { case <-ch: // a read from ch has occurred case <-timeout: // the read from ch has timed out } The `timeout` channel is buffered with space for 1 value, allowing the timeout goroutine to send to the channel and then exit. The goroutine doesn’t know (or care) whether the value is received. This means the goroutine won’t hang around forever if the `ch` receive happens before the timeout is reached. The `timeout` channel will eventually be deallocated by the garbage collector. (In this example we used `time.Sleep` to demonstrate the mechanics of goroutines and channels. In real programs you should use `[time.After](/pkg/time/#After)`, a function that returns a channel and sends on that channel after the specified duration.) Let’s look at another variation of this pattern. In this example we have a program that reads from multiple replicated databases simultaneously. The program needs only one of the answers, and it should accept the answer that arrives first. The function `Query` takes a slice of database connections and a `query` string. It queries each of the databases in parallel and returns the first response it receives: func Query(conns []Conn, query string) Result { ch := make(chan Result) for _, conn := range conns { go func(c Conn) { select { case ch <- c.DoQuery(query): default: } }(conn) } return <-ch } In this example, the closure does a non-blocking send, which it achieves by using the send operation in `select` statement with a `default` case. If the send cannot go through immediately the default case will be selected. Making the send non-blocking guarantees that none of the goroutines launched in the loop will hang around. However, if the result arrives before the main function has made it to the receive, the send could fail since no one is ready. This problem is a textbook example of what is known as a [race condition](https://en.wikipedia.org/wiki/Race_condition) , but the fix is trivial. We just make sure to buffer the channel `ch` (by adding the buffer length as the second argument to [make](https://go.dev/pkg/builtin/#make) ), guaranteeing that the first send has a place to put the value. This ensures the send will always succeed, and the first value to arrive will be retrieved regardless of the order of execution. These two examples demonstrate the simplicity with which Go can express complex interactions between goroutines. **Next article:** [Real Go Projects: SmartTwitter and web.go](https://go.dev/blog/smarttwitter) **Previous article:** [Introducing the Go Playground](https://go.dev/blog/playground-intro) **[Blog Index](https://go.dev/blog/all) ** go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Tutorial: Accessing a relational database - The Go Programming Language 1. [Documentation](https://go.dev/doc/) 2. [Tutorials](https://go.dev/doc/tutorial/) 3. [Tutorial: Accessing a relational database](https://go.dev/doc/tutorial/database-access) Tutorial: Accessing a relational database ========================================= This tutorial introduces the basics of accessing a relational database with Go and the `database/sql` package in its standard library. You’ll get the most out of this tutorial if you have a basic familiarity with Go and its tooling. If this is your first exposure to Go, please see [Tutorial: Get started with Go](https://go.dev/doc/tutorial/getting-started) for a quick introduction. The [`database/sql`](https://pkg.go.dev/database/sql) package you’ll be using includes types and functions for connecting to databases, executing transactions, canceling an operation in progress, and more. For more details on using the package, see [Accessing databases](https://go.dev/doc/database/index) . In this tutorial, you’ll create a database, then write code to access the database. Your example project will be a repository of data about vintage jazz records. In this tutorial, you’ll progress through the following sections: 1. Create a folder for your code. 2. Set up a database. 3. Import the database driver. 4. Get a database handle and connect. 5. Query for multiple rows. 6. Query for a single row. 7. Add data. **Note:** For other tutorials, see [Tutorials](https://go.dev/doc/tutorial/index.html) . Prerequisites ------------- * **An installation of the [MySQL](https://dev.mysql.com/doc/mysql-installation-excerpt/5.7/en/) relational database management system (DBMS).** * **An installation of Go.** For installation instructions, see [Installing Go](https://go.dev/doc/install) . * **A tool to edit your code.** Any text editor you have will work fine. * **A command terminal.** Go works well using any terminal on Linux and Mac, and on PowerShell or cmd in Windows. Create a folder for your code ----------------------------- To begin, create a folder for the code you’ll write. 1. Open a command prompt and change to your home directory. On Linux or Mac: $ cd On Windows: C:\> cd %HOMEPATH% For the rest of the tutorial we will show a $ as the prompt. The commands we use will work on Windows too. 2. From the command prompt, create a directory for your code called data-access. $ mkdir data-access $ cd data-access 3. Create a module in which you can manage dependencies you will add during this tutorial. Run the `go mod init` command, giving it your new code’s module path. $ go mod init example/data-access go: creating new go.mod: module example/data-access This command creates a go.mod file in which dependencies you add will be listed for tracking. For more, be sure to see [Managing dependencies](https://go.dev/doc/modules/managing-dependencies) . **Note:** In actual development, you’d specify a module path that’s more specific to your own needs. For more, see [Managing dependencies](https://go.dev/doc/modules/managing-dependencies#naming_module) . Next, you’ll create a database. Set up a database ----------------- In this step, you’ll create the database you’ll be working with. You’ll use the CLI for the DBMS itself to create the database and table, as well as to add data. You’ll be creating a database with data about vintage jazz recordings on vinyl. The code here uses the [MySQL CLI](https://dev.mysql.com/doc/refman/8.0/en/mysql.html) , but most DBMSes have their own CLI with similar features. 1. Open a new command prompt. 2. At the command line, log into your DBMS, as in the following example for MySQL. $ mysql -u root -p Enter password: mysql> 3. At the `mysql` command prompt, create a database. mysql> create database recordings; 4. Change to the database you just created so you can add tables. mysql> use recordings; Database changed 5. In your text editor, in the data-access folder, create a file called create-tables.sql to hold SQL script for adding tables. 6. Into the file, paste the following SQL code, then save the file. DROP TABLE IF EXISTS album; CREATE TABLE album ( id INT AUTO_INCREMENT NOT NULL, title VARCHAR(128) NOT NULL, artist VARCHAR(255) NOT NULL, price DECIMAL(5,2) NOT NULL, PRIMARY KEY (`id`) ); INSERT INTO album (title, artist, price) VALUES ('Blue Train', 'John Coltrane', 56.99), ('Giant Steps', 'John Coltrane', 63.99), ('Jeru', 'Gerry Mulligan', 17.99), ('Sarah Vaughan', 'Sarah Vaughan', 34.98); In this SQL code, you: * Delete (drop) a table called `album`. Executing this command first makes it easier for you to re-run the script later if you want to start over with the table. * Create an `album` table with four columns: `title`, `artist`, and `price`. Each row’s `id` value is created automatically by the DBMS. * Add four rows with values. 7. From the `mysql` command prompt, run the script you just created. You’ll use the `source` command in the following form: mysql> source /path/to/create-tables.sql 8. At your DBMS command prompt, use a `SELECT` statement to verify you’ve successfully created the table with data. mysql> select * from album; +----+---------------+----------------+-------+ | id | title | artist | price | +----+---------------+----------------+-------+ | 1 | Blue Train | John Coltrane | 56.99 | | 2 | Giant Steps | John Coltrane | 63.99 | | 3 | Jeru | Gerry Mulligan | 17.99 | | 4 | Sarah Vaughan | Sarah Vaughan | 34.98 | +----+---------------+----------------+-------+ 4 rows in set (0.00 sec) Next, you’ll write some Go code to connect so you can query. Find and import a database driver --------------------------------- Now that you’ve got a database with some data, get your Go code started. Locate and import a database driver that will translate requests you make through functions in the `database/sql` package into requests the database understands. 1. In your browser, visit the [SQLDrivers](https://go.dev/wiki/SQLDrivers) wiki page to identify a driver you can use. Use the list on the page to identify the driver you’ll use. For accessing MySQL in this tutorial, you’ll use [Go-MySQL-Driver](https://github.com/go-sql-driver/mysql/) . 2. Note the package name for the driver – here, `github.com/go-sql-driver/mysql`. 3. Using your text editor, create a file in which to write your Go code and save the file as main.go in the data-access directory you created earlier. 4. Into main.go, paste the following code to import the driver package. package main import "github.com/go-sql-driver/mysql" In this code, you: * Add your code to a `main` package so you can execute it independently. * Import the MySQL driver `github.com/go-sql-driver/mysql`. With the driver imported, you’ll start writing code to access the database. Get a database handle and connect --------------------------------- Now write some Go code that gives you database access with a database handle. You’ll use a pointer to an `sql.DB` struct, which represents access to a specific database. #### Write the code 1. Into main.go, beneath the `import` code you just added, paste the following Go code to create a database handle. var db *sql.DB func main() { // Capture connection properties. cfg := mysql.NewConfig() cfg.User = os.Getenv("DBUSER") cfg.Passwd = os.Getenv("DBPASS") cfg.Net = "tcp" cfg.Addr = "127.0.0.1:3306" cfg.DBName = "recordings" // Get a database handle. var err error db, err = sql.Open("mysql", cfg.FormatDSN()) if err != nil { log.Fatal(err) } pingErr := db.Ping() if pingErr != nil { log.Fatal(pingErr) } fmt.Println("Connected!") } In this code, you: * Declare a `db` variable of type [`*sql.DB`](https://pkg.go.dev/database/sql#DB) . This is your database handle. Making `db` a global variable simplifies this example. In production, you’d avoid the global variable, such as by passing the variable to functions that need it or by wrapping it in a struct. * Use the MySQL driver’s [`Config`](https://pkg.go.dev/github.com/go-sql-driver/mysql#Config) – and the type’s [`FormatDSN`](https://pkg.go.dev/github.com/go-sql-driver/mysql#Config.FormatDSN) -– to collect connection properties and format them into a DSN for a connection string. The `Config` struct makes for code that’s easier to read than a connection string would be. * Call [`sql.Open`](https://pkg.go.dev/database/sql#Open) to initialize the `db` variable, passing the return value of `FormatDSN`. * Check for an error from `sql.Open`. It could fail if, for example, your database connection specifics weren’t well-formed. To simplify the code, you’re calling `log.Fatal` to end execution and print the error to the console. In production code, you’ll want to handle errors in a more graceful way. * Call [`DB.Ping`](https://pkg.go.dev/database/sql#DB.Ping) to confirm that connecting to the database works. At run time, `sql.Open` might not immediately connect, depending on the driver. You’re using `Ping` here to confirm that the `database/sql` package can connect when it needs to. * Check for an error from `Ping`, in case the connection failed. * Print a message if `Ping` connects successfully. 2. Near the top of the main.go file, just beneath the package declaration, import the packages you’ll need to support the code you’ve just written. The top of the file should now look like this: package main import ( "database/sql" "fmt" "log" "os" "github.com/go-sql-driver/mysql" ) 3. Save main.go. #### Run the code 1. Begin tracking the MySQL driver module as a dependency. Use the [`go get`](https://go.dev/cmd/go/#hdr-Add_dependencies_to_current_module_and_install_them) to add the github.com/go-sql-driver/mysql module as a dependency for your own module. Use a dot argument to mean “get dependencies for code in the current directory.” $ go get . go: added filippo.io/edwards25519 v1.1.0 go: added github.com/go-sql-driver/mysql v1.8.1 Go downloaded this dependency because you added it to the `import` declaration in the previous step. For more about dependency tracking, see [Adding a dependency](https://go.dev/doc/modules/managing-dependencies#adding_dependency) . 2. From the command prompt, set the `DBUSER` and `DBPASS` environment variables for use by the Go program. On Linux or Mac: $ export DBUSER=username $ export DBPASS=password On Windows: C:\Users\you\data-access> set DBUSER=username C:\Users\you\data-access> set DBPASS=password 3. From the command line in the directory containing main.go, run the code by typing `go run` with a dot argument to mean “run the package in the current directory.” $ go run . Connected! You can connect! Next, you’ll query for some data. Query for multiple rows ----------------------- In this section, you’ll use Go to execute an SQL query designed to return multiple rows. For SQL statements that might return multiple rows, you use the `Query` method from the `database/sql` package, then loop through the rows it returns. (You’ll learn how to query for a single row later, in the section [Query for a single row](https://go.dev/doc/tutorial/database-access#single_row) .) #### Write the code 1. Into main.go, immediately above `func main`, paste the following definition of an `Album` struct. You’ll use this to hold row data returned from the query. type Album struct { ID int64 Title string Artist string Price float32 } 2. Beneath `func main`, paste the following `albumsByArtist` function to query the database. // albumsByArtist queries for albums that have the specified artist name. func albumsByArtist(name string) ([]Album, error) { // An albums slice to hold data from returned rows. var albums []Album rows, err := db.Query("SELECT * FROM album WHERE artist = ?", name) if err != nil { return nil, fmt.Errorf("albumsByArtist %q: %v", name, err) } defer rows.Close() // Loop through rows, using Scan to assign column data to struct fields. for rows.Next() { var alb Album if err := rows.Scan(&alb.ID, &alb.Title, &alb.Artist, &alb.Price); err != nil { return nil, fmt.Errorf("albumsByArtist %q: %v", name, err) } albums = append(albums, alb) } if err := rows.Err(); err != nil { return nil, fmt.Errorf("albumsByArtist %q: %v", name, err) } return albums, nil } In this code, you: * Declare an `albums` slice of the `Album` type you defined. This will hold data from returned rows. Struct field names and types correspond to database column names and types. * Use [`DB.Query`](https://pkg.go.dev/database/sql#DB.Query) to execute a `SELECT` statement to query for albums with the specified artist name. `Query`’s first parameter is the SQL statement. After the parameter, you can pass zero or more parameters of any type. These provide a place for you to specify the values for parameters in your SQL statement. By separating the SQL statement from parameter values (rather than concatenating them with, say, `fmt.Sprintf`), you enable the `database/sql` package to send the values separate from the SQL text, removing any SQL injection risk. * Defer closing `rows` so that any resources it holds will be released when the function exits. * Loop through the returned rows, using [`Rows.Scan`](https://pkg.go.dev/database/sql#Rows.Scan) to assign each row’s column values to `Album` struct fields. `Scan` takes a list of pointers to Go values, where the column values will be written. Here, you pass pointers to fields in the `alb` variable, created using the `&` operator. `Scan` writes through the pointers to update the struct fields. * Inside the loop, check for an error from scanning column values into the struct fields. * Inside the loop, append the new `alb` to the `albums` slice. * After the loop, check for an error from the overall query, using `rows.Err`. Note that if the query itself fails, checking for an error here is the only way to find out that the results are incomplete. 3. Update your `main` function to call `albumsByArtist`. To the end of `func main`, add the following code. albums, err := albumsByArtist("John Coltrane") if err != nil { log.Fatal(err) } fmt.Printf("Albums found: %v\n", albums) In the new code, you now: * Call the `albumsByArtist` function you added, assigning its return value to a new `albums` variable. * Print the result. #### Run the code From the command line in the directory containing main.go, run the code. $ go run . Connected! Albums found: [{1 Blue Train John Coltrane 56.99} {2 Giant Steps John Coltrane 63.99}] Next, you’ll query for a single row. Query for a single row ---------------------- In this section, you’ll use Go to query for a single row in the database. For SQL statements you know will return at most a single row, you can use `QueryRow`, which is simpler than using a `Query` loop. #### Write the code 1. Beneath `albumsByArtist`, paste the following `albumByID` function. // albumByID queries for the album with the specified ID. func albumByID(id int64) (Album, error) { // An album to hold data from the returned row. var alb Album row := db.QueryRow("SELECT * FROM album WHERE id = ?", id) if err := row.Scan(&alb.ID, &alb.Title, &alb.Artist, &alb.Price); err != nil { if err == sql.ErrNoRows { return alb, fmt.Errorf("albumsById %d: no such album", id) } return alb, fmt.Errorf("albumsById %d: %v", id, err) } return alb, nil } In this code, you: * Use [`DB.QueryRow`](https://pkg.go.dev/database/sql#DB.QueryRow) to execute a `SELECT` statement to query for an album with the specified ID. It returns an `sql.Row`. To simplify the calling code (your code!), `QueryRow` doesn’t return an error. Instead, it arranges to return any query error (such as `sql.ErrNoRows`) from `Rows.Scan` later. * Use [`Row.Scan`](https://pkg.go.dev/database/sql#Row.Scan) to copy column values into struct fields. * Check for an error from `Scan`. The special error `sql.ErrNoRows` indicates that the query returned no rows. Typically that error is worth replacing with more specific text, such as “no such album” here. 2. Update `main` to call `albumByID`. To the end of `func main`, add the following code. // Hard-code ID 2 here to test the query. alb, err := albumByID(2) if err != nil { log.Fatal(err) } fmt.Printf("Album found: %v\n", alb) In the new code, you now: * Call the `albumByID` function you added. * Print the album ID returned. #### Run the code From the command line in the directory containing main.go, run the code. $ go run . Connected! Albums found: [{1 Blue Train John Coltrane 56.99} {2 Giant Steps John Coltrane 63.99}] Album found: {2 Giant Steps John Coltrane 63.99} Next, you’ll add an album to the database. Add data -------- In this section, you’ll use Go to execute an SQL `INSERT` statement to add a new row to the database. You’ve seen how to use `Query` and `QueryRow` with SQL statements that return data. To execute SQL statements that _don’t_ return data, you use `Exec`. #### Write the code 1. Beneath `albumByID`, paste the following `addAlbum` function to insert a new album in the database, then save the main.go. // addAlbum adds the specified album to the database, // returning the album ID of the new entry func addAlbum(alb Album) (int64, error) { result, err := db.Exec("INSERT INTO album (title, artist, price) VALUES (?, ?, ?)", alb.Title, alb.Artist, alb.Price) if err != nil { return 0, fmt.Errorf("addAlbum: %v", err) } id, err := result.LastInsertId() if err != nil { return 0, fmt.Errorf("addAlbum: %v", err) } return id, nil } In this code, you: * Use [`DB.Exec`](https://pkg.go.dev/database/sql#DB.Exec) to execute an `INSERT` statement. Like `Query`, `Exec` takes an SQL statement followed by parameter values for the SQL statement. * Check for an error from the attempt to `INSERT`. * Retrieve the ID of the inserted database row using [`Result.LastInsertId`](https://pkg.go.dev/database/sql#Result.LastInsertId) . * Check for an error from the attempt to retrieve the ID. 2. Update `main` to call the new `addAlbum` function. To the end of `func main`, add the following code. albID, err := addAlbum(Album{ Title: "The Modern Sound of Betty Carter", Artist: "Betty Carter", Price: 49.99, }) if err != nil { log.Fatal(err) } fmt.Printf("ID of added album: %v\n", albID) In the new code, you now: * Call `addAlbum` with a new album, assigning the ID of the album you’re adding to an `albID` variable. #### Run the code From the command line in the directory containing main.go, run the code. $ go run . Connected! Albums found: [{1 Blue Train John Coltrane 56.99} {2 Giant Steps John Coltrane 63.99}] Album found: {2 Giant Steps John Coltrane 63.99} ID of added album: 5 Conclusion ---------- Congratulations! You’ve just used Go to perform simple actions with a relational database. Suggested next topics: * Take a look at the data access guide, which includes more information about the subjects only touched on here. * If you’re new to Go, you’ll find useful best practices described in [Effective Go](https://go.dev/doc/effective_go) and [How to write Go code](https://go.dev/doc/code) . * The [Go Tour](https://go.dev/tour/) is a great step-by-step introduction to Go fundamentals. Completed code -------------- This section contains the code for the application you build with this tutorial. package main import ( "database/sql" "fmt" "log" "os" "github.com/go-sql-driver/mysql" ) var db *sql.DB type Album struct { ID int64 Title string Artist string Price float32 } func main() { // Capture connection properties. cfg := mysql.NewConfig() cfg.User = os.Getenv("DBUSER") cfg.Passwd = os.Getenv("DBPASS") cfg.Net = "tcp" cfg.Addr = "127.0.0.1:3306" cfg.DBName = "recordings" // Get a database handle. var err error db, err = sql.Open("mysql", cfg.FormatDSN()) if err != nil { log.Fatal(err) } pingErr := db.Ping() if pingErr != nil { log.Fatal(pingErr) } fmt.Println("Connected!") albums, err := albumsByArtist("John Coltrane") if err != nil { log.Fatal(err) } fmt.Printf("Albums found: %v\n", albums) // Hard-code ID 2 here to test the query. alb, err := albumByID(2) if err != nil { log.Fatal(err) } fmt.Printf("Album found: %v\n", alb) albID, err := addAlbum(Album{ Title: "The Modern Sound of Betty Carter", Artist: "Betty Carter", Price: 49.99, }) if err != nil { log.Fatal(err) } fmt.Printf("ID of added album: %v\n", albID) } // albumsByArtist queries for albums that have the specified artist name. func albumsByArtist(name string) ([]Album, error) { // An albums slice to hold data from returned rows. var albums []Album rows, err := db.Query("SELECT * FROM album WHERE artist = ?", name) if err != nil { return nil, fmt.Errorf("albumsByArtist %q: %v", name, err) } defer rows.Close() // Loop through rows, using Scan to assign column data to struct fields. for rows.Next() { var alb Album if err := rows.Scan(&alb.ID, &alb.Title, &alb.Artist, &alb.Price); err != nil { return nil, fmt.Errorf("albumsByArtist %q: %v", name, err) } albums = append(albums, alb) } if err := rows.Err(); err != nil { return nil, fmt.Errorf("albumsByArtist %q: %v", name, err) } return albums, nil } // albumByID queries for the album with the specified ID. func albumByID(id int64) (Album, error) { // An album to hold data from the returned row. var alb Album row := db.QueryRow("SELECT * FROM album WHERE id = ?", id) if err := row.Scan(&alb.ID, &alb.Title, &alb.Artist, &alb.Price); err != nil { if err == sql.ErrNoRows { return alb, fmt.Errorf("albumsById %d: no such album", id) } return alb, fmt.Errorf("albumsById %d: %v", id, err) } return alb, nil } // addAlbum adds the specified album to the database, // returning the album ID of the new entry func addAlbum(alb Album) (int64, error) { result, err := db.Exec("INSERT INTO album (title, artist, price) VALUES (?, ?, ?)", alb.Title, alb.Artist, alb.Price) if err != nil { return 0, fmt.Errorf("addAlbum: %v", err) } id, err := result.LastInsertId() if err != nil { return 0, fmt.Errorf("addAlbum: %v", err) } return id, nil } go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Defer, Panic, and Recover - The Go Programming Language [The Go Blog](https://go.dev/blog/) ==================================== Defer, Panic, and Recover ========================= Andrew Gerrand 4 August 2010 Go has the usual mechanisms for control flow: if, for, switch, goto. It also has the go statement to run code in a separate goroutine. Here I’d like to discuss some of the less common ones: defer, panic, and recover. A **defer statement** pushes a function call onto a list. The list of saved calls is executed after the surrounding function returns. Defer is commonly used to simplify functions that perform various clean-up actions. For example, let’s look at a function that opens two files and copies the contents of one file to the other: func CopyFile(dstName, srcName string) (written int64, err error) { src, err := os.Open(srcName) if err != nil { return } dst, err := os.Create(dstName) if err != nil { return } written, err = io.Copy(dst, src) dst.Close() src.Close() return } This works, but there is a bug. If the call to os.Create fails, the function will return without closing the source file. This can be easily remedied by putting a call to src.Close before the second return statement, but if the function were more complex the problem might not be so easily noticed and resolved. By introducing defer statements we can ensure that the files are always closed: func CopyFile(dstName, srcName string) (written int64, err error) { src, err := os.Open(srcName) if err != nil { return } defer src.Close() dst, err := os.Create(dstName) if err != nil { return } defer dst.Close() return io.Copy(dst, src) } Defer statements allow us to think about closing each file right after opening it, guaranteeing that, regardless of the number of return statements in the function, the files _will_ be closed. The behavior of defer statements is straightforward and predictable. There are three simple rules: 1. _A deferred function’s arguments are evaluated when the defer statement is evaluated._ In this example, the expression “i” is evaluated when the Println call is deferred. The deferred call will print “0” after the function returns. func a() { i := 0 defer fmt.Println(i) i++ return } 2. _Deferred function calls are executed in Last In First Out order after the surrounding function returns._ This function prints “3210”: func b() { for i := 0; i < 4; i++ { defer fmt.Print(i) } } 3. _Deferred functions may read and assign to the returning function’s named return values._ In this example, a deferred function increments the return value i _after_ the surrounding function returns. Thus, this function returns 2: func c() (i int) { defer func() { i++ }() return 1 } This is convenient for modifying the error return value of a function; we will see an example of this shortly. **Panic** is a built-in function that stops the ordinary flow of control and begins _panicking_. When the function F calls panic, execution of F stops, any deferred functions in F are executed normally, and then F returns to its caller. To the caller, F then behaves like a call to panic. The process continues up the stack until all functions in the current goroutine have returned, at which point the program crashes. Panics can be initiated by invoking panic directly. They can also be caused by runtime errors, such as out-of-bounds array accesses. **Recover** is a built-in function that regains control of a panicking goroutine. Recover is only useful inside deferred functions. During normal execution, a call to recover will return nil and have no other effect. If the current goroutine is panicking, a call to recover will capture the value given to panic and resume normal execution. Here’s an example program that demonstrates the mechanics of panic and defer: package main import "fmt" func main() { f() fmt.Println("Returned normally from f.") } func f() { defer func() { if r := recover(); r != nil { fmt.Println("Recovered in f", r) } }() fmt.Println("Calling g.") g(0) fmt.Println("Returned normally from g.") } func g(i int) { if i > 3 { fmt.Println("Panicking!") panic(fmt.Sprintf("%v", i)) } defer fmt.Println("Defer in g", i) fmt.Println("Printing in g", i) g(i + 1) } The function g takes the int i, and panics if i is greater than 3, or else it calls itself with the argument i+1. The function f defers a function that calls recover and prints the recovered value (if it is non-nil). Try to picture what the output of this program might be before reading on. The program will output: Calling g. Printing in g 0 Printing in g 1 Printing in g 2 Printing in g 3 Panicking! Defer in g 3 Defer in g 2 Defer in g 1 Defer in g 0 Recovered in f 4 Returned normally from f. If we remove the deferred function from f the panic is not recovered and reaches the top of the goroutine’s call stack, terminating the program. This modified program will output: Calling g. Printing in g 0 Printing in g 1 Printing in g 2 Printing in g 3 Panicking! Defer in g 3 Defer in g 2 Defer in g 1 Defer in g 0 panic: 4 panic PC=0x2a9cd8 [stack trace omitted] For a real-world example of **panic** and **recover**, see the [json package](https://go.dev/pkg/encoding/json/) from the Go standard library. It encodes an interface with a set of recursive functions. If an error occurs when traversing the value, panic is called to unwind the stack to the top-level function call, which recovers from the panic and returns an appropriate error value (see the ’error’ and ‘marshal’ methods of the encodeState type in [encode.go](https://go.dev/src/pkg/encoding/json/encode.go) ). The convention in the Go libraries is that even when a package uses panic internally, its external API still presents explicit error return values. Other uses of **defer** (beyond the file.Close example given earlier) include releasing a mutex: mu.Lock() defer mu.Unlock() printing a footer: printHeader() defer printFooter() and more. In summary, the defer statement (with or without panic and recover) provides an unusual and powerful mechanism for control flow. It can be used to model a number of features implemented by special-purpose structures in other programming languages. Try it out. **Next article:** [Go Wins 2010 Bossie Award](https://go.dev/blog/bossie) **Previous article:** [Share Memory By Communicating](https://go.dev/blog/codelab-share) **[Blog Index](https://go.dev/blog/all) ** go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # About the go command - The Go Programming Language About the go command ==================== The Go distribution includes a command, named "`[go](https://go.dev/cmd/go/) `", that automates the downloading, building, installation, and testing of Go packages and commands. This document talks about why we wrote a new command, what it is, what it's not, and how to use it. Motivation ---------- You might have seen early Go talks in which Rob Pike jokes that the idea for Go arose while waiting for a large Google server to compile. That really was the motivation for Go: to build a language that worked well for building the large software that Google writes and runs. It was clear from the start that such a language must provide a way to express dependencies between code libraries clearly, hence the package grouping and the explicit import blocks. It was also clear from the start that you might want arbitrary syntax for describing the code being imported; this is why import paths are string literals. An explicit goal for Go from the beginning was to be able to build Go code using only the information found in the source itself, not needing to write a makefile or one of the many modern replacements for makefiles. If Go needed a configuration file to explain how to build your program, then Go would have failed. At first, there was no Go compiler, and the initial development focused on building one and then building libraries for it. For expedience, we postponed the automation of building Go code by using make and writing makefiles. When compiling a single package involved multiple invocations of the Go compiler, we even used a program to write the makefiles for us. You can find it if you dig through the repository history. The purpose of the new go command is our return to this ideal, that Go programs should compile without configuration or additional effort on the part of the developer beyond writing the necessary import statements. Configuration versus convention ------------------------------- The way to achieve the simplicity of a configuration-free system is to establish conventions. The system works only to the extent that those conventions are followed. When we first launched Go, many people published packages that had to be installed in certain places, under certain names, using certain build tools, in order to be used. That's understandable: that's the way it works in most other languages. Over the last few years we consistently reminded people about the `goinstall` command (now replaced by [`go get`](https://go.dev/cmd/go/#hdr-Download_and_install_packages_and_dependencies) ) and its conventions: first, that the import path is derived in a known way from the URL of the source code; second, that the place to store the sources in the local file system is derived in a known way from the import path; third, that each directory in a source tree corresponds to a single package; and fourth, that the package is built using only information in the source code. Today, the vast majority of packages follow these conventions. The Go ecosystem is simpler and more powerful as a result. We received many requests to allow a makefile in a package directory to provide just a little extra configuration beyond what's in the source code. But that would have introduced new rules. Because we did not accede to such requests, we were able to write the go command and eliminate our use of make or any other build system. It is important to understand that the go command is not a general build tool. It cannot be configured and it does not attempt to build anything but Go packages. These are important simplifying assumptions: they simplify not only the implementation but also, more important, the use of the tool itself. Go's conventions ---------------- The `go` command requires that code adheres to a few key, well-established conventions. First, the import path is derived in a known way from the URL of the source code. For Bitbucket, GitHub, Google Code, and Launchpad, the root directory of the repository is identified by the repository's main URL, without the `https://` prefix. Subdirectories are named by adding to that path. For example, the source code of the Google logging package `glog` is obtained by running git clone https://github.com/golang/glog and thus the import path of the [glog](https://pkg.go.dev/github.com/golang/glog) package is "`github.com/golang/glog`". These paths are on the long side, but in exchange we get an automatically managed name space for import paths and the ability for a tool like the go command to look at an unfamiliar import path and deduce where to obtain the source code. Second, the place to store sources in the local file system is derived in a known way from the import path, specifically `$GOPATH/src/`. If unset, `$GOPATH` defaults to a subdirectory named `go` in the user's home directory. If `$GOPATH` is set to a list of paths, the go command tries `/src/` for each of the directories in that list. Each of those trees contains, by convention, a top-level directory named "`bin`", for holding compiled executables, and a top-level directory named "`pkg`", for holding compiled packages that can be imported, and the "`src`" directory, for holding package source files. Imposing this structure lets us keep each of these directory trees self-contained: the compiled form and the sources are always near each other. These naming conventions also let us work in the reverse direction, from a directory name to its import path. This mapping is important for many of the go command's subcommands, as we'll see below. Third, each directory in a source tree corresponds to a single package. By restricting a directory to a single package, we don't have to create hybrid import paths that specify first the directory and then the package within that directory. Also, most file management tools and UIs work on directories as fundamental units. Tying the fundamental Go unit—the package—to file system structure means that file system tools become Go package tools. Copying, moving, or deleting a package corresponds to copying, moving, or deleting a directory. Fourth, each package is built using only the information present in the source files. This makes it much more likely that the tool will be able to adapt to changing build environments and conditions. For example, if we allowed extra configuration such as compiler flags or command line recipes, then that configuration would need to be updated each time the build tools changed; it would also be inherently tied to the use of a specific toolchain. Getting started with the go command ----------------------------------- Finally, a quick tour of how to use the go command. As mentioned above, the default `$GOPATH` on Unix is `$HOME/go`. We'll store our programs there. To use a different location, you can set `$GOPATH`; see [How to Write Go Code](https://go.dev/doc/code.html) for details. We first add some source code. Suppose we want to use the indexing library from the codesearch project along with a left-leaning red-black tree. We can install both with the "`go get`" subcommand: $ go get github.com/google/codesearch/index $ go get github.com/petar/GoLLRB/llrb $ Both of these projects are now downloaded and installed into `$HOME/go`, which contains the two directories `src/github.com/google/codesearch/index/` and `src/github.com/petar/GoLLRB/llrb/`, along with the compiled packages (in `pkg/`) for those libraries and their dependencies. Because we used version control systems (Mercurial and Git) to check out the sources, the source tree also contains the other files in the corresponding repositories, such as related packages. The "`go list`" subcommand lists the import paths corresponding to its arguments, and the pattern "`./...`" means start in the current directory ("`./`") and find all packages below that directory ("`...`"): $ cd $HOME/go/src $ go list ./... github.com/google/codesearch/cmd/cgrep github.com/google/codesearch/cmd/cindex github.com/google/codesearch/cmd/csearch github.com/google/codesearch/index github.com/google/codesearch/regexp github.com/google/codesearch/sparse github.com/petar/GoLLRB/example github.com/petar/GoLLRB/llrb $ We can also test those packages: $ go test ./... ? github.com/google/codesearch/cmd/cgrep \[no test files\] ? github.com/google/codesearch/cmd/cindex \[no test files\] ? github.com/google/codesearch/cmd/csearch \[no test files\] ok github.com/google/codesearch/index 0.203s ok github.com/google/codesearch/regexp 0.017s ? github.com/google/codesearch/sparse \[no test files\] ? github.com/petar/GoLLRB/example \[no test files\] ok github.com/petar/GoLLRB/llrb 0.231s $ If a go subcommand is invoked with no paths listed, it operates on the current directory: $ cd github.com/google/codesearch/regexp $ go list github.com/google/codesearch/regexp $ go test -v === RUN TestNstateEnc --- PASS: TestNstateEnc (0.00s) === RUN TestMatch --- PASS: TestMatch (0.00s) === RUN TestGrep --- PASS: TestGrep (0.00s) PASS ok github.com/google/codesearch/regexp 0.018s $ go install $ That "`go install`" subcommand installs the latest copy of the package into the pkg directory. Because the go command can analyze the dependency graph, "`go install`" also installs any packages that this package imports but that are out of date, recursively. Notice that "`go install`" was able to determine the name of the import path for the package in the current directory, because of the convention for directory naming. It would be a little more convenient if we could pick the name of the directory where we kept source code, and we probably wouldn't pick such a long name, but that ability would require additional configuration and complexity in the tool. Typing an extra directory name or two is a small price to pay for the increased simplicity and power. Limitations ----------- As mentioned above, the go command is not a general-purpose build tool. In particular, it does not have any facility for generating Go source files _during_ a build, although it does provide [`go` `generate`](https://go.dev/cmd/go/#hdr-Generate_Go_files_by_processing_source) , which can automate the creation of Go files _before_ the build. For more advanced build setups, you may need to write a makefile (or a configuration file for the build tool of your choice) to run whatever tool creates the Go files and then check those generated source files into your repository. This is more work for you, the package author, but it is significantly less work for your users, who can use "`go get`" without needing to obtain and build any additional tools. More information ---------------- For more information, read [How to Write Go Code](https://go.dev/doc/code.html) and see the [go command documentation](https://go.dev/cmd/go/) . go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Go Wiki: GoUserGroups - The Go Programming Language Go Wiki: GoUserGroups ===================== Africa ------ * [Golang Buea](https://meetup.com/Docker-Buea) - Buea, Cameroon * [Go Egypt](https://web.facebook.com/groups/563213043835298/) - Cairo, Egypt. * [Go Cape Town](http://www.meetup.com/gocapetown/) - Cape Town. * [golang-dakar](https://plus.google.com/u/0/communities/116552609416802012105) Dakar (Sénégal) * [Joburg Golang Group](http://www.meetup.com/Joburg-Golang-Group/) - Johannesburg * [angolang](https://plus.google.com/u/0/b/112645881964422842789/communities/111727796450195570970) Luanda, Angola * [Golang Nigeria](https://www.meetup.com/GolangNigeria/) - Lagos, Nigeria * [Golang Abuja](https://www.meetup.com/golang-abuja/) - Abuja, Nigeria * [Gophers Mauritius](https://www.gophers.mu/) - Mauritius * [cyberstorm.mu](https://cyberstorm.mu/) - Mauritius * [Tunisian Gophers](https://www.facebook.com/TunisiaGolangcommunity/) - Tunisia * [Nairobi Gophers](https://www.meetup.com/nairobi-gophers/) - Nairobi, Kenya Asia ---- * [Golang ِArabic Community](https://www.facebook.com/groups/111860102810970/) * [Golang Azerbaijan](https://www.facebook.com/groups/852343094928521/) - Baku, Azerbaijan 🇦🇿 * [Golang Iraq](https://www.facebook.com/groups/go.iraq/) - Iraq * [Indonesia](https://golang-id.org/) - Indonesia * [GoJakarta](https://gophers.id/GoJakarta) - Jakarta, Indonesia. * [Golang Surabaya](https://github.com/golangSurabaya) - Surabaya, Indonesia. * Persia * [Telegram Iranian Group](https://t.me/GolangFarsi) - Iran * [devheroes Iran](https://devheroes.club/c/go) - Iran * India * [Golang New Delhi](https://www.meetup.com/GolangNewDelhi) - New Delhi, India. * [Golang Bangalore](https://www.meetup.com/Golang-Bangalore) - Bangalore, India. * [Golang Chennai](http://www.meetup.com/Chennai-golang-Meetup/) - Chennai, India * [Golang Pune](https://www.meetup.com/Golang-Pune/) - Pune, India. * [Golang Ahmedabad](https://www.meetup.com/Golang-Ahmedabad) - Ahmedabad, India. * [Golang Kolkata](https://www.facebook.com/groups/125791539385850) - Kolkata, India. * Japan * [Gophers Japan](https://gocon.jp/) - Japan. * [golang.tokyo](https://golangtokyo.github.io/) - Tokyo, Japan. * [Akiba.go](https://akihabarago.connpass.com/) - Tokyo, Japan. * [Women Who Go Tokyo](https://web.womenwhogo.tokyo/) - Tokyo, Japan. * [kyoto.go](https://kyotogo.connpass.com/) - Kyoto, Japan. * [Umeda.go](https://umedago.connpass.com/) - Osaka, Japan. * [Sendai.go](https://techplay.jp/community/sendaigo) - Miyagi, Japan. * [Fukuoka.go](https://fukuokago.dev/) - Fukuoka, Japan. * [Okayama.go](https://okayamago.connpass.com/) - Okayama, Japan. * [nagoya.go](https://nagoyago.connpass.com/) - Aichi, Japan. * [Shizuoka.go](https://shizuoka-go.connpass.com/) - Shizuoka, Japan. * [Golang札幌](https://golang-sapporo.connpass.com/) - Hokkaido, Japan. * [Gopher道場](https://gopherdojo.org/) - Japan. * [Go Language Specification 輪読会](https://gospecreading.connpass.com/) - Japan. * [ゴリラ.Go](https://gorilla-go.connpass.com/) - Japan. * Other * [Golang China](http://groups.google.com/group/golang-china) - China. * [Golang Korea](https://www.facebook.com/groups/golangko/about/) - Korea. * [Golang Vietnam](https://www.facebook.com/golang.org.vn) - [github](https://github.com/golang-vietnam) - Vietnam * [Golang Taiwan](http://golang.tw/) - Taiwan. * [Golang Israel](http://www.meetup.com/Go-Israel) - Israel. * [Singapore Gophers](https://www.facebook.com/groups/golang.sg/) - [Meetup](http://www.meetup.com/golangsg/) - Singapore * [Golang UAE](https://plus.google.com/communities/114845275110994424259) - UAE. * [Golang Malaysia](https://www.facebook.com/groups/mygolang/) - [Homepage](https://golang.my/) - Malaysia. * [Go Developers Network Bangladesh](https://www.facebook.com/groups/godevnet/) - Bangladesh. * [Golang Pakistan](https://www.facebook.com/gopherpk/) - [Facebook Group](https://www.facebook.com/groups/2547735885480597) - Pakistan Europe ------ * [Bärner Go Meetup](https://www.meetup.com/Berner-Go-Meetup/) - Berne, Switzerland * [Budapest Go Meetup](https://www.meetup.com/go-budapest/) - Budapest, Hungary * [Athens Gophers](https://www.meetup.com/Athens-Gophers/) - Athens, Greece. * [Golang Türkiye](https://www.facebook.com/groups/GolangTurkiye) - Istanbul, Turkey. * [Gophers Aachen](https://www.meetup.com/Gophers-Aachen/) - Aachen, Germany. * [Amsterdam, NL](http://www.meetup.com/golang-amsterdam/) * [Belarus](http://gophers.by/) - Belarus. * [Belfast Gophers](https://www.meetup.com/Belfast-Gophers/) - Belfast, UK. * [Barcelona](http://golangbcn.org/) - Barcelona, Catalunya, Spain. * [Brno Golang](http://www.meetup.com/Golang-Brno/) - Brno, the Czech Republic * [Golang Prague](https://www.meetup.com/Prague-Golang-Meetup/) - Prague, Czech Republic. * [Cambridge Gophers](http://www.meetup.com/Cambridge-Gophers/) - Cambridge, UK * [Suffolk Gophers](https://suffolkgophers.github.io/) - Suffolk, UK * [Stuttgart Gophers](https://www.meetup.com/Stuttgart-Gophers/) - Stuttgart, Germany. * [GDG Berlin Golang](http://www.meetup.com/golang-users-berlin/) - Berlin, Germany. * [Go-User-Group-Hamburg](http://www.meetup.com/Go-User-Group-Hamburg) - Hamburg, Germany. * [Go-User-Group Rhein-Ruhr](https://www.meetup.com/Go-Usergroup-Rhein-Ruhr) - Rhine-Ruhr region, Germany. * [Go Lithuania User Group](http://gophers.lt/) - Lithuania, Kaunas. * [Go London User Group](http://www.meetup.com/Go-London-User-Group/) - London, UK. * [ManGo The Manchester Go User Group](http://mango.computer/) - Manchester, UK. * [Golang Dorset](https://www.meetup.com/Golang-Dorset/) - Bournemouth, UK. * [Golang Bristol++](https://www.meetup.com/golang-bristol/) - Bristol, UK. * [GoSheffield](https://www.meetup.com/GoSheffield/) - Sheffield, UK. * [GoMAD](https://www.meetup.com/go-mad/) - Madrid, Spain * [Go Valhalla](https://t.me/joinchat/AAAAAEJGQn7K_oRUBWaNng) - Valencia, Spain. 🇻🇳 * [Go Wales User Group](http://golang.cymru/) - Wales, UK. * [Golang Москва](http://www.meetup.com/Golang-Moscow/) - Moscow, Russia. * [Golang Питер](http://www.meetup.com/Golang-Peter/) - Saint Petersburg, Russia. * [Golang Новосибирск](http://www.meetup.com/GolangNSK/) - Novosibirsk, Russia. * [Golang Казань](https://www.meetup.com/Golang-Kazan/) - Kazan, Russia. * [Go Yola](https://golang-yola.timepad.ru/) - Yoshkar-Ola, Russia. * [Chisinau Golang Meetup](http://www.meetup.com/Chisinau-Golang-Meetup/) - Chisinau, Moldova * [Munich Gophers](http://www.meetup.com/Munich-Gophers-Go-User-Group/) - Munich, Germany * [Golang Paris](http://www.meetup.com/Golang-Paris) - Paris, France. * [Golang Rennes](http://www.meetup.com/Golang-Rennes/) - Rennes, France. * [Golang Lyon](http://www.meetup.com/fr-FR/Golang-Lyon/) - Lyon, France. * [Golang Lille](http://www.meetup.com/Golang-Lille) - Lille, France. * [Golang Marseille](https://www.meetup.com/Golang-Marseille/) - Marseille, France. * [golang-pl](https://groups.google.com/forum/?fromgroups#!forum/golang-pl) - Poland. * [Gophers Katowice](http://www.meetup.com/Gophers-Katowice) - Katowice, Poland. * [Golang Warsaw](http://www.meetup.com/Golang-Warsaw) - Warsaw, Poland. * [G.L.U.G. Wroclaw](http://www.meetup.com/GoLang-User-Group-Wroclaw/) - Wroclaw, Poland * [Golang User Group Trójmiasto](https://www.meetup.com/Golang-User-Group-Trojmiasto/) - Gdańsk/Gdynia/Sopot, Poland * [Go-Stockholm](http://www.meetup.com/Go-Stockholm/) - Stockholm, Sweden. * [Go-Uppsala](http://www.meetup.com/Go-Uppsala/) - Uppsala, Sweden. * [Go-Malmö](http://www.meetup.com/Go-Malmo/) - Malmö, Sweden. * [golang-greece](https://groups.google.com/forum/#!forum/golang-greece) - Greece * [GolangIT](http://golangit.github.io/) - Italy. * [Go-Turkey](https://plus.google.com/communities/101920753066440157216) - Turkey. * [Go Graz](http://gograz.org/) - Graz, Austria * [Meetup Belgium](http://www.meetup.com/Golang-Belgium/) - Belgium * [Meetup Lausanne](http://www.meetup.com/Lausanne-golang-Meetup) - Lausanne, Switzerland * [Meetup Zurich](http://www.meetup.com/Zurich-Gophers/) - Zurich, Switzerland * [Meetup Frankfurt](http://www.meetup.com/Frankfurt-Gophers-Meetup/) - Frankfurt am Main, Germany * [Go-ningen](http://www.meetup.com/Go-ningen/) - Groningen, The Netherlands * [Golang Ljubljana](http://www.meetup.com/Slovenian-Go-lang-User-Group/) - Ljubljana, Slovenia * [Go Euregio](https://plus.google.com/communities/116272759718686417490) - Maastricht, Netherlands; Liège/Hasselt, Belgium; Aachen, Germany * [Ukrainian Golang User Group](http://www.meetup.com/uagolang/) - Kyiv, Ukraine * [L’viv Golang Group](http://www.meetup.com/Lviv-Golang-Group/) - L’viv, Ukraine * [Go SXB, Go!](http://www.meetup.com/fr-FR/Go-SXB-Go/) - Strasbourg, France * [Hannover Gophers Meetup](http://www.meetup.com/de-DE/Hannover-Gophers-Meetup/) - Hannover, Germany * [Golang Nürnberg](http://www.meetup.com/de-DE/Golang-Nuernberg/) - Nürnberg, Germany * [Helsinki Gophers](http://www.meetup.com/Helsinki-Gophers/) - Helsinki, Finland * [Golang Cologne](http://www.meetup.com/Golang-Cologne/) - Cologne/Bonn, Germany * [Golang Vilnius](http://www.meetup.com/Vilnius-Golang/) - Vilnius, Lithuania * [Meetup Surrey](http://www.meetup.com/Surrey-Go-User-Group-Meetup/) - Surrey, UK * [Golang Zagreb](https://www.meetup.com/Golang-ZG/) - Zagreb, Croatia * [Meetup Edinburgh](https://www.meetup.com/Edinburgh-Golang-meetup/) - Edinburgh, UK * [Münster Gophers](https://www.meetup.com/de-DE/Munster-Gophers/) - Münster, Germany * [Vienna.go - Vienna Go User Group](https://www.meetup.com/Vienna-go-Vienna-Go-User-Group/events/242583645/) - Vienna, Austria * [Golang Bulgaria](https://www.meetup.com/Golang-Bulgaria/) - Sofia, Bulgaria 🇧🇬 * [Go Oslo User Group](https://www.meetup.com/Go-Oslo-User-Group/) - Oslo, Norway * [Gophers Linz](https://www.meetup.com/Gophers-Linz/) - Linz, Austria * [Golang Rotterdam](https://www.meetup.com/Golang-Rotterdam/) - Rotterdam, Netherlands * [Leipzig Gophers](https://golangleipzig.space/) - Leipzig, Germany * [Golang Estonia](https://www.meetup.com/Golang-Estonia/) - Estonia North America ------------- ### Canada * [Edmonton Go Meetup](https://edmontongo.org/) - Edmonton AB, Canada. * [vangophers](http://groups.google.com/group/vangophers) - Vancouver BC, Canada. * [Go Vancouver](https://plus.google.com/u/0/communities/106063002572645508555) - Vancouver BC, Canada. * [GolangVan meetup](http://www.meetup.com/golangvan/) - Vancouver BC, Canada. * [GolangVan](http://golangvan.org/) - Vancouver BC, Canada. * [GoTO](http://www.meetup.com/go-toronto) - Toronto ON, Canada. * [Ottawa Go](http://www.meetup.com/Ottawa-Go-Meetup/) - Ottawa ON, Canada. * [KW Go Developers](http://www.meetup.com/Golang-KW/) - Waterloo, ON, Canada. * [Golang Montréal](https://golangmontreal.org/) - Montréal, QC * [Golang Winnipeg](https://www.meetup.com/Golang-Wpg-Meetup/) - Winnipeg MB, Canada ### Dominican Republic * [Golang Dominicana](https://www.facebook.com/groups/golangdominicana) - República Dominicana. ### Latin America * [Golang-ES](https://www.facebook.com/groups/goenespanol) - Facebook Español-Latino. * [Gophers LATAM](https://discord.com/invite/AEarh2kSvn) - Discord Español-Latino. ### Mexico * [Golang - Go MX](https://www.facebook.com/groups/es.golang.mx) - All Mexico * [GophersMX](https://www.meetup.com/es/GophersMX/) - Ciudad de México, México * [Golang Monterrey](http://www.meetup.com/Golang-MTY/) - Monterrey, Mexico * [Golang Guadalajara](https://www.meetup.com/es/Golang-Guadalajara/) - Guadalajara, México ### United States * [Los-Angeles-Gophers](http://www.meetup.com/Los-Angeles-Gophers/) - Los Angeles CA, USA. * [Westside GoLang Meetup](https://www.meetup.com/Westside-GoLang-Meetup/) - Los Angeles CA, USA. * [OC Gophers](http://www.meetup.com/Orange-County-Gophers) - Orange County CA, USA. * [SDGophers](http://www.meetup.com/sdgophers/) — San Diego CA, USA * [GoSF](http://www.meetup.com/golangsf/) - San Francisco CA, USA. * [GoSV](http://www.meetup.com/GolangSV/) - San Mateo CA, USA. * [Boulder Gophers](https://www.meetup.com/Boulder-Gophers/) - Boulder CO, USA. * [Denver Go Language User Group](http://www.meetup.com/Denver-Go-Language-User-Group/) - Denver CO, USA. * [DTC Go Meetup](https://www.meetup.com/Denver-Go-Programming-Language-Meetup/) - Denver Tech Center CO, USA. * [Go-Miami](http://www.meetup.com/Go-Miami/) - Miami FL, USA. * [Orlando Go Users](https://www.meetup.com/OrlanGo/) - Orlando FL, USA * [Go-Users-Group-Atlanta](http://www.meetup.com/Go-Users-Group-Atlanta/) - Atlanta GA, USA. * [chicagolang](http://groups.google.com/group/chicagolang) - Chicago IL, USA. * [Chicago Ultimate Golang](https://www.meetup.com/Chicago-Ultimate-Golang/) - Chicago IL, USA. * [Boston Golang](http://bostongolang.org/) - Boston MA, USA. * [Framingham Go](https://www.meetup.com/Framingham-Golang-Meetup/) - Framingham MA, USA. * [GoMN](http://www.meetup.com/golangmn/) - Minneapolis MN, USA. * [Buffalo GoLang Meetup Group](https://www.meetup.com/Buffalo-GoLang-Meetup-Group/) - Buffalo NY, USA. * [New York Go Language Meetup](http://www.meetup.com/golangny/) - New York NY, USA. * [New Jersey Go Language Meetup](http://www.meetup.com/golangnj/) - New Jersey NY, USA. * [Bowery Golang Meetup](https://www.meetup.com/Bowery-Go/) - New York NY, USA. * [GoLangPhilly](http://www.meetup.com/GoLangPhilly/) - Philadelphia PA, USA. * [GoLangPhoenix](http://www.meetup.com/Golang-Phoenix/) - Phoenix AZ, USA. * [GoLangCleveland](https://www.golangcleveland.org/) - Cleveland OH, USA. * [Cincinnati Golang Meetup](https://www.meetup.com/Cincinnati-Golang-Meetup/) - Cincinnati, OH, USA. * [EUG-Go](http://www.meetup.com/EUG-Go/) - Eugene, OR, USA. * [PDX-Go](http://www.meetup.com/PDX-Go/) - Portland, OR, USA. * [GoLancaster](http://www.meetup.com/GoLancaster/) - Lancaster PA, USA. * [ATX-Golang](http://www.meetup.com/atxgolang/) - Austin TX, USA. * [GoDFW](http://www.meetup.com/GoCowboys/) - Dallas TX, USA. * [Golang Houston](http://www.meetup.com/Golang-Houston/) - Houston TX, USA. * [Utah Golang (#UTGO)](http://utahgolang.com/) - Salt Lake City UT, USA. * [Golang-DC](http://www.meetup.com/Golang-DC/) - Arlington VA, USA. * [Seattle Go Programmers](http://www.meetup.com/golang/) - Seattle WA, USA. * [Go Charlotte Meetup](http://www.meetup.com/golangclt/) - Charlotte NC, USA * [Triangle Golang](http://www.meetup.com/Triangle-Golang-Meetup/) - Raleigh NC, USA. * [Las Vegas Go Meetup](http://www.meetup.com/Las-Vegas-Go-Meetup/) - Las Vegas NV, USA. * [Ann Arbor Gophers](http://www.meetup.com/Ann-Arbor-Gophers/) - Ann Arbor MI, USA * [Baltimore Metro Area Golang](http://baltimoregolang.org/) - Baltimore MD, USA * [Nashville Gophers](https://nashgo.org/) - Nashville TN, USA * [Omaha Gophers](https://www.meetup.com/omaha-gophers/) - Omaha NE, USA * [St Louis Go Meetup](https://www.meetup.com/StL-Go/) - St Louis, MO, USA Oceania ------- * [Canberra Gophers](https://plus.google.com/u/1/communities/114036877112593565975) - Canberra, Australia. * [golang-sydney](http://www.meetup.com/golang-syd/) - Sydney, Australia. * [golang-nz](http://groups.google.com/group/golang-nz) - New Zealand. * [Golang Wellington](https://www.meetup.com/wellington-golang/) - Wellington, New Zealand. * [Melbourne-Go-Nuts](http://www.meetup.com/Melbourne-Go-Nuts) - Melbourne, Australia. * [Brisbane Gophers](http://www.meetup.com/Brisbane-Golang-Meetup/) - Brisbane, Australia. * [Honolulu Go Users](http://hnlgo.org/) - Oahu, Hawaii South America ------------- * [Golang Brasil](https://www.meetup.com/pt-BR/golangbr/) - Brazil. * [Golang Rio de Janeiro](https://www.meetup.com/pt-BR/Gophers-Rio) - Rio de Janeiro, Brazil. * [Golang Recife](https://linktr.ee/golang_recife) - Recife, Pernambuco, Brazil. * [Golang Argentina](http://www.meetup.com/es-ES/Golang-Argentina/) - Argentina. * [Golang Medellín](http://www.meetup.com/Golang-Medellin/) - Colombia. * [Golang Chile](https://groups.google.com/d/forum/golang-chile) - Chile. * [Golang Panamá](https://groups.google.com/d/forum/golang-panama) - Panamá. * [Golang Lima](http://www.meetup.com/es/Golang-Peru/) - Perú * [Golang Venezuela](https://t.me/golangve) - Venezuela * * * _This content is part of the [Go Wiki](https://go.dev/wiki/) ._ go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Command Documentation - The Go Programming Language 1. [Documentation](https://go.dev/doc/) 2. [Command Documentation](https://go.dev/doc/cmd) Command Documentation ===================== There is a suite of programs to build and process Go source code. Instead of being run directly, programs in the suite are usually invoked by the [go](https://go.dev/cmd/go/) program. The most common way to run these programs is as a subcommand of the go program, for instance as `go fmt`. Run like this, the command operates on complete packages of Go source code, with the go program invoking the underlying binary with arguments appropriate to package-level processing. The programs can also be run as stand-alone binaries, with unmodified arguments, using the go `tool` subcommand, such as `go tool cgo`. For most commands this is mainly useful for debugging. Some of the commands, such as `pprof`, are accessible only through the go `tool` subcommand. The Go installation process also installs an executable called `gofmt`, equivalent to `go fmt`, because it is so often referenced. Click on the links for more documentation, invocation methods, and usage details. | Name | | Synopsis | | --- | --- | --- | | [go](https://go.dev/cmd/go/) | | The `go` program manages Go source code and runs the other commands listed here. See the command docs for usage details. | | [cgo](https://go.dev/cmd/cgo/) | | Cgo enables the creation of Go packages that call C code. | | [cover](https://go.dev/cmd/cover/) | | Cover is a program for creating and analyzing the coverage profiles generated by `"go test -coverprofile"`. | | [fix](https://go.dev/cmd/fix/) | | Fix finds Go programs that use old features of the language and libraries and rewrites them to use newer ones. | | [fmt](https://go.dev/cmd/gofmt/) | | Fmt formats Go packages, it is also available as an independent [gofmt](https://go.dev/cmd/gofmt/)
command with more general options. | | [doc](https://go.dev/cmd/doc) | | Doc extracts and generates documentation for Go packages. | | [vet](https://go.dev/cmd/vet/) | | Vet examines Go source code and reports suspicious constructs, such as Printf calls whose arguments do not align with the format string. | This is an abridged list. See the [full command reference](https://go.dev/cmd/) for documentation of the compilers and more. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Go Wiki: NonEnglish - The Go Programming Language Go Wiki: NonEnglish =================== Some of this documentation may be out of date. Belarusian - Беларуская ----------------------- * [faq-be](http://www.designcontest.com/show/faq-be) - Frequently Asked Questions. Brazilian Portuguese - Português brasileiro ------------------------------------------- * [A Tour of Go](https://go-tour-br.appspot.com/) * [Go Project](https://pt.docs.dev.br/p/go/) - Go documentation and related pages. * [golangbr.org](http://golangbr.org/) - Go documentation and news. Chinese - 中文 ------------ * [Go 语言之旅](https://tour.go-zh.org/) * [Go 编程语言](https://go-zh.org/) - Chinese Translation of tip.golang.org * [Effective Go and (old) Tutorial (Deprecated)](http://code.google.com/p/ac-me/downloads/detail?name=fango.pdf) Czech - Čeština --------------- * [Pravidla reflexe](http://www.abclinuxu.cz/clanky/google-go-pravidla-reflexe) - a translation of [The Laws of Reflection](https://go.dev/blog/2011/09/laws-of-reflection.html) . French - Français ----------------- * [golang-france](http://code.google.com/p/golang-france/) - Go documentation. German - Deutsch ---------------- * [Deutschsprachige Go Themenseite - German Go resource page](https://github.com/hweidner/golang-de/wiki) * [Deutschsprachiges Diskussionsforum - German discussion forum](https://mewe.com/group/5c4b0dcb1c1ea52d0c45b6b0) on [MeWe](https://mewe.com/) ([Invitation link](https://mewe.com/join/golang-de) ) (requires free registration on MeWe.com). Indonesia --------- * [Komunitas Pengguna Go Indonesia](https://golang-id.org/) Japanese - 日本語 -------------- * [A Tour of Go](https://go-tour-jp.appspot.com/) * [CodeReviewComments](https://knsh14.github.io/translations/go-codereview-comments/) Korean - 한국어 ------------ * [A Tour of Go](http://go-tour-kr.appspot.com/) * [golang-kr wiki](http://github.com/golang-kr/golang-doc/wiki) - Korean Translation of golang.org/doc Russian - русский язык ---------------------- * [Effective Go](https://github.com/Konstantin8105/Effective_Go_RU/blob/master/README.md) * [Contribution Guide](https://github.com/Konstantin8105/Contribution_Guide_RU) Spanish - español ----------------- * [Video course: Go Course from 0 to 100](https://www.youtube.com/watch?v=7SIIyt5-XK0&list=PLl_hIu4u7P64MEJpR3eVwQ1l_FtJq4a5g) * [Workshop video: Creating a wiki with Go](https://www.youtube.com/watch?v=0fYb43gIl6I&list=PLfHn_OMWQAHDNxoA3BRWs5NHcstZMAY_B) * [Sitio web: Go con ejemplos](http://goconejemplos.com/) * [Apuntes: Aprender de Go desde Cero](https://apuntes.de/golang/) Thai - ไทย ---------- * [A Tour of Go](https://go-tour-th.appspot.com/) Turkish - Türkçe ---------------- * [A Tour of Go](https://go-tour-turkish.appspot.com/) Vietnamese - Tiếng Việt ----------------------- * [A Tour of Go](http://go-tour-vi.appspot.com/) Uzbek — Ўзбекча --------------- * [A Tour of Go](http://go-tour-uz.appspot.com/) * * * _This content is part of the [Go Wiki](https://go.dev/wiki/) ._ go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Container-aware GOMAXPROCS - The Go Programming Language [The Go Blog](https://go.dev/blog/) ==================================== Container-aware GOMAXPROCS ========================== Michael Pratt and Carlos Amedee 20 August 2025 Go 1.25 includes new container-aware `GOMAXPROCS` defaults, providing more sensible default behavior for many container workloads, avoiding throttling that can impact tail latency, and improving Go’s out-of-the-box production-readiness. In this post, we will dive into how Go schedules goroutines, how that scheduling interacts with container-level CPU controls, and how Go can perform better with awareness of container CPU controls. `GOMAXPROCS` ------------ One of Go’s strengths is its built-in and easy-to-use concurrency via goroutines. From a semantic perspective, goroutines appear very similar to operating system threads, enabling us to write simple, blocking code. On the other hand, goroutines are more lightweight than operating system threads, making it much cheaper to create and destroy them on the fly. While a Go implementation could map each goroutine to a dedicated operating system thread, Go keeps goroutines lightweight with a runtime scheduler that makes threads fungible. Any Go-managed thread can run any goroutine, so creating a new goroutine doesn’t require creating a new thread, and waking a goroutine doesn’t necessarily require waking another thread. That said, along with a scheduler comes scheduling questions. For example, exactly how many threads should we use to run goroutines? If 1,000 goroutines are runnable, should we schedule them on 1,000 different threads? This is where [`GOMAXPROCS`](https://go.dev/pkg/runtime#GOMAXPROCS) comes in. Semantically, `GOMAXPROCS` tells the Go runtime the “available parallelism” that Go should use. In more concrete terms, `GOMAXPROCS` is the maximum number of threads to use for running goroutines at once. So, if `GOMAXPROCS=8` and there are 1,000 runnable goroutines, Go will use 8 threads to run 8 goroutines at a time. Often, goroutines run for a very short time and then block, at which point Go will switch to running another goroutine on that same thread. Go will also preempt goroutines that don’t block on their own, ensuring all goroutines get a chance to run. From Go 1.5 through Go 1.24, `GOMAXPROCS` defaulted to the total number of CPU cores on the machine. Note that in this post, “core” more precisely means “logical CPU.” For example, a machine with 4 physical CPUs with hyperthreading has 8 logical CPUs. This typically makes a good default for “available parallelism” because it naturally matches the available parallelism of the hardware. That is, if there are 8 cores and Go runs more than 8 threads at a time, the operating system will have to multiplex these threads onto the 8 cores, much like how Go multiplexes goroutines onto threads. This extra layer of scheduling is not always a problem, but it is unnecessary overhead. Container Orchestration ----------------------- Another of Go’s core strengths is the convenience of deploying applications via a container, and managing the number of cores Go uses is especially important when deploying an application within a container orchestration platform. Container orchestration platforms like [Kubernetes](https://kubernetes.io/) take a set of machine resources and schedule containers within the available resources based on requested resources. Packing as many containers as possible within a cluster’s resources requires the platform to be able to predict the resource usage of each scheduled container. We want Go to adhere to the resource utilization constraints that the container orchestration platform sets. Let’s explore the effects of the `GOMAXPROCS` setting in the context of Kubernetes, as an example. Platforms like Kubernetes provide a mechanism to limit the resources consumed by a container. Kubernetes has the concept of CPU resource limits, which signal to the underlying operating system how many core resources a specific container or set of containers will be allocated. Setting a CPU limit translates to the creation of a Linux [control group](https://docs.kernel.org/admin-guide/cgroup-v2.html#cpu) CPU bandwidth limit. Before Go 1.25, Go was unaware of CPU limits set by orchestration platforms. Instead, it would set `GOMAXPROCS` to the number of cores on the machine it was deployed to. If there was a CPU limit in place, the application may try to use far more CPU than allowed by the limit. To prevent an application from exceeding its limit, the Linux kernel will [throttle](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#how-pods-with-resource-limits-are-run) the application. Throttling is a blunt mechanism for restricting containers that would otherwise exceed their CPU limit: it completely pauses application execution for the remainder of the throttling period. The throttling period is typically 100ms, so throttling can cause substantial tail latency impact compared to the softer scheduling multiplexing effects of a lower `GOMAXPROCS` setting. Even if the application never has much parallelism, tasks performed by the Go runtime—such as garbage collection—can still cause CPU spikes that trigger throttling. New default ----------- We want Go to provide efficient and reliable defaults when possible, so in Go 1.25, we have made `GOMAXPROCS` take into account its container environment by default. If a Go process is running inside a container with a CPU limit, `GOMAXPROCS` will default to the CPU limit if it is less than the core count. Container orchestration systems may adjust container CPU limits on the fly, so Go 1.25 will also periodically check the CPU limit and adjust `GOMAXPROCS` automatically if it changes. Both of these defaults only apply if `GOMAXPROCS` is otherwise unspecified. Setting the `GOMAXPROCS` environment variable or calling `runtime.GOMAXPROCS` continues to behave as before. The [`runtime.GOMAXPROCS`](https://go.dev/pkg/runtime#GOMAXPROCS) documentation covers the details of the new behavior. Slightly different models ------------------------- Both `GOMAXPROCS` and a container CPU limit place a limit on the maximum amount of CPU the process can use, but their models are subtly different. `GOMAXPROCS` is a parallelism limit. If `GOMAXPROCS=8` Go will never run more than 8 goroutines at a time. By contrast, CPU limits are a throughput limit. That is, they limit the total CPU time used in some period of wall time. The default period is 100ms. So an “8 CPU limit” is actually a limit of 800ms of CPU time every 100ms of wall time. This limit could be filled by running 8 threads continuously for the entire 100ms, which is equivalent to `GOMAXPROCS=8`. On the other hand, the limit could also be filled by running 16 threads for 50ms each, with each thread being idle or blocked for the other 50ms. In other words, a CPU limit doesn’t limit the total number of CPUs the container can run on. It only limits total CPU time. Most applications have fairly consistent CPU usage across 100ms periods, so the new `GOMAXPROCS` default is a pretty good match to the CPU limit, and certainly better than the total core count! However, it is worth noting that particularly spiky workloads may see a latency increase from this change due to `GOMAXPROCS` preventing short-lived spikes of additional threads beyond the CPU limit average. In addition, since CPU limits are a throughput limit, they can have a fractional component (e.g., 2.5 CPU). On the other hand, `GOMAXPROCS` must be a positive integer. Thus, Go must round the limit to a valid `GOMAXPROCS` value. Go always rounds up to enable use of the full CPU limit. CPU Requests ------------ Go’s new `GOMAXPROCS` default is based on the container’s CPU limit, but container orchestration systems also provide a “CPU request” control. While the CPU limit specifies the maximum CPU a container may use, the CPU request specifies the minimum CPU guaranteed to be available to the container at all times. It is common to create containers with a CPU request but no CPU limit, as this allows containers to utilize machine CPU resources beyond the CPU request that would otherwise be idle due to lack of load from other containers. Unfortunately, this means that Go cannot set `GOMAXPROCS` based on the CPU request, which would prevent utilization of additional idle resources. Containers with a CPU request are still [constrained](https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/#how-pods-with-resource-limits-are-run) when exceeding their request if the machine is busy. The weight-based constraint of exceeding requests is “softer” than the hard period-based throttling of CPU limits, but CPU spikes from high `GOMAXPROCS` can still have an adverse impact on application behavior. Should I set a CPU limit? ------------------------- We have learned about the problems caused by having `GOMAXPROCS` too high, and that setting a container CPU limit allows Go to automatically set an appropriate `GOMAXPROCS`, so an obvious next step is to wonder whether all containers should set a CPU limit. While that may be good advice to automatically get a reasonable `GOMAXPROCS` defaults, there are many other factors to consider when deciding whether to set a CPU limit, such as prioritizing utilization of idle resources by avoiding limits vs prioritizing predictable latency by setting limits. The worst behaviors from a mismatch between `GOMAXPROCS` and effective CPU limits occur when `GOMAXPROCS` is significantly higher than the effective CPU limit. For example, a small container receiving 2 CPUs running on a 128 core machine. These are the cases where it is most valuable to consider setting an explicit CPU limit, or, alternatively, explicitly setting `GOMAXPROCS`. Conclusion ---------- Go 1.25 provides more sensible default behavior for many container workloads by setting `GOMAXPROCS` based on container CPU limits. Doing so avoids throttling that can impact tail latency, improves efficiency, and generally tries to ensure Go is production-ready out-of-the-box. You can get the new defaults simply by setting the Go version to 1.25.0 or higher in your `go.mod`. Thanks to everyone in the community that contributed to the [long](https://go.dev/issue/33803) [discussions](https://go.dev/issue/73193) that made this a reality, and in particular to feedback from the maintainers of [`go.uber.org/automaxprocs`](https://pkg.go.dev/go.uber.org/automaxprocs) from Uber, which has long provided similar behavior to its users. **Next article:** [Testing Time (and other asynchronicities)](https://go.dev/blog/testing-time) **Previous article:** [Go 1.25 is released](https://go.dev/blog/go1.25) **[Blog Index](https://go.dev/blog/all) ** go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Flight Recorder in Go 1.25 - The Go Programming Language [The Go Blog](https://go.dev/blog/) ==================================== Flight Recorder in Go 1.25 ========================== Carlos Amedee and Michael Knyszek 26 September 2025 In 2024 we introduced the world to [more powerful Go execution traces](https://go.dev/blog/execution-traces-2024) . In that blog post we gave a sneak peek into some of the new functionality we could unlock with our new execution tracer, including _flight recording_. We’re happy to announce that flight recording is now available in Go 1.25, and it’s a powerful new tool in the Go diagnostics toolbox. Execution traces ---------------- First, a quick recap on Go execution traces. The Go runtime can be made to write a log recording many of the events that happen during the execution of a Go application. That log is called a runtime execution trace. Go execution traces contain a plethora of information about how goroutines interact with each other and the underlying system. This makes them very handy for debugging latency issues, since they tell you both when your goroutines are executing, and crucially, when they are not. The [runtime/trace](https://go.dev/pkg/runtime/trace) package provides an API for collecting an execution trace over a given time window by calling `runtime/trace.Start` and `runtime/trace.Stop`. This works well if the code you’re tracing is just a test, microbenchmark, or command line tool. You can collect a trace of the complete end-to-end execution, or just the parts you care about. However, in long-running web services, the kinds of applications Go is known for, that’s not good enough. Web servers might be up for days or even weeks, and collecting a trace of the entire execution would produce far too much data to sift through. Often just one part of the program’s execution goes wrong, like a request timing out or a failed health check. By the time it happens it’s already too late to call `Start`! One way to approach this problem is to randomly sample execution traces from across the fleet. While this approach is powerful, and can help find issues before they become outages, it requires a lot of infrastructure to get going. Large quantities of execution trace data would need to be stored, triaged, and processed, much of which won’t contain anything interesting at all. And when you’re trying to get to the bottom of a specific issue, it’s a non-starter. Flight recording ---------------- This brings us to the flight recorder. A program often knows when something has gone wrong, but the root cause may have happened long ago. The flight recorder lets you collect a trace of the last few seconds of execution leading up to the moment a program detects there’s been a problem. The flight recorder collects the execution trace as normal, but instead of writing it out to a socket or a file, it buffers the last few seconds of the trace in memory. At any point, the program can request the contents of the buffer and snapshot exactly the problematic window of time. The flight recorder is like a scalpel cutting directly to the problem area. Example ------- Let’s learn how to use the flight recorder with an example. Specifically, let’s use it to diagnose a performance problem with an HTTP server that implements a “guess the number” game. It exposes a `/guess-number` endpoint that accepts an integer and responds to the caller informing them if they guessed the right number. There is also a goroutine that, once per minute, sends a report of all the guessed numbers to another service via an HTTP request. // bucket is a simple mutex-protected counter. type bucket struct { mu sync.Mutex guesses int } func main() { // Make one bucket for each valid number a client could guess. // The HTTP handler will look up the guessed number in buckets by // using the number as an index into the slice. buckets := make([]bucket, 100) // Every minute, we send a report of how many times each number was guessed. go func() { for range time.Tick(1 * time.Minute) { sendReport(buckets) } }() // Choose the number to be guessed. answer := rand.Intn(len(buckets)) http.HandleFunc("/guess-number", func(w http.ResponseWriter, r *http.Request) { start := time.Now() // Fetch the number from the URL query variable "guess" and convert it // to an integer. Then, validate it. guess, err := strconv.Atoi(r.URL.Query().Get("guess")) if err != nil || !(0 <= guess && guess < len(buckets)) { http.Error(w, "invalid 'guess' value", http.StatusBadRequest) return } // Select the appropriate bucket and safely increment its value. b := &buckets[guess] b.mu.Lock() b.guesses++ b.mu.Unlock() // Respond to the client with the guess and whether it was correct. fmt.Fprintf(w, "guess: %d, correct: %t", guess, guess == answer) log.Printf("HTTP request: endpoint=/guess-number guess=%d duration=%s", guess, time.Since(start)) }) log.Fatal(http.ListenAndServe(":8090", nil)) } // sendReport posts the current state of buckets to a remote service. func sendReport(buckets []bucket) { counts := make([]int, len(buckets)) for index := range buckets { b := &buckets[index] b.mu.Lock() defer b.mu.Unlock() counts[index] = b.guesses } // Marshal the report data into a JSON payload. b, err := json.Marshal(counts) if err != nil { log.Printf("failed to marshal report data: error=%s", err) return } url := "http://localhost:8091/guess-number-report" if _, err := http.Post(url, "application/json", bytes.NewReader(b)); err != nil { log.Printf("failed to send report: %s", err) } } Here is the full code for the server: [https://go.dev/play/p/rX1eyKtVglF](https://go.dev/play/p/rX1eyKtVglF) , and for a simple client: [https://go.dev/play/p/2PjQ-1ORPiw](https://go.dev/play/p/2PjQ-1ORPiw) . To avoid a third process, the “client” also implements the report server, though in a real system this would be separate. Let’s suppose that after deploying the application in production, we received complaints from users that some `/guess-number` calls were taking longer than expected. When we look at our logs, we see that sometimes response times exceed 100 milliseconds, while the majority of calls are on the order of microseconds. 2025/09/19 16:52:02 HTTP request: endpoint=/guess-number guess=69 duration=625ns 2025/09/19 16:52:02 HTTP request: endpoint=/guess-number guess=62 duration=458ns 2025/09/19 16:52:02 HTTP request: endpoint=/guess-number guess=42 duration=1.417µs 2025/09/19 16:52:02 HTTP request: endpoint=/guess-number guess=86 duration=115.186167ms 2025/09/19 16:52:02 HTTP request: endpoint=/guess-number guess=0 duration=127.993375ms Before we continue, take a minute and see if you can spot what’s wrong! Regardless of whether you found the problem or not, let’s dive deeper and see how we can find the problem from first principles. In particular, it would be great if we could see what the application was doing in the time leading up to the slow response. This is exactly what the flight recorder was built for! We’ll use it to capture an execution trace once we see the first response exceeding 100 milliseconds. First, in `main`, we’ll configure and start the flight recorder: // Set up the flight recorder fr := trace.NewFlightRecorder(trace.FlightRecorderConfig{ MinAge: 200 * time.Millisecond, MaxBytes: 1 << 20, // 1 MiB }) fr.Start() `MinAge` configures the duration for which trace data is reliably retained, and we suggest setting it to around 2x the time window of the event. For example, if you are debugging a 5-second timeout, set it to 10 seconds. `MaxBytes` configures the size of the buffered trace so you don’t blow up your memory usage. On average, you can expect a few MB of trace data to be produced per second of execution, or 10 MB/s for a busy service. Next, we’ll add a helper function to capture the snapshot and write it to a file: var once sync.Once // captureSnapshot captures a flight recorder snapshot. func captureSnapshot(fr *trace.FlightRecorder) { // once.Do ensures that the provided function is executed only once. once.Do(func() { f, err := os.Create("snapshot.trace") if err != nil { log.Printf("opening snapshot file %s failed: %s", f.Name(), err) return } defer f.Close() // ignore error // WriteTo writes the flight recorder data to the provided io.Writer. _, err = fr.WriteTo(f) if err != nil { log.Printf("writing snapshot to file %s failed: %s", f.Name(), err) return } // Stop the flight recorder after the snapshot has been taken. fr.Stop() log.Printf("captured a flight recorder snapshot to %s", f.Name()) }) } And finally, just before logging a completed request, we’ll trigger the snapshot if the request took more than 100 milliseconds: // Capture a snapshot if the response takes more than 100ms. // Only the first call has any effect. if fr.Enabled() && time.Since(start) > 100*time.Millisecond { go captureSnapshot(fr) } Here’s the full code for the server, now instrumented with the flight recorder: [https://go.dev/play/p/3V33gfIpmjG](https://go.dev/play/p/3V33gfIpmjG) Now, we run the server again and send requests until we get a slow request that triggers a snapshot. Once we’ve gotten a trace, we’ll need a tool that will help us examine it. The Go toolchain provides a built-in execution trace analysis tool via the [`go tool trace` command](https://pkg.go.dev/cmd/trace) . Run `go tool trace snapshot.trace` to launch the tool, which starts a local web server, then open the displayed URL in your browser (if the tool doesn’t open your browser automatically). This tool gives us a few ways to look at the trace, but let’s focus on visualizing the trace to get a sense of what’s going on. Click “View trace by proc” to do so. In this view, the trace is presented as a timeline of events. At the top of the page, in the “STATS” section, we can see a summary of the application’s state, including the number of threads, the heap size, and the goroutine count. Below that, in the “PROCS” section, we can see how the execution of goroutines is mapped onto `GOMAXPROCS` (the number of operating system threads created by the Go application). We can see when and how each goroutine starts, runs, and finally stops executing. For now, let’s turn our attention to this massive gap in execution on the right side of the viewer. For a period of time, around 100ms, nothing is happening! [![](https://go.dev/blog/flight-recorder/flight_recorder_1.png)](https://go.dev/blog/flight-recorder/flight_recorder_1.png) By selecting the `zoom` tool (or pressing `3`), we can inspect the section of the trace right after the gap with more detail. [![](https://go.dev/blog/flight-recorder/flight_recorder_2.png)](https://go.dev/blog/flight-recorder/flight_recorder_2.png) In addition to the activity of each individual goroutine, we can see how goroutines interact via “flow events.” An incoming flow event indicates what happened to make a goroutine start running. An outgoing flow edge indicates what effect one goroutine had on another. Enabling the visualization of all flow events often provides clues that hint at the source of a problem. [![](https://go.dev/blog/flight-recorder/flight_recorder_3.png)](https://go.dev/blog/flight-recorder/flight_recorder_3.png) In this case, we can see that many of the goroutines have a direct connection to a single goroutine right after the pause in activity. Clicking on the single goroutine shows an event table filled with outgoing flow events, which matches what we saw when the flow view was enabled. What happened when this goroutine ran? Part of the information stored in the trace is a view of the stack trace at different points in time. When we look at the goroutine we can see that the start stack trace shows that it was waiting for the HTTP request to complete when the goroutine was scheduled to run. And the end stack trace shows that the `sendReport` function had already returned and it was waiting for the ticker for the next scheduled time to send the report. [![](https://go.dev/blog/flight-recorder/flight_recorder_4.png)](https://go.dev/blog/flight-recorder/flight_recorder_4.png) Between the start and the end of this goroutine running, we see a huge number of “outgoing flows,” where it interacts with other goroutines. Clicking on one of the `Outgoing flow` entries takes us to a view of the interaction. [![](https://go.dev/blog/flight-recorder/flight_recorder_5.png)](https://go.dev/blog/flight-recorder/flight_recorder_5.png) This flow implicates the `Unlock` in `sendReport`: for index := range buckets { b := &buckets[index] b.mu.Lock() defer b.mu.Unlock() counts[index] = b.guesses } In `sendReport`, we intended to acquire a lock on each bucket and release the lock after copying the value. But here’s the problem: we don’t actually release the lock immediately after copying the value contained in `bucket.guesses`. Because we used a `defer` statement to release the lock, that release doesn’t happen until the function returns. We hold the lock not just past the end of the loop, but until after the HTTP request completes. That’s a subtle error that may be difficult to track down in a large production system. Fortunately, execution tracing helped us pinpoint the problem. However, if we tried to use the execution tracer in a long-running server without the new flight-recording mode, it would likely amass a huge amount of execution trace data, which an operator would have to store, transmit, and sift through. The flight recorder gives us the power of hindsight. It lets us capture just what went wrong, after it’s already happened, and quickly zero in on the cause. The flight recorder is just the latest addition to the Go developer’s toolbox for diagnosing the inner workings of running applications. We’ve steadily been improving tracing over the past couple of releases. Go 1.21 greatly reduced the run-time overhead of tracing. The trace format became more robust and also splittable in the Go 1.22 release, leading to features like the flight recorder. Open-source tools like [gotraceui](https://gotraceui.dev/) , and the [forthcoming ability to programmatically parse execution traces](https://go.dev/issue/62627) are more ways to leverage the power of execution traces. The [Diagnostics page](https://go.dev/doc/diagnostics) lists many additional tools at your disposal. We hope you make use of them as you write and refine your Go applications. Thanks ------ We’d like to take a moment to thank those community members who have been active in the diagnostics meetings, contributed to the designs, and provided feedback over the years: Felix Geisendörfer ([@felixge.de](https://bsky.app/profile/felixge.de) ), Nick Ripley ([@nsrip-dd](https://github.com/nsrip-dd) ), Rhys Hiltner ([@rhysh](https://github.com/rhysh) ), Dominik Honnef ([@dominikh](https://github.com/dominikh) ), Bryan Boreham ([@bboreham](https://github.com/bboreham) ), and PJ Malloy ([@thepudds](https://github.com/thepudds) ). The discussions, feedback, and work you’ve all put in have been instrumental in pushing us to a better diagnostics future. Thank you! **Next article:** [The Green Tea Garbage Collector](https://go.dev/blog/greenteagc) **Previous article:** [It's survey time! How has Go has been working out for you?](https://go.dev/blog/survey2025-announce) **[Blog Index](https://go.dev/blog/all) ** go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # talks/2017 - The Go Programming Language Go talks ======== talks/2017 ---------- #### Slide decks: [state-of-go-may.slide](https://go.dev/talks/2017/state-of-go-may.slide) : The State of Go [state-of-go.slide](https://go.dev/talks/2017/state-of-go.slide) : The State of Go #### Files: [exporting-go.pdf](https://go.dev/talks/2017/exporting-go.pdf) [state-of-go-aug.pdf](https://go.dev/talks/2017/state-of-go-aug.pdf) #### Sub-directories: [state-of-go](https://go.dev/talks/2017/state-of-go) [state-of-go-may](https://go.dev/talks/2017/state-of-go-may) Opens in new window. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Migrating to Go Modules - The Go Programming Language [The Go Blog](https://go.dev/blog/) ==================================== Migrating to Go Modules ======================= Jean Barkhuysen 21 August 2019 Introduction ------------ This post is part 2 in a series. * Part 1 — [Using Go Modules](https://go.dev/blog/using-go-modules) * **Part 2 — Migrating To Go Modules** (this post) * Part 3 — [Publishing Go Modules](https://go.dev/blog/publishing-go-modules) * Part 4 — [Go Modules: v2 and Beyond](https://go.dev/blog/v2-go-modules) * Part 5 — [Keeping Your Modules Compatible](https://go.dev/blog/module-compatibility) **Note:** For documentation, see [Managing dependencies](https://go.dev/doc/modules/managing-dependencies) and [Developing and publishing modules](https://go.dev/doc/modules/developing) . Go projects use a wide variety of dependency management strategies. [Vendoring](https://go.dev/cmd/go/#hdr-Vendor_Directories) tools such as [dep](https://github.com/golang/dep) and [glide](https://github.com/Masterminds/glide) are popular, but they have wide differences in behavior and don’t always work well together. Some projects store their entire GOPATH directory in a single Git repository. Others simply rely on `go get` and expect fairly recent versions of dependencies to be installed in GOPATH. Go’s module system, introduced in Go 1.11, provides an official dependency management solution built into the `go` command. This article describes tools and techniques for converting a project to modules. Please note: if your project is already tagged at v2.0.0 or higher, you will need to update your module path when you add a `go.mod` file. We’ll explain how to do that without breaking your users in a future article focused on v2 and beyond. Migrating to Go modules in your project --------------------------------------- A project might be in one of three states when beginning the transition to Go modules: * A brand new Go project. * An established Go project with a non-modules dependency manager. * An established Go project without any dependency manager. The first case is covered in [Using Go Modules](https://go.dev/blog/using-go-modules) ; we’ll address the latter two in this post. With a dependency manager ------------------------- To convert a project that already uses a dependency management tool, run the following commands: $ git clone https://github.com/my/project [...] $ cd project $ cat Godeps/Godeps.json { "ImportPath": "github.com/my/project", "GoVersion": "go1.12", "GodepVersion": "v80", "Deps": [\ {\ "ImportPath": "rsc.io/binaryregexp",\ "Comment": "v0.2.0-1-g545cabd",\ "Rev": "545cabda89ca36b48b8e681a30d9d769a30b3074"\ },\ {\ "ImportPath": "rsc.io/binaryregexp/syntax",\ "Comment": "v0.2.0-1-g545cabd",\ "Rev": "545cabda89ca36b48b8e681a30d9d769a30b3074"\ }\ ] } $ go mod init github.com/my/project go: creating new go.mod: module github.com/my/project go: copying requirements from Godeps/Godeps.json $ cat go.mod module github.com/my/project go 1.12 require rsc.io/binaryregexp v0.2.1-0.20190524193500-545cabda89ca $ `go mod init` creates a new go.mod file and automatically imports dependencies from `Godeps.json`, `Gopkg.lock`, or a number of [other supported formats](https://go.googlesource.com/go/+/362625209b6cd2bc059b6b0a67712ddebab312d9/src/cmd/go/internal/modconv/modconv.go#9) . The argument to `go mod init` is the module path, the location where the module may be found. This is a good time to pause and run `go build ./...` and `go test ./...` before continuing. Later steps may modify your `go.mod` file, so if you prefer to take an iterative approach, this is the closest your `go.mod` file will be to your pre-modules dependency specification. $ go mod tidy go: downloading rsc.io/binaryregexp v0.2.1-0.20190524193500-545cabda89ca go: extracting rsc.io/binaryregexp v0.2.1-0.20190524193500-545cabda89ca $ cat go.sum rsc.io/binaryregexp v0.2.1-0.20190524193500-545cabda89ca h1:FKXXXJ6G2bFoVe7hX3kEX6Izxw5ZKRH57DFBJmHCbkU= rsc.io/binaryregexp v0.2.1-0.20190524193500-545cabda89ca/go.mod h1:qTv7/COck+e2FymRvadv62gMdZztPaShugOCi3I+8D8= $ `go mod tidy` finds all the packages transitively imported by packages in your module. It adds new module requirements for packages not provided by any known module, and it removes requirements on modules that don’t provide any imported packages. If a module provides packages that are only imported by projects that haven’t migrated to modules yet, the module requirement will be marked with an `// indirect` comment. It is always good practice to run `go mod tidy` before committing a `go.mod` file to version control. Let’s finish by making sure the code builds and tests pass: $ go build ./... $ go test ./... [...] $ Note that other dependency managers may specify dependencies at the level of individual packages or entire repositories (not modules), and generally do not recognize the requirements specified in the `go.mod` files of dependencies. Consequently, you may not get exactly the same version of every package as before, and there’s some risk of upgrading past breaking changes. Therefore, it’s important to follow the above commands with an audit of the resulting dependencies. To do so, run $ go list -m all go: finding rsc.io/binaryregexp v0.2.1-0.20190524193500-545cabda89ca github.com/my/project rsc.io/binaryregexp v0.2.1-0.20190524193500-545cabda89ca $ and compare the resulting versions with your old dependency management file to ensure that the selected versions are appropriate. If you find a version that wasn’t what you wanted, you can find out why using `go mod why -m` and/or `go mod graph`, and upgrade or downgrade to the correct version using `go get`. (If the version you request is older than the version that was previously selected, `go get` will downgrade other dependencies as needed to maintain compatibility.) For example, $ go mod why -m rsc.io/binaryregexp [...] $ go mod graph | grep rsc.io/binaryregexp [...] $ go get rsc.io/binaryregexp@v0.2.0 $ Without a dependency manager ---------------------------- For a Go project without a dependency management system, start by creating a `go.mod` file: $ git clone https://go.googlesource.com/blog [...] $ cd blog $ go mod init golang.org/x/blog go: creating new go.mod: module golang.org/x/blog $ cat go.mod module golang.org/x/blog go 1.12 $ Without a configuration file from a previous dependency manager, `go mod init` will create a `go.mod` file with only the `module` and `go` directives. In this example, we set the module path to `golang.org/x/blog` because that is its [custom import path](https://go.dev/cmd/go/#hdr-Remote_import_paths) . Users may import packages with this path, and we must be careful not to change it. The `module` directive declares the module path, and the `go` directive declares the expected version of the Go language used to compile the code within the module. Next, run `go mod tidy` to add the module’s dependencies: $ go mod tidy go: finding golang.org/x/website latest go: finding gopkg.in/tomb.v2 latest go: finding golang.org/x/net latest go: finding golang.org/x/tools latest go: downloading github.com/gorilla/context v1.1.1 go: downloading golang.org/x/tools v0.0.0-20190813214729-9dba7caff850 go: downloading golang.org/x/net v0.0.0-20190813141303-74dc4d7220e7 go: extracting github.com/gorilla/context v1.1.1 go: extracting golang.org/x/net v0.0.0-20190813141303-74dc4d7220e7 go: downloading gopkg.in/tomb.v2 v2.0.0-20161208151619-d5d1b5820637 go: extracting gopkg.in/tomb.v2 v2.0.0-20161208151619-d5d1b5820637 go: extracting golang.org/x/tools v0.0.0-20190813214729-9dba7caff850 go: downloading golang.org/x/website v0.0.0-20190809153340-86a7442ada7c go: extracting golang.org/x/website v0.0.0-20190809153340-86a7442ada7c $ cat go.mod module golang.org/x/blog go 1.12 require ( github.com/gorilla/context v1.1.1 golang.org/x/net v0.0.0-20190813141303-74dc4d7220e7 golang.org/x/text v0.3.2 golang.org/x/tools v0.0.0-20190813214729-9dba7caff850 golang.org/x/website v0.0.0-20190809153340-86a7442ada7c gopkg.in/tomb.v2 v2.0.0-20161208151619-d5d1b5820637 ) $ cat go.sum cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= cloud.google.com/go v0.34.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= git.apache.org/thrift.git v0.0.0-20180902110319-2566ecd5d999/go.mod h1:fPE2ZNJGynbRyZ4dJvy6G277gSllfV2HJqblrnkyeyg= git.apache.org/thrift.git v0.0.0-20181218151757-9b75e4fe745a/go.mod h1:fPE2ZNJGynbRyZ4dJvy6G277gSllfV2HJqblrnkyeyg= github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q= [...] $ `go mod tidy` added module requirements for all the packages transitively imported by packages in your module and built a `go.sum` with checksums for each library at a specific version. Let’s finish by making sure the code still builds and tests still pass: $ go build ./... $ go test ./... ok golang.org/x/blog 0.335s ? golang.org/x/blog/content/appengine [no test files] ok golang.org/x/blog/content/cover 0.040s ? golang.org/x/blog/content/h2push/server [no test files] ? golang.org/x/blog/content/survey2016 [no test files] ? golang.org/x/blog/content/survey2017 [no test files] ? golang.org/x/blog/support/racy [no test files] $ Note that when `go mod tidy` adds a requirement, it adds the latest version of the module. If your `GOPATH` included an older version of a dependency that subsequently published a breaking change, you may see errors in `go mod tidy`, `go build`, or `go test`. If this happens, try downgrading to an older version with `go get` (for example, `go get github.com/broken/module@v1.1.0`), or take the time to make your module compatible with the latest version of each dependency. ### Tests in module mode Some tests may need tweaks after migrating to Go modules. If a test needs to write files in the package directory, it may fail when the package directory is in the module cache, which is read-only. In particular, this may cause `go test all` to fail. The test should copy files it needs to write to a temporary directory instead. If a test relies on relative paths (`../package-in-another-module`) to locate and read files in another package, it will fail if the package is in another module, which will be located in a versioned subdirectory of the module cache or a path specified in a `replace` directive. If this is the case, you may need to copy the test inputs into your module, or convert the test inputs from raw files to data embedded in `.go` source files. If a test expects `go` commands within the test to run in GOPATH mode, it may fail. If this is the case, you may need to add a `go.mod` file to the source tree to be tested, or set `GO111MODULE=off` explicitly. Publishing a release -------------------- Finally, you should tag and publish a release version for your new module. This is optional if you haven’t released any versions yet, but without an official release, downstream users will depend on specific commits using [pseudo-versions](https://go.dev/cmd/go/#hdr-Pseudo_versions) , which may be more difficult to support. $ git tag v1.2.0 $ git push origin v1.2.0 Your new `go.mod` file defines a canonical import path for your module and adds new minimum version requirements. If your users are already using the correct import path, and your dependencies haven’t made breaking changes, then adding the `go.mod` file is backwards-compatible — but it’s a significant change, and may expose existing problems. If you have existing version tags, you should increment the [minor version](https://semver.org/#spec-item-7) . See [Publishing Go Modules](https://go.dev/blog/publishing-go-modules) to learn how to increment and publish versions. Imports and canonical module paths ---------------------------------- Each module declares its module path in its `go.mod` file. Each `import` statement that refers to a package within the module must have the module path as a prefix of the package path. However, the `go` command may encounter a repository containing the module through many different [remote import paths](https://go.dev/cmd/go/#hdr-Remote_import_paths) . For example, both `golang.org/x/lint` and `github.com/golang/lint` resolve to repositories containing the code hosted at [go.googlesource.com/lint](https://go.googlesource.com/lint) . The [`go.mod` file](https://go.googlesource.com/lint/+/refs/heads/master/go.mod) contained in that repository declares its path to be `golang.org/x/lint`, so only that path corresponds to a valid module. Go 1.4 provided a mechanism for declaring canonical import paths using [`// import` comments](https://go.dev/cmd/go/#hdr-Import_path_checking) , but package authors did not always provide them. As a result, code written prior to modules may have used a non-canonical import path for a module without surfacing an error for the mismatch. When using modules, the import path must match the canonical module path, so you may need to update `import` statements: for example, you may need to change `import "github.com/golang/lint"` to `import "golang.org/x/lint"`. Another scenario in which a module’s canonical path may differ from its repository path occurs for Go modules at major version 2 or higher. A Go module with a major version above 1 must include a major-version suffix in its module path: for example, version `v2.0.0` must have the suffix `/v2`. However, `import` statements may have referred to the packages within the module _without_ that suffix. For example, non-module users of `github.com/russross/blackfriday/v2` at `v2.0.1` may have imported it as `github.com/russross/blackfriday` instead, and will need to update the import path to include the `/v2` suffix. Conclusion ---------- Converting to Go modules should be a straightforward process for most users. Occasional issues may arise due to non-canonical import paths or breaking changes within a dependency. Future posts will explore [publishing new versions](https://go.dev/blog/publishing-go-modules) , v2 and beyond, and ways to debug strange situations. To provide feedback and help shape the future of dependency management in Go, please send us [bug reports](https://go.dev/issue/new) or [experience reports](https://go.dev/wiki/ExperienceReports) . Thanks for all your feedback and help improving modules. **Next article:** [Module Mirror and Checksum Database Launched](https://go.dev/blog/module-mirror-launch) **Previous article:** [Contributors Summit 2019](https://go.dev/blog/contributors-summit-2019) **[Blog Index](https://go.dev/blog/all) ** go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # A GIF decoder: an exercise in Go interfaces - The Go Programming Language [The Go Blog](https://go.dev/blog/) ==================================== A GIF decoder: an exercise in Go interfaces =========================================== Rob Pike 25 May 2011 Introduction ------------ At the Google I/O conference in San Francisco on May 10, 2011, we announced that the Go language is now available on Google App Engine. Go is the first language to be made available on App Engine that compiles directly to machine code, which makes it a good choice for CPU-intensive tasks such as image manipulation. In that vein, we demonstrated a program called [Moustachio](http://moustach-io.appspot.com/) that makes it easy to improve a picture such as this one: ![](https://go.dev/blog/gif-decoder/image00.jpg) by adding a moustache and sharing the result: ![](https://go.dev/blog/gif-decoder/image02.jpg) All the graphical processing, including rendering the antialiased moustache, is done by a Go program running on App Engine. (The source is available at [the appengine-go project](http://code.google.com/p/appengine-go/source/browse/example/moustachio/) .) Although most images on the web—at least those likely to be moustachioed—are JPEGs, there are countless other formats floating around, and it seemed reasonable for Moustachio to accept uploaded images in a few of them. JPEG and PNG decoders already existed in the Go image library, but the venerable GIF format was not represented, so we decided to write a GIF decoder in time for the announcement. That decoder contains a few pieces that demonstrate how Go’s interfaces make some problems easier to solve. The rest of this blog post describes a couple of instances. The GIF format -------------- First, a quick tour of the GIF format. A GIF image file is _paletted_, that is, each pixel value is an index into a fixed color map that is included in the file. The GIF format dates from a time when there were usually no more than 8 bits per pixel on the display, and a color map was used to convert the limited set of values into the RGB (red, green, blue) triples needed to light the screen. (This is in contrast to a JPEG, for example, which has no color map because the encoding represents the distinct color signals separately.) A GIF image can contain anywhere from 1 to 8 bits per pixel, inclusive, but 8 bits per pixel is the most common. Simplifying somewhat, a GIF file contains a header defining the pixel depth and image dimensions, a color map (256 RGB triples for an 8-bit image), and then the pixel data. The pixel data is stored as a one-dimensional bit stream, compressed using the LZW algorithm, which is quite effective for computer-generated graphics although not so good for photographic imagery. The compressed data is then broken into length-delimited blocks with a one-byte count (0-255) followed by that many bytes: ![](https://go.dev/blog/gif-decoder/image03.gif) Deblocking the pixel data ------------------------- To decode GIF pixel data in Go, we can use the LZW decompressor from the `compress/lzw` package. It has a NewReader function that returns an object that, as [the documentation](https://go.dev/pkg/compress/lzw/#NewReader) says, “satisfies reads by decompressing the data read from r”: func NewReader(r io.Reader, order Order, litWidth int) io.ReadCloser Here `order` defines the bit-packing order and `litWidth` is the word size in bits, which for a GIF file corresponds to the pixel depth, typically 8. But we can’t just give `NewReader` the input file as its first argument because the decompressor needs a stream of bytes but the GIF data is a stream of blocks that must be unpacked. To address this problem, we can wrap the input `io.Reader` with some code to deblock it, and make that code again implement `Reader`. In other words, we put the deblocking code into the `Read` method of a new type, which we call `blockReader`. Here’s the data structure for a `blockReader`. type blockReader struct { r reader // Input source; implements io.Reader and io.ByteReader. slice []byte // Buffer of unread data. tmp [256]byte // Storage for slice. } The reader, `r`, will be the source of the image data, perhaps a file or HTTP connection. The `slice` and `tmp` fields will be used to manage the deblocking. Here’s the `Read` method in its entirety. It’s a nice example of the use of slices and arrays in Go. 1 func (b *blockReader) Read(p []byte) (int, os.Error) { 2 if len(p) == 0 { 3 return 0, nil 4 } 5 if len(b.slice) == 0 { 6 blockLen, err := b.r.ReadByte() 7 if err != nil { 8 return 0, err 9 } 10 if blockLen == 0 { 11 return 0, os.EOF 12 } 13 b.slice = b.tmp[0:blockLen] 14 if _, err = io.ReadFull(b.r, b.slice); err != nil { 15 return 0, err 16 } 17 } 18 n := copy(p, b.slice) 19 b.slice = b.slice[n:] 20 return n, nil 21 } Lines 2-4 are just a sanity check: if there’s no place to put data, return zero. That should never happen, but it’s good to be safe. Line 5 asks if there’s data left over from a previous call by checking the length of `b.slice`. If there isn’t, the slice will have length zero and we need to read the next block from `r`. A GIF block starts with a byte count, read on line 6. If the count is zero, GIF defines this to be a terminating block, so we return `EOF` on line 11. Now we know we should read `blockLen` bytes, so we point `b.slice` to the first `blockLen` bytes of `b.tmp` and then use the helper function `io.ReadFull` to read that many bytes. That function will return an error if it can’t read exactly that many bytes, which should never happen. Otherwise we have `blockLen` bytes ready to read. Lines 18-19 copy the data from `b.slice` to the caller’s buffer. We are implementing `Read`, not `ReadFull`, so we are allowed to return fewer than the requested number of bytes. That makes it easy: we just copy the data from `b.slice` to the caller’s buffer (`p`), and the return value from copy is the number of bytes transferred. Then we reslice `b.slice` to drop the first `n` bytes, ready for the next call. It’s a nice technique in Go programming to couple a slice (`b.slice`) to an array (`b.tmp`). In this case, it means `blockReader` type’s `Read` method never does any allocations. It also means we don’t need to keep a count around (it’s implicit in the slice length), and the built-in `copy` function guarantees we never copy more than we should. (For more about slices, see [this post from the Go Blog](https://go.dev/blog/go-slices-usage-and-internals) .) Given the `blockReader` type, we can unblock the image data stream just by wrapping the input reader, say a file, like this: deblockingReader := &blockReader{r: imageFile} This wrapping turns a block-delimited GIF image stream into a simple stream of bytes accessible by calls to the `Read` method of the `blockReader`. Connecting the pieces --------------------- With `blockReader` implemented and the LZW compressor available from the library, we have all the pieces we need to decode the image data stream. We stitch them together with this thunderclap, straight from the code: lzwr := lzw.NewReader(&blockReader{r: d.r}, lzw.LSB, int(litWidth)) if _, err = io.ReadFull(lzwr, m.Pix); err != nil { break } That’s it. The first line creates a `blockReader` and passes it to `lzw.NewReader` to create a decompressor. Here `d.r` is the `io.Reader` holding the image data, `lzw.LSB` defines the byte order in the LZW decompressor, and `litWidth` is the pixel depth. Given the decompressor, the second line calls `io.ReadFull` to decompress the data and store it in the image, `m.Pix`. When `ReadFull` returns, the image data is decompressed and stored in the image, `m`, ready to be displayed. This code worked first time. Really. We could avoid the temporary variable `lzwr` by placing the `NewReader` call into the argument list for `ReadFull`, just as we built the `blockReader` inside the call to `NewReader`, but that might be packing too much into a single line of code. Conclusion ---------- Go’s interfaces make it easy to construct software by assembling piece parts like this to restructure data. In this example, we implemented GIF decoding by chaining together a deblocker and a decompressor using the `io.Reader` interface, analogous to a type-safe Unix pipeline. Also, we wrote the deblocker as an (implicit) implementation of a `Reader` interface, which then required no extra declaration or boilerplate to fit it into the processing pipeline. It’s hard to implement this decoder so compactly yet cleanly and safely in most languages, but the interface mechanism plus a few conventions make it almost natural in Go. That deserves another picture, a GIF this time: ![](https://go.dev/blog/gif-decoder/image01.gif) The GIF format is defined at [http://www.w3.org/Graphics/GIF/spec-gif89a.txt](http://www.w3.org/Graphics/GIF/spec-gif89a.txt) . **Next article:** [Spotlight on external Go libraries](https://go.dev/blog/external-libraries) **Previous article:** [Go at Google I/O 2011: videos](https://go.dev/blog/io2011) **[Blog Index](https://go.dev/blog/all) ** go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # talks/2016 - The Go Programming Language Go talks ======== talks/2016 ---------- #### Articles: [refactor.article](https://go.dev/talks/2016/refactor.article) : Codebase Refactoring (with help from Go) #### Slide decks: [applicative.slide](https://go.dev/talks/2016/applicative.slide) : Program your next server in Go [asm.slide](https://go.dev/talks/2016/asm.slide) : The Design of the Go Assembler [state-of-go.slide](https://go.dev/talks/2016/state-of-go.slide) : The State of Go [token.slide](https://go.dev/talks/2016/token.slide) : Stacks of Tokens #### Files: [prototype-your-design.pdf](https://go.dev/talks/2016/prototype-your-design.pdf) #### Sub-directories: [applicative](https://go.dev/talks/2016/applicative) [asm](https://go.dev/talks/2016/asm) [refactor](https://go.dev/talks/2016/refactor) [state-of-go](https://go.dev/talks/2016/state-of-go) Opens in new window. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Writing Web Applications - The Go Programming Language 1. [Documentation](https://go.dev/doc/) 2. [/doc/articles/](https://go.dev/doc/articles/) 3. [Writing Web Applications](https://go.dev/doc/articles/wiki/) Writing Web Applications ======================== Introduction ------------ Covered in this tutorial: * Creating a data structure with load and save methods * Using the `net/http` package to build web applications * Using the `html/template` package to process HTML templates * Using the `regexp` package to validate user input * Using closures Assumed knowledge: * Programming experience * Understanding of basic web technologies (HTTP, HTML) * Some UNIX/DOS command-line knowledge Getting Started --------------- At present, you need to have a FreeBSD, Linux, macOS, or Windows machine to run Go. We will use `$` to represent the command prompt. Install Go (see the [Installation Instructions](https://go.dev/doc/install) ). Make a new directory for this tutorial inside your `GOPATH` and cd to it: $ mkdir gowiki $ cd gowiki Create a file named `wiki.go`, open it in your favorite editor, and add the following lines: package main import ( "fmt" "os" ) We import the `fmt` and `os` packages from the Go standard library. Later, as we implement additional functionality, we will add more packages to this `import` declaration. Data Structures --------------- Let's start by defining the data structures. A wiki consists of a series of interconnected pages, each of which has a title and a body (the page content). Here, we define `Page` as a struct with two fields representing the title and body. type Page struct { Title string Body \[\]byte } The type `[]byte` means "a `byte` slice". (See [Slices: usage and internals](https://go.dev/doc/articles/slices_usage_and_internals.html) for more on slices.) The `Body` element is a `[]byte` rather than `string` because that is the type expected by the `io` libraries we will use, as you'll see below. The `Page` struct describes how page data will be stored in memory. But what about persistent storage? We can address that by creating a `save` method on `Page`: func (p \*Page) save() error { filename := p.Title + ".txt" return os.WriteFile(filename, p.Body, 0600) } This method's signature reads: "This is a method named `save` that takes as its receiver `p`, a pointer to `Page` . It takes no parameters, and returns a value of type `error`." This method will save the `Page`'s `Body` to a text file. For simplicity, we will use the `Title` as the file name. The `save` method returns an `error` value because that is the return type of `WriteFile` (a standard library function that writes a byte slice to a file). The `save` method returns the error value, to let the application handle it should anything go wrong while writing the file. If all goes well, `Page.save()` will return `nil` (the zero-value for pointers, interfaces, and some other types). The octal integer literal `0600`, passed as the third parameter to `WriteFile`, indicates that the file should be created with read-write permissions for the current user only. (See the Unix man page `open(2)` for details.) In addition to saving pages, we will want to load pages, too: func loadPage(title string) \*Page { filename := title + ".txt" body, \_ := os.ReadFile(filename) return &Page{Title: title, Body: body} } The function `loadPage` constructs the file name from the title parameter, reads the file's contents into a new variable `body`, and returns a pointer to a `Page` literal constructed with the proper title and body values. Functions can return multiple values. The standard library function `os.ReadFile` returns `[]byte` and `error`. In `loadPage`, error isn't being handled yet; the "blank identifier" represented by the underscore (`_`) symbol is used to throw away the error return value (in essence, assigning the value to nothing). But what happens if `ReadFile` encounters an error? For example, the file might not exist. We should not ignore such errors. Let's modify the function to return `*Page` and `error`. func loadPage(title string) (\*Page, error) { filename := title + ".txt" body, err := os.ReadFile(filename) if err != nil { return nil, err } return &Page{Title: title, Body: body}, nil } Callers of this function can now check the second parameter; if it is `nil` then it has successfully loaded a Page. If not, it will be an `error` that can be handled by the caller (see the [language specification](https://go.dev/ref/spec#Errors) for details). At this point we have a simple data structure and the ability to save to and load from a file. Let's write a `main` function to test what we've written: func main() { p1 := &Page{Title: "TestPage", Body: \[\]byte("This is a sample Page.")} p1.save() p2, \_ := loadPage("TestPage") fmt.Println(string(p2.Body)) } After compiling and executing this code, a file named `TestPage.txt` would be created, containing the contents of `p1`. The file would then be read into the struct `p2`, and its `Body` element printed to the screen. You can compile and run the program like this: $ go build wiki.go $ ./wiki This is a sample Page. (If you're using Windows you must type "`wiki`" without the "`./`" to run the program.) [Click here to view the code we've written so far.](https://go.dev/doc/articles/wiki/part1.go) Introducing the `net/http` package (an interlude) ------------------------------------------------- Here's a full working example of a simple web server: //go:build ignore package main import ( "fmt" "log" "net/http" ) func handler(w http.ResponseWriter, r \*http.Request) { fmt.Fprintf(w, "Hi there, I love %s!", r.URL.Path\[1:\]) } func main() { http.HandleFunc("/", handler) log.Fatal(http.ListenAndServe(":8080", nil)) } The `main` function begins with a call to `http.HandleFunc`, which tells the `http` package to handle all requests to the web root (`"/"`) with `handler`. It then calls `http.ListenAndServe`, specifying that it should listen on port 8080 on any interface (`":8080"`). (Don't worry about its second parameter, `nil`, for now.) This function will block until the program is terminated. `ListenAndServe` always returns an error, since it only returns when an unexpected error occurs. In order to log that error we wrap the function call with `log.Fatal`. The function `handler` is of the type `http.HandlerFunc`. It takes an `http.ResponseWriter` and an `http.Request` as its arguments. An `http.ResponseWriter` value assembles the HTTP server's response; by writing to it, we send data to the HTTP client. An `http.Request` is a data structure that represents the client HTTP request. `r.URL.Path` is the path component of the request URL. The trailing `[1:]` means "create a sub-slice of `Path` from the 1st character to the end." This drops the leading "/" from the path name. If you run this program and access the URL: http://localhost:8080/monkeys the program would present a page containing: Hi there, I love monkeys! Using `net/http` to serve wiki pages ------------------------------------ To use the `net/http` package, it must be imported: import ( "fmt" "os" "log" **"net/http"** ) Let's create a handler, `viewHandler` that will allow users to view a wiki page. It will handle URLs prefixed with "/view/". func viewHandler(w http.ResponseWriter, r \*http.Request) { title := r.URL.Path\[len("/view/"):\] p, \_ := loadPage(title) fmt.Fprintf(w, "

%s

%s
", p.Title, p.Body) } Again, note the use of `_` to ignore the `error` return value from `loadPage`. This is done here for simplicity and generally considered bad practice. We will attend to this later. First, this function extracts the page title from `r.URL.Path`, the path component of the request URL. The `Path` is re-sliced with `[len("/view/"):]` to drop the leading `"/view/"` component of the request path. This is because the path will invariably begin with `"/view/"`, which is not part of the page's title. The function then loads the page data, formats the page with a string of simple HTML, and writes it to `w`, the `http.ResponseWriter`. To use this handler, we rewrite our `main` function to initialize `http` using the `viewHandler` to handle any requests under the path `/view/`. func main() { http.HandleFunc("/view/", viewHandler) log.Fatal(http.ListenAndServe(":8080", nil)) } [Click here to view the code we've written so far.](https://go.dev/doc/articles/wiki/part2.go) Let's create some page data (as `test.txt`), compile our code, and try serving a wiki page. Open `test.txt` file in your editor, and save the string "Hello world" (without quotes) in it. $ go build wiki.go $ ./wiki (If you're using Windows you must type "`wiki`" without the "`./`" to run the program.) With this web server running, a visit to `[http://localhost:8080/view/test](http://localhost:8080/view/test) ` should show a page titled "test" containing the words "Hello world". Editing Pages ------------- A wiki is not a wiki without the ability to edit pages. Let's create two new handlers: one named `editHandler` to display an 'edit page' form, and the other named `saveHandler` to save the data entered via the form. First, we add them to `main()`: func main() { http.HandleFunc("/view/", viewHandler) http.HandleFunc("/edit/", editHandler) http.HandleFunc("/save/", saveHandler) log.Fatal(http.ListenAndServe(":8080", nil)) } The function `editHandler` loads the page (or, if it doesn't exist, create an empty `Page` struct), and displays an HTML form. func editHandler(w http.ResponseWriter, r \*http.Request) { title := r.URL.Path\[len("/edit/"):\] p, err := loadPage(title) if err != nil { p = &Page{Title: title} } fmt.Fprintf(w, "

Editing %s

"+ "
"+ "
"+ ""+ "
", p.Title, p.Title, p.Body) } This function will work fine, but all that hard-coded HTML is ugly. Of course, there is a better way. The `html/template` package --------------------------- The `html/template` package is part of the Go standard library. We can use `html/template` to keep the HTML in a separate file, allowing us to change the layout of our edit page without modifying the underlying Go code. First, we must add `html/template` to the list of imports. We also won't be using `fmt` anymore, so we have to remove that. import ( **"html/template"** "os" "net/http" ) Let's create a template file containing the HTML form. Open a new file named `edit.html`, and add the following lines:

Editing {{.Title}}

Modify `editHandler` to use the template, instead of the hard-coded HTML: func editHandler(w http.ResponseWriter, r \*http.Request) { title := r.URL.Path\[len("/edit/"):\] p, err := loadPage(title) if err != nil { p = &Page{Title: title} } t, \_ := template.ParseFiles("edit.html") t.Execute(w, p) } The function `template.ParseFiles` will read the contents of `edit.html` and return a `*template.Template`. The method `t.Execute` executes the template, writing the generated HTML to the `http.ResponseWriter`. The `.Title` and `.Body` dotted identifiers refer to `p.Title` and `p.Body`. Template directives are enclosed in double curly braces. The `printf "%s" .Body` instruction is a function call that outputs `.Body` as a string instead of a stream of bytes, the same as a call to `fmt.Printf`. The `html/template` package helps guarantee that only safe and correct-looking HTML is generated by template actions. For instance, it automatically escapes any greater than sign (`>`), replacing it with `>`, to make sure user data does not corrupt the form HTML. Since we're working with templates now, let's create a template for our `viewHandler` called `view.html`:

{{.Title}}

\[edit\]

{{printf "%s" .Body}}
Modify `viewHandler` accordingly: func viewHandler(w http.ResponseWriter, r \*http.Request) { title := r.URL.Path\[len("/view/"):\] p, \_ := loadPage(title) t, \_ := template.ParseFiles("view.html") t.Execute(w, p) } Notice that we've used almost exactly the same templating code in both handlers. Let's remove this duplication by moving the templating code to its own function: func renderTemplate(w http.ResponseWriter, tmpl string, p \*Page) { t, \_ := template.ParseFiles(tmpl + ".html") t.Execute(w, p) } And modify the handlers to use that function: func viewHandler(w http.ResponseWriter, r \*http.Request) { title := r.URL.Path\[len("/view/"):\] p, \_ := loadPage(title) renderTemplate(w, "view", p) } func editHandler(w http.ResponseWriter, r \*http.Request) { title := r.URL.Path\[len("/edit/"):\] p, err := loadPage(title) if err != nil { p = &Page{Title: title} } renderTemplate(w, "edit", p) } If we comment out the registration of our unimplemented save handler in `main`, we can once again build and test our program. [Click here to view the code we've written so far.](https://go.dev/doc/articles/wiki/part3.go) Handling non-existent pages --------------------------- What if you visit [`/view/APageThatDoesntExist`](http://localhost:8080/view/APageThatDoesntExist) ? You'll see a page containing HTML. This is because it ignores the error return value from `loadPage` and continues to try and fill out the template with no data. Instead, if the requested Page doesn't exist, it should redirect the client to the edit Page so the content may be created: func viewHandler(w http.ResponseWriter, r \*http.Request) { title := r.URL.Path\[len("/view/"):\] p, err := loadPage(title) if err != nil { http.Redirect(w, r, "/edit/"+title, http.StatusFound) return } renderTemplate(w, "view", p) } The `http.Redirect` function adds an HTTP status code of `http.StatusFound` (302) and a `Location` header to the HTTP response. Saving Pages ------------ The function `saveHandler` will handle the submission of forms located on the edit pages. After uncommenting the related line in `main`, let's implement the handler: func saveHandler(w http.ResponseWriter, r \*http.Request) { title := r.URL.Path\[len("/save/"):\] body := r.FormValue("body") p := &Page{Title: title, Body: \[\]byte(body)} p.save() http.Redirect(w, r, "/view/"+title, http.StatusFound) } The page title (provided in the URL) and the form's only field, `Body`, are stored in a new `Page`. The `save()` method is then called to write the data to a file, and the client is redirected to the `/view/` page. The value returned by `FormValue` is of type `string`. We must convert that value to `[]byte` before it will fit into the `Page` struct. We use `[]byte(body)` to perform the conversion. Error handling -------------- There are several places in our program where errors are being ignored. This is bad practice, not least because when an error does occur the program will have unintended behavior. A better solution is to handle the errors and return an error message to the user. That way if something does go wrong, the server will function exactly how we want and the user can be notified. First, let's handle the errors in `renderTemplate`: func renderTemplate(w http.ResponseWriter, tmpl string, p \*Page) { t, err := template.ParseFiles(tmpl + ".html") if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } err = t.Execute(w, p) if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) } } The `http.Error` function sends a specified HTTP response code (in this case "Internal Server Error") and error message. Already the decision to put this in a separate function is paying off. Now let's fix up `saveHandler`: func saveHandler(w http.ResponseWriter, r \*http.Request) { title := r.URL.Path\[len("/save/"):\] body := r.FormValue("body") p := &Page{Title: title, Body: \[\]byte(body)} err := p.save() if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } http.Redirect(w, r, "/view/"+title, http.StatusFound) } Any errors that occur during `p.save()` will be reported to the user. Template caching ---------------- There is an inefficiency in this code: `renderTemplate` calls `ParseFiles` every time a page is rendered. A better approach would be to call `ParseFiles` once at program initialization, parsing all templates into a single `*Template`. Then we can use the [`ExecuteTemplate`](https://go.dev/pkg/html/template/#Template.ExecuteTemplate) method to render a specific template. First we create a global variable named `templates`, and initialize it with `ParseFiles`. var templates = template.Must(template.ParseFiles("edit.html", "view.html")) The function `template.Must` is a convenience wrapper that panics when passed a non-nil `error` value, and otherwise returns the `*Template` unaltered. A panic is appropriate here; if the templates can't be loaded the only sensible thing to do is exit the program. The `ParseFiles` function takes any number of string arguments that identify our template files, and parses those files into templates that are named after the base file name. If we were to add more templates to our program, we would add their names to the `ParseFiles` call's arguments. We then modify the `renderTemplate` function to call the `templates.ExecuteTemplate` method with the name of the appropriate template: func renderTemplate(w http.ResponseWriter, tmpl string, p \*Page) { err := templates.ExecuteTemplate(w, tmpl+".html", p) if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) } } Note that the template name is the template file name, so we must append `".html"` to the `tmpl` argument. Validation ---------- As you may have observed, this program has a serious security flaw: a user can supply an arbitrary path to be read/written on the server. To mitigate this, we can write a function to validate the title with a regular expression. First, add `"regexp"` to the `import` list. Then we can create a global variable to store our validation expression: var validPath = regexp.MustCompile("^/(edit|save|view)/(\[a-zA-Z0-9\]+)$") The function `regexp.MustCompile` will parse and compile the regular expression, and return a `regexp.Regexp`. `MustCompile` is distinct from `Compile` in that it will panic if the expression compilation fails, while `Compile` returns an `error` as a second parameter. Now, let's write a function that uses the `validPath` expression to validate path and extract the page title: func getTitle(w http.ResponseWriter, r \*http.Request) (string, error) { m := validPath.FindStringSubmatch(r.URL.Path) if m == nil { http.NotFound(w, r) return "", errors.New("invalid Page Title") } return m\[2\], nil // The title is the second subexpression. } If the title is valid, it will be returned along with a `nil` error value. If the title is invalid, the function will write a "404 Not Found" error to the HTTP connection, and return an error to the handler. To create a new error, we have to import the `errors` package. Let's put a call to `getTitle` in each of the handlers: func viewHandler(w http.ResponseWriter, r \*http.Request) { title, err := getTitle(w, r) if err != nil { return } p, err := loadPage(title) if err != nil { http.Redirect(w, r, "/edit/"+title, http.StatusFound) return } renderTemplate(w, "view", p) } func editHandler(w http.ResponseWriter, r \*http.Request) { title, err := getTitle(w, r) if err != nil { return } p, err := loadPage(title) if err != nil { p = &Page{Title: title} } renderTemplate(w, "edit", p) } func saveHandler(w http.ResponseWriter, r \*http.Request) { title, err := getTitle(w, r) if err != nil { return } body := r.FormValue("body") p := &Page{Title: title, Body: \[\]byte(body)} err = p.save() if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } http.Redirect(w, r, "/view/"+title, http.StatusFound) } Introducing Function Literals and Closures ------------------------------------------ Catching the error condition in each handler introduces a lot of repeated code. What if we could wrap each of the handlers in a function that does this validation and error checking? Go's [function literals](https://go.dev/ref/spec#Function_literals) provide a powerful means of abstracting functionality that can help us here. First, we re-write the function definition of each of the handlers to accept a title string: func viewHandler(w http.ResponseWriter, r \*http.Request, title string) func editHandler(w http.ResponseWriter, r \*http.Request, title string) func saveHandler(w http.ResponseWriter, r \*http.Request, title string) Now let's define a wrapper function that _takes a function of the above type_, and returns a function of type `http.HandlerFunc` (suitable to be passed to the function `http.HandleFunc`): func makeHandler(fn func (http.ResponseWriter, \*http.Request, string)) http.HandlerFunc { return func(w http.ResponseWriter, r \*http.Request) { // Here we will extract the page title from the Request, // and call the provided handler 'fn' } } The returned function is called a closure because it encloses values defined outside of it. In this case, the variable `fn` (the single argument to `makeHandler`) is enclosed by the closure. The variable `fn` will be one of our save, edit, or view handlers. Now we can take the code from `getTitle` and use it here (with some minor modifications): func makeHandler(fn func(http.ResponseWriter, \*http.Request, string)) http.HandlerFunc { return func(w http.ResponseWriter, r \*http.Request) { m := validPath.FindStringSubmatch(r.URL.Path) if m == nil { http.NotFound(w, r) return } fn(w, r, m\[2\]) } } The closure returned by `makeHandler` is a function that takes an `http.ResponseWriter` and `http.Request` (in other words, an `http.HandlerFunc`). The closure extracts the `title` from the request path, and validates it with the `validPath` regexp. If the `title` is invalid, an error will be written to the `ResponseWriter` using the `http.NotFound` function. If the `title` is valid, the enclosed handler function `fn` will be called with the `ResponseWriter`, `Request`, and `title` as arguments. Now we can wrap the handler functions with `makeHandler` in `main`, before they are registered with the `http` package: func main() { http.HandleFunc("/view/", makeHandler(viewHandler)) http.HandleFunc("/edit/", makeHandler(editHandler)) http.HandleFunc("/save/", makeHandler(saveHandler)) log.Fatal(http.ListenAndServe(":8080", nil)) } Finally we remove the calls to `getTitle` from the handler functions, making them much simpler: func viewHandler(w http.ResponseWriter, r \*http.Request, title string) { p, err := loadPage(title) if err != nil { http.Redirect(w, r, "/edit/"+title, http.StatusFound) return } renderTemplate(w, "view", p) } func editHandler(w http.ResponseWriter, r \*http.Request, title string) { p, err := loadPage(title) if err != nil { p = &Page{Title: title} } renderTemplate(w, "edit", p) } func saveHandler(w http.ResponseWriter, r \*http.Request, title string) { body := r.FormValue("body") p := &Page{Title: title, Body: \[\]byte(body)} err := p.save() if err != nil { http.Error(w, err.Error(), http.StatusInternalServerError) return } http.Redirect(w, r, "/view/"+title, http.StatusFound) } Try it out! ----------- [Click here to view the final code listing.](https://go.dev/doc/articles/wiki/final.go) Recompile the code, and run the app: $ go build wiki.go $ ./wiki Visiting [http://localhost:8080/view/ANewPage](http://localhost:8080/view/ANewPage) should present you with the page edit form. You should then be able to enter some text, click 'Save', and be redirected to the newly created page. Other tasks ----------- Here are some simple tasks you might want to tackle on your own: * Store templates in `tmpl/` and page data in `data/`. * Add a handler to make the web root redirect to `/view/FrontPage`. * Spruce up the page templates by making them valid HTML and adding some CSS rules. * Implement inter-page linking by converting instances of `[PageName]` to `PageName`. (hint: you could use `regexp.ReplaceAllFunc` to do this) go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Go Wiki: Go talks - The Go Programming Language Go Wiki: Go talks ================= Check out [http://talks.golang.org](http://talks.golang.org/) for presentations for some of the talks. For a comprehensive, curated and searchable index, try [GopherVids](http://gophervids.appspot.com/) from Damian Gryski. Official -------- ### Introductory Talks An introduction to Go. #### Russ Cox’s Tour of Go \[[video and discussion](http://research.swtch.com/gotour)\ \] Three things that make Go fast, fun, and productive:interfaces, reflection, and concurrency. Builds a toy web crawler to demonstrate these. #### Go: a simple programming environment \[[video](http://vimeo.com/53221558)\ \] \[[another video](http://vimeo.com/69237265)\ \] \[[slides](http://talks.golang.org/2012/simple.slide)\ \] Go is a general-purpose language that bridges the gap between efficient statically typed languages and productive dynamic language. But it’s not just the language that makes Go special – Go has broad and consistent standard libraries and powerful but simple tools. This talk gives an introduction to Go, followed by a tour of some real programs that demonstrate the power, scope, and simplicity of the Go programming environment. #### Get Started with Go \[[video](http://www.youtube.com/watch?v=2KmHtgtEZ1s)\ \] Get a feel for the language and its standard libraries and tools in this session, where we go through installing Go and writing some simple but useful programs. #### Go Programming \[[video](http://www.youtube.com/watch?v=jgVhBThJdXc)\ \] \[[code](http://talks.golang.org/2010/io/)\ \] A presentation delivered by Rob Pike and Russ Cox at Google I/O 2010. It illustrates how programming in Go differs from other languages through a set of examples demonstrating features particular to Go. These include concurrency, embedded types, methods on any type, and program construction using interfaces. #### The Go Tech Talk \[[video](http://www.youtube.com/watch?v=rKnDgT73v8s)\ \] \[[slides](http://talks.golang.org/2009/go_talk-20091030.pdf)\ \] An hour-long talk delivered by Rob Pike at Google in October 2009. The language’s first public introduction. The language has changed since it was made, but it’s still a good introduction. ### Development in Go #### Writing Web Apps in Go \[[video](http://www.youtube.com/watch?v=-i0hat7pdpk)\ \] \[[slides](http://talks.golang.org/2011/Writing_Web_Apps_in_Go.pdf)\ \] A talk by Rob Pike and Andrew Gerrand presented at Google I/O 2011. It walks through the construction and deployment of a simple web application and unveils the [Go runtime for App Engine](https://go.dev/blog/2011/05/go-and-google-app-engine.html) . #### Real World Go \[[video](http://www.youtube.com/watch?v=7QDVRowyUQA)\ \] \[[slides](http://talks.golang.org/2011/Real_World_Go.pdf)\ \] A talk by Andrew Gerrand presented at Google I/O Bootcamp 2011. It gives a broad overview of Go’s type system and concurrency model and provides four examples of Go programs that solve real problems. #### Building Integrated Apps on Google’s Cloud Platform \[[video](http://www.youtube.com/watch?v=Mo1YKpIF1PQ)\ \] A talk by Andrew Gerrand presented at Google Developer Day Japan 2011. It discusses the development of a web application that runs on Google App Engine and renders raytraced that it stores on Google Cloud Storage. #### High Performance Apps with Go on App Engine Google I/O, May 2013 \[[video](http://www.youtube.com/watch?v=fc25ihfXhbg)\ \] \[[slides](http://talks.golang.org/2013/highperf.slide)\ \] #### Practical Go Programming \[[video](http://www.youtube.com/watch?v=2-pPAvqyluI)\ \] \[[slides](http://wh3rd.net/practical-go)\ \] \[[code](http://github.com/nf/goto)\ \] This talk presents the development of a complete web application in Go. It looks at design, storage, concurrency, and scaling issues in detail, using the simple example of an URL shortening service. #### Lexical Scanning in Go \[[video](http://www.youtube.com/watch?v=HxaD_trXwRE)\ \] This GTUG talk by Rob Pike discusses the detailed design of a lexical scanner that uses Go’s features in expressive combinations. (The discussion near the end about avoiding goroutines at initialization is obsolete: Go 1 allows goroutines in init functions so the extra complexity is unnecessary.) #### Go in Production Google I/O, June 2012 \[[video](http://www.youtube.com/watch?v=kKQLhGZVN4A)\ \] Since Go’s release in 2009 many companies (besides Google, of course) have used the language to build cool stuff. In this session programmers from several companies will share their first-hand experience using Go in production environments. #### Go: code that grows with grace \[[video](http://vimeo.com/53221560)\ \] \[[slides](http://talks.golang.org/2012/chat.slide)\ \] One of the Go Programming Language’s key design goals is code adaptability; that it should be easy to take a simple design and build upon it in a clean and natural way. In this talk I describe a simple “chat roulette” server that matches pairs of incoming TCP connections, and then use Go’s concurrency mechanisms, interfaces, and standard library to extend it with a web interface and other features. Although the function of the program changes dramatically, the inherent flexibility of Go allows the original design to remain intact as it grows. #### Implementing a bignum calculator \[[video](https://www.youtube.com/watch?v=PXoG0WX0r_E)\ \] \[[slides](http://go-talks.appspot.com/github.com/robpike/ivy/talks/ivy.slide)\ \] Rob Pike describes his interpreter for an APL-like calculator language. #### Go in Go \[[video](https://www.youtube.com/watch?v=cF1zJYkBW4A)\ \] \[[slides](https://talks.golang.org/2015/gogo.slide)\ \] Rob Pike speaks on moving the Go toolchain from C to Go ### Concurrency in Go #### Go concurrency patterns Google I/O, June 2012 \[[video](http://www.youtube.com/watch?v=f6kdp27TYZs)\ \] #### Advanced Concurrency Patterns \[[video](https://www.youtube.com/watch?v=QDDwwePbDtw)\ \] \[[slides](http://talks.golang.org/2013/advconc.slide)\ \] Google I/0, May 2013 Concurrency is the key to designing high performance network services. This talk expands on last year’s popular Go Concurrency Patterns talk to dive deeper into Go’s concurrency primitives, and see how tricky concurrency problems can be solved gracefully with simple Go code. ### Design of Go #### The Expressiveness Of Go \[[slides](http://talks.golang.org/2010/ExpressivenessOfGo-2010.pdf)\ \] A discussion of the qualities that make Go an expressive and comprehensible language. The talk was presented by Rob Pike at JAOO 2010. The recording of the event was lost due to a hardware error. #### Another Go at Language Design \[[video](http://sydney.edu.au/engineering/it/videos/seminar_pike)\ from Sydney University\] \[[slides](http://assets.en.oreilly.com/1/event/45/Another%20Go%20at%20Language%20Design%20Presentation.pdf)\ \] A tour, with some background, of the major features of Go, intended for an audience new to the language. The talk was presented at OSCON 2010. This talk was also delivered at Sydney University in September 2010. #### Go Emerging Languages Conference Talk \[[video](http://confreaks.com/videos/115-elcamp2010-go)\ \] \[[slides](http://assets.en.oreilly.com/1/event/45/Go%20Presentation.pdf)\ \] Rob Pike’s Emerging Languages Conference presentation delivered in July 2010. Talk abstract: > Go’s approach to concurrency differs from that of many languages, even those (such as Erlang) that make concurrency central, yet it has deep roots. The path from Hoare’s 1978 paper to Go provides insight into how and why Go works as it does. ### The State of Go #### June 2014 \[[video](https://www.youtube.com/watch?v=0KF44QtoByk)\ \] \[[slides](https://talks.golang.org/2014/state-of-go.slide)\ \] #### February 2015 \[[video](https://www.youtube.com/watch?v=Kd8EqTvW5EQ)\ \] \[[slides](https://talks.golang.org/2015/state-of-go.slide)\ \] #### May 2015 \[[video](https://www.youtube.com/watch?v=S9Bu6fZnLGM)\ \] \[[slides](https://talks.golang.org/2015/state-of-go-may.slide)\ \] ### Miscellaneous #### The Go frontend for GCC \[[paper](http://talks.golang.org/2010/gofrontend-gcc-summit-2010.pdf)\ \] A description of the Go language frontend for gcc. Ian Lance Taylor’s paper delivered at the GCC Summit 2010. #### The Go Promo Video \[[video](http://www.youtube.com/watch?v=wwoWei-GAPo)\ \] A short promotional video featuring Russ Cox demonstrating Go’s fast compiler. #### Meet the Go team Google I/O, June 2012 \[[video](http://www.youtube.com/watch?v=sln-gJaURzk)\ \] A panel discussion with David Symonds, Robert Griesemer, Rob Pike, Ken Thompson, Andrew Gerrand, and Brad Fitzpatrick. #### Fireside Chat with Go team Google I/0, May 2013 \[[video](http://www.youtube.com/watch?v=p9VUCp98ay4)\ \] A fireside chat with Andrew Gerrand, Brad Fitzpatrick, David Symonds, Ian Lance Taylor, Nigel Tao, Rob Pike, Robert Griesemer, Sameer Ajmani. #### The State of the Gopher \[[video](https://www.youtube.com/watch?v=4KFTacxqkcQ)\ \] \[[slides](https://talks.golang.org/2014/state-of-the-gopher.slide)\ \] Unofficial ---------- Talks by members of the community. #### Let’s Go, or introduction to Go \[[video (starting at 14:35)](http://live.digicast.ru/view/1582)\ \] \[[slides](http://talks.godoc.org/github.com/AlekSi/LetsGo/lets-go.slide)\ \] \[[source](https://github.com/AlekSi/LetsGo)\ \] This talk gives an introduction to Go in Russian. #### What are Go modules and how do I use them? _[Paul Jolly](https://twitter.com/_myitcv) at [**LondonGophers**](https://twitter.com/LondonGophers) _ \[[video](https://youtu.be/6MbIzJmLz6Q)\ \] \[[slides](https://talks.godoc.org/github.com/myitcv/talks/2018-08-15-glug-modules/main.slide#1)\ \] #### What else is in Go 1.11 \_[Daniel Martì](https://twitter.com/mvdan_) at [**LondonGophers**](https://twitter.com/LondonGophers) \_ \[[video](https://youtu.be/mQYjjVCGVJ8)\ \] \[[slides](https://talks.godoc.org/github.com/mvdan/talks/2018/go1.11.slide#1)\ \] Sneak peak at the Go 1.11 release #### Get Going with WebAssembly _[Johan Brandhorst](https://twitter.com/JohanBrandhorst) at [**LondonGophers**](https://twitter.com/LondonGophers) _ \[[video](https://youtu.be/iTrx0BbUXI4)\ \] \[[slides](https://talks.godoc.org/github.com/johanbrandhorst/presentations/wasm-lightning/wasm.slide#1)\ \] \[[code wasm](https://github.com/johanbrandhorst/wasm-experiments)\ \] \[[code grpc](https://github.com/johanbrandhorst/grpcweb-wasm-example)\ \] In this talk, Johan introduces you to the WebAssembly port in Go 1.11 and how it can help when dealing with JavaScript madness :) #### Go and Mongo - and how it’s changing _[DJ Walker-Morgan](https://twitter.com/codepope) at [**LondonGophers**](https://twitter.com/LondonGophers) _ \[[video](https://youtu.be/W22tZ5p3aDk)\ \] \[[slides](https://github.com/codepope/talks/blob/master/GoAndMongo.pdf)\ \] #### Building a simple concurrency teaching language with Go _[Nicholas Ng](https://twitter.com/nicholascwng) at [**LondonGophers**](https://twitter.com/LondonGophers) _ \[[video](https://youtu.be/7cEp98y6WCs)\ \] \[[slides](https://talks.godoc.org/github.com/nickng/londongophers-aug18/londongophers-aug18.slide#1)\ \] In this talk Nicholas presents the design and implementation of a simple language designed for teaching concurrency theory (process calculi), implemented in Go. He covers some of Go’s static analysis tools used in the implementation and show how you can use them too! #### Introducing Remoto _[Mat Ryer](https://twitter.com/matryer) at [**LondonGophers**](https://twitter.com/LondonGophers) _ \[[video](https://youtu.be/dhbq7R7h-C0)\ \] Mat shares the first glimpse of a new project that aims to make building RPC services easy. gRPC isn’t good for clients (especially web), and RESTful designs sometimes lead to confusing APIs. Remoto lets you define your service with a Go interface, and generate everything you need to build and consume the service. #### Go Swagger _[Simone Trubian](https://twitter.com/simone_trubian) at [**LondonGophers**](https://twitter.com/LondonGophers) _ \[[video](https://youtu.be/PUejMR82RgU)\ \] Simone gives an overview of the Go Swagger command line tool and briefly explain how he used it to improve productivity in designing REST API’s. #### ORMs in Go _Renato Serra at [**LondonGophers**](https://twitter.com/LondonGophers) _ \[[video](https://youtu.be/0XW6wI2FnPA)\ \] Renato explains where ORMs can help, what the options were and what it’s been like to use one. #### Unused parameters in Go code \_[Daniel Martì](https://twitter.com/mvdan_) at [**LondonGophers**](https://twitter.com/LondonGophers) \_ \[[video](https://youtu.be/VW5jI6V_Y2c)\ \] \[[slides](https://talks.godoc.org/github.com/mvdan/talks/2018/unparam.slide#1)\ \] Daniel talks about how to use SSA and callgraphs to write powerful code analysis tools. In particular, he demonstrates how to detect unused parameters in functions. #### Lies, Damn Lies, and Benchmarks _Amnon at [**LondonGophers**](https://twitter.com/LondonGophers) _ \[[video](https://youtu.be/YDPKUJndhw4)\ \] \[[slides](https://talks.godoc.org/github.com/amnonbc/talks/lies.slide#1)\ \] Amnon discusses why microbenchmarks can be misleading for optimising real world systems, why data layout is often more significant than code structure, and how Go can help us in the quest for performance. #### A debugger from scratch _[Liz Rice](https://twitter.com/LizRice) at [**LondonGophers**](https://twitter.com/LondonGophers) _ \[[video](https://youtu.be/tZ5PUKbGjO4)\ \] \[[slides](https://speakerdeck.com/lizrice/debuggers-from-scratch)\ \] \[[code](https://github.com/lizrice/debugger-from-scratch)\ \] Liz explains how a debugger works by building one in a few lines of Go. This includes mapping between Go source code and the machine code instructions it compiles to, and using the ptrace system call to set break points and examine and modify the running process. #### Fast Fractal Fun With SDL _[Sue Spence](https://twitter.com/virtualsue) at [**LondonGophers**](https://twitter.com/LondonGophers) _ \[[video](https://youtu.be/eTjL3grAYAM)\ \] \[[slides](https://gitlab.com/virtualsue/sdl-fractal/blob/master/Fast%20Fractal%20Fun.pdf)\ \] \[[code](https://gitlab.com/virtualsue/sdl-fractal)\ \] Go programs which create images such as the Mandelbrot & Julia sets often output an image file. I will show how to use Go bindings for the Simple Directmedia Layer library to output them on a display device instead. #### Concurrency: a Journey from Ruby to Go _[Mathilda Thompson](https://twitter.com/mathildathompso) at [**LondonGophers**](https://twitter.com/LondonGophers) _ \[[video](https://youtu.be/mK3r5PDED-0)\ \] #### Go in a Polyglot Environment _[Kevin McKelvin](https://twitter.com/kmckelvin) at [**LondonGophers**](https://twitter.com/LondonGophers) _ \[[video](https://youtu.be/kWAxBhsEayk)\ \] In this talk Kevin goes through his experience of adopting Go, moving to a polyglot environment, successes and challenges, and how Go fits into his company’s overall architecture and strategy. #### Delivering Go Services _[Zak Knill](https://twitter.com/zakknill) at [**LondonGophers**](https://twitter.com/LondonGophers) _ \[[video](https://youtu.be/pRdfJTuGxEw)\ \] Delivering Go Services: After introducing Go to your company, and deploying your first go service. What are the next steps? This talk focuses on some of the things that come next, touching on the fabled “New service to prod in X (10, 20, 30) mins”, as well as some gotchas along the way. #### Go-ing Lambda _[David Blooman](https://twitter.com/dblooman) at [**LondonGophers**](https://twitter.com/LondonGophers) _ \[[video](https://youtu.be/BBiIr19JOo4)\ \] Go-ing Lambda - A year in production: How we(FundApps) used Go in lambda functions to build a service for importing/scraping/parsing data for financial services to build API’s on top of. Tips and tricks of lambda functions in Go, limitations, performance and using the Apex framework. #### The RED method _[Tom Wilkie](https://twitter.com/tom_wilkie) at [**LondonGophers**](https://twitter.com/LondonGophers) _ \[[video](https://youtu.be/rc3V-k-JYAo)\ \] We’ll also have a section dedicated to those of you who are hiring or looking to get hired (if we’ll miss it like last time, please don’t be afraid to remind us). #### Abusing Go’s net package for fun and profit _[Michał Witkowski](https://twitter.com/MWitkow) at [**LondonGophers**](https://twitter.com/LondonGophers) _ \[[video](https://youtu.be/JDjHFmke0ZI)\ \] This talks into the details of how Go’s composition-based philosophy, as applied to the net package, can be creatively leveraged to beautiful and useful hacks that significantly augment the functionality of the stack. We’ll explore the net.Conn, and how one can (ab)use them in creative ways. We’ll take a peek into net/http, and explore how the http.Handler and http.Roundtripper interfaces can be creatively appropriated to build useful middleware. We’ll then dig even deeper into the net/http internals and how they related tls.Conn and x/net/http2, to understand how they work, and armed with that knowledge we’ll demonstrate some of our most beautiful hacks. #### 2018’s stringer \_[Daniel Martì](https://twitter.com/mvdan_) at [**LondonGophers**](https://twitter.com/LondonGophers) \_ \[[video](https://youtu.be/IyVEW19IkXE)\ \] \[[slides](https://talks.godoc.org/github.com/mvdan/talks/2018/stringer.slide)\ \] 2018’s stringer - a demonstration of new features you likely haven’t heard of. * * * _This content is part of the [Go Wiki](https://go.dev/wiki/) ._ go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Contribution Guide - The Go Programming Language 1. [Documentation](https://go.dev/doc/) 2. [Contribution Guide](https://go.dev/doc/contribute) Contribution Guide ================== The Go project welcomes all contributors. This document is a guide to help you through the process of contributing to the Go project, which is a little different from that used by other open source projects. We assume you have a basic understanding of Git and Go. In addition to the information here, the Go community maintains a [CodeReview](https://go.dev/wiki/CodeReview) wiki page. Feel free to contribute to the wiki as you learn the review process. Note that the `gccgo` front end lives elsewhere; see [Contributing to gccgo](https://go.dev/doc/gccgo_contribute.html) . Becoming a contributor ---------------------- ### Overview The first step is registering as a Go contributor and configuring your environment. Here is a checklist of the required steps to follow: * **Step 0**: Decide on a single Google Account you will be using to contribute to Go. Use that account for all the following steps and make sure that `git` is configured to create commits with that account's e-mail address. * **Step 1**: [Sign and submit](https://cla.developers.google.com/clas) a CLA (Contributor License Agreement). * **Step 2**: Configure authentication credentials for the Go Git repository. Visit [go.googlesource.com](https://go.googlesource.com/) , click "Generate Password" in the page's top right menu bar, and follow the instructions. * **Step 3**: Register for Gerrit, the code review tool used by the Go team, by [visiting this page](https://go-review.googlesource.com/login/) . The CLA and the registration need to be done only once for your account. * **Step 4**: Install `git-codereview` by running `go install golang.org/x/review/git-codereview@latest` If you prefer, there is an automated tool that walks through these steps. Just run: $ go install golang.org/x/tools/cmd/go-contrib-init@latest $ cd /code/to/edit $ go-contrib-init The rest of this chapter elaborates on these instructions. If you have completed the steps above (either manually or through the tool), jump to [Before contributing code](https://go.dev/doc/contribute#before_contributing) . ### Step 0: Select a Google Account A contribution to Go is made through a Google account with a specific e-mail address. Make sure to use the same account throughout the process and for all your subsequent contributions. You may need to decide whether to use a personal address or a corporate address. The choice will depend on who will own the copyright for the code that you will be writing and submitting. You might want to discuss this topic with your employer before deciding which account to use. Google accounts can either be Gmail e-mail accounts, G Suite organization accounts, or accounts associated with an external e-mail address. For instance, if you need to use an existing corporate e-mail that is not managed through G Suite, you can create an account associated [with your existing e-mail address](https://accounts.google.com/SignUpWithoutGmail) . You also need to make sure that your Git tool is configured to create commits using your chosen e-mail address. You can either configure Git globally (as a default for all projects), or locally (for a single specific project). You can check the current configuration with this command: $ git config --global user.email # check current global config $ git config user.email # check current local config To change the configured address: $ git config --global user.email name@example.com # change global config $ git config user.email name@example.com # change local config ### Step 1: Contributor License Agreement Before sending your first change to the Go project you must have completed one of the following two CLAs. Which CLA you should sign depends on who owns the copyright to your work. * If you are the copyright holder, you will need to agree to the [individual contributor license agreement](https://developers.google.com/open-source/cla/individual) , which can be completed online. * If your organization is the copyright holder, the organization will need to agree to the [corporate contributor license agreement](https://developers.google.com/open-source/cla/corporate) . You can check your currently signed agreements and sign new ones at the [Google Developers Contributor License Agreements](https://cla.developers.google.com/clas?pli=1&authuser=1) website. If the copyright holder for your contribution has already completed the agreement in connection with another Google open source project, it does not need to be completed again. If the copyright holder for the code you are submitting changes—for example, if you start contributing code on behalf of a new company—please send mail to the [`golang-dev` mailing list](mailto:golang-dev@googlegroups.com) . This will let us know the situation so we can make sure an appropriate agreement is completed. ### Step 2: Configure git authentication The main Go repository is located at [go.googlesource.com](https://go.googlesource.com/) , a Git server hosted by Google. Authentication on the web server is made through your Google account, but you also need to configure `git` on your computer to access it. Follow these steps: 1. Visit [go.googlesource.com](https://go.googlesource.com/) and click on "Generate Password" in the page's top right menu bar. You will be redirected to accounts.google.com to sign in. 2. After signing in, you will be taken to a page with the heading "Configure Git". This page contains a personalized script that when run locally will configure Git to hold your unique authentication key. This key is paired with one that is generated and stored on the server, analogous to how SSH keys work. 3. Copy and run this script locally in your terminal to store your secret authentication token in a `.gitcookies` file. If you are using a Windows computer and running `cmd`, you should instead follow the instructions in the yellow box to run the command; otherwise run the regular script. ### Step 3: Create a Gerrit account Gerrit is an open-source tool used by Go maintainers to discuss and review code submissions. To register your account, visit [go-review.googlesource.com/login/](https://go-review.googlesource.com/login/) and sign in once using the same Google Account you used above. ### Step 4: Install the git-codereview command Changes to Go must be reviewed before they are accepted, no matter who makes the change. A custom `git` command called `git-codereview` simplifies sending changes to Gerrit. Install the `git-codereview` command by running, $ go install golang.org/x/review/git-codereview@latest Make sure `git-codereview` is installed in your shell path, so that the `git` command can find it. Check that $ git codereview help prints help text, not an error. If it prints an error, make sure that `$GOPATH/bin` is in your `$PATH`. On Windows, when using git-bash you must make sure that `git-codereview.exe` is in your `git` exec-path. Run `git --exec-path` to discover the right location then create a symbolic link or just copy the executable from `$GOPATH/bin` to this directory. Before contributing code ------------------------ The project welcomes code patches, but to make sure things are well coordinated you should discuss any significant change before starting the work. It's recommended that you signal your intention to contribute in the issue tracker, either by [filing a new issue](https://go.dev/issue/new) or by claiming an [existing one](https://go.dev/issues) . ### Where to contribute The Go project consists of the main [go](https://go.googlesource.com/go) repository, which contains the source code for the Go language, as well as many golang.org/x/... repositories. These contain the various tools and infrastructure that support Go. For example, [golang.org/x/pkgsite](https://go.googlesource.com/pkgsite) is for [pkg.go.dev](https://pkg.go.dev/) , [golang.org/x/playground](https://go.googlesource.com/playground) is for the Go playground, and [golang.org/x/tools](https://go.googlesource.com/tools) contains a variety of Go tools, including the Go language server, [gopls](https://go.dev/s/gopls) . You can see a list of all the golang.org/x/... repositories on [go.googlesource.com](https://go.googlesource.com/) . ### Check the issue tracker Whether you already know what contribution to make, or you are searching for an idea, the [issue tracker](https://github.com/golang/go/issues) is always the first place to go. Issues are triaged to categorize them and manage the workflow. The majority of the golang.org/x/... repos also use the main Go issue tracker. However, a few of these repositories manage their issues separately, so please be sure to check the right tracker for the repository to which you would like to contribute. Most issues will be marked with one of the following workflow labels: * **NeedsInvestigation**: The issue is not fully understood and requires analysis to understand the root cause. * **NeedsDecision**: the issue is relatively well understood, but the Go team hasn't yet decided the best way to address it. It would be better to wait for a decision before writing code. If you are interested in working on an issue in this state, feel free to "ping" maintainers in the issue's comments if some time has passed without a decision. * **NeedsFix**: the issue is fully understood and code can be written to fix it. You can use GitHub's search functionality to find issues to help out with. Examples: * Issues that need investigation: [`is:issue is:open label:NeedsInvestigation`](https://github.com/golang/go/issues?q=is%3Aissue+is%3Aopen+label%3ANeedsInvestigation) * Issues that need a fix: [`is:issue is:open label:NeedsFix`](https://github.com/golang/go/issues?q=is%3Aissue+is%3Aopen+label%3ANeedsFix) * Issues that need a fix and have a suggested change: [`is:issue is:open label:NeedsFix ("golang.org/cl" OR "go.dev/cl")`](https://github.com/golang/go/issues?q=is%3Aissue+is%3Aopen+label%3ANeedsFix+%28%22golang.org%2Fcl%22+OR+%22go.dev%2Fcl%22%29) * Issues that need a fix and do not have a suggested change: [`is:issue is:open label:NeedsFix NOT "golang.org/cl" NOT "go.dev/cl"`](https://github.com/golang/go/issues?q=is%3Aissue+is%3Aopen+label%3ANeedsFix+NOT+%22golang.org%2Fcl%22+NOT+%22go.dev%2Fcl%22) ### Open an issue for any new problem Excluding very trivial changes, all contributions should be connected to an existing issue. Feel free to open one and discuss your plans. This process gives everyone a chance to validate the design, helps prevent duplication of effort, and ensures that the idea fits inside the goals for the language and tools. It also checks that the design is sound before code is written; the code review tool is not the place for high-level discussions. When planning work, please note that the Go project follows a [six-month development cycle](https://go.dev/wiki/Go-Release-Cycle) for the main Go repository. The latter half of each cycle is a three-month feature freeze during which only bug fixes and documentation updates are accepted. New contributions can be sent during a feature freeze, but they will not be merged until the freeze is over. The freeze applies to the entire main repository as well as to the code in golang.org/x/... repositories that is needed to build the binaries included in the release. See the lists of packages vendored into [the standard library](https://github.com/golang/go/blob/master/src/vendor/modules.txt) and the [`go` command](https://github.com/golang/go/blob/master/src/cmd/vendor/modules.txt) . Significant changes to the language, libraries, or tools (which includes API changes in the main repo and all golang.org/x repos, as well as command-line changes to the `go` command) must go through the [change proposal process](https://go.dev/s/proposal-process) before they can be accepted. Sensitive security-related issues (only!) should be reported to [security@golang.org](mailto:security@golang.org) . Sending a change via GitHub --------------------------- First-time contributors that are already familiar with the [GitHub flow](https://guides.github.com/introduction/flow/) are encouraged to use the same process for Go contributions. Even though Go maintainers use Gerrit for code review, a bot called Gopherbot has been created to sync GitHub pull requests to Gerrit. Open a GitHub pull request as you normally would. Gopherbot will create a corresponding Gerrit change list (a "CL") and post a link to it on your GitHub pull request; updates to the pull request will also get reflected in the Gerrit CL. When somebody comments on the CL, their comment will be also posted in your pull request, so you will get a notification. Some things to keep in mind: * You will need a [Gerrit account](https://go-review.googlesource.com/login/) to respond to your reviewers, including to [mark feedback as 'Done'](https://go.dev/doc/contribute#reviews) if implemented as suggested. It is a good idea to familiarize yourself with Gerrit, such as by [skimming the open CLs](https://go-review.googlesource.com/q/status:open+-is:wip) , subscribing to updates on interesting CLs (via the star icon), or [reviewing or giving a +1](https://go.dev/wiki/Gardening#pending-cls) to other people's CLs. * To update the pull request with new code, just push it to the branch; you can either add more commits, or rebase and force-push (both styles are accepted). * If the request is accepted, all commits will be squashed, and the final commit description will be composed by concatenating the pull request's title and description. The individual commits' descriptions will be discarded. See [Writing good commit messages](https://go.dev/doc/contribute#commit_messages) for some suggestions. * See the [FAQ](https://go.dev/wiki/GerritBot#frequently-asked-questions) for more details. Sending a change via Gerrit --------------------------- It is not possible to fully sync Gerrit and GitHub, at least at the moment, so we recommend learning Gerrit. It's different but powerful and familiarity with it will help you understand the flow. The following sections give a brief walkthrough of submitting a change via Gerrit. For more details on interacting with Gerrit, refer to [Gerrit's documentation](https://gerrit-review.googlesource.com/Documentation/index.html) . In particular, we recommend reading the [Review UI Overview](https://gerrit-review.googlesource.com/Documentation/user-review-ui.html) and [Basic Gerrit Walkthrough — For GitHub Users](https://gerrit-review.googlesource.com/Documentation/intro-gerrit-walkthrough-github.html) pages. ### Overview This is an overview of the overall process: * **Step 1:** Clone the source code from `go.googlesource.com` and make sure it's stable by compiling and testing it once. If you're making a change to the [main Go repository](https://go.googlesource.com/go) : $ git clone https://go.googlesource.com/go $ cd go/src $ ./all.bash # compile and test If you're making a change to one of the golang.org/x/... repositories ([golang.org/x/tools](https://go.googlesource.com/tools) , in this example): $ git clone https://go.googlesource.com/tools $ cd tools $ go test ./... # compile and test * **Step 2:** Prepare changes in a new branch, created from the master branch. To commit the changes, use `git` `codereview` `change`; that will create or amend a single commit in the branch. $ git checkout -b mybranch $ \[edit files...\] $ git add \[files...\] $ git codereview change # create commit in the branch $ \[edit again...\] $ git add \[files...\] $ git codereview change # amend the existing commit with new changes $ \[etc.\] * **Step 3:** Test your changes, either by running the tests in the package you edited or by re-running `all.bash`. In the main Go repository: $ ./all.bash # recompile and test In a golang.org/x/... repository: $ go test ./... # recompile and test * **Step 4:** Send the changes for review to Gerrit using `git` `codereview` `mail` (which doesn't use e-mail, despite the name). $ git codereview mail # send changes to Gerrit * **Step 5:** After a review, apply changes to the same single commit and mail them to Gerrit again: $ \[edit files...\] $ git add \[files...\] $ git codereview change # update same commit $ git codereview mail # send to Gerrit again The rest of this section describes these steps in more detail. ### Step 1: Clone the source code In addition to a recent Go installation, you need to have a local copy of the source checked out from the correct repository. You can check out the Go source repo onto your local file system anywhere you want as long as it's outside your `GOPATH` (defaults to the directory `go` in your home directory). Clone from `go.googlesource.com` (not GitHub): Main Go repository: $ git clone https://go.googlesource.com/go $ cd go golang.org/x/... repository ([golang.org/x/tools](https://go.googlesource.com/tools) in this example): $ git clone https://go.googlesource.com/tools $ cd tools ### Step 2: Prepare changes in a new branch Each Go change must be made in a separate branch, created from the master branch. You can use the normal `git` commands to create a branch and add changes to the staging area: $ git checkout -b mybranch $ \[edit files...\] $ git add \[files...\] To commit changes, instead of `git commit`, use `git codereview change`. $ git codereview change (open $EDITOR) You can edit the commit description in your favorite editor as usual. The `git` `codereview` `change` command will automatically add a unique Change-Id line near the bottom. That line is used by Gerrit to match successive uploads of the same change. Do not edit or delete it. A Change-Id looks like this: Change-Id: I2fbdbffb3aab626c4b6f56348861b7909e3e8990 The tool also checks that you've run `go` `fmt` over the source code, and that the commit message follows the [suggested format](https://go.dev/doc/contribute#commit_messages) . If you need to edit the files again, you can stage the new changes and re-run `git` `codereview` `change`: each subsequent run will amend the existing commit while preserving the Change-Id. Make sure that you always keep a single commit in each branch. If you add more commits by mistake, you can use `git` `rebase` to [squash them together](https://stackoverflow.com/questions/31668794/squash-all-your-commits-in-one-before-a-pull-request-in-github) into a single one. ### Step 3: Test your changes You've [written and tested your code](https://go.dev/doc/code.html) , but before sending code out for review, run _all the tests for the whole tree_ to make sure the changes don't break other packages or programs. #### In the main Go repository For standard library packages, all tests within the package must pass: $ go test The short test suite for the entire tree can be run with `all.bash` (To build under Windows use `all.bat`): $ cd go/src $ ./all.bash After running for a while and printing a lot of testing output, the command should finish by printing, ALL TESTS PASSED You can use `make.bash` instead of `all.bash` to just build the compiler and the standard library without running the test suite. Once the `go` tool is built, it will be installed as `bin/go` under the directory in which you cloned the Go repository, and you can run it directly from there. See also the section on how to [test your changes quickly](https://go.dev/doc/contribute#quick_test) . #### In the golang.org/x/... repositories Run the tests for the entire repository ([golang.org/x/tools](https://go.googlesource.com/tools) , in this example): $ cd tools $ go test ./... If you're concerned about the build status, you can check the [Build Dashboard](https://build.golang.org/) . Test failures may also be caught by the TryBots in code review. Some repositories, like [golang.org/x/vscode-go](https://go.googlesource.com/vscode-go) will have different testing infrastructures, so always check the documentation for the repository in which you are working. The README file in the root of the repository will usually have this information. ### Step 4: Send changes for review Once the change is ready and tested over the whole tree, send it for review. This is done with the `mail` sub-command which, despite its name, doesn't directly mail anything; it just sends the change to Gerrit: $ git codereview mail Gerrit assigns your change a number and URL, which `git` `codereview` `mail` will print, something like: remote: New Changes: remote: https://go-review.googlesource.com/99999 math: improved Sin, Cos and Tan precision for very large arguments If you get an error instead, check the [Troubleshooting mail errors](https://go.dev/doc/contribute#troubleshooting_mail) section. If your change relates to an open GitHub issue and you have followed the [suggested commit message format](https://go.dev/doc/contribute#commit_messages) , the issue will be updated in a few minutes by a bot, linking your Gerrit change to it in the comments. ### Step 5: Revise changes after a review Go maintainers will review your code on Gerrit, and you will get notifications via e-mail. You can see the review on Gerrit and comment on them there. You can also reply [using e-mail](https://gerrit-review.googlesource.com/Documentation/intro-user.html#reply-by-email) if you prefer. If you need to revise your change after the review, edit the files in the same branch you previously created, add them to the Git staging area, and then amend the commit with `git` `codereview` `change`: $ git codereview change # amend current commit (open $EDITOR) $ git codereview mail # send new changes to Gerrit If you don't need to change the commit description, just save and exit from the editor. Remember not to touch the special Change-Id line. Again, make sure that you always keep a single commit in each branch. If you add more commits by mistake, you can use `git rebase` to [squash them together](https://stackoverflow.com/questions/31668794/squash-all-your-commits-in-one-before-a-pull-request-in-github) into a single one. Good commit messages -------------------- Commit messages in Go follow a specific set of conventions, which we discuss in this section. Here is an example of a good one: math: improve Sin, Cos and Tan precision for very large arguments The existing implementation has poor numerical properties for large arguments, so use the McGillicutty algorithm to improve accuracy above 1e10. The algorithm is described at https://wikipedia.org/wiki/McGillicutty\_Algorithm Fixes #159 ### First line The first line of the change description is conventionally a short one-line summary of the change, prefixed by the primary affected package. A rule of thumb is that it should be written so to complete the sentence "This change modifies Go to \_\_\_\_\_." That means it does not start with a capital letter, is not a complete sentence, and actually summarizes the result of the change. Follow the first line by a blank line. ### Main content The rest of the description elaborates and should provide context for the change and explain what it does. Write in complete sentences with correct punctuation, just like for your comments in Go. Don't use HTML, Markdown, or any other markup language. The text should be wrapped at around 72 columns. See [Commit messages](https://go.dev/wiki/CommitMessage) for additional details. Add any relevant information, such as benchmark data if the change affects performance. The [benchstat](https://godoc.org/golang.org/x/perf/cmd/benchstat) tool is conventionally used to format benchmark data for change descriptions. ### Referencing issues The special notation "Fixes #12345" associates the change with issue 12345 in the [Go issue tracker](https://go.dev/issue/12345) . When this change is eventually applied, the issue tracker will automatically mark the issue as fixed. If the change is a partial step towards the resolution of the issue, write "Updates #12345" instead. This will leave a comment in the issue linking back to the change in Gerrit, but it will not close the issue when the change is applied. If you are sending a change against a golang.org/x/... repository, you must use the fully-qualified syntax supported by GitHub to make sure the change is linked to the issue in the main repository, not the x/ repository. Most issues are tracked in the main repository's issue tracker. The correct form is "Fixes golang/go#159". The review process ------------------ This section explains the review process in detail and how to approach reviews after a change has been mailed. ### Common beginner mistakes When a change is sent to Gerrit, it is usually triaged within a few days. A maintainer will have a look and provide some initial review that for first-time contributors usually focuses on basic cosmetics and common mistakes. These include things like: * Commit message not following the [suggested format](https://go.dev/doc/contribute#commit_messages) . * The lack of a linked GitHub issue. The vast majority of changes require a linked issue that describes the bug or the feature that the change fixes or implements, and consensus should have been reached on the tracker before proceeding with it. Gerrit reviews do not discuss the merit of the change, just its implementation. Only trivial or cosmetic changes will be accepted without an associated issue. * Change sent during the freeze phase of the development cycle, when the tree is closed for general changes. In this case, a maintainer might review the code with a line such as `R=go1.12`, which means that it will be reviewed later when the tree opens for a new development window. You can add `R=go1.XX` as a comment yourself if you know that it's not the correct time frame for the change. ### Trybots After an initial reading of your change, maintainers will trigger trybots, a cluster of servers that will run the full test suite on several different architectures. Most trybots complete in a few minutes, at which point a link will be posted in Gerrit where you can see the results. If the trybot run fails, follow the link and check the full logs of the platforms on which the tests failed. Try to understand what broke, update your patch to fix it, and upload again. Maintainers will trigger a new trybot run to see if the problem was fixed. Sometimes, the tree can be broken on some platforms for a few hours; if the failure reported by the trybot doesn't seem related to your patch, go to the [Build Dashboard](https://build.golang.org/) and check if the same failure appears in other recent commits on the same platform. In this case, feel free to write a comment in Gerrit to mention that the failure is unrelated to your change, to help maintainers understand the situation. You can also search GitHub issues for the failing error message or browse [recently updated `watchflakes` issues](https://github.com/golang/go/issues?q=is%3Aissue%20watchflakes%20sort%3Aupdated-desc) . If your change is based on an older commit or it looks like someone else might have already fixed the problem, try rebasing to the latest master commit via `git rebase`. ### Reviews The Go community values very thorough reviews. Think of each review comment like a ticket: you are expected to somehow "close" it by acting on it, either by implementing the suggestion or convincing the reviewer otherwise. After you update the change, go through the review comments and make sure to reply to every one. You can click the "Done" button to reply indicating that you've implemented the reviewer's suggestion; otherwise, click on "Reply" and explain why you have not, or what you have done instead. It is perfectly normal for changes to go through several round of reviews, with one or more reviewers making new comments every time and then waiting for an updated change before reviewing again. This cycle happens even for experienced contributors, so don't be discouraged by it. ### Voting conventions As they near a decision, reviewers will apply a Code-Review “vote” to your change. There are two possible votes: * **+2** The change is approved for being merged. Only Go maintainers (also referred to as “approvers”) can cast a +2 vote. * **+1** The change looks good, but either the reviewer is requesting minor changes before approving it, or they are not a maintainer and cannot approve it, but would like to encourage an approval. To be submitted, a change must have a Code-Review +2 from a maintainer. Maintainers can also apply a Hold +1 vote to the change, to mark a change that should not be submitted now (for example, because the [proposal review](https://go.dev/s/proposal-process) for new API in the change has not completed). To be submitted, a change must not have any Hold +1 votes from a maintainer. Finally, to be submitted, a change must have the involvement of two Google employees, either as the uploader of the change or as a reviewer voting at least Code-Review +1. This requirement is for compliance and supply chain security reasons. ### Submitting an approved change When a change is ready, a maintainer will submit the change, which adds it as a commit to the Gerrit repository. The two steps (approving and submitting) are separate because in some cases maintainers may want to approve it but not to submit it right away (for instance, the tree could be temporarily frozen). Submitting a change checks it into the repository. The change description will include a link to the code review, which will be updated with a link to the change in the repository. Since the method used to integrate the changes is Git's "Cherry Pick", the commit hashes in the repository will be changed by the submit operation. If your change has been approved for a few days without being submitted, feel free to write a comment in Gerrit requesting submission. ### More information In addition to the information here, the Go community maintains a [CodeReview](https://go.dev/wiki/CodeReview) wiki page. Feel free to contribute to this page as you learn more about the review process. Miscellaneous topics -------------------- This section collects a number of other comments that are outside the issue/edit/code review/submit process itself. ### Gopls When working on the main Go repository and using `gopls` with your editor, the `go` command invoked by `gopls` must correspond to the version of the source code you are working on. The `go` command can be built with `make.bash` and the `bin` directory should be added to your `PATH`. See [Gopls: Advanced topics](https://go.dev/gopls/advanced) for additional details. Complete documentation for Gopls can be found at https://go.dev/gopls. ### Copyright headers Files in the Go repository don't list author names, both to avoid clutter and to avoid having to keep the lists up to date. Instead, your name will appear in the [change log](https://go.dev/change) . New files that you contribute should use the standard copyright header: // Copyright 2026 The Go Authors. All rights reserved. // Use of this source code is governed by a BSD-style // license that can be found in the LICENSE file. Files in the repository are copyrighted the year they are added. Do not update the copyright year on files that you change. ### Troubleshooting mail errors The most common way that the `git` `codereview` `mail` command fails is because the e-mail address in the commit does not match the one that you used during [the registration process](https://go.dev/doc/contribute#google_account) . If you see something like... remote: Processing changes: refs: 1, done remote: remote: ERROR: In commit ab13517fa29487dcf8b0d48916c51639426c5ee9 remote: ERROR: author email address XXXXXXXXXXXXXXXXXXX remote: ERROR: does not match your user account. you need to configure Git for this repository to use the e-mail address that you registered with. To change the e-mail address to ensure this doesn't happen again, run: $ git config user.email email@address.com Then change the commit to use this alternative e-mail address with this command: $ git commit --amend --author="Author Name " Then retry by running: $ git codereview mail ### Quickly testing your changes Running `all.bash` for every single change to the code tree is burdensome. Even though it is strongly suggested to run it before sending a change, during the normal development cycle you may want to compile and test only the package you are developing. * In general, you can run `make.bash` instead of `all.bash` to only rebuild the Go tool chain without running the whole test suite. Or you can run `run.bash` to only run the whole test suite without rebuilding the tool chain. You can think of `all.bash` as `make.bash` followed by `run.bash`. * In this section, we'll call the directory into which you cloned the Go repository `$GOROOT`. The `go` tool built by `$GOROOT/src/make.bash` will be installed in `$GOROOT/bin/go` and you can invoke it to test your code. For instance, if you have modified the compiler and you want to test how it affects the test suite of your own project, just run `go` `test` using it: $ cd $ $GOROOT/bin/go test * If you're changing the standard library, you probably don't need to rebuild the compiler: you can just run the tests for the package you've changed. You can do that either with the Go version you normally use, or with the Go compiler built from your clone (which is sometimes required because the standard library code you're modifying might require a newer version than the stable one you have installed). $ cd $GOROOT/src/crypto/sha1 $ \[make changes...\] $ $GOROOT/bin/go test . * If you're modifying the compiler itself, you can just recompile the `compile` tool (which is the internal binary invoked by `go` `build` to compile each single package). After that, you will want to test it by compiling or running something. $ cd $GOROOT/src $ \[make changes...\] $ $GOROOT/bin/go install cmd/compile $ $GOROOT/bin/go build \[something...\] # test the new compiler $ $GOROOT/bin/go run \[something...\] # test the new compiler $ $GOROOT/bin/go test \[something...\] # test the new compiler The same applies to other internal tools of the Go tool chain, such as `asm`, `cover`, `link`, and so on. Just recompile and install the tool using `go` `install` `cmd/` and then use the built Go binary to test it. * In addition to the standard per-package tests, there is a top-level test suite in `$GOROOT/test` that contains several black-box and regression tests. The test suite is run by `all.bash` but you can also run it manually: $ $GOROOT/bin/go test cmd/internal/testdir ### Specifying a reviewer / CCing others Unless explicitly told otherwise, such as in the discussion leading up to sending in the change, it's better not to specify a reviewer. All changes are automatically CC'ed to the [golang-codereviews@googlegroups.com](https://groups.google.com/group/golang-codereviews) mailing list. If this is your first ever change, there may be a moderation delay before it appears on the mailing list, to prevent spam. You can specify a reviewer or CC interested parties using the `-r` or `-cc` options. Both accept a comma-separated list of e-mail addresses: $ git codereview mail -r joe@golang.org -cc mabel@example.com,math-nuts@swtch.com ### Synchronize your client While you were working, others might have submitted changes to the repository. To update your local branch, run $ git codereview sync (Under the covers this runs `git` `pull` `-r`.) ### Reviewing code by others As part of the review process reviewers can propose changes directly (in the GitHub workflow this would be someone else attaching commits to a pull request). Gerrit provides access to commands that will help you import changes proposed by another developer so you can review/test them locally. From the Gerrit page for the CL you want to import, open the "⋮" menu, click the "Download patch" link. Depending on your preferred git workflow, choose the appropriate command. The options will look something like this: $ git fetch https://go.googlesource.com/review refs/changes/21/13245/1 && git checkout FETCH\_HEAD To revert, change back to the branch you were working in. ### Set up git aliases The `git-codereview` command can be run directly from the shell by typing, for instance, $ git codereview sync but it is more convenient to set up aliases for `git-codereview`'s own subcommands, so that the above becomes, $ git sync The `git-codereview` subcommands have been chosen to be distinct from Git's own, so it's safe to define these aliases. To install them, copy this text into your Git configuration file (usually `.gitconfig` in your home directory): \[alias\] change = codereview change gofmt = codereview gofmt mail = codereview mail pending = codereview pending submit = codereview submit sync = codereview sync ### Sending multiple dependent changes Advanced users may want to stack up related commits in a single branch. Gerrit allows for changes to be dependent on each other, forming such a dependency chain. Each change will need to be approved and submitted separately but the dependency will be visible to reviewers. To send out a group of dependent changes, keep each change as a different commit under the same branch, and then run: $ git codereview mail HEAD Make sure to explicitly specify `HEAD`, which is usually not required when sending single changes. More details can be found in the [git-codereview documentation](https://pkg.go.dev/golang.org/x/review/git-codereview?tab=doc#hdr-Multiple_Commit_Work_Branches) . ### Minor Releases If you want to make a change to a release branch for backporting, see [Minor Releases](https://go.dev/wiki/MinorReleases) . go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Tutorial: Getting started with generics - The Go Programming Language 1. [Documentation](https://go.dev/doc/) 2. [Tutorials](https://go.dev/doc/tutorial/) 3. [Tutorial: Getting started with generics](https://go.dev/doc/tutorial/generics) Tutorial: Getting started with generics ======================================= This tutorial introduces the basics of generics in Go. With generics, you can declare and use functions or types that are written to work with any of a set of types provided by calling code. In this tutorial, you’ll declare two simple non-generic functions, then capture the same logic in a single generic function. You’ll progress through the following sections: 1. Create a folder for your code. 2. Add non-generic functions. 3. Add a generic function to handle multiple types. 4. Remove type arguments when calling the generic function. 5. Declare a type constraint. **Note:** For other tutorials, see [Tutorials](https://go.dev/doc/tutorial/index.html) . **Note:** If you prefer, you can use [the Go playground in “Go dev branch” mode](https://go.dev/play/?v=gotip) to edit and run your program instead. Prerequisites ------------- * **An installation of Go 1.18 or later.** For installation instructions, see [Installing Go](https://go.dev/doc/install) . * **A tool to edit your code.** Any text editor you have will work fine. * **A command terminal.** Go works well using any terminal on Linux and Mac, and on PowerShell or cmd in Windows. Create a folder for your code ----------------------------- To begin, create a folder for the code you’ll write. 1. Open a command prompt and change to your home directory. On Linux or Mac: $ cd On Windows: C:\> cd %HOMEPATH% The rest of the tutorial will show a $ as the prompt. The commands you use will work on Windows too. 2. From the command prompt, create a directory for your code called generics. $ mkdir generics $ cd generics 3. Create a module to hold your code. Run the `go mod init` command, giving it your new code’s module path. $ go mod init example/generics go: creating new go.mod: module example/generics **Note:** For production code, you’d specify a module path that’s more specific to your own needs. For more, be sure to see [Managing dependencies](https://go.dev/doc/modules/managing-dependencies) . Next, you’ll add some simple code to work with maps. Add non-generic functions ------------------------- In this step, you’ll add two functions that each add together the values of a map and return the total. You’re declaring two functions instead of one because you’re working with two different types of maps: one that stores `int64` values, and one that stores `float64` values. #### Write the code 1. Using your text editor, create a file called main.go in the generics directory. You’ll write your Go code in this file. 2. Into main.go, at the top of the file, paste the following package declaration. package main A standalone program (as opposed to a library) is always in package `main`. 3. Beneath the package declaration, paste the following two function declarations. // SumInts adds together the values of m. func SumInts(m map[string]int64) int64 { var s int64 for _, v := range m { s += v } return s } // SumFloats adds together the values of m. func SumFloats(m map[string]float64) float64 { var s float64 for _, v := range m { s += v } return s } In this code, you: * Declare two functions to add together the values of a map and return the sum. * `SumFloats` takes a map of `string` to `float64` values. * `SumInts` takes a map of `string` to `int64` values. 4. At the top of main.go, beneath the package declaration, paste the following `main` function to initialize the two maps and use them as arguments when calling the functions you declared in the preceding step. func main() { // Initialize a map for the integer values ints := map[string]int64{ "first": 34, "second": 12, } // Initialize a map for the float values floats := map[string]float64{ "first": 35.98, "second": 26.99, } fmt.Printf("Non-Generic Sums: %v and %v\n", SumInts(ints), SumFloats(floats)) } In this code, you: * Initialize a map of `float64` values and a map of `int64` values, each with two entries. * Call the two functions you declared earlier to find the sum of each map’s values. * Print the result. 5. Near the top of main.go, just beneath the package declaration, import the package you’ll need to support the code you’ve just written. The first lines of code should look like this: package main import "fmt" 6. Save main.go. #### Run the code From the command line in the directory containing main.go, run the code. $ go run . Non-Generic Sums: 46 and 62.97 With generics, you can write one function here instead of two. Next, you’ll add a single generic function for maps containing either integer or float values. Add a generic function to handle multiple types ----------------------------------------------- In this section, you’ll add a single generic function that can receive a map containing either integer or float values, effectively replacing the two functions you just wrote with a single function. To support values of either type, that single function will need a way to declare what types it supports. Calling code, on the other hand, will need a way to specify whether it is calling with an integer or float map. To support this, you’ll write a function that declares _type parameters_ in addition to its ordinary function parameters. These type parameters make the function generic, enabling it to work with arguments of different types. You’ll call the function with _type arguments_ and ordinary function arguments. Each type parameter has a _type constraint_ that acts as a kind of meta-type for the type parameter. Each type constraint specifies the permissible type arguments that calling code can use for the respective type parameter. While a type parameter’s constraint typically represents a set of types, at compile time the type parameter stands for a single type – the type provided as a type argument by the calling code. If the type argument’s type isn’t allowed by the type parameter’s constraint, the code won’t compile. Keep in mind that a type parameter must support all the operations the generic code is performing on it. For example, if your function’s code were to try to perform `string` operations (such as indexing) on a type parameter whose constraint included numeric types, the code wouldn’t compile. In the code you’re about to write, you’ll use a constraint that allows either integer or float types. #### Write the code 1. Beneath the two functions you added previously, paste the following generic function. // SumIntsOrFloats sums the values of map m. It supports both int64 and float64 // as types for map values. func SumIntsOrFloats[K comparable, V int64 | float64](m map[K]V) V { var s V for _, v := range m { s += v } return s } In this code, you: * Declare a `SumIntsOrFloats` function with two type parameters (inside the square brackets), `K` and `V`, and one argument that uses the type parameters, `m` of type `map[K]V`. The function returns a value of type `V`. * Specify for the `K` type parameter the type constraint `comparable`. Intended specifically for cases like these, the `comparable` constraint is predeclared in Go. It allows any type whose values may be used as an operand of the comparison operators `==` and `!=`. Go requires that map keys be comparable. So declaring `K` as `comparable` is necessary so you can use `K` as the key in the map variable. It also ensures that calling code uses an allowable type for map keys. * Specify for the `V` type parameter a constraint that is a union of two types: `int64` and `float64`. Using `|` specifies a union of the two types, meaning that this constraint allows either type. Either type will be permitted by the compiler as an argument in the calling code. * Specify that the `m` argument is of type `map[K]V`, where `K` and `V` are the types already specified for the type parameters. Note that we know `map[K]V` is a valid map type because `K` is a comparable type. If we hadn’t declared `K` comparable, the compiler would reject the reference to `map[K]V`. 2. In main.go, beneath the code you already have, paste the following code. fmt.Printf("Generic Sums: %v and %v\n", SumIntsOrFloats[string, int64](ints), SumIntsOrFloats[string, float64](floats)) In this code, you: * Call the generic function you just declared, passing each of the maps you created. * Specify type arguments – the type names in square brackets – to be clear about the types that should replace type parameters in the function you’re calling. As you’ll see in the next section, you can often omit the type arguments in the function call. Go can often infer them from your code. * Print the sums returned by the function. #### Run the code From the command line in the directory containing main.go, run the code. $ go run . Non-Generic Sums: 46 and 62.97 Generic Sums: 46 and 62.97 To run your code, in each call the compiler replaced the type parameters with the concrete types specified in that call. In calling the generic function you wrote, you specified type arguments that told the compiler what types to use in place of the function’s type parameters. As you’ll see in the next section, in many cases you can omit these type arguments because the compiler can infer them. Remove type arguments when calling the generic function ------------------------------------------------------- In this section, you’ll add a modified version of the generic function call, making a small change to simplify the calling code. You’ll remove the type arguments, which aren’t needed in this case. You can omit type arguments in calling code when the Go compiler can infer the types you want to use. The compiler infers type arguments from the types of function arguments. Note that this isn’t always possible. For example, if you needed to call a generic function that had no arguments, you would need to include the type arguments in the function call. #### Write the code * In main.go, beneath the code you already have, paste the following code. fmt.Printf("Generic Sums, type parameters inferred: %v and %v\n", SumIntsOrFloats(ints), SumIntsOrFloats(floats)) In this code, you: * Call the generic function, omitting the type arguments. #### Run the code From the command line in the directory containing main.go, run the code. $ go run . Non-Generic Sums: 46 and 62.97 Generic Sums: 46 and 62.97 Generic Sums, type parameters inferred: 46 and 62.97 Next, you’ll further simplify the function by capturing the union of integers and floats into a type constraint you can reuse, such as from other code. Declare a type constraint ------------------------- In this last section, you’ll move the constraint you defined earlier into its own interface so that you can reuse it in multiple places. Declaring constraints in this way helps streamline code, such as when a constraint is more complex. You declare a _type constraint_ as an interface. The constraint allows any type implementing the interface. For example, if you declare a type constraint interface with three methods, then use it with a type parameter in a generic function, type arguments used to call the function must have all of those methods. Constraint interfaces can also refer to specific types, as you’ll see in this section. #### Write the code 1. Just above `main`, immediately after the import statements, paste the following code to declare a type constraint. type Number interface { int64 | float64 } In this code, you: * Declare the `Number` interface type to use as a type constraint. * Declare a union of `int64` and `float64` inside the interface. Essentially, you’re moving the union from the function declaration into a new type constraint. That way, when you want to constrain a type parameter to either `int64` or `float64`, you can use this `Number` type constraint instead of writing out `int64 | float64`. 2. Beneath the functions you already have, paste the following generic `SumNumbers` function. // SumNumbers sums the values of map m. It supports both integers // and floats as map values. func SumNumbers[K comparable, V Number](m map[K]V) V { var s V for _, v := range m { s += v } return s } In this code, you: * Declare a generic function with the same logic as the generic function you declared previously, but with the new interface type instead of the union as the type constraint. As before, you use the type parameters for the argument and return types. 3. In main.go, beneath the code you already have, paste the following code. fmt.Printf("Generic Sums with Constraint: %v and %v\n", SumNumbers(ints), SumNumbers(floats)) In this code, you: * Call `SumNumbers` with each map, printing the sum from the values of each. As in the preceding section, you omit the type arguments (the type names in square brackets) in calls to the generic function. The Go compiler can infer the type argument from other arguments. #### Run the code From the command line in the directory containing main.go, run the code. $ go run . Non-Generic Sums: 46 and 62.97 Generic Sums: 46 and 62.97 Generic Sums, type parameters inferred: 46 and 62.97 Generic Sums with Constraint: 46 and 62.97 Conclusion ---------- Nicely done! You’ve just introduced yourself to generics in Go. Suggested next topics: * The [Go Tour](https://go.dev/tour/) is a great step-by-step introduction to Go fundamentals. * You’ll find useful Go best practices described in [Effective Go](https://go.dev/doc/effective_go) and [How to write Go code](https://go.dev/doc/code) . Completed code -------------- You can run this program in the [Go playground](https://go.dev/play/p/apNmfVwogK0?v=gotip) . On the playground simply click the **Run** button. package main import "fmt" type Number interface { int64 | float64 } func main() { // Initialize a map for the integer values ints := map[string]int64{ "first": 34, "second": 12, } // Initialize a map for the float values floats := map[string]float64{ "first": 35.98, "second": 26.99, } fmt.Printf("Non-Generic Sums: %v and %v\n", SumInts(ints), SumFloats(floats)) fmt.Printf("Generic Sums: %v and %v\n", SumIntsOrFloats[string, int64](ints), SumIntsOrFloats[string, float64](floats)) fmt.Printf("Generic Sums, type parameters inferred: %v and %v\n", SumIntsOrFloats(ints), SumIntsOrFloats(floats)) fmt.Printf("Generic Sums with Constraint: %v and %v\n", SumNumbers(ints), SumNumbers(floats)) } // SumInts adds together the values of m. func SumInts(m map[string]int64) int64 { var s int64 for _, v := range m { s += v } return s } // SumFloats adds together the values of m. func SumFloats(m map[string]float64) float64 { var s float64 for _, v := range m { s += v } return s } // SumIntsOrFloats sums the values of map m. It supports both floats and integers // as map values. func SumIntsOrFloats[K comparable, V int64 | float64](m map[K]V) V { var s V for _, v := range m { s += v } return s } // SumNumbers sums the values of map m. Its supports both integers // and floats as map values. func SumNumbers[K comparable, V Number](m map[K]V) V { var s V for _, v := range m { s += v } return s } go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Codewalk: Generating arbitrary text: a Markov chain algorithm - The Go Programming Language Codewalk: Generating arbitrary text: a Markov chain algorithm ============================================================= [![Pop Out Code](https://go.dev/doc/codewalk/popout.png "View code in new window")](https://go.dev/doc/codewalk/markov/) doc/codewalk/markov.go code on [left](https://go.dev/doc/codewalk/markov/#) • [right](https://go.dev/doc/codewalk/markov/#) code width 70% filepaths [shown](https://go.dev/doc/codewalk/markov/#) • [hidden](https://go.dev/doc/codewalk/markov/#) [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fmarkov.go&lo=6&hi=44#mark) Introduction This codewalk describes a program that generates random text using a Markov chain algorithm. The package comment describes the algorithm and the operation of the program. Please read it before continuing. doc/codewalk/markov.go:6,44 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fmarkov.go&lo=77&hi=77#mark) Modeling Markov chains A chain consists of a prefix and a suffix. Each prefix is a set number of words, while a suffix is a single word. A prefix can have an arbitrary number of suffixes. To model this data, we use a `map[string][]string`. Each map key is a prefix (a `string`) and its values are lists of suffixes (a slice of strings, `[]string`). Here is the example table from the package comment as modeled by this data structure: map\[string\]\[\]string{ " ": {"I"}, " I": {"am"}, "I am": {"a", "not"}, "a free": {"man!"}, "am a": {"free"}, "am not": {"a"}, "a number!": {"I"}, "number! I": {"am"}, "not a": {"number!"}, } While each prefix consists of multiple words, we store prefixes in the map as a single `string`. It would seem more natural to store the prefix as a `[]string`, but we can't do this with a map because the key type of a map must implement equality (and slices do not). Therefore, in most of our code we will model prefixes as a `[]string` and join the strings together with a space to generate the map key: Prefix Map key \[\]string{"", ""} " " \[\]string{"", "I"} " I" \[\]string{"I", "am"} "I am" doc/codewalk/markov.go:77 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fmarkov.go&lo=76&hi=79#mark) The Chain struct The complete state of the chain table consists of the table itself and the word length of the prefixes. The `Chain` struct stores this data. doc/codewalk/markov.go:76,79 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fmarkov.go&lo=82&hi=84#mark) The NewChain constructor function The `Chain` struct has two unexported fields (those that do not begin with an upper case character), and so we write a `NewChain` constructor function that initializes the `chain` map with `make` and sets the `prefixLen` field. This is constructor function is not strictly necessary as this entire program is within a single package (`main`) and therefore there is little practical difference between exported and unexported fields. We could just as easily write out the contents of this function when we want to construct a new Chain. But using these unexported fields is good practice; it clearly denotes that only methods of Chain and its constructor function should access those fields. Also, structuring `Chain` like this means we could easily move it into its own package at some later date. doc/codewalk/markov.go:82,84 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fmarkov.go&lo=60&hi=60#mark) The Prefix type Since we'll be working with prefixes often, we define a `Prefix` type with the concrete type `[]string`. Defining a named type clearly allows us to be explicit when we are working with a prefix instead of just a `[]string`. Also, in Go we can define methods on any named type (not just structs), so we can add methods that operate on `Prefix` if we need to. doc/codewalk/markov.go:60 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fmarkov.go&lo=63&hi=65#mark) The String method The first method we define on `Prefix` is `String`. It returns a `string` representation of a `Prefix` by joining the slice elements together with spaces. We will use this method to generate keys when working with the chain map. doc/codewalk/markov.go:63,65 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fmarkov.go&lo=88&hi=100#mark) Building the chain The `Build` method reads text from an `io.Reader` and parses it into prefixes and suffixes that are stored in the `Chain`. The `[io.Reader](https://go.dev/pkg/io/#Reader) ` is an interface type that is widely used by the standard library and other Go code. Our code uses the `[fmt.Fscan](https://go.dev/pkg/fmt/#Fscan) ` function, which reads space-separated values from an `io.Reader`. The `Build` method returns once the `Reader`'s `Read` method returns `io.EOF` (end of file) or some other read error occurs. doc/codewalk/markov.go:88,100 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fmarkov.go&lo=89&hi=89#mark) Buffering the input This function does many small reads, which can be inefficient for some `Readers`. For efficiency we wrap the provided `io.Reader` with `[bufio.NewReader](https://go.dev/pkg/bufio/) ` to create a new `io.Reader` that provides buffering. doc/codewalk/markov.go:89 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fmarkov.go&lo=90&hi=90#mark) The Prefix variable At the top of the function we make a `Prefix` slice `p` using the `Chain`'s `prefixLen` field as its length. We'll use this variable to hold the current prefix and mutate it with each new word we encounter. doc/codewalk/markov.go:90 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fmarkov.go&lo=92&hi=95#mark) Scanning words In our loop we read words from the `Reader` into a `string` variable `s` using `fmt.Fscan`. Since `Fscan` uses space to separate each input value, each call will yield just one word (including punctuation), which is exactly what we need. `Fscan` returns an error if it encounters a read error (`io.EOF`, for example) or if it can't scan the requested value (in our case, a single string). In either case we just want to stop scanning, so we `break` out of the loop. doc/codewalk/markov.go:92,95 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fmarkov.go&lo=96&hi=97#mark) Adding a prefix and suffix to the chain The word stored in `s` is a new suffix. We add the new prefix/suffix combination to the `chain` map by computing the map key with `p.String` and appending the suffix to the slice stored under that key. The built-in `append` function appends elements to a slice and allocates new storage when necessary. When the provided slice is `nil`, `append` allocates a new slice. This behavior conveniently ties in with the semantics of our map: retrieving an unset key returns the zero value of the value type and the zero value of `[]string` is `nil`. When our program encounters a new prefix (yielding a `nil` value in the map) `append` will allocate a new slice. For more information about the `append` function and slices in general see the [Slices: usage and internals](https://go.dev/doc/articles/slices_usage_and_internals.html) article. doc/codewalk/markov.go:96,97 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fmarkov.go&lo=98&hi=98#mark) Pushing the suffix onto the prefix Before reading the next word our algorithm requires us to drop the first word from the prefix and push the current suffix onto the prefix. When in this state p == Prefix{"I", "am"} s == "not" the new value for `p` would be p == Prefix{"am", "not"} This operation is also required during text generation so we put the code to perform this mutation of the slice inside a method on `Prefix` named `Shift`. doc/codewalk/markov.go:98 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fmarkov.go&lo=68&hi=71#mark) The Shift method The `Shift` method uses the built-in `copy` function to copy the last len(p)-1 elements of `p` to the start of the slice, effectively moving the elements one index to the left (if you consider zero as the leftmost index). p := Prefix{"I", "am"} copy(p, p\[1:\]) // p == Prefix{"am", "am"} We then assign the provided `word` to the last index of the slice: // suffix == "not" p\[len(p)-1\] = suffix // p == Prefix{"am", "not"} doc/codewalk/markov.go:68,71 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fmarkov.go&lo=103&hi=116#mark) Generating text The `Generate` method is similar to `Build` except that instead of reading words from a `Reader` and storing them in a map, it reads words from the map and appends them to a slice (`words`). `Generate` uses a conditional for loop to generate up to `n` words. doc/codewalk/markov.go:103,116 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fmarkov.go&lo=107&hi=110#mark) Getting potential suffixes At each iteration of the loop we retrieve a list of potential suffixes for the current prefix. We access the `chain` map at key `p.String()` and assign its contents to `choices`. If `len(choices)` is zero we break out of the loop as there are no potential suffixes for that prefix. This test also works if the key isn't present in the map at all: in that case, `choices` will be `nil` and the length of a `nil` slice is zero. doc/codewalk/markov.go:107,110 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fmarkov.go&lo=111&hi=113#mark) Choosing a suffix at random To choose a suffix we use the `[rand.Intn](https://go.dev/pkg/math/rand/#Intn) ` function. It returns a random integer up to (but not including) the provided value. Passing in `len(choices)` gives us a random index into the full length of the list. We use that index to pick our new suffix, assign it to `next` and append it to the `words` slice. Next, we `Shift` the new suffix onto the prefix just as we did in the `Build` method. doc/codewalk/markov.go:111,113 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fmarkov.go&lo=115&hi=115#mark) Returning the generated text Before returning the generated text as a string, we use the `strings.Join` function to join the elements of the `words` slice together, separated by spaces. doc/codewalk/markov.go:115 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fmarkov.go&lo=119&hi=121#mark) Command-line flags To make it easy to tweak the prefix and generated text lengths we use the `[flag](https://go.dev/pkg/flag/) ` package to parse command-line flags. These calls to `flag.Int` register new flags with the `flag` package. The arguments to `Int` are the flag name, its default value, and a description. The `Int` function returns a pointer to an integer that will contain the user-supplied value (or the default value if the flag was omitted on the command-line). doc/codewalk/markov.go:119,121 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fmarkov.go&lo=123&hi=124#mark) Program set up The `main` function begins by parsing the command-line flags with `flag.Parse` and seeding the `rand` package's random number generator with the current time. If the command-line flags provided by the user are invalid the `flag.Parse` function will print an informative usage message and terminate the program. doc/codewalk/markov.go:123,124 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fmarkov.go&lo=126&hi=127#mark) Creating and building a new Chain To create the new `Chain` we call `NewChain` with the value of the `prefix` flag. To build the chain we call `Build` with `os.Stdin` (which implements `io.Reader`) so that it will read its input from standard input. doc/codewalk/markov.go:126,127 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fmarkov.go&lo=128&hi=129#mark) Generating and printing text Finally, to generate text we call `Generate` with the value of the `words` flag and assigning the result to the variable `text`. Then we call `fmt.Println` to write the text to standard output, followed by a carriage return. doc/codewalk/markov.go:128,129 [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fmarkov.go&lo=0&hi=0#mark) Using this program To use this program, first build it with the [go](https://go.dev/cmd/go/) command: $ go build markov.go And then execute it while piping in some input text: $ echo "a man a plan a canal panama" \\ | ./markov -prefix=1 a plan a man a plan a canal panama Here's a transcript of generating some text using the Go distribution's README file as source material: $ ./markov -words=10 < $GOROOT/README This is the source code repository for the Go source $ ./markov -prefix=1 -words=10 < $GOROOT/README This is the go directory (the one containing this README). $ ./markov -prefix=1 -words=10 < $GOROOT/README This is the variable if you have just untarred a doc/codewalk/markov.go [](https://go.dev/doc/codewalk/?fileprint=/doc%2fcodewalk%2fmarkov.go&lo=0&hi=0#mark) An exercise for the reader The `Generate` function does a lot of allocations when it builds the `words` slice. As an exercise, modify it to take an `io.Writer` to which it incrementally writes the generated text with `Fprint`. Aside from being more efficient this makes `Generate` more symmetrical to `Build`. doc/codewalk/markov.go [previous step](https://go.dev/doc/codewalk/markov/#) • [next step](https://go.dev/doc/codewalk/markov/#) go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Gobs of data - The Go Programming Language [The Go Blog](https://go.dev/blog/) ==================================== Gobs of data ============ Rob Pike 24 March 2011 Introduction ------------ To transmit a data structure across a network or to store it in a file, it must be encoded and then decoded again. There are many encodings available, of course: [JSON](http://www.json.org/) , [XML](http://www.w3.org/XML/) , Google’s [protocol buffers](http://code.google.com/p/protobuf) , and more. And now there’s another, provided by Go’s [gob](https://go.dev/pkg/encoding/gob/) package. Why define a new encoding? It’s a lot of work and redundant at that. Why not just use one of the existing formats? Well, for one thing, we do! Go has [packages](https://go.dev/pkg/) supporting all the encodings just mentioned (the [protocol buffer package](https://github.com/golang/protobuf) is in a separate repository but it’s one of the most frequently downloaded). And for many purposes, including communicating with tools and systems written in other languages, they’re the right choice. But for a Go-specific environment, such as communicating between two servers written in Go, there’s an opportunity to build something much easier to use and possibly more efficient. Gobs work with the language in a way that an externally-defined, language-independent encoding cannot. At the same time, there are lessons to be learned from the existing systems. Goals ----- The gob package was designed with a number of goals in mind. First, and most obvious, it had to be very easy to use. First, because Go has reflection, there is no need for a separate interface definition language or “protocol compiler”. The data structure itself is all the package should need to figure out how to encode and decode it. On the other hand, this approach means that gobs will never work as well with other languages, but that’s OK: gobs are unashamedly Go-centric. Efficiency is also important. Textual representations, exemplified by XML and JSON, are too slow to put at the center of an efficient communications network. A binary encoding is necessary. Gob streams must be self-describing. Each gob stream, read from the beginning, contains sufficient information that the entire stream can be parsed by an agent that knows nothing a priori about its contents. This property means that you will always be able to decode a gob stream stored in a file, even long after you’ve forgotten what data it represents. There were also some things to learn from our experiences with Google protocol buffers. Protocol buffer misfeatures --------------------------- Protocol buffers had a major effect on the design of gobs, but have three features that were deliberately avoided. (Leaving aside the property that protocol buffers aren’t self-describing: if you don’t know the data definition used to encode a protocol buffer, you might not be able to parse it.) First, protocol buffers only work on the data type we call a struct in Go. You can’t encode an integer or array at the top level, only a struct with fields inside it. That seems a pointless restriction, at least in Go. If all you want to send is an array of integers, why should you have to put it into a struct first? Next, a protocol buffer definition may specify that fields `T.x` and `T.y` are required to be present whenever a value of type `T` is encoded or decoded. Although such required fields may seem like a good idea, they are costly to implement because the codec must maintain a separate data structure while encoding and decoding, to be able to report when required fields are missing. They’re also a maintenance problem. Over time, one may want to modify the data definition to remove a required field, but that may cause existing clients of the data to crash. It’s better not to have them in the encoding at all. (Protocol buffers also have optional fields. But if we don’t have required fields, all fields are optional and that’s that. There will be more to say about optional fields a little later.) The third protocol buffer misfeature is default values. If a protocol buffer omits the value for a “defaulted” field, then the decoded structure behaves as if the field were set to that value. This idea works nicely when you have getter and setter methods to control access to the field, but is harder to handle cleanly when the container is just a plain idiomatic struct. Required fields are also tricky to implement: where does one define the default values, what types do they have (is text UTF-8? uninterpreted bytes? how many bits in a float?) and despite the apparent simplicity, there were a number of complications in their design and implementation for protocol buffers. We decided to leave them out of gobs and fall back to Go’s trivial but effective defaulting rule: unless you set something otherwise, it has the “zero value” for that type - and it doesn’t need to be transmitted. So gobs end up looking like a sort of generalized, simplified protocol buffer. How do they work? Values ------ The encoded gob data isn’t about types like `int8` and `uint16`. Instead, somewhat analogous to constants in Go, its integer values are abstract, sizeless numbers, either signed or unsigned. When you encode an `int8`, its value is transmitted as an unsized, variable-length integer. When you encode an `int64`, its value is also transmitted as an unsized, variable-length integer. (Signed and unsigned are treated distinctly, but the same unsized-ness applies to unsigned values too.) If both have the value 7, the bits sent on the wire will be identical. When the receiver decodes that value, it puts it into the receiver’s variable, which may be of arbitrary integer type. Thus an encoder may send a 7 that came from an `int8`, but the receiver may store it in an `int64`. This is fine: the value is an integer and as a long as it fits, everything works. (If it doesn’t fit, an error results.) This decoupling from the size of the variable gives some flexibility to the encoding: we can expand the type of the integer variable as the software evolves, but still be able to decode old data. This flexibility also applies to pointers. Before transmission, all pointers are flattened. Values of type `int8`, `*int8`, `**int8`, `****int8`, etc. are all transmitted as an integer value, which may then be stored in `int` of any size, or `*int`, or `******int`, etc. Again, this allows for flexibility. Flexibility also happens because, when decoding a struct, only those fields that are sent by the encoder are stored in the destination. Given the value type T struct{ X, Y, Z int } // Only exported fields are encoded and decoded. var t = T{X: 7, Y: 0, Z: 8} the encoding of `t` sends only the 7 and 8. Because it’s zero, the value of `Y` isn’t even sent; there’s no need to send a zero value. The receiver could instead decode the value into this structure: type U struct{ X, Y *int8 } // Note: pointers to int8s var u U and acquire a value of `u` with only `X` set (to the address of an `int8` variable set to 7); the `Z` field is ignored - where would you put it? When decoding structs, fields are matched by name and compatible type, and only fields that exist in both are affected. This simple approach finesses the “optional field” problem: as the type `T` evolves by adding fields, out of date receivers will still function with the part of the type they recognize. Thus gobs provide the important result of optional fields - extensibility - without any additional mechanism or notation. From integers we can build all the other types: bytes, strings, arrays, slices, maps, even floats. Floating-point values are represented by their IEEE 754 floating-point bit pattern, stored as an integer, which works fine as long as you know their type, which we always do. By the way, that integer is sent in byte-reversed order because common values of floating-point numbers, such as small integers, have a lot of zeros at the low end that we can avoid transmitting. One nice feature of gobs that Go makes possible is that they allow you to define your own encoding by having your type satisfy the [GobEncoder](https://go.dev/pkg/encoding/gob/#GobEncoder) and [GobDecoder](https://go.dev/pkg/encoding/gob/#GobDecoder) interfaces, in a manner analogous to the [JSON](https://go.dev/pkg/encoding/json/) package’s [Marshaler](https://go.dev/pkg/encoding/json/#Marshaler) and [Unmarshaler](https://go.dev/pkg/encoding/json/#Unmarshaler) and also to the [Stringer](https://go.dev/pkg/fmt/#Stringer) interface from [package fmt](https://go.dev/pkg/fmt/) . This facility makes it possible to represent special features, enforce constraints, or hide secrets when you transmit data. See the [documentation](https://go.dev/pkg/encoding/gob/) for details. Types on the wire ----------------- The first time you send a given type, the gob package includes in the data stream a description of that type. In fact, what happens is that the encoder is used to encode, in the standard gob encoding format, an internal struct that describes the type and gives it a unique number. (Basic types, plus the layout of the type description structure, are predefined by the software for bootstrapping.) After the type is described, it can be referenced by its type number. Thus when we send our first type `T`, the gob encoder sends a description of `T` and tags it with a type number, say 127. All values, including the first, are then prefixed by that number, so a stream of `T` values looks like: ("define type id" 127, definition of type T)(127, T value)(127, T value), ... These type numbers make it possible to describe recursive types and send values of those types. Thus gobs can encode types such as trees: type Node struct { Value int Left, Right *Node } (It’s an exercise for the reader to discover how the zero-defaulting rule makes this work, even though gobs don’t represent pointers.) With the type information, a gob stream is fully self-describing except for the set of bootstrap types, which is a well-defined starting point. Compiling a machine ------------------- The first time you encode a value of a given type, the gob package builds a little interpreted machine specific to that data type. It uses reflection on the type to construct that machine, but once the machine is built it does not depend on reflection. The machine uses package unsafe and some trickery to convert the data into the encoded bytes at high speed. It could use reflection and avoid unsafe, but would be significantly slower. (A similar high-speed approach is taken by the protocol buffer support for Go, whose design was influenced by the implementation of gobs.) Subsequent values of the same type use the already-compiled machine, so they can be encoded right away. \[Update: As of Go 1.4, package unsafe is no longer use by the gob package, with a modest performance drop.\] Decoding is similar but harder. When you decode a value, the gob package holds a byte slice representing a value of a given encoder-defined type to decode, plus a Go value into which to decode it. The gob package builds a machine for that pair: the gob type sent on the wire crossed with the Go type provided for decoding. Once that decoding machine is built, though, it’s again a reflectionless engine that uses unsafe methods to get maximum speed. Use --- There’s a lot going on under the hood, but the result is an efficient, easy-to-use encoding system for transmitting data. Here’s a complete example showing differing encoded and decoded types. Note how easy it is to send and receive values; all you need to do is present values and variables to the [gob package](https://go.dev/pkg/encoding/gob/) and it does all the work. package main import ( "bytes" "encoding/gob" "fmt" "log" ) type P struct { X, Y, Z int Name string } type Q struct { X, Y *int32 Name string } func main() { // Initialize the encoder and decoder. Normally enc and dec would be // bound to network connections and the encoder and decoder would // run in different processes. var network bytes.Buffer // Stand-in for a network connection enc := gob.NewEncoder(&network) // Will write to network. dec := gob.NewDecoder(&network) // Will read from network. // Encode (send) the value. err := enc.Encode(P{3, 4, 5, "Pythagoras"}) if err != nil { log.Fatal("encode error:", err) } // Decode (receive) the value. var q Q err = dec.Decode(&q) if err != nil { log.Fatal("decode error:", err) } fmt.Printf("%q: {%d,%d}\n", q.Name, *q.X, *q.Y) } You can compile and run this example code in the [Go Playground](https://go.dev/play/p/_-OJV-rwMq) . The [rpc package](https://go.dev/pkg/net/rpc/) builds on gobs to turn this encode/decode automation into transport for method calls across the network. That’s a subject for another article. Details ------- The [gob package documentation](https://go.dev/pkg/encoding/gob/) , especially the file [doc.go](https://go.dev/src/pkg/encoding/gob/doc.go) , expands on many of the details described here and includes a full worked example showing how the encoding represents data. If you are interested in the innards of the gob implementation, that’s a good place to start. **Next article:** [Godoc: documenting Go code](https://go.dev/blog/godoc) **Previous article:** [C? Go? Cgo!](https://go.dev/blog/cgo) **[Blog Index](https://go.dev/blog/all) ** go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # The Go image package - The Go Programming Language [The Go Blog](https://go.dev/blog/) ==================================== The Go image package ==================== Nigel Tao 21 September 2011 Introduction ------------ The [image](https://go.dev/pkg/image/) and [image/color](https://go.dev/pkg/image/color/) packages define a number of types: `color.Color` and `color.Model` describe colors, `image.Point` and `image.Rectangle` describe basic 2-D geometry, and `image.Image` brings the two concepts together to represent a rectangular grid of colors. A [separate article](https://go.dev/doc/articles/image_draw.html) covers image composition with the [image/draw](https://go.dev/pkg/image/draw/) package. Colors and Color Models ----------------------- [Color](https://go.dev/pkg/image/color/#Color) is an interface that defines the minimal method set of any type that can be considered a color: one that can be converted to red, green, blue and alpha values. The conversion may be lossy, such as converting from CMYK or YCbCr color spaces. type Color interface { // RGBA returns the alpha-premultiplied red, green, blue and alpha values // for the color. Each value ranges within [0, 0xFFFF], but is represented // by a uint32 so that multiplying by a blend factor up to 0xFFFF will not // overflow. RGBA() (r, g, b, a uint32) } There are three important subtleties about the return values. First, the red, green and blue are alpha-premultiplied: a fully saturated red that is also 25% transparent is represented by RGBA returning a 75% r. Second, the channels have a 16-bit effective range: 100% red is represented by RGBA returning an r of 65535, not 255, so that converting from CMYK or YCbCr is not as lossy. Third, the type returned is `uint32`, even though the maximum value is 65535, to guarantee that multiplying two values together won’t overflow. Such multiplications occur when blending two colors according to an alpha mask from a third color, in the style of [Porter and Duff’s](https://en.wikipedia.org/wiki/Alpha_compositing) classic algebra: dstr, dstg, dstb, dsta := dst.RGBA() srcr, srcg, srcb, srca := src.RGBA() _, _, _, m := mask.RGBA() const M = 1<<16 - 1 // The resultant red value is a blend of dstr and srcr, and ranges in [0, M]. // The calculation for green, blue and alpha is similar. dstr = (dstr*(M-m) + srcr*m) / M The last line of that code snippet would have been more complicated if we worked with non-alpha-premultiplied colors, which is why `Color` uses alpha-premultiplied values. The image/color package also defines a number of concrete types that implement the `Color` interface. For example, [`RGBA`](https://go.dev/pkg/image/color/#RGBA) is a struct that represents the classic “8 bits per channel” color. type RGBA struct { R, G, B, A uint8 } Note that the `R` field of an `RGBA` is an 8-bit alpha-premultiplied color in the range \[0, 255\]. `RGBA` satisfies the `Color` interface by multiplying that value by 0x101 to generate a 16-bit alpha-premultiplied color in the range \[0, 65535\]. Similarly, the [`NRGBA`](https://go.dev/pkg/image/color/#NRGBA) struct type represents an 8-bit non-alpha-premultiplied color, as used by the PNG image format. When manipulating an `NRGBA`’s fields directly, the values are non-alpha-premultiplied, but when calling the `RGBA` method, the return values are alpha-premultiplied. A [`Model`](https://go.dev/pkg/image/color/#Model) is simply something that can convert `Color`s to other `Color`s, possibly lossily. For example, the `GrayModel` can convert any `Color` to a desaturated [`Gray`](https://go.dev/pkg/image/color/#Gray) . A `Palette` can convert any `Color` to one from a limited palette. type Model interface { Convert(c Color) Color } type Palette []Color Points and Rectangles --------------------- A [`Point`](https://go.dev/pkg/image/#Point) is an (x, y) co-ordinate on the integer grid, with axes increasing right and down. It is neither a pixel nor a grid square. A `Point` has no intrinsic width, height or color, but the visualizations below use a small colored square. type Point struct { X, Y int } ![](https://go.dev/blog/image/image-package-01.png) p := image.Point{2, 1} A [`Rectangle`](https://go.dev/pkg/image/#Rectangle) is an axis-aligned rectangle on the integer grid, defined by its top-left and bottom-right `Point`. A `Rectangle` also has no intrinsic color, but the visualizations below outline rectangles with a thin colored line, and call out their `Min` and `Max` `Point`s. type Rectangle struct { Min, Max Point } For convenience, `image.Rect(x0, y0, x1, y1)` is equivalent to `image.Rectangle{image.Point{x0, y0}, image.Point{x1, y1}}`, but is much easier to type. A `Rectangle` is inclusive at the top-left and exclusive at the bottom-right. For a `Point p` and a `Rectangle r`, `p.In(r)` if and only if `r.Min.X <= p.X && p.X < r.Max.X`, and similarly for `Y`. This is analogous to how a slice `s[i0:i1]` is inclusive at the low end and exclusive at the high end. (Unlike arrays and slices, a `Rectangle` often has a non-zero origin.) ![](https://go.dev/blog/image/image-package-02.png) r := image.Rect(2, 1, 5, 5) // Dx and Dy return a rectangle's width and height. fmt.Println(r.Dx(), r.Dy(), image.Pt(0, 0).In(r)) // prints 3 4 false Adding a `Point` to a `Rectangle` translates the `Rectangle`. Points and Rectangles are not restricted to be in the bottom-right quadrant. ![](https://go.dev/blog/image/image-package-03.png) r := image.Rect(2, 1, 5, 5).Add(image.Pt(-4, -2)) fmt.Println(r.Dx(), r.Dy(), image.Pt(0, 0).In(r)) // prints 3 4 true Intersecting two Rectangles yields another Rectangle, which may be empty. ![](https://go.dev/blog/image/image-package-04.png) r := image.Rect(0, 0, 4, 3).Intersect(image.Rect(2, 2, 5, 5)) // Size returns a rectangle's width and height, as a Point. fmt.Printf("%#v\n", r.Size()) // prints image.Point{X:2, Y:1} Points and Rectangles are passed and returned by value. A function that takes a `Rectangle` argument will be as efficient as a function that takes two `Point` arguments, or four `int` arguments. Images ------ An [Image](https://go.dev/pkg/image/#Image) maps every grid square in a `Rectangle` to a `Color` from a `Model`. “The pixel at (x, y)” refers to the color of the grid square defined by the points (x, y), (x+1, y), (x+1, y+1) and (x, y+1). type Image interface { // ColorModel returns the Image's color model. ColorModel() color.Model // Bounds returns the domain for which At can return non-zero color. // The bounds do not necessarily contain the point (0, 0). Bounds() Rectangle // At returns the color of the pixel at (x, y). // At(Bounds().Min.X, Bounds().Min.Y) returns the upper-left pixel of the grid. // At(Bounds().Max.X-1, Bounds().Max.Y-1) returns the lower-right one. At(x, y int) color.Color } A common mistake is assuming that an `Image`’s bounds start at (0, 0). For example, an animated GIF contains a sequence of Images, and each `Image` after the first typically only holds pixel data for the area that changed, and that area doesn’t necessarily start at (0, 0). The correct way to iterate over an `Image` m’s pixels looks like: b := m.Bounds() for y := b.Min.Y; y < b.Max.Y; y++ { for x := b.Min.X; x < b.Max.X; x++ { doStuffWith(m.At(x, y)) } } `Image` implementations do not have to be based on an in-memory slice of pixel data. For example, a [`Uniform`](https://go.dev/pkg/image/#Uniform) is an `Image` of enormous bounds and uniform color, whose in-memory representation is simply that color. type Uniform struct { C color.Color } Typically, though, programs will want an image based on a slice. Struct types like [`RGBA`](https://go.dev/pkg/image/#RGBA) and [`Gray`](https://go.dev/pkg/image/#Gray) (which other packages refer to as `image.RGBA` and `image.Gray`) hold slices of pixel data and implement the `Image` interface. type RGBA struct { // Pix holds the image's pixels, in R, G, B, A order. The pixel at // (x, y) starts at Pix[(y-Rect.Min.Y)*Stride + (x-Rect.Min.X)*4]. Pix []uint8 // Stride is the Pix stride (in bytes) between vertically adjacent pixels. Stride int // Rect is the image's bounds. Rect Rectangle } These types also provide a `Set(x, y int, c color.Color)` method that allows modifying the image one pixel at a time. m := image.NewRGBA(image.Rect(0, 0, 640, 480)) m.Set(5, 5, color.RGBA{255, 0, 0, 255}) If you’re reading or writing a lot of pixel data, it can be more efficient, but more complicated, to access these struct type’s `Pix` field directly. The slice-based `Image` implementations also provide a `SubImage` method, which returns an `Image` backed by the same array. Modifying the pixels of a sub-image will affect the pixels of the original image, analogous to how modifying the contents of a sub-slice `s[i0:i1]` will affect the contents of the original slice `s`. ![](https://go.dev/blog/image/image-package-05.png) m0 := image.NewRGBA(image.Rect(0, 0, 8, 5)) m1 := m0.SubImage(image.Rect(1, 2, 5, 5)).(*image.RGBA) fmt.Println(m0.Bounds().Dx(), m1.Bounds().Dx()) // prints 8, 4 fmt.Println(m0.Stride == m1.Stride) // prints true For low-level code that works on an image’s `Pix` field, be aware that ranging over `Pix` can affect pixels outside an image’s bounds. In the example above, the pixels covered by `m1.Pix` are shaded in blue. Higher-level code, such as the `At` and `Set` methods or the [image/draw package](https://go.dev/pkg/image/draw/) , will clip their operations to the image’s bounds. Image Formats ------------- The standard package library supports a number of common image formats, such as GIF, JPEG and PNG. If you know the format of a source image file, you can decode from an [`io.Reader`](https://go.dev/pkg/io/#Reader) directly. import ( "image/jpeg" "image/png" "io" ) // convertJPEGToPNG converts from JPEG to PNG. func convertJPEGToPNG(w io.Writer, r io.Reader) error { img, err := jpeg.Decode(r) if err != nil { return err } return png.Encode(w, img) } If you have image data of unknown format, the [`image.Decode`](https://go.dev/pkg/image/#Decode) function can detect the format. The set of recognized formats is constructed at run time and is not limited to those in the standard package library. An image format package typically registers its format in an init function, and the main package will “underscore import” such a package solely for the side effect of format registration. import ( "image" "image/png" "io" _ "code.google.com/p/vp8-go/webp" _ "image/jpeg" ) // convertToPNG converts from any recognized format to PNG. func convertToPNG(w io.Writer, r io.Reader) error { img, _, err := image.Decode(r) if err != nil { return err } return png.Encode(w, img) } **Next article:** [The Go image/draw package](https://go.dev/blog/image-draw) **Previous article:** [The Laws of Reflection](https://go.dev/blog/laws-of-reflection) **[Blog Index](https://go.dev/blog/all) ** go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Go Wiki: Learn - The Go Programming Language Go Wiki: Learn ============== In addition to the resources available [at golang.org](https://go.dev/doc/#learning) there are a range of community-driven initiatives: * [Boot.dev’s “Learn Go”](https://boot.dev/learn/learn-golang) - Code in the browser exercises with answer checking * [Essential Go](https://www.programming-books.io/essential/go/) – a free book about Go programming language. * [Go in 5 Minutes](https://gifm.dev/) * [The Little Go Book](http://openmymind.net/The-Little-Go-Book/) * [Tutorials Point](https://www.tutorialspoint.com/go/index.htm) * [Exercism.io - Go](http://exercism.io/languages/go) - Online code exercises for Go for practice and mentorship. * [EXLskills - A$AP Learn Go](https://exlskills.com/learn-en/courses/aap-learn-go-golang--learn_golang_asap) - Fast-paced free and open course for getting going with Go * [Learn Go in an Hour - Video](https://www.youtube.com/watch?v=CF9S4QZuV30) _2015-02-15_ * [Learning to Program in Go](https://www.youtube.com/playlist?list=PLei96ZX_m9sVSEXWwZi8uwd2vqCpEm4m6) , a multi-part video training class. * [Learn Go with Tests](https://quii.gitbook.io/learn-go-with-tests/) - An introduction to Go using a TDD approach. * [Pluralsight Classes for Go](http://www.pluralsight.com/tag/golang) - A growing collection of (paid) online classes. * [Ardan Labs Training](https://www.ardanlabs.com/) - Commercial, live instruction for Go programming. * [O’Reilly Go Fundamentals](http://shop.oreilly.com/category/learning-path/go-fundamentals.do) - Video learning path for Go programming. * [Go By Example](http://gobyexample.com/) provides a series of annotated code snippets. * [Learn Go in Y minutes](http://learnxinyminutes.com/docs/go/) is a top-to-bottom walk-through of the language. * [Workshop-Go](https://github.com/sendwithus/workshop-go) - Startup Slam Go Workshop - examples and slides. * [Go Fragments](http://www.gofragments.net/) - A collection of annotated Go code examples. * [50 Shades of Go: Traps, Gotchas, Common Mistakes for New Golang Devs](http://devs.cloudimmunity.com/gotchas-and-common-mistakes-in-go-golang/index.html) * [Free Go Language Workshop](https://www.frameworktraining.co.uk/go-language-free-training-workshop/) Framework Training is running regular free BYOD workshops in London, UK * [Golang Tutorials](http://golangtutorials.blogspot.com/2011/05/table-of-contents.html) - A free online class. * Rob Pike’s 2011 three day course - [Day 1](http://go.googlecode.com/hg-history/release-branch.r60/doc/GoCourseDay1.pdf) , [Day 2](http://go.googlecode.com/hg-history/release-branch.r60/doc/GoCourseDay2.pdf) , [Day 3](http://go.googlecode.com/hg-history/release-branch.r60/doc/GoCourseDay3.pdf) (_links are broken_, use the archived links from the wayback machine. [Day 1](http://web.archive.org/web/20160305024536/http://go.googlecode.com/hg-history/release-branch.r60/doc/GoCourseDay1.pdf) , [Day 2](http://web.archive.org/web/20160305081012/http://go.googlecode.com/hg-history/release-branch.r60/doc/GoCourseDay2.pdf) , [Day 3](http://web.archive.org/web/20160305075151/http://go.googlecode.com/hg-history/release-branch.r60/doc/GoCourseDay3.pdf) ) * [The Go Bridge Foundry](https://github.com/gobridge) - A member of the [Bridge Foundry](http://bridgefoundry.org/) family, offering a complete set of free Go training materials with the goal of bringing Go to under-served communities. * [Golangbot](https://golangbot.com/learn-golang-series/) - Tutorials to get started with programming in Go. * [Master Go](https://appliedgo.com/courses/mastergo/) - A paid online video course on Go for developers * [Learn to Create Web Applications using Go](https://www.usegolang.com/) - A paid online video course and book about Web programming with Go * [Learn Go Programming](https://blog.learngoprogramming.com/) - Weekly visual and concise tutorials for programming in Go. * [Gophercises](https://gophercises.com/) - coding exercises for budding gophers * [Algorithms to Go](http://yourbasic.org/) - Texts about algorithms and Go, with plenty of code examples. * [Games With Go](http://gameswithgo.org/) - Video series teaching programming fundamentals with game related projects. * [Go Language Tutorials](https://www.cybrhome.com/topic/go-language-tutorials) - List of popular sites, blogs and tutorials for learning Go language. * [Golang Development Video Course](https://www.youtube.com/playlist?list=PLzUGFf4GhXBL4GHXVcMMvzgtO8-WEJIoY) - A growing list of videos focused purely on Go development. * [Go: The Complete Bootcamp Course](https://www.udemy.com/learn-go-the-complete-bootcamp-course-golang/?couponCode=GOWIKI) - Step by step and intuitive explanations for every aspect of Go using animations (Paid) * [Learning Golang - TutorialEdge](https://tutorialedge.net/course/golang/) - A growing list of articles and courses on the fundamentals of Go. * [Go Discourse](https://github.com/godiscourse/godiscourse) - Another forum base on Go, without framework and ORM. * [دورة لغة غو بالعربي](https://argolang.com/learn-go-lang-beginners-course) * [Apuntes de Golang en Español](https://apuntes.de/golang/) - Apuntes para aprender Go en Español desde Cero. * [Go Classes at Codecademy](https://www.codecademy.com/learn/learn-go) - Online courses introducing the basics of Go. * [Go Tutorial - W3Basic](https://www.w3basic.com/golang/) - A well organized and structured Golang Tutorials for Beginners and Professionals Learning resources for specific topics: * [LearnConcurrency](https://go.dev/wiki/LearnConcurrency) outlines a course of study of Go’s concurrency model and patterns. * [LearnErrorHandling](https://go.dev/wiki/LearnErrorHandling) links to resources about error handling in Go. * [LearnTesting](https://go.dev/wiki/LearnTesting) links to resources about testing in Go. * [LearnServerProgramming](https://go.dev/wiki/LearnServerProgramming) links to resources about server programming in Go. * [Golang Online Courses](https://classpert.com/go-programming) - A collection of Go online courses from several providers at Classpert * [Hackr.io Golang Tutorials](https://hackr.io/tutorials/learn-golang) - Best Golang tutorials recommended by the programming community. * * * _This content is part of the [Go Wiki](https://go.dev/wiki/) ._ go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # A Guide to the Go Garbage Collector - The Go Programming Language 1. [Documentation](https://go.dev/doc/) 2. [A Guide to the Go Garbage Collector](https://go.dev/doc/gc-guide) A Guide to the Go Garbage Collector =================================== Introduction ------------ This guide is intended to aid advanced Go users in better understanding their application costs by providing insights into the Go garbage collector. It also provides guidance on how Go users may use these insights to improve their applications' resource utilization. It does not assume any knowledge of garbage collection, but does assume familiarity with the Go programming language. The Go language takes responsibility for arranging the storage of Go values; in most cases, a Go developer need not care about where these values are stored, or why, if at all. In practice, however, these values often need to be stored in computer **physical memory** and physical memory is a finite resource. Because it is finite, memory must be managed carefully and recycled in order to avoid running out of it while executing a Go program. It's the job of a Go implementation to allocate and recycle memory as needed. Another term for automatically recycling memory is **garbage collection**. At a high level, a garbage _collector_ (or GC, for short) is a system that recycles memory on behalf of the application by identifying which parts of memory are no longer needed. The Go standard toolchain provides a runtime library that ships with every application, and this runtime library includes a garbage collector. Note that the existence of a garbage collector as described by this guide is not guaranteed by the [Go specification](https://go.dev/ref/spec) , only that the underlying storage for Go values is managed by the language itself. This omission is intentional and enables the use of radically different memory management techniques. Therefore, this guide is about a specific implementation of the Go programming language and _may not apply to other implementations_. Specifically, this following guide applies to the standard toolchain (the `gc` Go compiler and tools). Gccgo and Gollvm both use a very similar GC implementation so many of the same concepts apply, but details may vary. Furthermore, this is a living document and will change over time to best reflect the latest release of Go. This document currently describes the garbage collector as of Go 1.19. ### Where Go Values Live Before we dive into the GC, let's first discuss the memory that doesn't need to be managed by the GC. For instance, non-pointer Go values stored in local variables will likely not be managed by the Go GC at all, and Go will instead arrange for memory to be allocated that's tied to the [lexical scope](https://go.dev/ref/spec#Declarations_and_scope) in which it's created. In general, this is more efficient than relying on the GC, because the Go compiler is able to predetermine when that memory may be freed and emit machine instructions that clean up. Typically, we refer to allocating memory for Go values this way as "stack allocation," because the space is stored on the goroutine stack. Go values whose memory cannot be allocated this way, because the Go compiler cannot determine its lifetime, are said to _escape to the heap_. "The heap" can be thought of as a catch-all for memory allocation, for when Go values need to be placed _somewhere_. The act of allocating memory on the heap is typically referred to as "dynamic memory allocation" because both the compiler and the runtime can make very few assumptions as to how this memory is used and when it can be cleaned up. That's where a GC comes in: it's a system that specifically identifies and cleans up dynamic memory allocations. There are many reasons why a Go value might need to escape to the heap. One reason could be that its size is dynamically determined. Consider for instance the backing array of a slice whose initial size is determined by a variable, rather than a constant. Note that escaping to the heap must also be transitive: if a reference to a Go value is written into another Go value that has already been determined to escape, that value must also escape. Whether a Go value escapes or not is a function of the context in which it is used and the Go compiler's escape analysis algorithm. It would be fragile and difficult to try to enumerate precisely when values escape: the algorithm itself is fairly sophisticated and changes between Go releases. For more details on how to identify which values escape and which do not, see the section on [eliminating heap allocations](https://go.dev/doc/gc-guide#Eliminating_heap_allocations) . ### Tracing Garbage Collection Garbage collection may refer to many different methods of automatically recycling memory; for example, reference counting. In the context of this document, garbage collection refers to **tracing** garbage collection, which identifies in-use, so-called **live**, objects by following pointers transitively. Let's define these terms more rigorously. * **Object**—An object is a dynamically allocated piece of memory that contains one or more Go values. * **Pointer**—A memory address that references any value within an object. This naturally includes Go values of the form `*T`, but also includes parts of built-in Go values. Strings, slices, channels, maps, and interface values all contain memory addresses that the GC must trace. Together, objects and pointers to other objects form the **object graph**. To identify live memory, the GC walks the object graph starting at the program's **roots**, pointers that identify objects that are definitely in-use by the program. Two examples of roots are local variables and global variables. The process of walking the object graph is referred to as **scanning**. Another phrase you might see in the Go documentation is whether an object is **reachable**, which just means that the object can be discovered by the scanning process. Note also that, [with one exception](https://go.dev/doc/gc-guide#Finalizers_cleanups_and_weak_pointers) , once memory becomes unreachable, it stays unreachable. This basic algorithm is common to all tracing GCs. Where tracing GCs differ is what they do once they discover memory is live. Go's GC uses the mark-sweep technique, which means that in order to keep track of its progress, the GC also **marks** the values it encounters as live. Once tracing is complete, the GC then walks over all memory in the heap and makes all memory that is _not_ marked available for allocation. This process is called **sweeping**. One alternative technique you may be familiar with is to actually _move_ the objects to a new part of memory and leave behind a forwarding pointer that is later used to update all the application's pointers. We call a GC that moves objects in this way a **moving** GC; Go has a **non-moving** GC. The GC cycle ------------ Because the Go GC is a mark-sweep GC, it broadly operates in two phases: the mark phase, and the sweep phase. While this statement might seem tautological, it contains an important insight: it's not possible to release memory back to be allocated until _all_ memory has been traced, because there may still be an un-scanned pointer keeping an object alive. As a result, the act of sweeping must be entirely separated from the act of marking. Furthermore, the GC may also not be active at all, when there's no GC-related work to do. The GC continuously rotates through these three phases of sweeping, off, and marking in what's known as the **GC cycle**. For the purposes of this document, consider the GC cycle starting with sweeping, turning off, then marking. The next few sections will focus on building intuition for the costs of the GC to aid users in tweaking GC parameters for their own benefit. ### Understanding costs The GC is inherently a complex piece of software built on even more complex systems. It's easy to become mired in detail when trying to understand the GC and tweak its behavior. This section is intended to provide a framework for reasoning about the cost of the Go GC and its tuning parameters. To begin with, consider this model of GC cost based on three simple axioms. 1. The GC involves only two resources: physical memory, and CPU time. 2. The GC's memory costs consist of live heap memory, new heap memory allocated before the mark phase, and space for metadata that, even if proportional to the previous costs, are small in comparison. _GC memory cost for cycle N = live heap from cycle N-1 + new heap_ Live heap memory is memory that was determined to be live by the previous GC cycle, while new heap memory is any memory allocated in the current cycle, which may or may not be live by the end. How much memory is live at any given point in time is a property of the program, and not something the GC can directly control. 3. The GC's CPU costs are modeled as a fixed cost per cycle, and a marginal cost that scales proportionally with the size of the live heap. _GC CPU time for cycle N = Fixed CPU time cost per cycle + average CPU time cost per byte \* live heap memory found in cycle N_ The fixed CPU time cost per cycle includes things that happen a constant number of times each cycle, like initializing data structures for the next GC cycle. This cost is typically small, and is included just for completeness. Most of the CPU cost of the GC is marking and scanning, which is captured by the marginal cost. The average cost of marking and scanning depends on the GC implementation, but also on the behavior of the program. For example, more pointers means more GC work, because at minimum the GC needs to visit all the pointers in the program. Structures like linked lists and trees are also more difficult for the GC to walk in parallel, increasing the average cost per byte. This model ignores sweeping costs, which are proportional to total heap memory, including memory that is dead (it must be made available for allocation). For Go's current GC implementation, sweeping is so much faster than marking and scanning that the cost is negligible in comparison. This model is simple but effective: it accurately categorizes the dominant costs of the GC. It also tells us that the _total CPU cost_ of the garbage collector depends on the total number of GC cycles in a given time frame. Finally, embedded in this model is a fundamental time/space trade-off for the GC. To see why, let's explore a constrained but useful scenario: the **steady state**. The steady state of an application, from the GC's perspective, is defined by the following properties: * The rate at which the application allocates new memory (in bytes per second) is constant. This means that, from the GC's perspective, the application's workload looks approximately the same over time. For example, for a web service, this would be a constant request rate with, on average, the same kinds of requests being made, with the average lifetime of each request staying roughly constant. * The marginal costs of the GC are constant. This means that statistics of the object graph, such as the distribution of object sizes, the number of pointers, and the average depth of data structures, remain the same from cycle to cycle. Let's work through an example. Assume some application is operating in a steady-state, allocating 10 MiB/s, while the GC can scan memory at a rate of 100 MiB/cpu-second (this is made up). The steady state makes no assumptions about the size of the live heap, but for simplicity, let's say this application's live heap is always 10 MiB. Let's also assume, again, for simplicity, that the fixed GC costs are zero. Let's play around with the GC cycle period. Suppose each GC cycle happens after exactly 1 cpu-second. Then, by the end of each GC cycle our example application will have allocated 10 MiB of additional memory, resulting in a 20 MiB total heap size. And with every GC cycle, the GC will spend 0.1 cpu-seconds scanning the 10 MiB live heap, resulting in a 10% CPU overhead. Recall that the GC only needs to walk the live heap, not the whole heap. (Note: a constant live heap does not mean that all newly allocated memory is dead. It means that, after the GC runs, _some mix_ of old and new heap memory dies, and only that the end result is 10 MiB found live each cycle.) Now suppose each GC cycle happens less often, once every 2 cpu-seconds. Then, our example application, in the steady state, will have a 30 MiB total heap size on each GC cycle, since it'll allocate 20 MiB in that time. But with every GC cycle, the GC will _still only need 0.1 cpu-seconds_ to scan the 10 MiB of live memory. Again, we're assuming the live heap size stays the same, regardless of how much memory is allocated. So this means that our GC overhead just went down, from 10% to 5%, at the cost of 50% more memory being used. This change in overheads is the fundamental time/space trade-off mentioned earlier. And **GC frequency** is at the center of this trade-off: if we execute the GC more frequently, then we use less memory, and vice versa. But how often does the GC actually execute? In Go, deciding when the GC should start is the main parameter which the user has control over. ### GOGC At a high level, GOGC determines the trade-off between GC CPU and memory. It works by determining the target heap size after each GC cycle, a target value for the total heap size in the next cycle. The GC's goal is to finish a collection cycle before the total heap size exceeds the target heap size. Total heap size is defined as the live heap size at the end of the previous cycle, plus any new heap memory allocated by the application since the previous cycle. Meanwhile, target heap memory is defined as: _Target heap memory = Live heap + (Live heap + GC roots) \* GOGC / 100_ As an example, consider a Go program with a live heap size of 8 MiB, 1 MiB of goroutine stacks, and 1 MiB of pointers in global variables. Then, with a GOGC value of 100, the amount of new memory that will be allocated before the next GC runs will be 10 MiB, or 100% of the 10 MiB of work, for a total heap footprint of 18 MiB. With a GOGC value of 50, then it'll be 50%, or 5 MiB. With a GOGC value of 200, it'll be 200%, or 20 MiB. Note: GOGC includes the root set only as of Go 1.18. Previously, it would only count the live heap. Often, the amount of memory in goroutine stacks is quite small and the live heap size dominates all other sources of GC work, but in cases where programs had hundreds of thousands of goroutines, the GC was making poor judgements. The heap target controls GC frequency: the bigger the target, the longer the GC can wait to start another mark phase and vice versa. While the precise formula is useful for making estimates, it's best to think of GOGC in terms of its fundamental purpose: a parameter that picks a point in the GC CPU and memory trade-off. The key takeaway is that **doubling GOGC will double heap memory overheads and roughly halve GC CPU cost**, and vice versa. (To see a full explanation as to why, see the [appendix](https://go.dev/doc/gc-guide#Additional_notes_on_GOGC) .) Note: the target heap size is just a target, and there are several reasons why the GC cycle might not finish right at that target. For one, a large enough heap allocation can simply exceed the target. However, other reasons appear in GC implementations that go beyond the [GC model](https://go.dev/doc/gc-guide#Understanding_costs) this guide has been using thus far. For some more detail, see the [latency section](https://go.dev/doc/gc-guide#Latency) , but the complete details may be found in the [additional resources](https://go.dev/doc/gc-guide#Additional_resources) . GOGC may be configured through either the `GOGC` environment variable (which all Go programs recognize), or through the [`SetGCPercent`](https://pkg.go.dev/runtime/debug#SetGCPercent) API in the `runtime/debug` package. Note that GOGC may also be used to turn off the GC entirely (provided the [memory limit](https://go.dev/doc/gc-guide#Memory_limit) does not apply) by setting `GOGC=off` or calling `SetGCPercent(-1)`. Conceptually, this setting is equivalent to setting GOGC to a value of infinity, as the amount of new memory before a GC is triggered is unbounded. To better understand everything we've discussed so far, try out the interactive visualization below that is built on the [GC cost model](https://go.dev/doc/gc-guide#Understanding_costs) discussed earlier. This visualization depicts the execution of some program whose non-GC work takes 10 seconds of CPU time to complete. In the first second it performs some initialization step (growing its live heap) before settling into a steady state. The application allocates 200 MiB in total, with 20 MiB live at a time. It assumes that the only relevant GC work to complete comes from the live heap, and that (unrealistically) the application uses no additional memory. Use the slider to adjust the value of GOGC to see how the application responds in terms of total duration and GC overhead. Each GC cycle ends while the new heap drops to zero. The time taken while the new heap drops to zero is the combined time for the mark phase for cycle N, and the sweep phase for the cycle N+1. Note that this visualization (and all the visualizations in this guide) assume the application is paused while the GC executes, so GC CPU costs are fully represented by the time it takes for new heap memory to drop to zero. This is only to make visualization simpler; the same intuition still applies. The X axis shifts to always show the full CPU-time duration of the program. Notice that additional CPU time used by the GC increases the overall duration. GOGC Notice that the GC always incurs some CPU and peak memory overhead. As GOGC increases, CPU overhead decreases, but peak memory increases proportionally to the live heap size. As GOGC decreases, the peak memory requirement decreases at the expense of additional CPU overhead. Note: the graph displays CPU time, not wall-clock time to complete the program. If the program runs on 1 CPU and fully utilizes its resources, then these are equivalent. A real-world program likely runs on a multi-core system and does not 100% utilize the CPUs at all times. In these cases the wall-time impact of the GC will be lower. Note: the Go GC has a minimum total heap size of 4 MiB, so if the GOGC-set target is ever below that, it gets rounded up. The visualization reflects this detail. Here's another example that's a little bit more dynamic and realistic. Once again, the application takes 10 CPU-seconds to complete without the GC, but the steady state allocation rate increases dramatically half-way through, and the live heap size shifts around a bit in the first phase. This example demonstrates how the steady state might look when the live heap size is actually changing, and how a higher allocation rate leads to more frequent GC cycles. GOGC ### Memory limit Until Go 1.19, GOGC was the sole parameter that could be used to modify the GC's behavior. While it works great as a way to set a trade-off, it doesn't take into account that available memory is finite. Consider what happens when there's a transient spike in the live heap size: because the GC will pick a total heap size proportional to that live heap size, GOGC must be configured such for the _peak_ live heap size, even if in the usual case a higher GOGC value provides a better trade-off. The visualization below demonstrates this transient heap spike situation. GOGC If the example workload is running in a container with a bit over 60 MiB of memory available, then GOGC can't be increased beyond 100, even though the rest of the GC cycles have the available memory to make use of that extra memory. Furthermore, in some applications, these transient peaks can be rare and hard to predict, leading to occasional, unavoidable, and potentially costly out-of-memory conditions. That's why in the 1.19 release, Go added support for setting a runtime memory limit. The memory limit may be configured either via the `GOMEMLIMIT` environment variable which all Go programs recognize, or through the `SetMemoryLimit` function available in the `runtime/debug` package. This memory limit sets a maximum on the _total amount of memory that the Go runtime can use_. The specific set of memory included is defined in terms of [`runtime.MemStats`](https://pkg.go.dev/runtime#MemStats) as the expression `Sys` `-` `HeapReleased` or equivalently in terms of the [`runtime/metrics`](https://pkg.go.dev/runtime/metrics) package, `/memory/classes/total:bytes` `-` `/memory/classes/heap/released:bytes` Because the Go GC has explicit control over how much heap memory it uses, it sets the total heap size based on this memory limit and how much other memory the Go runtime uses. The visualization below depicts the same single-phase steady state workload from the GOGC section, but this time with an extra 10 MiB of overhead from the Go runtime and with an adjustable memory limit. Try shifting around both GOGC and the memory limit and see what happens. GOGC Memory Limit Notice that when the memory limit is lowered below the peak memory that's determined by GOGC (42 MiB for a GOGC of 100), the GC runs more frequently to keep the peak memory within the limit. Returning to our previous example of the transient heap spike, by setting a memory limit and turning up GOGC, we can get the best of both worlds: no memory limit breach, and better resource economy. Try out the interactive visualization below. GOGC Memory Limit Notice that with some values of GOGC and the memory limit, peak memory use stops at whatever the memory limit is, but that the rest of the program's execution still obeys the total heap size rule set by GOGC. This observation leads to another interesting detail: even when GOGC is set to off, the memory limit is still respected! In fact, this particular configuration represents a _maximization of resource economy_ because it sets the minimum GC frequency required to maintain some memory limit. In this case, _all_ of the program's execution has the heap size rise to meet the memory limit. Now, while the memory limit is clearly a powerful tool, **the use of a memory limit does not come without a cost**, and certainly doesn't invalidate the utility of GOGC. Consider what happens when the live heap grows large enough to bring total memory use close to the memory limit. In the steady state visualization above, try turning GOGC off and then slowly lowering the memory limit further and further to see what happens. Notice that the total time the application takes will start to grow in an unbounded manner as the GC is constantly executing to maintain an impossible memory limit. This situation, where the program fails to make reasonable progress due to constant GC cycles, is called **thrashing**. It's particularly dangerous because it effectively stalls the program. Even worse, it can happen for exactly the same situation we were trying to avoid with GOGC: a large enough transient heap spike can cause a program to stall indefinitely! Try reducing the memory limit (around 30 MiB or lower) in the transient heap spike visualization and notice how the worst behavior specifically starts with the heap spike. In many cases, an indefinite stall is worse than an out-of-memory condition, which tends to result in a much faster failure. For this reason, the memory limit is defined to be **soft**. The Go runtime makes no guarantees that it will maintain this memory limit under all circumstances; it only promises some reasonable amount of effort. This relaxation of the memory limit is critical to avoiding thrashing behavior, because it gives the GC a way out: let memory use surpass the limit to avoid spending too much time in the GC. How this works internally is the GC sets an upper limit on the amount of CPU time it can use over some time window (with some hysteresis for very short transient spikes in CPU use). This limit is currently set at roughly 50%, with a `2 * GOMAXPROCS` CPU-second window. The consequence of limiting GC CPU time is that the GC's work is delayed, meanwhile the Go program may continue allocating new heap memory, even beyond the memory limit. The intuition behind the 50% GC CPU limit is based on the worst-case impact on a program with ample available memory. In the case of a misconfiguration of the memory limit, where it is set too low mistakenly, the program will slow down at most by 2x, because the GC can't take more than 50% of its CPU time away. Note: the visualizations on this page do not simulate the GC CPU limit. #### Suggested uses While the memory limit is a powerful tool, and the Go runtime takes steps to mitigate the worst behaviors from misuse, it's still important to use it thoughtfully. Below is a collection of tidbits of advice about where the memory limit is most useful and applicable, and where it might cause more harm than good. * **Do** take advantage of the memory limit when the execution environment of your Go program is entirely within your control, and the Go program is the only program with access to some set of resources (i.e. some kind of memory reservation, like a container memory limit). A good example is the deployment of a web service into containers with a fixed amount of available memory. **In this case, a good rule of thumb is to leave an additional 5-10% of headroom to account for memory sources the Go runtime is unaware of.** * **Do** feel free to adjust the memory limit in real time to adapt to changing conditions. A good example is a cgo program where C libraries temporarily need to use substantially more memory. * **Don't** set GOGC to off with a memory limit if the Go program might share some of its limited memory with other programs, and those programs are generally decoupled from the Go program. Instead, keep the memory limit since it may help to curb undesirable transient behavior, but set GOGC to some smaller, reasonable value for the average case. While it may be tempting to try and "reserve" memory for co-tenant programs, unless the programs are fully synchronized (e.g. the Go program calls some subprocess and blocks while its callee executes), the result will be less reliable as inevitably both programs will need more memory. Letting the Go program use less memory when it doesn't need it will generate a more reliable result overall. This advice also applies to overcommit situations, where the sum of memory limits of containers running on one machine may exceed the actual physical memory available to the machine. * **Don't** use the memory limit when deploying to an execution environment you don't control, especially when your program's memory use is proportional to its inputs. A good example is a CLI tool or a desktop application. Baking a memory limit into the program when it's unclear what kind of inputs it might be fed, or how much memory might be available on the system can lead to confusing crashes and poor performance. Plus, an advanced end-user can always set a memory limit if they wish. * **Don't** set a memory limit to avoid out-of-memory conditions when a program is already close to its environment's memory limits. This effectively replaces an out-of-memory risk with a risk of severe application slowdown, which is often not a favorable trade, even with the efforts Go makes to mitigate thrashing. In such a case, it would be much more effective to either increase the environment's memory limits (and _then_ potentially set a memory limit) or decrease GOGC (which provides a much cleaner trade-off than thrashing-mitigation does). ### Latency The visualizations in this document have modeled the application as paused while the GC is executing. GC implementations do exist that behave this way, and they're referred to as "stop-the-world" GCs. The Go GC, however, is not fully stop-the-world and does most of its work concurrently with the application. This is primarily to reduce application _latencies_. Specifically, the end-to-end duration of a single unit of computation (e.g. a web request). Thus far, this document mainly considered application _throughput_ (e.g. web requests handled per second). Note that each example in the [GC cycle](https://go.dev/doc/gc-guide#The_GC_cycle) section focused on the total CPU duration of an executing program. However, such a duration is far less meaningful for say, a web service. While throughput is still important for a web service (i.e. queries per second), often the latency of each individual request matters even more. In terms of latency, a stop-the-world GC may require a considerable length of time to execute both its mark and sweep phases, during which the application, and in the context of a web service, any in-flight request, is unable to make further progress. Instead, the Go GC avoids making the length of any global application pauses proportional to the size of the heap, and that the core tracing algorithm is performed while the application is actively executing. (The pauses are more strongly proportional to GOMAXPROCS algorithmically, but most commonly are dominated by the time it takes to stop running goroutines.) Collecting concurrently is not without cost: in practice it often leads to a design with lower throughput than an equivalent stop-the-world garbage collector. However, it's important to note that _lower latency does not inherently mean lower throughput_, and the performance of the Go garbage collector has steadily improved over time, in both latency and throughput. The concurrent nature of Go's current GC does not invalidate anything discussed in this document so far: none of the statements relied on this design choice. GC frequency is still the primary way the GC trades off between CPU time and memory for throughput, and in fact, it also takes on this role for latency. This is because most of the costs for the GC are incurred while the mark phase is active. The key takeaway then, is that **reducing GC frequency may also lead to latency improvements**. This applies not only to reductions in GC frequency from modifying tuning parameters, like increasing GOGC and/or the memory limit, but also applies to the optimizations described in the [optimization guide](https://go.dev/doc/gc-guide#Optimization_guide) . However, latency is often more complex to understand than throughput, because it is a product of the moment-to-moment execution of the program and not just an aggregation of costs. As a result, the connection between latency and GC frequency is less direct. Below is a list of possible sources of latency for those inclined to dig deeper. 1. Brief stop-the-world pauses when the GC transitions between the mark and sweep phases, 2. Scheduling delays because the GC takes 25% of CPU resources when in the mark phase, 3. User goroutines assisting the GC in response to a high allocation rate, 4. Pointer writes requiring additional work while the GC is in the mark phase, and 5. Running goroutines must be suspended for their roots to be scanned. These latency sources are visible in [execution traces](https://go.dev/doc/diagnostics#execution-tracer) , except for pointer writes requiring additional work. ### Finalizers, cleanups, and weak pointers Garbage collection provides the illusion of infinite memory using only finite memory. Memory is allocated but never explicitly freed, which enables simpler APIs and concurrent algorithms compared to bare-bones manual memory management. (Some languages with manually managed memory use alternative approaches such as "smart pointers" and compile-time ownership tracking to ensure that objects are freed, but these features are deeply embedded into the API design conventions in these languages.) Only the live objects—those reachable from a global variable or a computation in some goroutine—can affect the behavior of the program. Any time after an object becomes unreachable ("dead"), it may be safely recycled by the GC. This allows for a wide variety of GC designs, such as the tracing design used by Go today. The death of an object is not an observable event at the language level. However, Go's runtime library provides three features that break that illusion: [cleanups](https://go.dev/pkg/runtime#AddCleanup) , [weak pointers](https://go.dev/pkg/weak#Pointer) , and [finalizers](https://go.dev/pkg/runtime#SetFinalizer) . Each of these features provides some way to observe and react to object death, and in the case of finalizers, even reverse it. This of course complicates Go programs and adds an additional burden to the GC implementation. Nonetheless, these features exist because they are useful in a variety of circumstances, and Go programs use them and benefit from them all the time. For the details of each feature, refer to its package documentation ([runtime.AddCleanup](https://go.dev/pkg/runtime#AddCleanup) , [weak.Pointer](https://go.dev/pkg/weak#Pointer) , [runtime.SetFinalizer](https://go.dev/pkg/runtime#SetFinalizer) ). Below is some general advice for using these features, outlines of common issues you can run into with each feature, and advice for testing uses of these features. #### General advice * **Write unit tests.** The exact timing of cleanups, weak pointers, and finalizers can be difficult to predict, and it's easy to convince yourself that everything works, even after many consecutive executions. But it's also easy to make subtle mistakes. [Writing tests](https://go.dev/doc/gc-guide#Testing_object_death) for them can be tricky, but given that they're so subtle to use, testing is even more important usual. * **Avoid using these features directly in typical Go code.** These are low-level features with subtle restrictions and behaviors. For instance, there's no guarantee cleanups or finalizers will be run at program exit, or at all for that matter. The long comments in their API documentation should be seen as a warning. The vast majority of Go code does not benefit from using these features directly, only indirectly. * **Encapsulate the use of these mechanisms within a package.** Where possible, do not allow the use of these mechanisms to leak into the public API of your package; provide interfaces that make it hard or impossible for users to misuse them. For example, instead of asking the user to set up a cleanup on some C-allocated memory to free it, write a wrapper package and hide that detail inside. * **Restrict access to objects that have finalizers, cleanups, and weak pointers to the package that created and applied them.** This is related to the previous point, but is worth calling out explicitly, since it's a very powerful pattern for using these features in a less error-prone way. For example, the [unique package](https://go.dev/pkg/unique) uses weak pointers under the hood, but completely encasulates the objects that are weakly pointed-to. Those values can never be mutated by the rest of the application, it can only be copied through the [Value method](https://go.dev/pkg/unique#Handle.Value) , preserving the illusion of infinite memory for package users. * **Prefer cleaning up non-memory resources deterministically when possible, with finalizers and cleanups as a fallback.** Cleanups and finalizers are a good fit for memory resources such as memory allocated externally, like from C, or references to an `mmap` mapping. Memory allocated by C's malloc must eventually be freed by C's free. A finalizer that calls `free`, attached to a wrapper object for the C memory, is a reasonable way to ensure that C memory is eventually reclaimed as a consequence of garbage collection. However, non-memory resources, like file descriptors, tend to be subject to system limits that the Go runtime is generally unaware of. In addition, the timing of the garbage collector in a given Go program is usually something a package author has little control over (for instance, how often the GC runs is controlled by [GOGC](https://go.dev/doc/gc-guide#GOGC) , which can be set by operators to a variety of different values in practice). These two facts conspire to make cleanups and finalizers a bad fit to use as the only mechanism for releasing non-memory resources. If you're a package author exposing an API that wraps some non-memory resource, consider providing an explicit API for releasing the resource deterministically (through a `Close` method, or something similar), rather than relying on the garbage collector through cleanups or finalizers. Instead, prefer to use cleanups and finalizers as a best-effort handler for programmer mistakes, either by cleaning up the resource anyway like [os.File](https://go.dev/pkg/os#File) does, or by reporting the failure to deterministically clean up back to the user. * **Prefer cleanups to finalizers.** Historically, finalizers were added to simplify the interface between Go code and C code and to clean up non-memory resources. The intended use was to apply them to wrapper objects that owned C memory or some other non-memory resource, so that the resource could be released once Go code was done using it. These reasons at least partially explain why finalizers are narrowly scoped, why any given object can only have one finalizer, and why that finalizer must be attached to the first byte of the object only. This limitation already stifles some use-cases. For example, any package that wishes to internally cache some information about an object passed to it cannot clean up that information once the object is gone. But worse than that, finalizers are inefficient and error-prone due to the fact that they [resurrect the object](https://en.wikipedia.org/wiki/Object_resurrection) they're attached to, so that it can be passed to the finalizer function (and even continue to live beyond that, too). This simple fact means that if the object is part of a reference cycle it can never be freed, and the memory backing the object cannot be reused until at least until the following garbage collection cycle. Because finalizers resurrect objects, though, they do have a better-defined execution order than cleanups. For this reason, finalizers are still potentially (but rarely) useful for cleaning up structures that have complex destruction ordering requirements. But for all other uses in Go 1.24 and beyond, we recommend you use cleanups because they are more flexible, less error-prone, and more efficient than finalizers. #### Common cleanup issues * Objects with attached cleanups must not be reachable from the cleanup function (for example, through a captured local variable). This will prevent the object from being reclaimed and the cleanup from ever running. f := new(myFile) f.fd = syscall.Open(...) runtime.AddCleanup(f, func(fd int) { syscall.Close(f.fd) // Mistake: We reference f, so this cleanup won't run! }, f.fd) * Objects with attached cleanups must not be reachable from the argument to the cleanup function. This will prevent the object from being reclaimed and the cleanup from ever running. f := new(myFile) f.fd = syscall.Open(...) runtime.AddCleanup(f, func(f \*myFile) { syscall.Close(f.fd) }, f) // Mistake: We reference f, so this cleanup wouldn't ever run. This specific case also panics. * Finalizers have a well-defined execution order, but cleanups do not. Cleanups can also run concurrently with one another. * Long running cleanups should create a goroutine to avoid blocking the execution of other cleanups. * `runtime.GC` will not wait until cleanups for unreachable objects are executed, only until they are all queued. #### Common weak pointer issues * Weak pointers can begin returning `nil` from their `Value` method at unexpected times. Always guard the call to `Value` with a `nil` check and have a backup plan. * When weak pointers are used as map keys, they do not affect the reachability of map values. Therefore, if a weak pointer map key points to an object that is also reachable from the map value, that object will still be considered reachable. #### Common finalizer issues * Objects with attached finalizers must not be reachable from themselves by any path (in other words, they cannot be in a reference cycle). This will prevent the object from being reclaimed and the finalizer from ever running. f := new(myCycle) f.self = f // Mistake: f is reachable from f, so this finalizer would never run. runtime.SetFinalizer(f, func(f \*myCycle) { ... }) * Objects with attached finalizers must not be reachable from the finalizer function (for example, through a captured local variable). This will prevent the object from being reclaimed and the finalizer from ever running. f := new(myFile) f.fd = syscall.Open(...) runtime.SetFinalizer(f, func(\_ \*myFile) { syscall.Close(f.fd) // Mistake: We reference the outer f, so this cleanup won't run! }) * Reference chains of objects with attached finalizers (say, in a linked list) take, at minimum, as many GC cycles as there are objects in the chain to clean them all up. Keep finalizers shallow! // Mistake: reclaiming this linked list will take at least 10 GC cycles. node := new(linkedListNode) for range 10 { tmp := new(linkedListNode) tmp.next = node node = tmp runtime.SetFinalizer(node, func(node \*linkedListNode) { ... }) } * Avoid placing finalizers on objects returned at package boundaries. This makes it possible for users of your package to call `runtime.SetFinalizer` to mutate the finalizer on the object you return, which can be an unexpected behavior that users of your package may end up relying on. * Long running finalizers should create a new goroutine to avoid blocking the execution of other finalizers. * `runtime.GC` will not wait until finalizers for unreachable objects are executed, only until they are all queued. #### Testing object death When using these features, it can sometimes be tricky to write tests for code that uses them. Here are some tips for writing robust tests for code that uses these features. * Avoid running such tests in parallel with other tests. It helps a lot to increase determinism as much as possible and to have a good handle on the state of the world at any given time. * Use `runtime.GC` to establish a baseline upon entering the test. Use `runtime.GC` to force weak pointers to `nil`, and to queue up cleanups and finalizers to run. * `runtime.GC` does not wait for cleanups and finalizers to run, it only queues them. To write the most robust tests possible, inject a way to block on a cleanup or finalizer from your test (for example, pass an optional channel to the cleanup and/or finalizer from the test, and write to the channel once it has finished executing). If this is too hard or impossible, an alternative is to spin on a particular post-cleanup state to be true. For example, the `os` tests call `runtime.Gosched` in a loop that checks whether a file has been closed, once it becomes unreachable. * If writing tests for using finalizers, and you have a chain of objects that use finalizers, you will need at minimum the length of the deepest chain the test can create of `runtime.GC` calls to ensure all the finalizers run. * Test in race mode to discover races between concurrent cleanups, and between cleanup and finalizer code and the rest of the codebase. ### Additional resources While the information presented above is accurate, it lacks the detail to fully understand costs and trade-offs in the Go GC's design. For more information, see the following additional resources. * [The GC Handbook](https://gchandbook.org/) —An excellent general resource and reference on garbage collector design. * [TCMalloc](https://google.github.io/tcmalloc/design.html) —Design document for the C/C++ memory allocator TCMalloc, which the Go memory allocator is based on. * [Go 1.5 GC announcement](https://go.dev/blog/go15gc) —The blog post announcing the Go 1.5 concurrent GC, which describes the algorithm in more detail. * [Getting to Go](https://go.dev/blog/ismmkeynote) —An in-depth presentation about the evolution of Go's GC design up to 2018. * [Go 1.5 concurrent GC pacing](https://docs.google.com/document/d/1wmjrocXIWTr1JxU-3EQBI6BK6KgtiFArkG47XK73xIQ/edit) —Design document for determining when to start a concurrent mark phase. * [Smarter scavenging](https://go.dev/issue/30333) —Design document for revising the way the Go runtime returns memory to the operating system. * [Scalable page allocator](https://go.dev/issue/35112) —Design document for revising the way the Go runtime manages memory it gets from the operating system. * [GC pacer redesign (Go 1.18)](https://go.dev/issue/44167) —Design document for revising the algorithm to determine when to start a concurrent mark phase. * [Soft memory limit (Go 1.19)](https://go.dev/issue/48409) —Design document for the soft memory limit. A note about virtual memory --------------------------- This guide has largely focused on the physical memory use of the GC, but a question that comes up regularly is what exactly that means and how it compares to virtual memory (typically presented in programs like `top` as "VSS"). Physical memory is memory housed in the actual physical RAM chip in most computers. [Virtual memory](https://en.wikipedia.org/wiki/Virtual_memory) is an abstraction over physical memory provided by the operating system to isolate programs from one another. It's also typically acceptable for programs to reserve virtual address space that doesn't map to any physical addresses at all. **Because virtual memory is just a mapping maintained by the operating system, it is typically very cheap to make large virtual memory reservations that don't map to physical memory.** The Go runtime generally relies upon this view of the cost of virtual memory in a few ways: * The Go runtime never deletes virtual memory that it maps. Instead, it uses special operations that most operating systems provide to explicitly release any physical memory resources associated with some virtual memory range. This technique is used explicitly to manage the [memory limit](https://go.dev/doc/gc-guide#Memory_limit) and return memory to the operating system that the Go runtime no longer needs. The Go runtime also releases memory it no longer needs continuously in the background. See [the additional resources](https://go.dev/doc/gc-guide#Additional_resources) for more information. * On 32-bit platforms, the Go runtime reserves between 128 MiB and 512 MiB of address space up-front for the heap to limit fragmentation issues. * The Go runtime uses large virtual memory address space reservations in the implementation of several internal data structures. On 64-bit platforms, these typically have a minimum virtual memory footprint of about 700 MiB. On 32-bit platforms, their footprint is negligible. As a result, virtual memory metrics such as "VSS" in `top` are typically not very useful in understanding a Go program's memory footprint. Instead, focus on "RSS" and similar measurements, which more directly reflect physical memory usage. Optimization guide ------------------ ### Identifying costs Before trying to optimize how your Go application interacts with the GC, it's important to first identify that the GC is a major cost in the first place. The Go ecosystem provides a number of tools for identifying costs and optimizing Go applications. For a brief overview of these tools, see the [guide on diagnostics](https://go.dev/doc/diagnostics) . Here, we'll focus on a subset of these tools and a reasonable order to apply them in in order to understand GC impact and behavior. 1. **CPU profiles** A good place to start is with [CPU profiling](https://pkg.go.dev/runtime/pprof#hdr-Profiling_a_Go_program) . CPU profiling provides an overview of where CPU time is spent, though to the untrained eye it may be difficult to identify the magnitude of the role the GC plays in a particular application. Luckily, understanding how the GC fits in mostly boils down to knowing what different functions in the \`runtime\` package mean. Below is a useful subset of these functions for interpreting CPU profiles. Note that the functions listed below are not leaf functions, so they may not show up in the default the `pprof` tool provides with the `top` command. Instead, use the `top -cum` command or use the `list` command on these functions directly and focus on the cumulative percent column. * **`runtime.gcBgMarkWorker`**: Entrypoint to the background mark worker goroutines. Time spent here scales with GC frequency and the complexity and size of the object graph. It represents a baseline for how much time the application spends marking and scanning. Note that within these goroutines, you will find calls to `runtime.gcDrainMarkWorkerDedicated`, `runtime.gcDrainMarkWorkerFractional`, and `runtime.gcDrainMarkWorkerIdle`, which indicate worker type. In a largely idle Go application, the Go GC is going to use up additional (idle) CPU resources to get its job done faster, which is indicated with the `runtime.gcDrainMarkWorkerIdle` symbol. As a result, time here may represent a large fraction of CPU samples, which the Go GC believes are free. If the application becomes more active, CPU time in idle workers will drop. One common reason this can happen is if an application runs entirely in one goroutine but `GOMAXPROCS` is >1. * **`runtime.mallocgc`**: Entrypoint to the memory allocator for heap memory. A large amount of cumulative time spent here (>15%) typically indicates a lot of memory being allocated. * **`runtime.gcAssistAlloc`**: Function goroutines enter to yield some of their time to assist the GC with scanning and marking. A large amount of cumulative time spent here (>5%) indicates that the application is likely out-pacing the GC with respect to how fast it's allocating. It indicates a particularly high degree of impact from the GC, and also represents time the application spend marking and scanning. Note that this is included in the `runtime.mallocgc` call tree, so it will inflate that as well. 3. **Execution traces** While CPU profiles are great for identifying where time is spent in aggregate, they're less useful for indicating performance costs that are more subtle, rare, or related to latency specifically. Execution traces on the other hand provide a rich and deep view into a short window of a Go program's execution. They contain a variety of events related to the Go GC and specific execution paths can be directly observed, along with how the application might interact with the Go GC. All the GC events tracked are conveniently labeled as such in the trace viewer. See the [documentation for the `runtime/trace`](https://pkg.go.dev/runtime/trace) package for how to get started with execution traces. 4. **GC traces** When all else fails, the Go GC provides a few different specific traces that provide much deeper insights into GC behavior. These traces are always printed directly to STDERR, one line per GC cycle, and are configured through the `GODEBUG` environment variable that all Go programs recognize. They're mostly useful for debugging the Go GC itself since they require some familiarity with the specifics of the GC's implementation, but nonetheless can occasionally be useful to gain a better understanding of GC behavior. The core GC trace is enabled by setting `GODEBUG=gctrace=1`. The output produced by this trace is documented in the [environment variables section in the documentation for the `runtime` package.](https://pkg.go.dev/runtime#hdr-Environment_Variables) A supplementary GC trace called the "pacer trace" provides even deeper insights and is enabled by setting `GODEBUG=gcpacertrace=1`. Interpreting this output requires an understanding of the GC's "pacer" (see [additional resources](https://go.dev/doc/gc-guide#Additional_resources) ), which is outside the scope of this guide. ### Eliminating heap allocations One way to reduce costs from the GC is to have the GC manage fewer values to begin with. The techniques described below can produce some of the largest improvements in performance, because as the [GOGC section](https://go.dev/doc/gc-guide#GOGC) demonstrated, the allocation rate of a Go program is a major factor in GC frequency, the key cost metric used by this guide. #### Heap profiling After [identifying that the GC is a source of significant costs](https://go.dev/doc/gc-guide#Identifying_costs) , the next step in eliminating heap allocations is to find out where most of them are coming from. For this purpose, memory profiles (really, heap memory profiles) are very useful. Check out the [documentation](https://pkg.go.dev/runtime/pprof#hdr-Profiling_a_Go_program) for how to get started with them. Memory profiles describe where in the program heap allocations come from, identifying them by the stack trace at the point they were allocated. Each memory profile can break down memory in four ways. * `inuse_objects`—Breaks down the number of objects that are live. * `inuse_space`—Breaks down live objects by how much memory they use in bytes. * `alloc_objects`—Breaks down the number of objects that have been allocated since the Go program began executing. * `alloc_space`—Breaks down the total amount of memory allocated since the Go program began executing. Switching between these different views of heap memory may be done with either the `-sample_index` flag to the `pprof` tool, or via the `sample_index` option when the tool is used interactively. Note: memory profiles by default only sample a subset of heap objects so they will not contain information about every single heap allocation. However, this is sufficient to find hot-spots. To change the sampling rate, see [`runtime.MemProfileRate`](https://pkg.go.dev/runtime#pkg-variables) . For the purposes of reducing GC costs, `alloc_space` is typically the most useful view as it directly corresponds to the allocation rate. This view will indicate allocation hot spots that would provide the most benefit. #### Escape analysis Once candidate heap allocation sites have been identified with the help of [heap profiles](https://go.dev/doc/gc-guide#Heap_profiling) , how can they be eliminated? The key is to leverage the Go compiler's escape analysis to have the Go compiler find alternative, and more efficient storage for this memory, for example in the goroutine stack. Luckily, the Go compiler has the ability to describe why it decides to escape a Go value to the heap. With that knowledge, it becomes a matter of reorganizing your source code to change the outcome of the analysis (which is often the hardest part, but outside the scope of this guide). As for how to access the information from the Go compiler's escape analysis, the simplest way is through a debug flag supported by the Go compiler that describes all optimizations it applied or did not apply to some package in a text format. This includes whether or not values escape. Try the following command, where `[package]` is some Go package path. $ go build -gcflags=-m=3 \[package\] This information can also be visualized as an overlay in an LSP-capable editor; it is exposed as a code action. For example, in VS Code, invoke the "Source Action... > Show compiler optimization details" command to enable diagnostics for the current package. (You can also run the "Go: Toggle compiler optimization details" command.) Use this configuration setting to control which annotations are displayed: 1. Enable the overlay for escape analysis by [setting `ui.diagnostic.annotations` to include `escape`](https://github.com/golang/vscode-go/wiki/settings#uidiagnosticannotations) . Finally, the Go compiler provides this information in a machine-readable (JSON) format that may be used to build additional custom tooling. For more information on that, see the [documentation in the source Go code](https://cs.opensource.google/go/go/+/master:src/cmd/compile/internal/logopt/log_opts.go;l=25;drc=351e0f4083779d8ac91c05afebded42a302a6893) . ### Implementation-specific optimizations The Go GC is sensitive to the demographics of live memory, because a complex graph of objects and pointers both limits parallelism and generates more work for the GC. As a result, the GC contains a few optimizations for specific common structures. The most directly useful ones for performance optimization are listed below. Note: Applying the optimizations below may reduce the readability of your code by obscuring intent, and may fail to hold up across Go releases. Prefer to apply these optimizations only in the places they matter most. Such places may be identified by using the tools listed in the [section on identifying costs](https://go.dev/doc/gc-guide#Identifying_costs) . * Pointer-free values are segregated from other values. As a result, it may be advantageous to eliminate pointers from data structures that do not strictly need them, as this reduces the cache pressure the GC exerts on the program. As a result, data structures that rely on indices over pointer values, while less well-typed, may perform better. This is only worth doing if it's clear that the object graph is complex and the GC is spending a lot of time marking and scanning. * The GC will stop scanning values at the last pointer in the value. As a result, it may be advantageous to group pointer fields in struct-typed values at the beginning of the value. This is only worth doing if it's clear the application spends a lot of its time marking and scanning. (In theory the compiler can do this automatically, but it is not yet implemented, and struct fields are arranged as written in the source code.) Furthermore, the GC must interact with nearly every pointer it sees, so using indices into an slice, for example, instead of pointers, can aid in reducing GC costs. ### Linux transparent huge pages (THP) When a program accesses memory, the CPU needs to translate the [virtual memory](https://go.dev/doc/gc-guide#A_note_about_virtual_memory) addresses it uses into physical memory addresses that refer to the data it was trying to access. To do this, the CPU consults the "page table," a data structure that represents the mapping from virtual to physical memory, managed by the operating system. Each entry in the page table represents an indivisible block of physical memory called a page, hence the name. Transparent huge pages (THP) is a Linux feature that transparently replaces pages of physical memory backing contiguous virtual memory regions with bigger blocks of memory called huge pages. By using bigger blocks, fewer page table entries are needed to represent the same memory region, improving page table lookup times. However, bigger blocks mean more waste if only a small part of the huge page is used by the system. When running Go programs in production, enabling transparent huge pages on Linux can improve throughput and latency at the cost of additional memory use. Applications with small heaps tend not to benefit from THP and may end up using a substantial amount of additional memory (as high as 50%). However, applications with big heaps (1 GiB or more) tend to benefit quite a bit (up to 10% throughput) without very much additional memory overhead (1-2% or less). Being aware of your THP settings in either case can be helpful, and experimentation is always recommended. One can enable or disable transparent huge pages in a Linux environment by modifying `/sys/kernel/mm/transparent_hugepage/enabled`. See the [official Linux admin guide](https://www.kernel.org/doc/html/next/admin-guide/mm/transhuge.html) for more details. If you choose to have your Linux production environment enable transparent huge pages, we recommend the following additional settings for Go programs. * Set `/sys/kernel/mm/transparent_hugepage/defrag` to `defer` or `defer+madvise`. This setting controls how aggressively a Linux kernel coalesces regular pages into huge pages. `defer` tells the kernel to coalesce huge pages lazily and in the background. A more aggressive setting can induce stalls in memory constrained systems and can often hurt application latencies. `defer+madvise` is like `defer`, but is friendlier to other applications on the system that request huge pages explicitly and require them for performance. * Set `/sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none` to `0`. This setting controls how many additional pages the Linux kernel daemon can allocate when trying to allocate a huge page. The default setting is maximally aggressive, and can often [undo work the Go runtime does to return memory to the OS](https://bugzilla.kernel.org/show_bug.cgi?id=93111) . Before Go 1.21, the Go runtime tried to mitigate the negative effects of the default setting, but it came with a CPU cost. With Go 1.21+ and Linux 6.2+, the Go runtime no longer mutates huge page state. If you experience an increase in memory usage when upgrading to Go 1.21.1 or later, try applying this setting; it will likely resolve your issue. As an additional workaround, you can call [the `Prctl` function](https://go.dev/pkg/golang.org/x/sys/unix#Prctl) with `PR_SET_THP_DISABLE` to disable huge pages at the process level, or you can set `GODEBUG=disablethp=1` (to be added in Go 1.21.6 and Go 1.22) to disable huge pages for heap memory. Note that the `GODEBUG` setting may be removed in a future release. Appendix -------- ### Additional notes on GOGC The [GOGC section](https://go.dev/doc/gc-guide#GOGC) claimed that doubling GOGC doubles heap memory overheads and halves GC CPU costs. To see why, let's break it down mathematically. Firstly, the heap target sets a target for the total heap size. This target, however, mainly influences the new heap memory, because the live heap is fundamental to the application. _Target heap memory = Live heap + (Live heap + GC roots) \* GOGC / 100_ _Total heap memory = Live heap + New heap memory_ _⇒_ _New heap memory = (Live heap + GC roots) \* GOGC / 100_ From this we can see that doubling GOGC would also double the amount of new heap memory that application will allocate each cycle, which captures heap memory overheads. Note that _Live heap + GC roots_ is an approximation of the amount of memory the GC needs to scan. Next, let's look at GC CPU cost. Total cost can be broken down as the cost per cycle, times GC frequency over some time period T. _Total GC CPU cost = (GC CPU cost per cycle) \* (GC frequency) \* T_ GC CPU cost per cycle can be derived from the [GC model](https://go.dev/doc/gc-guide#Understanding_costs) : _GC CPU cost per cycle = (Live heap + GC roots) \* (Cost per byte) + Fixed cost_ Note that sweep phase costs are ignored here as mark and scan costs dominate. The steady state is defined by a constant allocation rate and a constant cost per byte, so in the steady state we can derive a GC frequency from this new heap memory: _GC frequency = (Allocation rate) / (New heap memory) = (Allocation rate) / ((Live heap + GC roots) \* GOGC / 100)_ Putting this together, we get the full equation for the total cost: _Total GC CPU cost = (Allocation rate) / ((Live heap + GC roots) \* GOGC / 100) \* ((Live heap + GC roots) \* (Cost per byte) + Fixed cost) \* T_ For a sufficiently large heap (which represents most cases), the marginal costs of a GC cycle dominate the fixed costs. This allows for a significant simplification of the total GC CPU cost formula. _Total GC CPU cost = (Allocation rate) / (GOGC / 100) \* (Cost per byte) \* T_ From this simplified formula, we can see that if we double GOGC, we halve total GC CPU cost. (Note that the visualizations in this guide do simulate fixed costs, so the GC CPU overheads reported by them will not exactly halve when GOGC doubles.) Furthermore, GC CPU costs are largely determined by allocation rate and the cost per byte to scan memory. For more information on how to reduce these costs specifically, see the [optimization guide](https://go.dev/doc/gc-guide#Optimization_guide) . Note: there exists a discrepancy between the size of the live heap, and the amount of that memory the GC actually needs to scan: the same size live heap but with a different structure will result in a different CPU cost, but the same memory cost, resulting a different trade-off. This is why the structure of the heap is part of the definition of the steady state. The heap target should arguably only include the scannable live heap as a closer approximation of memory the GC needs to scan, but this leads to degenerate behavior when there's a very small amount of scannable live heap but the live heap is otherwise large. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Go Vulnerability Database - The Go Programming Language Go Vulnerability Database ========================= [Back to Go Vulnerability Management](https://go.dev/security/vuln) Overview -------- The Go vulnerability database ([https://vuln.go.dev](https://vuln.go.dev/) ) serves Go vulnerability information in the [Open Source Vulnerability (OSV) schema](https://ossf.github.io/osv-schema/) . You can also browse vulnerabilities in the database at [pkg.go.dev/vuln](https://pkg.go.dev/vuln) . **Do not** rely on the contents of the x/vulndb Git repository. The YAML files in that repository are maintained using an internal format that may change without warning. Contributing ------------ We would love for all Go package maintainers to [contribute](https://go.dev/s/vulndb-report-new) information about public vulnerabilities in their own projects, and [update](https://go.dev/s/vulndb-report-feedback) existing information about vulnerabilities in their Go packages. We aim to make reporting a low friction process, so feel free to [send us your suggestions](https://go.dev/s/vuln-feedback) . Please **do not** use the forms above to report a vulnerability in the Go standard library or sub-repositories. Instead, follow the process at [go.dev/security/policy](https://go.dev/security/policy) for vulnerabilities about the Go project. API --- The canonical Go vulnerability database, [https://vuln.go.dev](https://vuln.go.dev/) , is an HTTP server that can respond to GET requests for the endpoints specified below. The endpoints have no query parameters, and no specific headers are required. Because of this, even a site serving from a fixed file system (including a `file://` URL) can implement this API. Each endpoint returns a JSON-encoded response, in either uncompressed (if requested as `.json`) or gzipped form (if requested as `.json.gz`). The endpoints are: * `/index/db.json[.gz]` Returns metadata about the database: { // The latest time the database should be considered // to have been modified, as an RFC3339-formatted UTC // timestamp ending in "Z". "modified": string } Note that the modified time _should not_ be compared to wall clock time, e.g. for purposes of cache invalidation, as there may a delay in making database modifications live. See [/index/db.json](https://vuln.go.dev/index/db.json) for a live example. * `/index/modules.json[.gz]` Returns a list containing metadata about each module in the database: [ {\ // The module path.\ "path": string,\ // The vulnerabilities that affect this module.\ "vulns":\ [ {\ // The vulnerability ID.\ "id": string,\ // The latest time the vulnerability should be considered\ // to have been modified, as an RFC3339-formatted UTC\ // timestamp ending in "Z".\ "modified": string,\ // (Optional) The module version (in SemVer 2.0.0 format)\ // that contains the latest fix for the vulnerability.\ // If unknown or unavailable, this should be omitted.\ "fixed": string,\ } ]\ } ] See [/index/modules.json](https://vuln.go.dev/index/modules.json) for a live example. * `/index/vulns.json[.gz]` Returns a list containing metadata about each vulnerability in the database: [ {\ // The vulnerability ID.\ "id": string,\ // The latest time the vulnerability should be considered\ // to have been modified, as an RFC3339-formatted UTC\ // timestamp ending in "Z".\ "modified": string,\ // A list of IDs of the same vulnerability in other databases.\ "aliases": [ string ]\ } ] See [/index/vulns.json](https://vuln.go.dev/index/vulns.json) for a live example. * `/ID/$id.json[.gz]` Returns the individual report for the vulnerability with ID `$id`, in OSV format (described below in [Schema](https://go.dev/doc/security/vuln/database#schema) ). See [/ID/GO-2022-0191.json](https://vuln.go.dev/ID/GO-2022-0191.json) for a live example. ### Bulk download To make it easier to download the entire Go vulnerability database, a zip file containing all the index and OSV files is available at [vuln.go.dev/vulndb.zip](https://vuln.go.dev/vulndb.zip) . ### Usage in `govulncheck` By default, `govulncheck` uses the canonical Go vulnerability database at [vuln.go.dev](https://vuln.go.dev/) . The command can be configured to contact a different vulnerability database using the `-db` flag,which accepts a vulnerability database URL with protocol `http://`, `https://`, or `file://`. To work correctly with `govulncheck`, the vulnerability database specified must implement the API described above. The `govulncheck` command uses compressed “.json.gz” endpoints when reading from an http(s) source, and the “.json” endpoints when reading from a file source. ### Legacy API The canonical database contains some additional endpoints that are part of a legacy API. We plan to remove support for these endpoints soon. If you are relying on the legacy API and need additional time to migrate, [please let us know](https://go.dev/s/govulncheck-feedback) . Schema ------ Reports use the [Open Source Vulnerability (OSV) schema](https://ossf.github.io/osv-schema/) . The Go vulnerability database assigns the following meanings to the fields: ### id The id field is a unique identifier for the vulnerability entry. It is a string of the format GO--. ### affected The [affected](https://ossf.github.io/osv-schema/#affected-fields) field is a JSON array containing objects that describes the module versions that contain the vulnerability. #### affected\[\].package The [affected\[\].package](https://ossf.github.io/osv-schema/#affectedpackage-field) field is a JSON object identifying the affected _module._ The object has two required fields: * **ecosystem**: this will always be “Go” * **name**: this is the Go module path * Importable packages in the standard library will have the name _stdlib_. * The go command will have the name _toolchain_. #### affected\[\].ecosystem\_specific The [affected\[\].ecosystem\_specific](https://ossf.github.io/osv-schema/#affectedecosystem_specific-field) field is a JSON object with additional information about the vulnerability, which is used by Go’s vulnerability detection tools. For now, ecosystem specific will always be an object with a single field, `imports`. ##### affected\[\].ecosystem\_specific.imports The `affected[].ecosystem_specific.imports` field is a JSON array containing the packages and symbols affected by the vulnerability. Each object in the array will have these two fields: * **path:** a string with the import path of the package containing the vulnerability * **symbols:** a string array with the names of the symbols (function or method) that contains the vulnerability * **goos**: a string array with the execution operating system where the symbols appear, if known * **goarch**: a string array with the architecture where the symbols appear, if known ### database\_specific The `database_specific` field contains custom fields specific to the Go vulnerability database. #### database\_specific.url The `database_specific.url` field is a string representing the fully-qualified URL of the Go vulnerability report, e.g, “[https://pkg.go.dev/vuln/GO-2023-1621"](https://pkg.go.dev/vuln/GO-2023-1621%22) . #### database\_specific.review\_status The `database_specific.review_status` field is a string representing the review status of the vulnerability report. If not present, the report should be considered `REVIEWED`. The possible values are: * `UNREVIEWED`: The report was automatically generated based on another source, such as a CVE or GHSA. Its data may be limited and has not been verified by the Go team. * `REVIEWED`: The report originated from the Go team, or was generated based on an external source. A member of the Go team has reviewed the report, and where appropriate, added additional data. For information on other fields in the schema, refer to the [OSV spec](https://ossf.github.io/osv-schema) . Note on Versions ---------------- Our tooling attempts to automatically map modules and versions in source advisories to canonical Go modules and versions, in accordance with standard [Go module version numbers](https://go.dev/doc/modules/version-numbers) . Tools like `govulncheck` are designed to rely on these standard versions to determine whether a Go project is affected by a vulnerability in a dependency or not. In some cases, such as when a Go project uses its own versioning scheme, the mapping to standard Go versions can fail. When this happens, the Go vulnerability database report may conservatively list all Go versions as affected. This ensures that tools such as `govulncheck` do not fail to report vulnerabilities due to unrecognized version ranges (false negatives). However, conservatively listing all versions as affected may cause tools to incorrectly report a fixed version of a module as containing the vulnerability (false positives). If you believe `govulncheck` is incorrectly reporting (or failing to report) a vulnerability, please [suggest an edit](https://github.com/golang/vulndb/issues/new?assignees=&labels=Needs+Triage%2CSuggested+Edit&template=suggest_edit.yaml&title=x%2Fvulndb%3A+suggestion+regarding+GO-2024-2965&report=GO-XXXX-YYYY) to the vulnerability report and we will review it. Examples -------- All vulnerabilities in the Go vulnerability database use the OSV schema described above. See the links below for examples of different Go vulnerabilities: * **Go standard library vulnerability** (GO-2022-0191): [JSON](https://vuln.go.dev/ID/GO-2022-0191.json) , [HTML](https://pkg.go.dev/vuln/GO-2022-0191) * **Go toolchain vulnerability** (GO-2022-0189): [JSON](https://vuln.go.dev/ID/GO-2022-0189.json) , [HTML](https://pkg.go.dev/vuln/GO-2022-0189) * **Vulnerability in Go module** (GO-2020-0015): [JSON](https://vuln.go.dev/ID/GO-2020-0015.json) , [HTML](https://pkg.go.dev/vuln/GO-2020-0015) Excluded Reports ---------------- The reports in the Go vulnerability database are collected from different sources and curated by the Go Security team. We may come across a vulnerability advisory (for example, a CVE or GHSA) and choose to exclude it for a variety of reasons. In these cases, a minimal report will be created in the x/vulndb repository, under [x/vulndb/data/excluded](https://github.com/golang/vulndb/tree/master/data/excluded) . Reports may be excluded for these reasons: * `NOT_GO_CODE`: The vulnerability is not in a Go package, but it was marked as a security advisory for the Go ecosystem by another source. This vulnerability cannot affect any Go packages. (For example, a vulnerability in a C++ library.) * `NOT_IMPORTABLE`: The vulnerability occurs in package `main`, an `internal/` package only imported by package `main`, or some other location which can never be imported by another module. * `EFFECTIVELY_PRIVATE`: While the vulnerability occurs in a Go package which can be imported by another module, the package is not intended for external use and is not likely to ever be imported outside the module in which it is defined. * `DEPENDENT_VULNERABILITY`: This vulnerability is a subset of another vulnerability in the database. For example, if package A contains a vulnerability, package B depends on package A, and there are separate CVE IDs for packages A and B, we might mark the report for B as a dependent vulnerability entirely superseded by the report for A. * `NOT_A_VULNERABILITY`: While a CVE ID or GHSA has been assigned, there is no known vulnerability associated with it. * `WITHDRAWN`: The vulnerability has been withdrawn by its source. At the moment, excluded reports are not served via [vuln.go.dev](https://vuln.go.dev/) API. However, if you have a specific use case and it would be helpful to have access to this information through the API, [please let us know](https://go.dev/s/govulncheck-feedback) . go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # It's survey time! How has Go has been working out for you? - The Go Programming Language [The Go Blog](https://go.dev/blog/) ==================================== It's survey time! How has Go has been working out for you? ========================================================== Todd Kulesza, on behalf of the Go team 16 September 2025 Hi Gophers! Today we’re excited to announce our [2025 Go Developer Survey](https://google.qualtrics.com/jfe/form/SV_3wwSstC8vv4Ymkm?s=b) . The Go Team uses the results of this annual survey to better understand the needs and concerns of Go developers across the world. Your feedback helps us brainstorm, plan, and prioritize work on Go. You can [take the survey here](https://google.qualtrics.com/jfe/form/SV_3wwSstC8vv4Ymkm?s=b) . It will be open through **September 30th**. The survey should take 10 – 20 minutes to complete, and every question is optional. We’ll share aggregated survey results on this blog in early November. This year we’ll also share the raw dataset of survey responses, so that the entire Go community can benefit from this knowledge and conduct your own analyses on the data. Similar to our approach with [Go Telemetry](https://go.dev/blog/gotelemetry) , we’re using an opt-in model: the survey will ask for your permission to include your responses in the dataset. If you don’t give us permission, your survey responses will not be shared. Please help us reach as many Go developers as possible! We love it when you share the survey with your colleagues, friends, and online communities. The more voices we hear, the better we can understand the diverse needs of Go developers everywhere. We’re looking forward to hearing your feedback! **Next article:** [Flight Recorder in Go 1.25](https://go.dev/blog/flight-recorder) **Previous article:** [A new experimental Go API for JSON](https://go.dev/blog/jsonv2-exp) **[Blog Index](https://go.dev/blog/all) ** go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Testing Time (and other asynchronicities) - The Go Programming Language [The Go Blog](https://go.dev/blog/) ==================================== Testing Time (and other asynchronicities) ========================================= Damien Neil 26 August 2025 In Go 1.24, we introduced the [`testing/synctest`](https://go.dev/pkg/testing/synctest) package as an experimental package. This package can significantly simplify writing tests for concurrent, asynchronous code. In Go 1.25, the `testing/synctest` package has graduated from experiment to general availability. What follows is the blog version of my talk on the [`testing/synctest`](https://go.dev/pkg/testing/synctest) package at GopherCon Europe 2025 in Berlin. What is an asynchronous function? --------------------------------- A synchronous function is pretty simple. You call it, it does something, and it returns. An asynchronous function is different. You call it, it returns, and then it does something. As a concrete, if somewhat artificial, example, the following `Cleanup` function is synchronous. You call it, it deletes a cache directory, and it returns. func (c *Cache) Cleanup() { os.RemoveAll(c.cacheDir) } `CleanupInBackground` is an asynchronous function. You call it, it returns, and the cache directory is deleted…sooner or later. func (c *Cache) CleanupInBackground() { go os.RemoveAll(c.cacheDir) } Sometimes an asynchronous function does something in the future. For example, the `context` package’s `WithDeadline` function returns a context which will be canceled in the future. package context // WithDeadline returns a derived context // with a deadline no later than d. func WithDeadline(parent Context, d time.Time) (Context, CancelFunc) When I talk about testing concurrent code, I mean testing these sorts of asynchronous operations, both ones which use real time and ones which do not. Tests ----- A test verifies that a system behaves as we expect. There’s a lot of terminology describing types of test–unit tests, integration tests, and so on–but for our purposes here every kind of test reduces to three steps: 1. Set up some initial conditions. 2. Tell the system under test to do something. 3. Verify the result. Testing a synchronous function is straightforward: * You call the function; * the function does something and returns; * you verify the result. Testing an asynchronous function, however, is tricky: * You call the function; * it returns; * you wait for it to finish doing whatever it does; * you verify the result. If you don’t wait for the correct amount of time, you may find yourself verifying the result of an operation that hasn’t happened yet or has only happened partially. This never ends well. Testing an asynchronous function is especially tricky when you want to assert that something has _not_ happened. You can verify that the thing has not happened yet, but how do you know with certainty that it isn’t going to happen later? An example ---------- To make things a little more concrete, let’s work with a real-world example. Consider the `context` package’s `WithDeadline` function again. package context // WithDeadline returns a derived context // with a deadline no later than d. func WithDeadline(parent Context, d time.Time) (Context, CancelFunc) There are two obvious tests to write for `WithDeadline`. 1. The context is _not_ canceled _before_ the deadline. 2. The context _is_ canceled _after_ the deadline. Let’s write a test. To keep the amount of code marginally less overwhelming, we’ll just test the second case: After the deadline expires, the context is canceled. func TestWithDeadlineAfterDeadline(t *testing.T) { deadline := time.Now().Add(1 * time.Second) ctx, _ := context.WithDeadline(t.Context(), deadline) time.Sleep(time.Until(deadline)) if err := ctx.Err(); err != context.DeadlineExceeded { t.Fatalf("context not canceled after deadline") } } This test is simple: 1. Use `context.WithDeadline` to create a context with a deadline one second in the future. 2. Wait until the deadline. 3. Verify that the context is canceled. Unfortunately, this test obviously has a problem. It sleeps until the exact moment the deadline expires. Odds are good that the context has not been canceled yet by the time we examine it. At best, this test will be very flaky. Let’s fix it. time.Sleep(time.Until(deadline) + 100*time.Millisecond) We can sleep until 100ms after the deadline. A hundred milliseconds is an eternity in computer terms. This should be fine. Unfortunately, we still have two problems. First, this test takes 1.1 seconds to execute. That’s slow. This is a simple test. It should execute in milliseconds at the most. Second, this test is flaky. A hundred milliseconds is an eternity in computer terms, but on an overloaded continuous integration (CI) system it isn’t unusual to see pauses much longer than that. This test will probably pass consistently on a developer’s workstation, but I would expect occasional failures in a CI system. Slow or flaky: Pick two ----------------------- Tests that use real time are always slow or flaky. Usually they’re both. If the test waits longer than necessary, it is slow. If it doesn’t wait long enough, it is flaky. You can make the test more slow and less flaky, or less slow and more flaky, but you can’t make it fast and reliable. We have a lot of tests in the `net/http` package which use this approach. They’re all slow and/or flaky, which is what started me down the road which brings us here today. Write synchronous functions? ---------------------------- The simplest way to test an asynchronous function is not to do it. Synchronous functions are easy to test. If you can transform an asynchronous function into a synchronous one, it will be easier to test. For example, if we consider our cache cleanup functions from earlier, the synchronous `Cleanup` is obviously better than the asynchronous `CleanupInBackground`. The synchronous function is easier to test, and the caller can easily start a new goroutine to run it in the background if needed. As a general rule, the higher up the call stack you can push your concurrency, the better. // CleanupInBackground is hard to test. cache.CleanupInBackground() // Cleanup is easy to test, // and easy to run in the background when needed. go cache.Cleanup() Unfortunately, this sort of transformation isn’t always possible. For example, `context.WithDeadline` is an inherently asynchronous API. Instrument code for testability? -------------------------------- A better approach is to make our code more testable. Here’s an example of what this might look like for our `WithDeadline` test: func TestWithDeadlineAfterDeadline(t *testing.T) { clock := fakeClock() timeout := 1 * time.Second deadline := clock.Now().Add(timeout) ctx, _ := context.WithDeadlineClock( t.Context(), deadline, clock) clock.Advance(timeout) context.WaitUntilIdle(ctx) if err := ctx.Err(); err != context.DeadlineExceeded { t.Fatalf("context not canceled after deadline") } } Instead of using real time, we use a fake time implementation. Using fake time avoids unnecessarily slow tests, because we never wait around doing nothing. It also helps avoid test flakiness, since the current time only changes when the test adjusts it. There are various fake time packages out there, or you can write your own. To use fake time, we need to modify our API to accept a fake clock. I’ve added a `context.WithDeadlineClock` function here, that takes an additional clock parameter: ctx, _ := context.WithDeadlineClock( t.Context(), deadline, clock) When we advance our fake clock, we have a problem. Advancing time is an asynchrounous operation. Sleeping goroutines may wake up, timers may send on their channels, and timer functions may run. We need to wait for that work to finish before we can test the expected behavior of the system. I’ve added a `context.WaitUntilIdle` function here, which waits for any background work related to a context to complete: clock.Advance(timeout) context.WaitUntilIdle(ctx) This is a simple example, but it demonstrates the two fundamental principles of writing testable concurrent code: 1. Use fake time (if you use time). 2. Have some way to wait for quiescence, which is a fancy way of saying “all background activity has stopped and the system is stable”. The interesting question, of course, is how we do this. I’ve glossed over the details in this example because there are some big downsides to this approach. It’s hard. Using a fake clock isn’t difficult, but identifying when background concurrent work is finished and it is safe to examine the state of the system is. Your code becomes less idiomatic. You can’t use standard time package functions. You need to be very careful to keep track of everything happening in the background. You need to instrument not just your code, but any other packages you use. If you call any third-party concurrent code, you’re probably out of luck. Worst of all, it can be just about impossible to retrofit this approach into an existing codebase. I attempted to apply this approach to Go’s HTTP implementation, and while I had some success at doing so in places, the HTTP/2 server simply defeated me. In particular, adding instrumentation to detect quiescence without extensive rewriting proved infeasible, or at least beyond my skills. Horrible runtime hacks? ----------------------- What do we do if we can’t make our code testable? What if instead of instrumenting our code, we had a way to observe the behavior of the uninstrumented system? A Go program consists of a set of goroutines. Those goroutines have states. We just need to wait until all the goroutines have stopped running. Unfortunately, the Go runtime doesn’t provide any way to tell what those goroutines are doing. Or does it? The `runtime` package contains a function that gives us a stack trace for every running goroutine, as well as their states. This is text intended for human consumption, but we could parse that output. Could we use this to detect quiescence? Now, of course this is a terrible idea. There is no guarantee that the format of these stack traces will be stable over time. You should not do this. I did it. And it worked. In fact, it worked surprisingly well. With a simple implementation of a fake clock, a small amount of instrumentation to keep track of what goroutines were part of the test, and some horrifying abuse of `runtime.Stack`, I finally had a way to write fast, reliable tests for the `http` package. The underlying implementation of these tests was horrible, but it demonstrated that there was a useful concept here. A better way ------------ Go may have built-in concurrency, but testing programs that use that concurrency is hard. We’re faced with an unfortunate choice: We can write simple, idiomatic code, but it will be impossible to test quickly and reliably; or we can write testable code, but it will be complicated and unidiomatic. So we asked ourselves what we can do to make this better. As we saw earlier, the two fundamental features required to write testable concurrent code are fake time and a way to wait for quiescence. We need a better way to to wait for quiescence. We should be able to ask the runtime when background goroutines have finished their work. We also want to be able to limit the scope of this query to a single test, so that unrelated tests do not interfere with each other. We also need better support for testing programs using fake time. It isn’t hard to make a fake time implementation, but code which uses an implementation like this is not idiomatic. Idiomatic code will use a `time.Timer`, but it is not possible to create a fake `Timer`. We asked ourselves whether we should provide a way for tests to create a fake `Timer`, where the test controls when the timer fires. A testing implementation of time needs to define an entirely new version of the `time` package, and pass that to every function that operates on time. We considered whether we should define a common time interface, in the same way that `net.Conn` is a common interface describing a network connection. What we realized, however, is that unlike network connections, there is only one possible implementation of fake time. A fake network may want to introduce latency or errors. Time, in contrast, does only one thing: It moves forward. Tests need to control the rate at which time progresses, but a timer scheduled to fire ten seconds in the future should always fire ten (possibly fake) seconds in the future. In addition, we don’t want to upset the entire Go ecosystem. Most programs today use functions in the time package. We want to keep those programs not only working, but idiomatic. This led to the conclusion that what we need is a way for a test to tell the time package to use a fake clock, in much the same way that the Go playground uses a fake clock. Unlike the playground, we need to limit the scope of that change to a single test. (It may not be obvious that the Go playground uses a fake clock, because we turn any fake delays into real delays on the front end, but it does.) The `synctest` experiment ------------------------- And so in Go 1.24 we introduced [`testing/synctest`](https://go.dev/pkg/testing/synctest) , a new, experimental package to simplify testing concurrent programs. Over the months following the release of Go 1.24 we gathered feedback from early adopters. (Thank you to everyone who tried it out!) We made a number of changes to address problems and shortcomings. And now, in Go 1.25, we’ve released the `testing/synctest` package as part of the standard library. It lets you run a function in what we’re calling a “bubble”. Within the bubble, the time package uses a fake clock, and the `synctest` package provides a function to wait for the bubble to quiesce. The `synctest` package ---------------------- The `synctest` package contains just two functions. package synctest // Test executes f in a new bubble. // Goroutines in the bubble use a fake clock. func Test(t *testing.T, f func(*testing.T)) // Wait waits for background activity in the bubble to complete. func Wait() [`Test`](https://go.dev/pkg/testing/synctest#Test) executes a function in a new bubble. [`Wait`](https://go.dev/pkg/testing/synctest#Wait) blocks until every goroutine in the bubble is blocked waiting for some other goroutine in the bubble. We call that state being “durably blocked”. Testing with synctest --------------------- Let’s look at an example of synctest in action. func TestWithDeadlineAfterDeadline(t *testing.T) { synctest.Test(t, func(t *testing.T) { deadline := time.Now().Add(1 * time.Second) ctx, _ := context.WithDeadline(t.Context(), deadline) time.Sleep(time.Until(deadline)) synctest.Wait() if err := ctx.Err(); err != context.DeadlineExceeded { t.Fatalf("context not canceled after deadline") } }) } This might look a little familiar. This is the naïve test for `context.WithDeadline` that we looked at earlier. The only changes are that we’ve wrapped the test in a `synctest.Test` call to execute it in a bubble and we have added a `synctest.Wait` call. This test is fast and reliable. It runs almost instantaneously. It precisely tests the expected behavior of the system under test. It also requires no modification of the `context` package. Using the `synctest` package, we can write simple, idiomatic code and test it reliably. This is a very simple example, of course, but this is a real test of real production code. If `synctest` had existed when the `context` package was written, we would have had a much easier time writing tests for it. Time ---- Time in the bubble behaves much the same as the fake time in the Go playground. Time starts at midnight, January 1, 2000 UTC. If you need to run a test at some specific point in time for some reason, you can just sleep until then. func TestAtSpecificTime(t *testing.T) { synctest.Test(t, func(t *testing.T) { // 2000-01-01 00:00:00 +0000 UTC t.Log(time.Now().In(time.UTC)) // This does not take 25 years. time.Sleep(time.Until( time.Date(2025, 1, 1, 0, 0, 0, 0, time.UTC))) // 2025-01-01 00:00:00 +0000 UTC t.Log(time.Now().In(time.UTC)) }) } Time only passes when every goroutine in the bubble has blocked. You can think of the bubble as simulating an infinitely fast computer: Any amount of computation takes no time. The following test will always print that zero seconds of fake time have elapsed since the start of the test, no matter how much real time has passed. func TestExpensiveWork(t *testing.T) { synctest.Test(t, func(t *testing.T) { start := time.Now() for range 1e7 { // do expensive work } t.Log(time.Since(start)) // 0s }) } In the next test, the `time.Sleep` call will return immediately, rather than waiting for ten real seconds. The test will always print that exactly ten fake seconds have passed since the start of the test. func TestSleep(t *testing.T) { synctest.Test(t, func(t *testing.T) { start := time.Now() time.Sleep(10 * time.Second) t.Log(time.Since(start)) // 10s }) } Waiting for quiescence ---------------------- The [`synctest.Wait`](https://go.dev/pkg/testing/synctest#Wait) function lets us wait for background activity to complete. func TestWait(t *testing.T) { synctest.Test(t, func(t *testing.T) { done := false go func() { done = true }() // Wait for the above goroutine to finish. synctest.Wait() t.Log(done) // true }) } If we didn’t have the `Wait` call in the above test, we would have a race condition: One goroutine modifies the `done` variable while another reads from it without synchronization. The `Wait` call provides that synchronization. You may be familiar with the `-race` test flag, which enables the data race detector. The race detector is aware of the synchronization provided by `Wait`, and does not complain about this test. If we forgot the `Wait` call, the race detector would correctly complain. The `synctest.Wait` function provides synchronization, but the passage of time does not. In the next example, one goroutine writes to the `done` variable while another sleeps for one nanosecond before reading from it. It should be obvious that when run with a real clock outside a synctest bubble, this code contains a race condition. Inside a synctest bubble, while the fake clock ensures that the goroutine completes before `time.Sleep` returns, the race detector will still report the data race, just like it would if this code were run outside a synctest bubble. func TestTimeDataRace(t *testing.T) { synctest.Test(t, func(t *testing.T) { done := false go func() { done = true // write }() time.Sleep(1 * time.Nanosecond) t.Log(done) // read (unsynchronized) }) } Adding a `Wait` call provides explicit synchronization and fixes the data race: time.Sleep(1 * time.Nanosecond) synctest.Wait() // synchronize t.Log(done) // read Example: `io.Copy` ------------------ Taking advantage of the synchronization provided by `synctest.Wait` allows us to write simpler tests with less explicit synchronization. For example, consider this test of [`io.Copy`](https://go.dev/pkg/io#Copy) . func TestIOCopy(t *testing.T) { synctest.Test(t, func(t *testing.T) { srcReader, srcWriter := io.Pipe() defer srcWriter.Close() var dst bytes.Buffer go io.Copy(&dst, srcReader) data := "1234" srcWriter.Write([]byte("1234")) synctest.Wait() if got, want := dst.String(), data; got != want { t.Errorf("Copy wrote %q, want %q", got, want) } }) } The `io.Copy` function copies data from an `io.Reader` to an `io.Writer`. You might not immediately think of `io.Copy` as a concurrent function, since it blocks until the copy has completed. However, providing data to `io.Copy`’s reader is an asynchronous operation: * `Copy` calls the reader’s `Read` method; * `Read` returns some data; * and the data is written to the writer at a later time. In this test, we are verifying that `io.Copy` writes new data to the writer without waiting to fill its buffer. Looking at the test step by step, we first create an `io.Pipe` to serve as the source `io.Copy` reads from: srcReader, srcWriter := io.Pipe() defer srcWriter.Close() We call `io.Copy` in a new goroutine, copying from the read end of the pipe into a `bytes.Buffer`: var dst bytes.Buffer go io.Copy(&dst, srcReader) We write to the other end of the pipe, and wait for `io.Copy` to handle the data: data := "1234" srcWriter.Write([]byte("1234")) synctest.Wait() Finally, we verify that the destination buffer contains the desired data: if got, want := dst.String(), data; got != want { t.Errorf("Copy wrote %q, want %q", got, want) } We don’t need to add a mutex or other synchronization around the destination buffer, because `synctest.Wait` ensures that it is never accessed concurrently. This test demonstrates a few important points. Even synchronous functions like `io.Copy`, which do not perform additional background work after they return, may exhibit asynchronous behaviors. Using `synctest.Wait`, we can test those behaviors. Note also that this test does not work with time. Many asynchronous systems involve time, but not all. Bubble exit ----------- The `synctest.Test` function waits for all goroutines in the bubble to exit before returning. Time stops advancing after the root goroutine (the goroutine started by `Test`) returns. In the next example, `Test` waits for the background goroutine to run and exit before it returns: func TestWaitForGoroutine(t *testing.T) { synctest.Test(t, func(t *testing.T) { go func() { // This runs before synctest.Test returns. }() }) } In this example, we schedule a `time.AfterFunc` for a time in the future. The bubble’s root goroutine returns before that time is reached, so the `AfterFunc` never runs: func TestDoNotWaitForTimer(t *testing.T) { synctest.Test(t, func(t *testing.T) { time.AfterFunc(1 * time.Nanosecond, func() { // This never runs. }) }) } In the next example, we start a goroutine that sleeps. The root goroutine returns and time stops advancing. The bubble is now deadlocked, because `Test` is waiting for all goroutines in the bubble to finish but the sleeping goroutine is waiting for time to advance. func TestDeadlock(t *testing.T) { synctest.Test(t, func(t *testing.T) { go func() { // This sleep never returns and the test deadlocks. time.Sleep(1 * time.Nanosecond) }() }) } Deadlocks --------- The `synctest` package panics when a bubble is deadlocked due to every goroutine in the bubble being durably blocked on another goroutine in the bubble. --- FAIL: Test (0.00s) --- FAIL: TestDeadlock (0.00s) panic: deadlock: main bubble goroutine has exited but blocked goroutines remain [recovered, repanicked] goroutine 7 [running]: (stacks elided for clarity) goroutine 10 [sleep (durable), synctest bubble 1]: time.Sleep(0x1) /Users/dneil/src/go/src/runtime/time.go:361 +0x130 _.TestDeadlock.func1.1() /tmp/s/main_test.go:13 +0x20 created by _.TestDeadlock.func1 in goroutine 9 /tmp/s/main_test.go:11 +0x24 FAIL _ 0.173s FAIL The runtime will print stack traces for every goroutine in the deadlocked bubble. When printing the status of a bubbled goroutine, the runtime indicates when the goroutine is durably blocked. You can see that the sleeping goroutine in this test is durably blocked. Durable blocking ---------------- “Durably blocking” is a core concept in synctest. A goroutine is durably blocked when it is not only blocked, but when it can only be unblocked by another goroutine in the same bubble. When every goroutine in a bubble is durably blocked: 1. `synctest.Wait` returns. 2. If there is no `synctest.Wait` call in progress, fake time advances instantly to the next point that will wake a goroutine. 3. If there is no goroutine that can be woken by advancing time, the bubble is deadlocked and the test fails. It is important for us to make a distinction between a goroutine which is merely blocked and one which is _durably_ blocked. We don’t want to declare a deadlock when a goroutine is temporarily blocked on some event arising outside its bubble. Let’s look at some ways in which a goroutine can block non-durably. ### Not durably blocking: I/O (files, pipes, network connections, etc.) The most important limitation is that I/O is not durably blocking, including network I/O. A goroutine reading from a network connection may be blocked, but it will be unblocked by data arriving on that connection. This is obviously true for a connection to some network service, but it is also true for a loopback connection, even when the reader and writer are both in the same bubble. When we write data to a network socket, even a loopback socket, the data is passed to the kernel for delivery. There is a period of time between the write system call returning and the kernel notifying the other side of the connection that data is available. The Go runtime cannot distinguish between a goroutine blocked waiting for data that is already in the kernel’s buffers and one blocked waiting for data that will not arrive. This means that tests of networked programs using synctest usually cannot use real network connections. Instead, they should use an in-memory fake. I’m not going to go over the process of creating a fake network here, but the `synctest` package documentation contains [a complete worked example](https://go.dev/pkg/testing/synctest#hdr-Example__HTTP_100_Continue) of testing an HTTP client and server communicating over a fake network. ### Not durably blocking: syscalls, cgo calls, anything that isn’t Go Syscalls and cgo calls are not durably blocking. We can only reason about the state of goroutines executing Go code. ### Not durably blocking: Mutexes Perhaps surprisingly, mutexes are not durably blocking. This is a decision born of practicality: Mutexes are often used to guard global state, so a bubbled goroutine will often need to acquire a mutex held outside its bubble. Mutexes are highly performance-sensitive, so adding additional instrumentation to them risks slowing down non-test programs. We can test programs that use mutexes with synctest, but the fake clock will not advance while a goroutine is blocked on mutex acquisition. This hasn’t posed a problem in any case we’ve encountered, but it is something to be aware of. ### Durably blocking: `time.Sleep` So what is durably blocking? `time.Sleep` is obviously durable, since time can only advance when every goroutine in the bubble is durably blocked. ### Durably blocking: send or receive on channels created in the same bubble Channel operations on channels created within the same bubble are durable. We make a distinction between bubbled channels (created in a bubble) and unbubbled channels (created outside any bubble). This means that a function using a global channel for synchronization, for example to control access to a globally cached resource, can be safely called from within a bubble. Trying to operate on a bubbled channel from outside its bubble is an error. ### Durably blocking: `sync.WaitGroup` belonging to the same bubble We also associate `sync.WaitGroup`s with bubbles. `WaitGroup` doesn’t have a constructor, so we make the association with the bubble implicitly on the first call to `Go` or `Add`. As with channels, waiting on a `WaitGroup` belonging to the same bubble is durably blocking, and waiting on one from outside the bubble is not. Calling `Go` or `Add` on a `WaitGroup` belonging to a different bubble is an error. ### Durably blocking: `sync.Cond.Wait` Waiting on a `sync.Cond` is always durably blocking. Waking up a goroutine waiting on a `Cond` in a different bubble is an error. ### Durably blocking: `select{}` Finally, an empty select is durably blocking. (A select with cases is durably blocking if all the operations in it are so.) That’s the complete list of durably blocking operations. It isn’t very long, but it’s enough to handle almost all real-world programs. The rule is that a goroutine is durably blocked when it is blocked, and we can guarantee that it can only be unblocked by another goroutine in its bubble. In cases where it is possible to attempt to wake a bubbled goroutine from outside its bubble, we panic. For example, it is an error to operate on a bubbled channel from outside its bubble. Changes from 1.24 to 1.25 ------------------------- We released an experimental version of the `synctest` package in Go 1.24. To ensure that early adopters were aware of the experimental status of the package, you needed to set a GOEXPERIMENT flag to make the package visible. The feedback we received from those early adopters was invaluable, both in demonstrating that the package is useful and in uncovering areas where the API needed work. These are some of the changes made between the experimental version and the version released in Go 1.25. ### Replaced Run with Test The original version of the API created a bubble with a `Run` function: // Run executes f in a new bubble. func Run(f func()) It became clear that we needed a way to create a `*testing.T` that is scoped to a bubble. For example, `t.Cleanup` should run cleanup functions in the same bubble they are registered in, not after the bubble exits. We renamed `Run` to `Test` and made it create a `T` scoped to the lifetime of the new bubble. ### Time stops when a bubble’s root goroutine returns We originally continued to advance time within a bubble for so long as the bubble contained any goroutines waiting for future events. This turned out to be very confusing when a long-lived goroutine never returned, such as a goroutine reading forever from a `time.Ticker`. We now stop advancing time when a bubble’s root goroutine returns. If the bubble is blocked waiting for time to advance, this results in a deadlock and a panic which can be analyzed. ### Removed cases where “durable” wasn’t We cleaned up the definition of “durably blocking”. The original implementation had cases where a durably blocked goroutine could be unblocked from outside the bubble. For example, channels recorded whether they were created in a bubble, but not which in which bubble they were created, so one bubble could unblock a channel in a different bubble. The current implementation contains no cases we know of where a durably blocked goroutine can be unblocked from outside its bubble. ### Better stack traces We made improvements to the information printed in stack traces. When a bubble deadlocks, we by default now only print stacks for the goroutines in that bubble. Stack traces also clearly indicate which goroutines in a bubble are durably blocked. ### Randomized events happening at the same time We made improvements to the randomization of events happening at the same time. Originally, timers scheduled to fire at the same instant would always do so in the order they were created. This ordering is now randomized. Future work ----------- We’re pretty happy with the synctest package at the moment. Aside from the inevitable bug fixes, we don’t currently expect any major changes to it in the future. Of course, with wider adoption it is always possible that we’ll discover something that needs doing. One possible area of work is to improve the detection of durably blocked goroutines. It would be nice if we could make mutex operations durably blocking, with a restriction that a mutex acquired in a bubble must be released from within the same bubble. Testing networked code with synctest requires a fake network. The `net.Pipe` function can create a fake `net.Conn`, but there is currently no standard library function that creates a fake `net.Listener` or `net.PacketConn`. In addition, the `net.Conn` returned by `net.Pipe` is synchronous–every write blocks until a read consumes the data–which is not representative of real network behavior. Perhaps we should add a good fake implementations of common network interfaces to the standard library. Conclusion ---------- That’s the `synctest` package. I can’t say that it makes testing concurrent code simple, because concurrency is never simple. What it does is let you write the simplest possible concurrent code, using idiomatic Go, and the standard time package, and then write fast, reliable tests for it. I hope you find it useful. **Next article:** [A new experimental Go API for JSON](https://go.dev/blog/jsonv2-exp) **Previous article:** [Container-aware GOMAXPROCS](https://go.dev/blog/container-aware-gomaxprocs) **[Blog Index](https://go.dev/blog/all) ** go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # The Go image/draw package - The Go Programming Language [The Go Blog](https://go.dev/blog/) ==================================== The Go image/draw package ========================= Nigel Tao 29 September 2011 Introduction ------------ [Package image/draw](https://go.dev/pkg/image/draw/) defines only one operation: drawing a source image onto a destination image, through an optional mask image. This one operation is surprisingly versatile and can perform a number of common image manipulation tasks elegantly and efficiently. Composition is performed pixel by pixel in the style of the Plan 9 graphics library and the X Render extension. The model is based on the classic “Compositing Digital Images” paper by Porter and Duff, with an additional mask parameter: `dst = (src IN mask) OP dst`. For a fully opaque mask, this reduces to the original Porter-Duff formula: `dst = src OP dst`. In Go, a nil mask image is equivalent to an infinitely sized, fully opaque mask image. The Porter-Duff paper presented [12 different composition operators](http://www.w3.org/TR/SVGCompositing/examples/compop-porterduff-examples.png) , but with an explicit mask, only 2 of these are needed in practice: source-over-destination and source. In Go, these operators are represented by the `Over` and `Src` constants. The `Over` operator performs the natural layering of a source image over a destination image: the change to the destination image is smaller where the source (after masking) is more transparent (that is, has lower alpha). The `Src` operator merely copies the source (after masking) with no regard for the destination image’s original content. For fully opaque source and mask images, the two operators produce the same output, but the `Src` operator is usually faster. Geometric Alignment ------------------- Composition requires associating destination pixels with source and mask pixels. Obviously, this requires destination, source and mask images, and a composition operator, but it also requires specifying what rectangle of each image to use. Not every drawing should write to the entire destination: when updating an animating image, it is more efficient to only draw the parts of the image that have changed. Not every drawing should read from the entire source: when using a sprite that combines many small images into one large one, only a part of the image is needed. Not every drawing should read from the entire mask: a mask image that collects a font’s glyphs is similar to a sprite. Thus, drawing also needs to know three rectangles, one for each image. Since each rectangle has the same width and height, it suffices to pass a destination rectangle `r` and two points `sp` and `mp`: the source rectangle is equal to `r` translated so that `r.Min` in the destination image aligns with `sp` in the source image, and similarly for `mp`. The effective rectangle is also clipped to each image’s bounds in their respective co-ordinate space. ![](https://go.dev/blog/image-draw/20.png) The [`DrawMask`](https://go.dev/pkg/image/draw/#DrawMask) function takes seven arguments, but an explicit mask and mask-point are usually unnecessary, so the [`Draw`](https://go.dev/pkg/image/draw/#Draw) function takes five: // Draw calls DrawMask with a nil mask. func Draw(dst Image, r image.Rectangle, src image.Image, sp image.Point, op Op) func DrawMask(dst Image, r image.Rectangle, src image.Image, sp image.Point, mask image.Image, mp image.Point, op Op) The destination image must be mutable, so the image/draw package defines a [`draw.Image`](https://go.dev/pkg/image/draw/#Image) interface which has a `Set` method. type Image interface { image.Image Set(x, y int, c color.Color) } Filling a Rectangle ------------------- To fill a rectangle with a solid color, use an `image.Uniform` source. The `ColorImage` type re-interprets a `Color` as a practically infinite-sized `Image` of that color. For those familiar with the design of Plan 9’s draw library, there is no need for an explicit “repeat bit” in Go’s slice-based image types; the concept is subsumed by `Uniform`. // image.ZP is the zero point -- the origin. draw.Draw(dst, r, &image.Uniform{c}, image.ZP, draw.Src) To initialize a new image to all-blue: m := image.NewRGBA(image.Rect(0, 0, 640, 480)) blue := color.RGBA{0, 0, 255, 255} draw.Draw(m, m.Bounds(), &image.Uniform{blue}, image.ZP, draw.Src) To reset an image to transparent (or black, if the destination image’s color model cannot represent transparency), use `image.Transparent`, which is an `image.Uniform`: draw.Draw(m, m.Bounds(), image.Transparent, image.ZP, draw.Src) ![](https://go.dev/blog/image-draw/2a.png) Copying an Image ---------------- To copy from a rectangle `sr` in the source image to a rectangle starting at a point `dp` in the destination, convert the source rectangle into the destination image’s co-ordinate space: r := image.Rectangle{dp, dp.Add(sr.Size())} draw.Draw(dst, r, src, sr.Min, draw.Src) Alternatively: r := sr.Sub(sr.Min).Add(dp) draw.Draw(dst, r, src, sr.Min, draw.Src) To copy the entire source image, use `sr = src.Bounds()`. ![](https://go.dev/blog/image-draw/2b.png) Scrolling an Image ------------------ Scrolling an image is just copying an image to itself, with different destination and source rectangles. Overlapping destination and source images are perfectly valid, just as Go’s built-in copy function can handle overlapping destination and source slices. To scroll an image m by 20 pixels: b := m.Bounds() p := image.Pt(0, 20) // Note that even though the second argument is b, // the effective rectangle is smaller due to clipping. draw.Draw(m, b, m, b.Min.Add(p), draw.Src) dirtyRect := b.Intersect(image.Rect(b.Min.X, b.Max.Y-20, b.Max.X, b.Max.Y)) ![](https://go.dev/blog/image-draw/2c.png) Converting an Image to RGBA --------------------------- The result of decoding an image format might not be an `image.RGBA`: decoding a GIF results in an `image.Paletted`, decoding a JPEG results in a `ycbcr.YCbCr`, and the result of decoding a PNG depends on the image data. To convert any image to an `image.RGBA`: b := src.Bounds() m := image.NewRGBA(image.Rect(0, 0, b.Dx(), b.Dy())) draw.Draw(m, m.Bounds(), src, b.Min, draw.Src) ![](https://go.dev/blog/image-draw/2d.png) Drawing Through a Mask ---------------------- To draw an image through a circular mask with center `p` and radius `r`: type circle struct { p image.Point r int } func (c *circle) ColorModel() color.Model { return color.AlphaModel } func (c *circle) Bounds() image.Rectangle { return image.Rect(c.p.X-c.r, c.p.Y-c.r, c.p.X+c.r, c.p.Y+c.r) } func (c *circle) At(x, y int) color.Color { xx, yy, rr := float64(x-c.p.X)+0.5, float64(y-c.p.Y)+0.5, float64(c.r) if xx*xx+yy*yy < rr*rr { return color.Alpha{255} } return color.Alpha{0} } draw.DrawMask(dst, dst.Bounds(), src, image.ZP, &circle{p, r}, image.ZP, draw.Over) ![](https://go.dev/blog/image-draw/2e.png) Drawing Font Glyphs ------------------- To draw a font glyph in blue starting from a point `p`, draw with an `image.ColorImage` source and an `image.Alpha mask`. For simplicity, we aren’t performing any sub-pixel positioning or rendering, or correcting for a font’s height above a baseline. src := &image.Uniform{color.RGBA{0, 0, 255, 255}} mask := theGlyphImageForAFont() mr := theBoundsFor(glyphIndex) draw.DrawMask(dst, mr.Sub(mr.Min).Add(p), src, image.ZP, mask, mr.Min, draw.Over) ![](https://go.dev/blog/image-draw/2f.png) Performance ----------- The image/draw package implementation demonstrates how to provide an image manipulation function that is both general purpose, yet efficient for common cases. The `DrawMask` function takes arguments of interface types, but immediately makes type assertions that its arguments are of specific struct types, corresponding to common operations like drawing one `image.RGBA` image onto another, or drawing an `image.Alpha` mask (such as a font glyph) onto an `image.RGBA` image. If a type assertion succeeds, that type information is used to run a specialized implementation of the general algorithm. If the assertions fail, the fallback code path uses the generic `At` and `Set` methods. The fast-paths are purely a performance optimization; the resultant destination image is the same either way. In practice, only a small number of special cases are necessary to support typical applications. **Next article:** [Learn Go from your browser](https://go.dev/blog/tour) **Previous article:** [The Go image package](https://go.dev/blog/image) **[Blog Index](https://go.dev/blog/all) ** go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Using Go Modules - The Go Programming Language [The Go Blog](https://go.dev/blog/) ==================================== Using Go Modules ================ Tyler Bui-Palsulich and Eno Compton 19 March 2019 Introduction ------------ This post is part 1 in a series. * **Part 1 — Using Go Modules** (this post) * Part 2 — [Migrating To Go Modules](https://go.dev/blog/migrating-to-go-modules) * Part 3 — [Publishing Go Modules](https://go.dev/blog/publishing-go-modules) * Part 4 — [Go Modules: v2 and Beyond](https://go.dev/blog/v2-go-modules) * Part 5 — [Keeping Your Modules Compatible](https://go.dev/blog/module-compatibility) **Note:** For documentation on managing dependencies with modules, see [Managing dependencies](https://go.dev/doc/modules/managing-dependencies) . Go 1.11 and 1.12 include preliminary [support for modules](https://go.dev/doc/go1.11#modules) , Go’s [new dependency management system](https://go.dev/blog/versioning-proposal) that makes dependency version information explicit and easier to manage. This blog post is an introduction to the basic operations needed to get started using modules. A module is a collection of [Go packages](https://go.dev/ref/spec#Packages) stored in a file tree with a `go.mod` file at its root. The `go.mod` file defines the module’s _module path_, which is also the import path used for the root directory, and its _dependency requirements_, which are the other modules needed for a successful build. Each dependency requirement is written as a module path and a specific [semantic version](http://semver.org/) . As of Go 1.11, the go command enables the use of modules when the current directory or any parent directory has a `go.mod`, provided the directory is _outside_ `$GOPATH/src`. (Inside `$GOPATH/src`, for compatibility, the go command still runs in the old GOPATH mode, even if a `go.mod` is found. See the [go command documentation](https://go.dev/cmd/go/#hdr-Preliminary_module_support) for details.) Starting in Go 1.13, module mode will be the default for all development. This post walks through a sequence of common operations that arise when developing Go code with modules: * Creating a new module. * Adding a dependency. * Upgrading dependencies. * Adding a dependency on a new major version. * Upgrading a dependency to a new major version. * Removing unused dependencies. Creating a new module --------------------- Let’s create a new module. Create a new, empty directory somewhere outside `$GOPATH/src`, `cd` into that directory, and then create a new source file, `hello.go`: package hello func Hello() string { return "Hello, world." } Let’s write a test, too, in `hello_test.go`: package hello import "testing" func TestHello(t *testing.T) { want := "Hello, world." if got := Hello(); got != want { t.Errorf("Hello() = %q, want %q", got, want) } } At this point, the directory contains a package, but not a module, because there is no `go.mod` file. If we were working in `/home/gopher/hello` and ran `go test` now, we’d see: $ go test PASS ok _/home/gopher/hello 0.020s $ The last line summarizes the overall package test. Because we are working outside `$GOPATH` and also outside any module, the `go` command knows no import path for the current directory and makes up a fake one based on the directory name: `_/home/gopher/hello`. Let’s make the current directory the root of a module by using `go mod init` and then try `go test` again: $ go mod init example.com/hello go: creating new go.mod: module example.com/hello $ go test PASS ok example.com/hello 0.020s $ Congratulations! You’ve written and tested your first module. The `go mod init` command wrote a `go.mod` file: $ cat go.mod module example.com/hello go 1.12 $ The `go.mod` file only appears in the root of the module. Packages in subdirectories have import paths consisting of the module path plus the path to the subdirectory. For example, if we created a subdirectory `world`, we would not need to (nor want to) run `go mod init` there. The package would automatically be recognized as part of the `example.com/hello` module, with import path `example.com/hello/world`. Adding a dependency ------------------- The primary motivation for Go modules was to improve the experience of using (that is, adding a dependency on) code written by other developers. Let’s update our `hello.go` to import `rsc.io/quote` and use it to implement `Hello`: package hello import "rsc.io/quote" func Hello() string { return quote.Hello() } Now let’s run the test again: $ go test go: finding rsc.io/quote v1.5.2 go: downloading rsc.io/quote v1.5.2 go: extracting rsc.io/quote v1.5.2 go: finding rsc.io/sampler v1.3.0 go: finding golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c go: downloading rsc.io/sampler v1.3.0 go: extracting rsc.io/sampler v1.3.0 go: downloading golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c go: extracting golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c PASS ok example.com/hello 0.023s $ The `go` command resolves imports by using the specific dependency module versions listed in `go.mod`. When it encounters an `import` of a package not provided by any module in `go.mod`, the `go` command automatically looks up the module containing that package and adds it to `go.mod`, using the latest version. (“Latest” is defined as the latest tagged stable (non-[prerelease](https://semver.org/#spec-item-9) ) version, or else the latest tagged prerelease version, or else the latest untagged version.) In our example, `go test` resolved the new import `rsc.io/quote` to the module `rsc.io/quote v1.5.2`. It also downloaded two dependencies used by `rsc.io/quote`, namely `rsc.io/sampler` and `golang.org/x/text`. Only direct dependencies are recorded in the `go.mod` file: $ cat go.mod module example.com/hello go 1.12 require rsc.io/quote v1.5.2 $ A second `go test` command will not repeat this work, since the `go.mod` is now up-to-date and the downloaded modules are cached locally (in `$GOPATH/pkg/mod`): $ go test PASS ok example.com/hello 0.020s $ Note that while the `go` command makes adding a new dependency quick and easy, it is not without cost. Your module now literally _depends_ on the new dependency in critical areas such as correctness, security, and proper licensing, just to name a few. For more considerations, see Russ Cox’s blog post, “[Our Software Dependency Problem](https://research.swtch.com/deps) .” As we saw above, adding one direct dependency often brings in other indirect dependencies too. The command `go list -m all` lists the current module and all its dependencies: $ go list -m all example.com/hello golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c rsc.io/quote v1.5.2 rsc.io/sampler v1.3.0 $ In the `go list` output, the current module, also known as the _main module_, is always the first line, followed by dependencies sorted by module path. The `golang.org/x/text` version `v0.0.0-20170915032832-14c0d48ead0c` is an example of a [pseudo-version](https://go.dev/ref/mod#pseudo-versions) , which is the `go` command’s version syntax for a specific untagged commit. In addition to `go.mod`, the `go` command maintains a file named `go.sum` containing the expected [cryptographic hashes](https://go.dev/cmd/go/#hdr-Module_downloading_and_verification) of the content of specific module versions: $ cat go.sum golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c h1:qgOY6WgZO... golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:Nq... rsc.io/quote v1.5.2 h1:w5fcysjrx7yqtD/aO+QwRjYZOKnaM9Uh2b40tElTs3... rsc.io/quote v1.5.2/go.mod h1:LzX7hefJvL54yjefDEDHNONDjII0t9xZLPX... rsc.io/sampler v1.3.0 h1:7uVkIFmeBqHfdjD+gZwtXXI+RODJ2Wc4O7MPEh/Q... rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9... $ The `go` command uses the `go.sum` file to ensure that future downloads of these modules retrieve the same bits as the first download, to ensure the modules your project depends on do not change unexpectedly, whether for malicious, accidental, or other reasons. Both `go.mod` and `go.sum` should be checked into version control. Upgrading dependencies ---------------------- With Go modules, versions are referenced with semantic version tags. A semantic version has three parts: major, minor, and patch. For example, for `v0.1.2`, the major version is 0, the minor version is 1, and the patch version is 2. Let’s walk through a couple minor version upgrades. In the next section, we’ll consider a major version upgrade. From the output of `go list -m all`, we can see we’re using an untagged version of `golang.org/x/text`. Let’s upgrade to the latest tagged version and test that everything still works: $ go get golang.org/x/text go: finding golang.org/x/text v0.3.0 go: downloading golang.org/x/text v0.3.0 go: extracting golang.org/x/text v0.3.0 $ go test PASS ok example.com/hello 0.013s $ Woohoo! Everything passes. Let’s take another look at `go list -m all` and the `go.mod` file: $ go list -m all example.com/hello golang.org/x/text v0.3.0 rsc.io/quote v1.5.2 rsc.io/sampler v1.3.0 $ cat go.mod module example.com/hello go 1.12 require ( golang.org/x/text v0.3.0 // indirect rsc.io/quote v1.5.2 ) $ The `golang.org/x/text` package has been upgraded to the latest tagged version (`v0.3.0`). The `go.mod` file has been updated to specify `v0.3.0` too. The `indirect` comment indicates a dependency is not used directly by this module, only indirectly by other module dependencies. See `go help modules` for details. Now let’s try upgrading the `rsc.io/sampler` minor version. Start the same way, by running `go get` and running tests: $ go get rsc.io/sampler go: finding rsc.io/sampler v1.99.99 go: downloading rsc.io/sampler v1.99.99 go: extracting rsc.io/sampler v1.99.99 $ go test --- FAIL: TestHello (0.00s) hello_test.go:8: Hello() = "99 bottles of beer on the wall, 99 bottles of beer, ...", want "Hello, world." FAIL exit status 1 FAIL example.com/hello 0.014s $ Uh, oh! The test failure shows that the latest version of `rsc.io/sampler` is incompatible with our usage. Let’s list the available tagged versions of that module: $ go list -m -versions rsc.io/sampler rsc.io/sampler v1.0.0 v1.2.0 v1.2.1 v1.3.0 v1.3.1 v1.99.99 $ We had been using v1.3.0; v1.99.99 is clearly no good. Maybe we can try using v1.3.1 instead: $ go get rsc.io/sampler@v1.3.1 go: finding rsc.io/sampler v1.3.1 go: downloading rsc.io/sampler v1.3.1 go: extracting rsc.io/sampler v1.3.1 $ go test PASS ok example.com/hello 0.022s $ Note the explicit `@v1.3.1` in the `go get` argument. In general each argument passed to `go get` can take an explicit version; the default is `@latest`, which resolves to the latest version as defined earlier. Adding a dependency on a new major version ------------------------------------------ Let’s add a new function to our package: `func Proverb` returns a Go concurrency proverb, by calling `quote.Concurrency`, which is provided by the module `rsc.io/quote/v3`. First we update `hello.go` to add the new function: package hello import ( "rsc.io/quote" quoteV3 "rsc.io/quote/v3" ) func Hello() string { return quote.Hello() } func Proverb() string { return quoteV3.Concurrency() } Then we add a test to `hello_test.go`: func TestProverb(t *testing.T) { want := "Concurrency is not parallelism." if got := Proverb(); got != want { t.Errorf("Proverb() = %q, want %q", got, want) } } Then we can test our code: $ go test go: finding rsc.io/quote/v3 v3.1.0 go: downloading rsc.io/quote/v3 v3.1.0 go: extracting rsc.io/quote/v3 v3.1.0 PASS ok example.com/hello 0.024s $ Note that our module now depends on both `rsc.io/quote` and `rsc.io/quote/v3`: $ go list -m rsc.io/q... rsc.io/quote v1.5.2 rsc.io/quote/v3 v3.1.0 $ Each different major version (`v1`, `v2`, and so on) of a Go module uses a different module path: starting at `v2`, the path must end in the major version. In the example, `v3` of `rsc.io/quote` is no longer `rsc.io/quote`: instead, it is identified by the module path `rsc.io/quote/v3`. This convention is called [semantic import versioning](https://research.swtch.com/vgo-import) , and it gives incompatible packages (those with different major versions) different names. In contrast, `v1.6.0` of `rsc.io/quote` should be backwards-compatible with `v1.5.2`, so it reuses the name `rsc.io/quote`. (In the previous section, `rsc.io/sampler` `v1.99.99` _should_ have been backwards-compatible with `rsc.io/sampler` `v1.3.0`, but bugs or incorrect client assumptions about module behavior can both happen.) The `go` command allows a build to include at most one version of any particular module path, meaning at most one of each major version: one `rsc.io/quote`, one `rsc.io/quote/v2`, one `rsc.io/quote/v3`, and so on. This gives module authors a clear rule about possible duplication of a single module path: it is impossible for a program to build with both `rsc.io/quote v1.5.2` and `rsc.io/quote v1.6.0`. At the same time, allowing different major versions of a module (because they have different paths) gives module consumers the ability to upgrade to a new major version incrementally. In this example, we wanted to use `quote.Concurrency` from `rsc/quote/v3 v3.1.0` but are not yet ready to migrate our uses of `rsc.io/quote v1.5.2`. The ability to migrate incrementally is especially important in a large program or codebase. Upgrading a dependency to a new major version --------------------------------------------- Let’s complete our conversion from using `rsc.io/quote` to using only `rsc.io/quote/v3`. Because of the major version change, we should expect that some APIs may have been removed, renamed, or otherwise changed in incompatible ways. Reading the docs, we can see that `Hello` has become `HelloV3`: $ go doc rsc.io/quote/v3 package quote // import "rsc.io/quote/v3" Package quote collects pithy sayings. func Concurrency() string func GlassV3() string func GoV3() string func HelloV3() string func OptV3() string $ We can update our use of `quote.Hello()` in `hello.go` to use `quoteV3.HelloV3()`: package hello import quoteV3 "rsc.io/quote/v3" func Hello() string { return quoteV3.HelloV3() } func Proverb() string { return quoteV3.Concurrency() } And then at this point, there’s no need for the renamed import anymore, so we can undo that: package hello import "rsc.io/quote/v3" func Hello() string { return quote.HelloV3() } func Proverb() string { return quote.Concurrency() } Let’s re-run the tests to make sure everything is working: $ go test PASS ok example.com/hello 0.014s Removing unused dependencies ---------------------------- We’ve removed all our uses of `rsc.io/quote`, but it still shows up in `go list -m all` and in our `go.mod` file: $ go list -m all example.com/hello golang.org/x/text v0.3.0 rsc.io/quote v1.5.2 rsc.io/quote/v3 v3.1.0 rsc.io/sampler v1.3.1 $ cat go.mod module example.com/hello go 1.12 require ( golang.org/x/text v0.3.0 // indirect rsc.io/quote v1.5.2 rsc.io/quote/v3 v3.0.0 rsc.io/sampler v1.3.1 // indirect ) $ Why? Because building a single package, like with `go build` or `go test`, can easily tell when something is missing and needs to be added, but not when something can safely be removed. Removing a dependency can only be done after checking all packages in a module, and all possible build tag combinations for those packages. An ordinary build command does not load this information, and so it cannot safely remove dependencies. The `go mod tidy` command cleans up these unused dependencies: $ go mod tidy $ go list -m all example.com/hello golang.org/x/text v0.3.0 rsc.io/quote/v3 v3.1.0 rsc.io/sampler v1.3.1 $ cat go.mod module example.com/hello go 1.12 require ( golang.org/x/text v0.3.0 // indirect rsc.io/quote/v3 v3.1.0 rsc.io/sampler v1.3.1 // indirect ) $ go test PASS ok example.com/hello 0.020s $ Conclusion ---------- Go modules are the future of dependency management in Go. Module functionality is now available in all supported Go versions (that is, in Go 1.11 and Go 1.12). This post introduced these workflows using Go modules: * `go mod init` creates a new module, initializing the `go.mod` file that describes it. * `go build`, `go test`, and other package-building commands add new dependencies to `go.mod` as needed. * `go list -m all` prints the current module’s dependencies. * `go get` changes the required version of a dependency (or adds a new dependency). * `go mod tidy` removes unused dependencies. We encourage you to start using modules in your local development and to add `go.mod` and `go.sum` files to your projects. To provide feedback and help shape the future of dependency management in Go, please send us [bug reports](https://go.dev/issue/new) or [experience reports](https://go.dev/wiki/ExperienceReports) . Thanks for all your feedback and help improving modules. **Next article:** [Debugging what you deploy in Go 1.12](https://go.dev/blog/debug-opt) **Previous article:** [The New Go Developer Network](https://go.dev/blog/go-developer-network) **[Blog Index](https://go.dev/blog/all) ** go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Tutorial: Developing a RESTful API with Go and Gin - The Go Programming Language 1. [Documentation](https://go.dev/doc/) 2. [Tutorials](https://go.dev/doc/tutorial/) 3. [Tutorial: Developing a RESTful API with Go and Gin](https://go.dev/doc/tutorial/web-service-gin) Tutorial: Developing a RESTful API with Go and Gin ================================================== This tutorial introduces the basics of writing a RESTful web service API with Go and the [Gin Web Framework](https://gin-gonic.com/en/docs/) (Gin). You’ll get the most out of this tutorial if you have a basic familiarity with Go and its tooling. If this is your first exposure to Go, please see [Tutorial: Get started with Go](https://go.dev/doc/tutorial/getting-started) for a quick introduction. Gin simplifies many coding tasks associated with building web applications, including web services. In this tutorial, you’ll use Gin to route requests, retrieve request details, and marshal JSON for responses. In this tutorial, you will build a RESTful API server with two endpoints. Your example project will be a repository of data about vintage jazz records. The tutorial includes the following sections: 1. Design API endpoints. 2. Create a folder for your code. 3. Create the data. 4. Write a handler to return all items. 5. Write a handler to add a new item. 6. Write a handler to return a specific item. **Note:** For other tutorials, see [Tutorials](https://go.dev/doc/tutorial/index.html) . To try this as an interactive tutorial you complete in Google Cloud Shell, click the button below. [![Open in Cloud Shell](https://gstatic.com/cloudssh/images/open-btn.png)](https://ide.cloud.google.com/?cloudshell_workspace=~&walkthrough_tutorial_url=https://raw.githubusercontent.com/golang/tour/master/tutorial/web-service-gin.md) Prerequisites ------------- * **An installation of Go 1.16 or later.** For installation instructions, see [Installing Go](https://go.dev/doc/install) . * **A tool to edit your code.** Any text editor you have will work fine. * **A command terminal.** Go works well using any terminal on Linux and Mac, and on PowerShell or cmd in Windows. * **The curl tool.** On Linux and Mac, this should already be installed. On Windows, it’s included on Windows 10 Insider build 17063 and later. For earlier Windows versions, you might need to install it. For more, see [Tar and Curl Come to Windows](https://docs.microsoft.com/en-us/virtualization/community/team-blog/2017/20171219-tar-and-curl-come-to-windows) . Design API endpoints -------------------- You’ll build an API that provides access to a store selling vintage recordings on vinyl. So you’ll need to provide endpoints through which a client can get and add albums for users. When developing an API, you typically begin by designing the endpoints. Your API’s users will have more success if the endpoints are easy to understand. Here are the endpoints you’ll create in this tutorial. /albums * `GET` – Get a list of all albums, returned as JSON. * `POST` – Add a new album from request data sent as JSON. /albums/:id * `GET` – Get an album by its ID, returning the album data as JSON. Next, you’ll create a folder for your code. Create a folder for your code ----------------------------- To begin, create a project for the code you’ll write. 1. Open a command prompt and change to your home directory. On Linux or Mac: $ cd On Windows: C:\> cd %HOMEPATH% 2. Using the command prompt, create a directory for your code called web-service-gin. $ mkdir web-service-gin $ cd web-service-gin 3. Create a module in which you can manage dependencies. Run the `go mod init` command, giving it the path of the module your code will be in. $ go mod init example/web-service-gin go: creating new go.mod: module example/web-service-gin This command creates a go.mod file in which dependencies you add will be listed for tracking. For more about naming a module with a module path, see [Managing dependencies](https://go.dev/doc/modules/managing-dependencies#naming_module) . Next, you’ll design data structures for handling data. Create the data --------------- To keep things simple for the tutorial, you’ll store data in memory. A more typical API would interact with a database. Note that storing data in memory means that the set of albums will be lost each time you stop the server, then recreated when you start it. #### Write the code 1. Using your text editor, create a file called main.go in the web-service directory. You’ll write your Go code in this file. 2. Into main.go, at the top of the file, paste the following package declaration. package main A standalone program (as opposed to a library) is always in package `main`. 3. Beneath the package declaration, paste the following declaration of an `album` struct. You’ll use this to store album data in memory. Struct tags such as `json:"artist"` specify what a field’s name should be when the struct’s contents are serialized into JSON. Without them, the JSON would use the struct’s capitalized field names – a style not as common in JSON. // album represents data about a record album. type album struct { ID string `json:"id"` Title string `json:"title"` Artist string `json:"artist"` Price float64 `json:"price"` } 4. Beneath the struct declaration you just added, paste the following slice of `album` structs containing data you’ll use to start. // albums slice to seed record album data. var albums = []album{ {ID: "1", Title: "Blue Train", Artist: "John Coltrane", Price: 56.99}, {ID: "2", Title: "Jeru", Artist: "Gerry Mulligan", Price: 17.99}, {ID: "3", Title: "Sarah Vaughan and Clifford Brown", Artist: "Sarah Vaughan", Price: 39.99}, } Next, you’ll write code to implement your first endpoint. Write a handler to return all items ----------------------------------- When the client makes a request at `GET /albums`, you want to return all the albums as JSON. To do this, you’ll write the following: * Logic to prepare a response * Code to map the request path to your logic Note that this is the reverse of how they’ll be executed at runtime, but you’re adding dependencies first, then the code that depends on them. #### Write the code 1. Beneath the struct code you added in the preceding section, paste the following code to get the album list. This `getAlbums` function creates JSON from the slice of `album` structs, writing the JSON into the response. // getAlbums responds with the list of all albums as JSON. func getAlbums(c *gin.Context) { c.IndentedJSON(http.StatusOK, albums) } In this code, you: * Write a `getAlbums` function that takes a [`gin.Context`](https://pkg.go.dev/github.com/gin-gonic/gin#Context) parameter. Note that you could have given this function any name – neither Gin nor Go require a particular function name format. `gin.Context` is the most important part of Gin. It carries request details, validates and serializes JSON, and more. (Despite the similar name, this is different from Go’s built-in [`context`](https://go.dev/pkg/context/) package.) * Call [`Context.IndentedJSON`](https://pkg.go.dev/github.com/gin-gonic/gin#Context.IndentedJSON) to serialize the struct into JSON and add it to the response. The function’s first argument is the HTTP status code you want to send to the client. Here, you’re passing the [`StatusOK`](https://pkg.go.dev/net/http#StatusOK) constant from the `net/http` package to indicate `200 OK`. Note that you can replace `Context.IndentedJSON` with a call to [`Context.JSON`](https://pkg.go.dev/github.com/gin-gonic/gin#Context.JSON) to send more compact JSON. In practice, the indented form is much easier to work with when debugging and the size difference is usually small. 2. Near the top of main.go, just beneath the `albums` slice declaration, paste the code below to assign the handler function to an endpoint path. This sets up an association in which `getAlbums` handles requests to the `/albums` endpoint path. func main() { router := gin.Default() router.GET("/albums", getAlbums) router.Run("localhost:8080") } In this code, you: * Initialize a Gin router using [`Default`](https://pkg.go.dev/github.com/gin-gonic/gin#Default) . * Use the [`GET`](https://pkg.go.dev/github.com/gin-gonic/gin#RouterGroup.GET) function to associate the `GET` HTTP method and `/albums` path with a handler function. Note that you’re passing the _name_ of the `getAlbums` function. This is different from passing the _result_ of the function, which you would do by passing `getAlbums()` (note the parenthesis). * Use the [`Run`](https://pkg.go.dev/github.com/gin-gonic/gin#Engine.Run) function to attach the router to an `http.Server` and start the server. 3. Near the top of main.go, just beneath the package declaration, import the packages you’ll need to support the code you’ve just written. The first lines of code should look like this: package main import ( "net/http" "github.com/gin-gonic/gin" ) 4. Save main.go. #### Run the code 1. Begin tracking the Gin module as a dependency. At the command line, use [`go get`](https://go.dev/cmd/go/#hdr-Add_dependencies_to_current_module_and_install_them) to add the github.com/gin-gonic/gin module as a dependency for your module. Use a dot argument to mean “get dependencies for code in the current directory.” $ go get . go get: added github.com/gin-gonic/gin v1.7.2 Go resolved and downloaded this dependency to satisfy the `import` declaration you added in the previous step. 2. From the command line in the directory containing main.go, run the code. Use a dot argument to mean “run code in the current directory.” $ go run . Once the code is running, you have a running HTTP server to which you can send requests. 3. From a new command line window, use `curl` to make a request to your running web service. $ curl http://localhost:8080/albums The command should display the data you seeded the service with. [\ {\ "id": "1",\ "title": "Blue Train",\ "artist": "John Coltrane",\ "price": 56.99\ },\ {\ "id": "2",\ "title": "Jeru",\ "artist": "Gerry Mulligan",\ "price": 17.99\ },\ {\ "id": "3",\ "title": "Sarah Vaughan and Clifford Brown",\ "artist": "Sarah Vaughan",\ "price": 39.99\ }\ ] You’ve started an API! In the next section, you’ll create another endpoint with code to handle a `POST` request to add an item. Write a handler to add a new item --------------------------------- When the client makes a `POST` request at `/albums`, you want to add the album described in the request body to the existing albums’ data. To do this, you’ll write the following: * Logic to add the new album to the existing list. * A bit of code to route the `POST` request to your logic. #### Write the code 1. Add code to add albums data to the list of albums. Somewhere after the `import` statements, paste the following code. (The end of the file is a good place for this code, but Go doesn’t enforce the order in which you declare functions.) // postAlbums adds an album from JSON received in the request body. func postAlbums(c *gin.Context) { var newAlbum album // Call BindJSON to bind the received JSON to // newAlbum. if err := c.BindJSON(&newAlbum); err != nil { return } // Add the new album to the slice. albums = append(albums, newAlbum) c.IndentedJSON(http.StatusCreated, newAlbum) } In this code, you: * Use [`Context.BindJSON`](https://pkg.go.dev/github.com/gin-gonic/gin#Context.BindJSON) to bind the request body to `newAlbum`. * Append the `album` struct initialized from the JSON to the `albums` slice. * Add a `201` status code to the response, along with JSON representing the album you added. 2. Change your `main` function so that it includes the `router.POST` function, as in the following. func main() { router := gin.Default() router.GET("/albums", getAlbums) router.POST("/albums", postAlbums) router.Run("localhost:8080") } In this code, you: * Associate the `POST` method at the `/albums` path with the `postAlbums` function. With Gin, you can associate a handler with an HTTP method-and-path combination. In this way, you can separately route requests sent to a single path based on the method the client is using. #### Run the code 1. If the server is still running from the last section, stop it. 2. From the command line in the directory containing main.go, run the code. $ go run . 3. From a different command line window, use `curl` to make a request to your running web service. $ curl http://localhost:8080/albums \ --include \ --header "Content-Type: application/json" \ --request "POST" \ --data '{"id": "4","title": "The Modern Sound of Betty Carter","artist": "Betty Carter","price": 49.99}' The command should display headers and JSON for the added album. HTTP/1.1 201 Created Content-Type: application/json; charset=utf-8 Date: Wed, 02 Jun 2021 00:34:12 GMT Content-Length: 116 { "id": "4", "title": "The Modern Sound of Betty Carter", "artist": "Betty Carter", "price": 49.99 } 4. As in the previous section, use `curl` to retrieve the full list of albums, which you can use to confirm that the new album was added. $ curl http://localhost:8080/albums \ --header "Content-Type: application/json" \ --request "GET" The command should display the album list. [\ {\ "id": "1",\ "title": "Blue Train",\ "artist": "John Coltrane",\ "price": 56.99\ },\ {\ "id": "2",\ "title": "Jeru",\ "artist": "Gerry Mulligan",\ "price": 17.99\ },\ {\ "id": "3",\ "title": "Sarah Vaughan and Clifford Brown",\ "artist": "Sarah Vaughan",\ "price": 39.99\ },\ {\ "id": "4",\ "title": "The Modern Sound of Betty Carter",\ "artist": "Betty Carter",\ "price": 49.99\ }\ ] In the next section, you’ll add code to handle a `GET` for a specific item. Write a handler to return a specific item ----------------------------------------- When the client makes a request to `GET /albums/[id]`, you want to return the album whose ID matches the `id` path parameter. To do this, you will: * Add logic to retrieve the requested album. * Map the path to the logic. #### Write the code 1. Beneath the `postAlbums` function you added in the preceding section, paste the following code to retrieve a specific album. This `getAlbumByID` function will extract the ID in the request path, then locate an album that matches. // getAlbumByID locates the album whose ID value matches the id // parameter sent by the client, then returns that album as a response. func getAlbumByID(c *gin.Context) { id := c.Param("id") // Loop over the list of albums, looking for // an album whose ID value matches the parameter. for _, a := range albums { if a.ID == id { c.IndentedJSON(http.StatusOK, a) return } } c.IndentedJSON(http.StatusNotFound, gin.H{"message": "album not found"}) } In this code, you: * Use [`Context.Param`](https://pkg.go.dev/github.com/gin-gonic/gin#Context.Param) to retrieve the `id` path parameter from the URL. When you map this handler to a path, you’ll include a placeholder for the parameter in the path. * Loop over the `album` structs in the slice, looking for one whose `ID` field value matches the `id` parameter value. If it’s found, you serialize that `album` struct to JSON and return it as a response with a `200 OK` HTTP code. As mentioned above, a real-world service would likely use a database query to perform this lookup. * Return an HTTP `404` error with [`http.StatusNotFound`](https://pkg.go.dev/net/http#StatusNotFound) if the album isn’t found. 2. Finally, change your `main` so that it includes a new call to `router.GET`, where the path is now `/albums/:id`, as shown in the following example. func main() { router := gin.Default() router.GET("/albums", getAlbums) router.GET("/albums/:id", getAlbumByID) router.POST("/albums", postAlbums) router.Run("localhost:8080") } In this code, you: * Associate the `/albums/:id` path with the `getAlbumByID` function. In Gin, the colon preceding an item in the path signifies that the item is a path parameter. #### Run the code 1. If the server is still running from the last section, stop it. 2. From the command line in the directory containing main.go, run the code to start the server. $ go run . 3. From a different command line window, use `curl` to make a request to your running web service. $ curl http://localhost:8080/albums/2 The command should display JSON for the album whose ID you used. If the album wasn’t found, you’ll get JSON with an error message. { "id": "2", "title": "Jeru", "artist": "Gerry Mulligan", "price": 17.99 } Conclusion ---------- Congratulations! You’ve just used Go and Gin to write a simple RESTful web service. Suggested next topics: * If you’re new to Go, you’ll find useful best practices described in [Effective Go](https://go.dev/doc/effective_go) and [How to write Go code](https://go.dev/doc/code) . * The [Go Tour](https://go.dev/tour/) is a great step-by-step introduction to Go fundamentals. * For more about Gin, see the [Gin Web Framework package documentation](https://pkg.go.dev/github.com/gin-gonic/gin) or the [Gin Web Framework docs](https://gin-gonic.com/en/docs/) . Completed code -------------- This section contains the code for the application you build with this tutorial. package main import ( "net/http" "github.com/gin-gonic/gin" ) // album represents data about a record album. type album struct { ID string `json:"id"` Title string `json:"title"` Artist string `json:"artist"` Price float64 `json:"price"` } // albums slice to seed record album data. var albums = []album{ {ID: "1", Title: "Blue Train", Artist: "John Coltrane", Price: 56.99}, {ID: "2", Title: "Jeru", Artist: "Gerry Mulligan", Price: 17.99}, {ID: "3", Title: "Sarah Vaughan and Clifford Brown", Artist: "Sarah Vaughan", Price: 39.99}, } func main() { router := gin.Default() router.GET("/albums", getAlbums) router.GET("/albums/:id", getAlbumByID) router.POST("/albums", postAlbums) router.Run("localhost:8080") } // getAlbums responds with the list of all albums as JSON. func getAlbums(c *gin.Context) { c.IndentedJSON(http.StatusOK, albums) } // postAlbums adds an album from JSON received in the request body. func postAlbums(c *gin.Context) { var newAlbum album // Call BindJSON to bind the received JSON to // newAlbum. if err := c.BindJSON(&newAlbum); err != nil { return } // Add the new album to the slice. albums = append(albums, newAlbum) c.IndentedJSON(http.StatusCreated, newAlbum) } // getAlbumByID locates the album whose ID value matches the id // parameter sent by the client, then returns that album as a response. func getAlbumByID(c *gin.Context) { id := c.Param("id") // Loop through the list of albums, looking for // an album whose ID value matches the parameter. for _, a := range albums { if a.ID == id { c.IndentedJSON(http.StatusOK, a) return } } c.IndentedJSON(http.StatusNotFound, gin.H{"message": "album not found"}) } go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Go Modules: v2 and Beyond - The Go Programming Language [The Go Blog](https://go.dev/blog/) ==================================== Go Modules: v2 and Beyond ========================= Jean Barkhuysen and Tyler Bui-Palsulich 7 November 2019 Introduction ------------ This post is part 4 in a series. * Part 1 — [Using Go Modules](https://go.dev/blog/using-go-modules) * Part 2 — [Migrating To Go Modules](https://go.dev/blog/migrating-to-go-modules) * Part 3 — [Publishing Go Modules](https://go.dev/blog/publishing-go-modules) * **Part 4 — Go Modules: v2 and Beyond** (this post) * Part 5 — [Keeping Your Modules Compatible](https://go.dev/blog/module-compatibility) **Note:** For documentation on developing modules, see [Developing and publishing modules](https://go.dev/doc/modules/developing) . As a successful project matures and new requirements are added, past features and design decisions might stop making sense. Developers may want to integrate lessons they’ve learned by removing deprecated functions, renaming types, or splitting complicated packages into manageable pieces. These kinds of changes require effort by downstream users to migrate their code to the new API, so they should not be made without careful consideration that the benefits outweigh the costs. For projects that are still experimental — at major version `v0` — occasional breaking changes are expected by users. For projects which are declared stable — at major version `v1` or higher — breaking changes must be done in a new major version. This post explores major version semantics, how to create and publish a new major version, and how to maintain multiple major versions of a module. Major versions and module paths ------------------------------- Modules formalized an important principle in Go, the [**import compatibility rule**](https://research.swtch.com/vgo-import) : If an old package and a new package have the same import path, the new package must be backwards compatible with the old package. By definition, a new major version of a package is not backwards compatible with the previous version. This means a new major version of a module must have a different module path than the previous version. Starting with `v2`, the major version must appear at the end of the module path (declared in the `module` statement in the `go.mod` file). For example, when the authors of the module `github.com/googleapis/gax-go` developed `v2`, they used the new module path `github.com/googleapis/gax-go/v2`. Users who wanted to use `v2` had to change their package imports and module requirements to `github.com/googleapis/gax-go/v2`. The need for major version suffixes is one of the ways Go modules differs from most other dependency management systems. Suffixes are needed to solve the [diamond dependency problem](https://research.swtch.com/vgo-import#dependency_story) . Before Go modules, [gopkg.in](http://gopkg.in/) allowed package maintainers to follow what we now refer to as the import compatibility rule. With gopkg.in, if you depend on a package that imports `gopkg.in/yaml.v1` and another package that imports `gopkg.in/yaml.v2`, there is no conflict because the two `yaml` packages have different import paths — they use a version suffix, as with Go modules. Since gopkg.in shares the same version suffix methodology as Go modules, the Go command accepts the `.v2` in `gopkg.in/yaml.v2` as a valid major version suffix. This is a special case for compatibility with gopkg.in: modules hosted at other domains need a slash suffix like `/v2`. Major version strategies ------------------------ The recommended strategy is to develop `v2+` modules in a directory named after the major version suffix. github.com/googleapis/gax-go @ master branch /go.mod → module github.com/googleapis/gax-go /v2/go.mod → module github.com/googleapis/gax-go/v2 This approach is compatible with tools that aren’t aware of modules: file paths within the repository match the paths expected by `go get` in `GOPATH` mode. This strategy also allows all major versions to be developed together in different directories. Other strategies may keep major versions on separate branches. However, if `v2+` source code is on the repository’s default branch (usually `master`), tools that are not version-aware — including the `go` command in `GOPATH` mode — may not distinguish between major versions. The examples in this post will follow the major version subdirectory strategy, since it provides the most compatibility. We recommend that module authors follow this strategy as long as they have users developing in `GOPATH` mode. Publishing v2 and beyond ------------------------ This post uses `github.com/googleapis/gax-go` as an example: $ pwd /tmp/gax-go $ ls CODE_OF_CONDUCT.md call_option.go internal CONTRIBUTING.md gax.go invoke.go LICENSE go.mod tools.go README.md go.sum RELEASING.md header.go $ cat go.mod module github.com/googleapis/gax-go go 1.9 require ( github.com/golang/protobuf v1.3.1 golang.org/x/exp v0.0.0-20190221220918-438050ddec5e golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3 golang.org/x/tools v0.0.0-20190114222345-bf090417da8b google.golang.org/grpc v1.19.0 honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099 ) $ To start development on `v2` of `github.com/googleapis/gax-go`, we’ll create a new `v2/` directory and copy our package into it. $ mkdir v2 $ cp -v *.go v2 'call_option.go' -> 'v2/call_option.go' 'gax.go' -> 'v2/gax.go' 'header.go' -> 'v2/header.go' 'invoke.go' -> 'v2/invoke.go' $ Now, let’s create a v2 `go.mod` file by copying the current `go.mod` file and adding a `/v2` suffix to the module path: $ cp go.mod v2/go.mod $ go mod edit -module github.com/googleapis/gax-go/v2 v2/go.mod $ Note that the `v2` version is treated as a separate module from the `v0 / v1` versions: both may coexist in the same build. So, if your `v2+` module has multiple packages, you should update them to use the new `/v2` import path: otherwise, your `v2+` module will depend on your `v0 / v1` module. For example, to update all `github.com/my/project` references to `github.com/my/project/v2`, you can use `find` and `sed`: $ find . -type f \ -name '*.go' \ -exec sed -i -e 's,github.com/my/project,github.com/my/project/v2,g' {} \; $ Now we have a `v2` module, but we want to experiment and make changes before publishing a release. Until we release `v2.0.0` (or any version without a pre-release suffix), we can develop and make breaking changes as we decide on the new API. If we want users to be able to experiment with the new API before we officially make it stable, we can publish a `v2` pre-release version: $ git tag v2.0.0-alpha.1 $ git push origin v2.0.0-alpha.1 $ Once we are happy with our `v2` API and are sure we don’t need any other breaking changes, we can tag `v2.0.0`: $ git tag v2.0.0 $ git push origin v2.0.0 $ At that point, there are now two major versions to maintain. Backwards compatible changes and bug fixes will lead to new minor and patch releases (for example, `v1.1.0`, `v2.0.1`, etc.). Conclusion ---------- Major version changes result in development and maintenance overhead and require investment from downstream users to migrate. The larger the project, the larger these overheads tend to be. A major version change should only come after identifying a compelling reason. Once a compelling reason has been identified for a breaking change, we recommend developing multiple major versions in the master branch because it is compatible with a wider variety of existing tools. Breaking changes to a `v1+` module should always happen in a new, `vN+1` module. When a new module is released, it means additional work for the maintainers and for the users who need to migrate to the new package. Maintainers should therefore validate their APIs before making a stable release, and consider carefully whether breaking changes are really necessary beyond `v1`. **Next article:** [Go Turns 10](https://go.dev/blog/10years) **Previous article:** [Working with Errors in Go 1.13](https://go.dev/blog/go1.13-errors) **[Blog Index](https://go.dev/blog/all) ** go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Publishing Go Modules - The Go Programming Language [The Go Blog](https://go.dev/blog/) ==================================== Publishing Go Modules ===================== Tyler Bui-Palsulich 26 September 2019 Introduction ------------ This post is part 3 in a series. * Part 1 — [Using Go Modules](https://go.dev/blog/using-go-modules) * Part 2 — [Migrating To Go Modules](https://go.dev/blog/migrating-to-go-modules) * **Part 3 — Publishing Go Modules** (this post) * Part 4 — [Go Modules: v2 and Beyond](https://go.dev/blog/v2-go-modules) * Part 5 — [Keeping Your Modules Compatible](https://go.dev/blog/module-compatibility) **Note:** For documentation on developing modules, see [Developing and publishing modules](https://go.dev/doc/modules/developing) . This post discusses how to write and publish modules so other modules can depend on them. Please note: this post covers development up to and including `v1`. If you are interested in `v2`, please see [Go Modules: v2 and Beyond](https://go.dev/blog/v2-go-modules) . This post uses [Git](https://git-scm.com/) in examples. [Mercurial](https://www.mercurial-scm.org/) , [Bazaar](http://wiki.bazaar.canonical.com/) , and others are supported as well. Project setup ------------- For this post, you’ll need an existing project to use as an example. So, start with the files from the end of the [Using Go Modules](https://go.dev/blog/using-go-modules) article: $ cat go.mod module example.com/hello go 1.12 require rsc.io/quote/v3 v3.1.0 $ cat go.sum golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c h1:qgOY6WgZOaTkIIMiVjBQcw93ERBE4m30iBm00nkL0i8= golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= rsc.io/quote/v3 v3.1.0 h1:9JKUTTIUgS6kzR9mK1YuGKv6Nl+DijDNIc0ghT58FaY= rsc.io/quote/v3 v3.1.0/go.mod h1:yEA65RcK8LyAZtP9Kv3t0HmxON59tX3rD+tICJqUlj0= rsc.io/sampler v1.3.0 h1:7uVkIFmeBqHfdjD+gZwtXXI+RODJ2Wc4O7MPEh/QiW4= rsc.io/sampler v1.3.0/go.mod h1:T1hPZKmBbMNahiBKFy5HrXp6adAjACjK9JXDnKaTXpA= $ cat hello.go package hello import "rsc.io/quote/v3" func Hello() string { return quote.HelloV3() } func Proverb() string { return quote.Concurrency() } $ cat hello_test.go package hello import ( "testing" ) func TestHello(t *testing.T) { want := "Hello, world." if got := Hello(); got != want { t.Errorf("Hello() = %q, want %q", got, want) } } func TestProverb(t *testing.T) { want := "Concurrency is not parallelism." if got := Proverb(); got != want { t.Errorf("Proverb() = %q, want %q", got, want) } } $ Next, create a new `git` repository and add an initial commit. If you’re publishing your own project, be sure to include a `LICENSE` file. Change to the directory containing the `go.mod` then create the repo: $ git init $ git add LICENSE go.mod go.sum hello.go hello_test.go $ git commit -m "hello: initial commit" $ Semantic versions and modules ----------------------------- Every required module in a `go.mod` has a [semantic version](https://semver.org/) , the minimum version of that dependency to use to build the module. A semantic version has the form `vMAJOR.MINOR.PATCH`. * Increment the `MAJOR` version when you make a [backwards incompatible](https://go.dev/doc/go1compat) change to the public API of your module. This should only be done when absolutely necessary. * Increment the `MINOR` version when you make a backwards compatible change to the API, like changing dependencies or adding a new function, method, struct field, or type. * Increment the `PATCH` version after making minor changes that don’t affect your module’s public API or dependencies, like fixing a bug. You can specify pre-release versions by appending a hyphen and dot separated identifiers (for example, `v1.0.1-alpha` or `v2.2.2-beta.2`). Normal releases are preferred by the `go` command over pre-release versions, so users must ask for pre-release versions explicitly (for example, `go get example.com/hello@v1.0.1-alpha`) if your module has any normal releases. `v0` major versions and pre-release versions do not guarantee backwards compatibility. They let you refine your API before making stability commitments to your users. However, `v1` major versions and beyond require backwards compatibility within that major version. The version referenced in a `go.mod` may be an explicit release tagged in the repository (for example, `v1.5.2`), or it may be a [pseudo-version](https://go.dev/ref/mod#pseudo-versions) based on a specific commit (for example, `v0.0.0-20170915032832-14c0d48ead0c`). Pseudo-versions are a special type of pre-release version. Pseudo-versions are useful when a user needs to depend on a project that has not published any semantic version tags, or develop against a commit that hasn’t been tagged yet, but users should not assume that pseudo-versions provide a stable or well-tested API. Tagging your modules with explicit versions signals to your users that specific versions are fully tested and ready to use. Once you start tagging your repo with versions, it’s important to keep tagging new releases as you develop your module. When users request a new version of your module (with `go get -u` or `go get example.com/hello`), the `go` command will choose the greatest semantic release version available, even if that version is several years old and many changes behind the primary branch. Continuing to tag new releases will make your ongoing improvements available to your users. Do not delete version tags from your repo. If you find a bug or a security issue with a version, release a new version. If people depend on a version that you have deleted, their builds may fail. Similarly, once you release a version, do not change or overwrite it. The [module mirror and checksum database](https://go.dev/blog/module-mirror-launch) store modules, their versions, and signed cryptographic hashes to ensure that the build of a given version remains reproducible over time. v0: the initial, unstable version --------------------------------- Let’s tag the module with a `v0` semantic version. A `v0` version does not make any stability guarantees, so nearly all projects should start with `v0` as they refine their public API. Tagging a new version has a few steps: 1. Run `go mod tidy`, which removes any dependencies the module might have accumulated that are no longer necessary. 2. Run `go test ./...` a final time to make sure everything is working. 3. Tag the project with a new version using [`git tag`](https://git-scm.com/docs/git-tag) . 4. Push the new tag to the origin repository. $ go mod tidy $ go test ./... ok example.com/hello 0.015s $ git add go.mod go.sum hello.go hello_test.go $ git commit -m "hello: changes for v0.1.0" $ git tag v0.1.0 $ git push origin v0.1.0 $ Now other projects can depend on `v0.1.0` of `example.com/hello`. For your own module, you can run `go list -m example.com/hello@v0.1.0` to confirm the latest version is available (this example module does not exist, so no versions are available). If you don’t see the latest version immediately and you’re using the Go module proxy (the default since Go 1.13), try again in a few minutes to give the proxy time to load the new version. If you add to the public API, make a breaking change to a `v0` module, or upgrade the minor or version of one of your dependencies, increment the `MINOR` version for your next release. For example, the next release after `v0.1.0` would be `v0.2.0`. If you fix a bug in an existing version, increment the `PATCH` version. For example, the next release after `v0.1.0` would be `v0.1.1`. v1: the first stable version ---------------------------- Once you are absolutely sure your module’s API is stable, you can release `v1.0.0`. A `v1` major version communicates to users that no incompatible changes will be made to the module’s API. They can upgrade to new `v1` minor and patch releases, and their code should not break. Function and method signatures will not change, exported types will not be removed, and so on. If there are changes to the API, they will be backwards compatible (for example, adding a new field to a struct) and will be included in a new minor release. If there are bug fixes (for example, a security fix), they will be included in a patch release (or as part of a minor release). Sometimes, maintaining backwards compatibility can lead to awkward APIs. That’s OK. An imperfect API is better than breaking users’ existing code. The standard library’s `strings` package is a prime example of maintaining backwards compatibility at the cost of API consistency. * [`Split`](https://godoc.org/strings#Split) slices a string into all substrings separated by a separator and returns a slice of the substrings between those separators. * [`SplitN`](https://godoc.org/strings#SplitN) can be used to control the number of substrings to return. However, [`Replace`](https://godoc.org/strings#Replace) took a count of how many instances of the string to replace from the beginning (unlike `Split`). Given `Split` and `SplitN`, you would expect functions like `Replace` and `ReplaceN`. But, we couldn’t change the existing `Replace` without breaking callers, which we promised not to do. So, in Go 1.12, we added a new function, [`ReplaceAll`](https://godoc.org/strings#ReplaceAll) . The resulting API is a little odd, since `Split` and `Replace` behave differently, but that inconsistency is better than a breaking change. Let’s say you’re happy with the API of `example.com/hello` and you want to release `v1` as the first stable version. Tagging `v1` uses the same process as tagging a `v0` version: run `go mod tidy` and `go test ./...`, tag the version, and push the tag to the origin repository: $ go mod tidy $ go test ./... ok example.com/hello 0.015s $ git add go.mod go.sum hello.go hello_test.go $ git commit -m "hello: changes for v1.0.0" $ git tag v1.0.0 $ git push origin v1.0.0 $ At this point, the `v1` API of `example.com/hello` is solidified. This communicates to everyone that our API is stable and they should feel comfortable using it. Conclusion ---------- This post walked through the process of tagging a module with semantic versions and when to release `v1`. A future post will cover how to maintain and publish modules at `v2` and beyond. To provide feedback and help shape the future of dependency management in Go, please send us [bug reports](https://go.dev/issue/new) or [experience reports](https://go.dev/wiki/ExperienceReports) . Thanks for all your feedback and help improving Go modules. **Next article:** [Working with Errors in Go 1.13](https://go.dev/blog/go1.13-errors) **Previous article:** [Go 1.13 is released](https://go.dev/blog/go1.13) **[Blog Index](https://go.dev/blog/all) ** go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Profiling Go Programs - The Go Programming Language [The Go Blog](https://go.dev/blog/) ==================================== Profiling Go Programs ===================== Russ Cox, July 2011; updated by Shenghou Ma, May 2013 24 June 2011 At Scala Days 2011, Robert Hundt presented a paper titled [Loop Recognition in C++/Java/Go/Scala.](http://research.google.com/pubs/pub37122.html) The paper implemented a specific loop finding algorithm, such as you might use in a flow analysis pass of a compiler, in C++, Go, Java, Scala, and then used those programs to draw conclusions about typical performance concerns in these languages. The Go program presented in that paper runs quite slowly, making it an excellent opportunity to demonstrate how to use Go’s profiling tools to take a slow program and make it faster. _By using Go’s profiling tools to identify and correct specific bottlenecks, we can make the Go loop finding program run an order of magnitude faster and use 6x less memory._ (Update: Due to recent optimizations of `libstdc++` in `gcc`, the memory reduction is now 3.7x.) Hundt’s paper does not specify which versions of the C++, Go, Java, and Scala tools he used. In this blog post, we will be using the most recent weekly snapshot of the `6g` Go compiler and the version of `g++` that ships with the Ubuntu Natty distribution. (We will not be using Java or Scala, because we are not skilled at writing efficient programs in either of those languages, so the comparison would be unfair. Since C++ was the fastest language in the paper, the comparisons here with C++ should suffice.) (Update: In this updated post, we will be using the most recent development snapshot of the Go compiler on amd64 and the most recent version of `g++` – 4.8.0, which was released in March 2013.) $ go version go version devel +08d20469cc20 Tue Mar 26 08:27:18 2013 +0100 linux/amd64 $ g++ --version g++ (GCC) 4.8.0 Copyright (C) 2013 Free Software Foundation, Inc. ... $ The programs are run on a computer with a 3.4GHz Core i7-2600 CPU and 16 GB of RAM running Gentoo Linux’s 3.8.4-gentoo kernel. The machine is running with CPU frequency scaling disabled via $ sudo bash # for i in /sys/devices/system/cpu/cpu[0-7] do echo performance > $i/cpufreq/scaling_governor done # We’ve taken [Hundt’s benchmark programs](https://github.com/hundt98847/multi-language-bench) in C++ and Go, combined each into a single source file, and removed all but one line of output. We’ll time the program using Linux’s `time` utility with a format that shows user time, system time, real time, and maximum memory usage: $ cat xtime #!/bin/sh /usr/bin/time -f '%Uu %Ss %er %MkB %C' "$@" $ $ make havlak1cc g++ -O3 -o havlak1cc havlak1.cc $ ./xtime ./havlak1cc # of loops: 76002 (total 3800100) loop-0, nest: 0, depth: 0 17.70u 0.05s 17.80r 715472kB ./havlak1cc $ $ make havlak1 go build havlak1.go $ ./xtime ./havlak1 # of loops: 76000 (including 1 artificial root node) 25.05u 0.11s 25.20r 1334032kB ./havlak1 $ The C++ program runs in 17.80 seconds and uses 700 MB of memory. The Go program runs in 25.20 seconds and uses 1302 MB of memory. (These measurements are difficult to reconcile with the ones in the paper, but the point of this post is to explore how to use `go tool pprof`, not to reproduce the results from the paper.) To start tuning the Go program, we have to enable profiling. If the code used the [Go testing package](https://go.dev/pkg/testing/) ’s benchmarking support, we could use gotest’s standard `-cpuprofile` and `-memprofile` flags. In a standalone program like this one, we have to import `runtime/pprof` and add a few lines of code: var cpuprofile = flag.String("cpuprofile", "", "write cpu profile to file") func main() { flag.Parse() if *cpuprofile != "" { f, err := os.Create(*cpuprofile) if err != nil { log.Fatal(err) } pprof.StartCPUProfile(f) defer pprof.StopCPUProfile() } ... The new code defines a flag named `cpuprofile`, calls the [Go flag library](https://go.dev/pkg/flag/) to parse the command line flags, and then, if the `cpuprofile` flag has been set on the command line, [starts CPU profiling](https://go.dev/pkg/runtime/pprof/#StartCPUProfile) redirected to that file. The profiler requires a final call to [`StopCPUProfile`](https://go.dev/pkg/runtime/pprof/#StopCPUProfile) to flush any pending writes to the file before the program exits; we use `defer` to make sure this happens as `main` returns. After adding that code, we can run the program with the new `-cpuprofile` flag and then run `go tool pprof` to interpret the profile. $ make havlak1.prof ./havlak1 -cpuprofile=havlak1.prof # of loops: 76000 (including 1 artificial root node) $ go tool pprof havlak1 havlak1.prof Welcome to pprof! For help, type 'help'. (pprof) The `go tool pprof` program is a slight variant of [Google’s `pprof` C++ profiler](https://github.com/gperftools/gperftools) . The most important command is `topN`, which shows the top `N` samples in the profile: (pprof) top10 Total: 2525 samples 298 11.8% 11.8% 345 13.7% runtime.mapaccess1_fast64 268 10.6% 22.4% 2124 84.1% main.FindLoops 251 9.9% 32.4% 451 17.9% scanblock 178 7.0% 39.4% 351 13.9% hash_insert 131 5.2% 44.6% 158 6.3% sweepspan 119 4.7% 49.3% 350 13.9% main.DFS 96 3.8% 53.1% 98 3.9% flushptrbuf 95 3.8% 56.9% 95 3.8% runtime.aeshash64 95 3.8% 60.6% 101 4.0% runtime.settype_flush 88 3.5% 64.1% 988 39.1% runtime.mallocgc When CPU profiling is enabled, the Go program stops about 100 times per second and records a sample consisting of the program counters on the currently executing goroutine’s stack. The profile has 2525 samples, so it was running for a bit over 25 seconds. In the `go tool pprof` output, there is a row for each function that appeared in a sample. The first two columns show the number of samples in which the function was running (as opposed to waiting for a called function to return), as a raw count and as a percentage of total samples. The `runtime.mapaccess1_fast64` function was running during 298 samples, or 11.8%. The `top10` output is sorted by this sample count. The third column shows the running total during the listing: the first three rows account for 32.4% of the samples. The fourth and fifth columns show the number of samples in which the function appeared (either running or waiting for a called function to return). The `main.FindLoops` function was running in 10.6% of the samples, but it was on the call stack (it or functions it called were running) in 84.1% of the samples. To sort by the fourth and fifth columns, use the `-cum` (for cumulative) flag: (pprof) top5 -cum Total: 2525 samples 0 0.0% 0.0% 2144 84.9% gosched0 0 0.0% 0.0% 2144 84.9% main.main 0 0.0% 0.0% 2144 84.9% runtime.main 0 0.0% 0.0% 2124 84.1% main.FindHavlakLoops 268 10.6% 10.6% 2124 84.1% main.FindLoops (pprof) top5 -cum In fact the total for `main.FindLoops` and `main.main` should have been 100%, but each stack sample only includes the bottom 100 stack frames; during about a quarter of the samples, the recursive `main.DFS` function was more than 100 frames deeper than `main.main` so the complete trace was truncated. The stack trace samples contain more interesting data about function call relationships than the text listings can show. The `web` command writes a graph of the profile data in SVG format and opens it in a web browser. (There is also a `gv` command that writes PostScript and opens it in Ghostview. For either command, you need [graphviz](http://www.graphviz.org/) installed.) (pprof) web A small fragment of [the full graph](https://rawgit.com/rsc/benchgraffiti/master/havlak/havlak1.svg) looks like: ![](https://go.dev/blog/pprof/havlak1a-75.png) Each box in the graph corresponds to a single function, and the boxes are sized according to the number of samples in which the function was running. An edge from box X to box Y indicates that X calls Y; the number along the edge is the number of times that call appears in a sample. If a call appears multiple times in a single sample, such as during recursive function calls, each appearance counts toward the edge weight. That explains the 21342 on the self-edge from `main.DFS` to itself. Just at a glance, we can see that the program spends much of its time in hash operations, which correspond to use of Go’s `map` values. We can tell `web` to use only samples that include a specific function, such as `runtime.mapaccess1_fast64`, which clears some of the noise from the graph: (pprof) web mapaccess1 ![](https://go.dev/blog/pprof/havlak1-hash_lookup-75.png) If we squint, we can see that the calls to `runtime.mapaccess1_fast64` are being made by `main.FindLoops` and `main.DFS`. Now that we have a rough idea of the big picture, it’s time to zoom in on a particular function. Let’s look at `main.DFS` first, just because it is a shorter function: (pprof) list DFS Total: 2525 samples ROUTINE ====================== main.DFS in /home/rsc/g/benchgraffiti/havlak/havlak1.go 119 697 Total samples (flat / cumulative) 3 3 240: func DFS(currentNode *BasicBlock, nodes []*UnionFindNode, number map[*BasicBlock]int, last []int, current int) int { 1 1 241: nodes[current].Init(currentNode, current) 1 37 242: number[currentNode] = current . . 243: 1 1 244: lastid := current 89 89 245: for _, target := range currentNode.OutEdges { 9 152 246: if number[target] == unvisited { 7 354 247: lastid = DFS(target, nodes, number, last, lastid+1) . . 248: } . . 249: } 7 59 250: last[number[currentNode]] = lastid 1 1 251: return lastid (pprof) The listing shows the source code for the `DFS` function (really, for every function matching the regular expression `DFS`). The first three columns are the number of samples taken while running that line, the number of samples taken while running that line or in code called from that line, and the line number in the file. The related command `disasm` shows a disassembly of the function instead of a source listing; when there are enough samples this can help you see which instructions are expensive. The `weblist` command mixes the two modes: it shows [a source listing in which clicking a line shows the disassembly](https://rawgit.com/rsc/benchgraffiti/master/havlak/havlak1.html) . Since we already know that the time is going into map lookups implemented by the hash runtime functions, we care most about the second column. A large fraction of time is spent in recursive calls to `DFS` (line 247), as would be expected from a recursive traversal. Excluding the recursion, it looks like the time is going into the accesses to the `number` map on lines 242, 246, and 250. For that particular lookup, a map is not the most efficient choice. Just as they would be in a compiler, the basic block structures have unique sequence numbers assigned to them. Instead of using a `map[*BasicBlock]int` we can use a `[]int`, a slice indexed by the block number. There’s no reason to use a map when an array or slice will do. Changing `number` from a map to a slice requires editing seven lines in the program and cut its run time by nearly a factor of two: $ make havlak2 go build havlak2.go $ ./xtime ./havlak2 # of loops: 76000 (including 1 artificial root node) 16.55u 0.11s 16.69r 1321008kB ./havlak2 $ (See the [diff between `havlak1` and `havlak2`](https://github.com/rsc/benchgraffiti/commit/58ac27bcac3ffb553c29d0b3fb64745c91c95948) ) We can run the profiler again to confirm that `main.DFS` is no longer a significant part of the run time: $ make havlak2.prof ./havlak2 -cpuprofile=havlak2.prof # of loops: 76000 (including 1 artificial root node) $ go tool pprof havlak2 havlak2.prof Welcome to pprof! For help, type 'help'. (pprof) (pprof) top5 Total: 1652 samples 197 11.9% 11.9% 382 23.1% scanblock 189 11.4% 23.4% 1549 93.8% main.FindLoops 130 7.9% 31.2% 152 9.2% sweepspan 104 6.3% 37.5% 896 54.2% runtime.mallocgc 98 5.9% 43.5% 100 6.1% flushptrbuf (pprof) The entry `main.DFS` no longer appears in the profile, and the rest of the program runtime has dropped too. Now the program is spending most of its time allocating memory and garbage collecting (`runtime.mallocgc`, which both allocates and runs periodic garbage collections, accounts for 54.2% of the time). To find out why the garbage collector is running so much, we have to find out what is allocating memory. One way is to add memory profiling to the program. We’ll arrange that if the `-memprofile` flag is supplied, the program stops after one iteration of the loop finding, writes a memory profile, and exits: var memprofile = flag.String("memprofile", "", "write memory profile to this file") ... FindHavlakLoops(cfgraph, lsgraph) if *memprofile != "" { f, err := os.Create(*memprofile) if err != nil { log.Fatal(err) } pprof.WriteHeapProfile(f) f.Close() return } We invoke the program with `-memprofile` flag to write a profile: $ make havlak3.mprof go build havlak3.go ./havlak3 -memprofile=havlak3.mprof $ (See the [diff from havlak2](https://github.com/rsc/benchgraffiti/commit/b78dac106bea1eb3be6bb3ca5dba57c130268232) ) We use `go tool pprof` exactly the same way. Now the samples we are examining are memory allocations, not clock ticks. $ go tool pprof havlak3 havlak3.mprof Adjusting heap profiles for 1-in-524288 sampling rate Welcome to pprof! For help, type 'help'. (pprof) top5 Total: 82.4 MB 56.3 68.4% 68.4% 56.3 68.4% main.FindLoops 17.6 21.3% 89.7% 17.6 21.3% main.(*CFG).CreateNode 8.0 9.7% 99.4% 25.6 31.0% main.NewBasicBlockEdge 0.5 0.6% 100.0% 0.5 0.6% itab 0.0 0.0% 100.0% 0.5 0.6% fmt.init (pprof) The command `go tool pprof` reports that `FindLoops` has allocated approximately 56.3 of the 82.4 MB in use; `CreateNode` accounts for another 17.6 MB. To reduce overhead, the memory profiler only records information for approximately one block per half megabyte allocated (the “1-in-524288 sampling rate”), so these are approximations to the actual counts. To find the memory allocations, we can list those functions. (pprof) list FindLoops Total: 82.4 MB ROUTINE ====================== main.FindLoops in /home/rsc/g/benchgraffiti/havlak/havlak3.go 56.3 56.3 Total MB (flat / cumulative) ... 1.9 1.9 268: nonBackPreds := make([]map[int]bool, size) 5.8 5.8 269: backPreds := make([][]int, size) . . 270: 1.9 1.9 271: number := make([]int, size) 1.9 1.9 272: header := make([]int, size, size) 1.9 1.9 273: types := make([]int, size, size) 1.9 1.9 274: last := make([]int, size, size) 1.9 1.9 275: nodes := make([]*UnionFindNode, size, size) . . 276: . . 277: for i := 0; i < size; i++ { 9.5 9.5 278: nodes[i] = new(UnionFindNode) . . 279: } ... . . 286: for i, bb := range cfgraph.Blocks { . . 287: number[bb.Name] = unvisited 29.5 29.5 288: nonBackPreds[i] = make(map[int]bool) . . 289: } ... It looks like the current bottleneck is the same as the last one: using maps where simpler data structures suffice. `FindLoops` is allocating about 29.5 MB of maps. As an aside, if we run `go tool pprof` with the `--inuse_objects` flag, it will report allocation counts instead of sizes: $ go tool pprof --inuse_objects havlak3 havlak3.mprof Adjusting heap profiles for 1-in-524288 sampling rate Welcome to pprof! For help, type 'help'. (pprof) list FindLoops Total: 1763108 objects ROUTINE ====================== main.FindLoops in /home/rsc/g/benchgraffiti/havlak/havlak3.go 720903 720903 Total objects (flat / cumulative) ... . . 277: for i := 0; i < size; i++ { 311296 311296 278: nodes[i] = new(UnionFindNode) . . 279: } . . 280: . . 281: // Step a: . . 282: // - initialize all nodes as unvisited. . . 283: // - depth-first traversal and numbering. . . 284: // - unreached BB's are marked as dead. . . 285: // . . 286: for i, bb := range cfgraph.Blocks { . . 287: number[bb.Name] = unvisited 409600 409600 288: nonBackPreds[i] = make(map[int]bool) . . 289: } ... (pprof) Since the ~200,000 maps account for 29.5 MB, it looks like the initial map allocation takes about 150 bytes. That’s reasonable when a map is being used to hold key-value pairs, but not when a map is being used as a stand-in for a simple set, as it is here. Instead of using a map, we can use a simple slice to list the elements. In all but one of the cases where maps are being used, it is impossible for the algorithm to insert a duplicate element. In the one remaining case, we can write a simple variant of the `append` built-in function: func appendUnique(a []int, x int) []int { for _, y := range a { if x == y { return a } } return append(a, x) } In addition to writing that function, changing the Go program to use slices instead of maps requires changing just a few lines of code. $ make havlak4 go build havlak4.go $ ./xtime ./havlak4 # of loops: 76000 (including 1 artificial root node) 11.84u 0.08s 11.94r 810416kB ./havlak4 $ (See the [diff from havlak3](https://github.com/rsc/benchgraffiti/commit/245d899f7b1a33b0c8148a4cd147cb3de5228c8a) ) We’re now at 2.11x faster than when we started. Let’s look at a CPU profile again. $ make havlak4.prof ./havlak4 -cpuprofile=havlak4.prof # of loops: 76000 (including 1 artificial root node) $ go tool pprof havlak4 havlak4.prof Welcome to pprof! For help, type 'help'. (pprof) top10 Total: 1173 samples 205 17.5% 17.5% 1083 92.3% main.FindLoops 138 11.8% 29.2% 215 18.3% scanblock 88 7.5% 36.7% 96 8.2% sweepspan 76 6.5% 43.2% 597 50.9% runtime.mallocgc 75 6.4% 49.6% 78 6.6% runtime.settype_flush 74 6.3% 55.9% 75 6.4% flushptrbuf 64 5.5% 61.4% 64 5.5% runtime.memmove 63 5.4% 66.8% 524 44.7% runtime.growslice 51 4.3% 71.1% 51 4.3% main.DFS 50 4.3% 75.4% 146 12.4% runtime.MCache_Alloc (pprof) Now memory allocation and the consequent garbage collection (`runtime.mallocgc`) accounts for 50.9% of our run time. Another way to look at why the system is garbage collecting is to look at the allocations that are causing the collections, the ones that spend most of the time in `mallocgc`: (pprof) web mallocgc ![](https://go.dev/blog/pprof/havlak4a-mallocgc.png) It’s hard to tell what’s going on in that graph, because there are many nodes with small sample numbers obscuring the big ones. We can tell `go tool pprof` to ignore nodes that don’t account for at least 10% of the samples: $ go tool pprof --nodefraction=0.1 havlak4 havlak4.prof Welcome to pprof! For help, type 'help'. (pprof) web mallocgc ![](https://go.dev/blog/pprof/havlak4a-mallocgc-trim.png) We can follow the thick arrows easily now, to see that `FindLoops` is triggering most of the garbage collection. If we list `FindLoops` we can see that much of it is right at the beginning: (pprof) list FindLoops ... . . 270: func FindLoops(cfgraph *CFG, lsgraph *LSG) { . . 271: if cfgraph.Start == nil { . . 272: return . . 273: } . . 274: . . 275: size := cfgraph.NumNodes() . . 276: . 145 277: nonBackPreds := make([][]int, size) . 9 278: backPreds := make([][]int, size) . . 279: . 1 280: number := make([]int, size) . 17 281: header := make([]int, size, size) . . 282: types := make([]int, size, size) . . 283: last := make([]int, size, size) . . 284: nodes := make([]*UnionFindNode, size, size) . . 285: . . 286: for i := 0; i < size; i++ { 2 79 287: nodes[i] = new(UnionFindNode) . . 288: } ... (pprof) Every time `FindLoops` is called, it allocates some sizable bookkeeping structures. Since the benchmark calls `FindLoops` 50 times, these add up to a significant amount of garbage, so a significant amount of work for the garbage collector. Having a garbage-collected language doesn’t mean you can ignore memory allocation issues. In this case, a simple solution is to introduce a cache so that each call to `FindLoops` reuses the previous call’s storage when possible. (In fact, in Hundt’s paper, he explains that the Java program needed just this change to get anything like reasonable performance, but he did not make the same change in the other garbage-collected implementations.) We’ll add a global `cache` structure: var cache struct { size int nonBackPreds [][]int backPreds [][]int number []int header []int types []int last []int nodes []*UnionFindNode } and then have `FindLoops` consult it as a replacement for allocation: if cache.size < size { cache.size = size cache.nonBackPreds = make([][]int, size) cache.backPreds = make([][]int, size) cache.number = make([]int, size) cache.header = make([]int, size) cache.types = make([]int, size) cache.last = make([]int, size) cache.nodes = make([]*UnionFindNode, size) for i := range cache.nodes { cache.nodes[i] = new(UnionFindNode) } } nonBackPreds := cache.nonBackPreds[:size] for i := range nonBackPreds { nonBackPreds[i] = nonBackPreds[i][:0] } backPreds := cache.backPreds[:size] for i := range nonBackPreds { backPreds[i] = backPreds[i][:0] } number := cache.number[:size] header := cache.header[:size] types := cache.types[:size] last := cache.last[:size] nodes := cache.nodes[:size] Such a global variable is bad engineering practice, of course: it means that concurrent calls to `FindLoops` are now unsafe. For now, we are making the minimal possible changes in order to understand what is important for the performance of our program; this change is simple and mirrors the code in the Java implementation. The final version of the Go program will use a separate `LoopFinder` instance to track this memory, restoring the possibility of concurrent use. $ make havlak5 go build havlak5.go $ ./xtime ./havlak5 # of loops: 76000 (including 1 artificial root node) 8.03u 0.06s 8.11r 770352kB ./havlak5 $ (See the [diff from havlak4](https://github.com/rsc/benchgraffiti/commit/2d41d6d16286b8146a3f697dd4074deac60d12a4) ) There’s more we can do to clean up the program and make it faster, but none of it requires profiling techniques that we haven’t already shown. The work list used in the inner loop can be reused across iterations and across calls to `FindLoops`, and it can be combined with the separate “node pool” generated during that pass. Similarly, the loop graph storage can be reused on each iteration instead of reallocated. In addition to these performance changes, the [final version](https://github.com/rsc/benchgraffiti/blob/master/havlak/havlak6.go) is written using idiomatic Go style, using data structures and methods. The stylistic changes have only a minor effect on the run time: the algorithm and constraints are unchanged. The final version runs in 2.29 seconds and uses 351 MB of memory: $ make havlak6 go build havlak6.go $ ./xtime ./havlak6 # of loops: 76000 (including 1 artificial root node) 2.26u 0.02s 2.29r 360224kB ./havlak6 $ That’s 11 times faster than the program we started with. Even if we disable reuse of the generated loop graph, so that the only cached memory is the loop finding bookkeeping, the program still runs 6.7x faster than the original and uses 1.5x less memory. $ ./xtime ./havlak6 -reuseloopgraph=false # of loops: 76000 (including 1 artificial root node) 3.69u 0.06s 3.76r 797120kB ./havlak6 -reuseloopgraph=false $ Of course, it’s no longer fair to compare this Go program to the original C++ program, which used inefficient data structures like `set`s where `vector`s would be more appropriate. As a sanity check, we translated the final Go program into [equivalent C++ code](https://github.com/rsc/benchgraffiti/blob/master/havlak/havlak6.cc) . Its execution time is similar to the Go program’s: $ make havlak6cc g++ -O3 -o havlak6cc havlak6.cc $ ./xtime ./havlak6cc # of loops: 76000 (including 1 artificial root node) 1.99u 0.19s 2.19r 387936kB ./havlak6cc The Go program runs almost as fast as the C++ program. As the C++ program is using automatic deletes and allocation instead of an explicit cache, the C++ program a bit shorter and easier to write, but not dramatically so: $ wc havlak6.cc; wc havlak6.go 401 1220 9040 havlak6.cc 461 1441 9467 havlak6.go $ (See [havlak6.cc](https://github.com/rsc/benchgraffiti/blob/master/havlak/havlak6.cc) and [havlak6.go](https://github.com/rsc/benchgraffiti/blob/master/havlak/havlak6.go) ) Benchmarks are only as good as the programs they measure. We used `go tool pprof` to study an inefficient Go program and then to improve its performance by an order of magnitude and to reduce its memory usage by a factor of 3.7. A subsequent comparison with an equivalently optimized C++ program shows that Go can be competitive with C++ when programmers are careful about how much garbage is generated by inner loops. The program sources, Linux x86-64 binaries, and profiles used to write this post are available in the [benchgraffiti project on GitHub](https://github.com/rsc/benchgraffiti/) . As mentioned above, [`go test`](https://go.dev/cmd/go/#Test_packages) includes these profiling flags already: define a [benchmark function](https://go.dev/pkg/testing/) and you’re all set. There is also a standard HTTP interface to profiling data. In an HTTP server, adding import _ "net/http/pprof" will install handlers for a few URLs under `/debug/pprof/`. Then you can run `go tool pprof` with a single argument—the URL to your server’s profiling data and it will download and examine a live profile. go tool pprof http://localhost:6060/debug/pprof/profile # 30-second CPU profile go tool pprof http://localhost:6060/debug/pprof/heap # heap profile go tool pprof http://localhost:6060/debug/pprof/block # goroutine blocking profile The goroutine blocking profile will be explained in a future post. Stay tuned. **Next article:** [First Class Functions in Go](https://go.dev/blog/functions-codewalk) **Previous article:** [Spotlight on external Go libraries](https://go.dev/blog/external-libraries) **[Blog Index](https://go.dev/blog/all) ** go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Tutorial: Getting started with fuzzing - The Go Programming Language 1. [Documentation](https://go.dev/doc/) 2. [Tutorials](https://go.dev/doc/tutorial/) 3. [Tutorial: Getting started with fuzzing](https://go.dev/doc/tutorial/fuzz) Tutorial: Getting started with fuzzing ====================================== This tutorial introduces the basics of fuzzing in Go. With fuzzing, random data is run against your test in an attempt to find vulnerabilities or crash-causing inputs. Some examples of vulnerabilities that can be found by fuzzing are SQL injection, buffer overflow, denial of service and cross-site scripting attacks. In this tutorial, you’ll write a fuzz test for a simple function, run the go command, and debug and fix issues in the code. For help with terminology throughout this tutorial, see the [Go Fuzzing glossary](https://go.dev/security/fuzz/#glossary) . You’ll progress through the following sections: 1. [Create a folder for your code.](https://go.dev/doc/tutorial/fuzz#create_folder) 2. [Add code to test.](https://go.dev/doc/tutorial/fuzz#code_to_test) 3. [Add a unit test.](https://go.dev/doc/tutorial/fuzz#unit_test) 4. [Add a fuzz test.](https://go.dev/doc/tutorial/fuzz#fuzz_test) 5. [Fix two bugs.](https://go.dev/doc/tutorial/fuzz#fix_invalid_string_error) 6. [Explore additional resources.](https://go.dev/doc/tutorial/fuzz#conclusion) **Note:** For other tutorials, see [Tutorials](https://go.dev/doc/tutorial/index.html) . **Note:** Go fuzzing currently supports a subset of built-in types, listed in the [Go Fuzzing docs](https://go.dev/security/fuzz/#requirements) , with support for more built-in types to be added in the future. Prerequisites ------------- * **An installation of Go 1.18 or later.** For installation instructions, see [Installing Go](https://go.dev/doc/install) . * **A tool to edit your code.** Any text editor you have will work fine. * **A command terminal.** Go works well using any terminal on Linux and Mac, and on PowerShell or cmd in Windows. * **An environment that supports fuzzing.** Go fuzzing with coverage instrumentation is only available on AMD64 and ARM64 architectures currently. Create a folder for your code ----------------------------- To begin, create a folder for the code you’ll write. 1. Open a command prompt and change to your home directory. On Linux or Mac: $ cd On Windows: C:\> cd %HOMEPATH% The rest of the tutorial will show a $ as the prompt. The commands you use will work on Windows too. 2. From the command prompt, create a directory for your code called fuzz. $ mkdir fuzz $ cd fuzz 3. Create a module to hold your code. Run the `go mod init` command, giving it your new code’s module path. $ go mod init example/fuzz go: creating new go.mod: module example/fuzz **Note:** For production code, you’d specify a module path that’s more specific to your own needs. For more, be sure to see [Managing dependencies](https://go.dev/doc/modules/managing-dependencies) . Next, you’ll add some simple code to reverse a string, which we’ll fuzz later. Add code to test ---------------- In this step, you’ll add a function to reverse a string. ### Write the code 1. Using your text editor, create a file called main.go in the fuzz directory. 2. Into main.go, at the top of the file, paste the following package declaration. package main A standalone program (as opposed to a library) is always in package `main`. 3. Beneath the package declaration, paste the following function declaration. func Reverse(s string) string { b := []byte(s) for i, j := 0, len(b)-1; i < len(b)/2; i, j = i+1, j-1 { b[i], b[j] = b[j], b[i] } return string(b) } This function will accept a `string`, loop over it a `byte` at a time, and return the reversed string at the end. _Note:_ This code is based on the `stringutil.Reverse` function within golang.org/x/example. 4. At the top of main.go, beneath the package declaration, paste the following `main` function to initialize a string, reverse it, print the output, and repeat. func main() { input := "The quick brown fox jumped over the lazy dog" rev := Reverse(input) doubleRev := Reverse(rev) fmt.Printf("original: %q\n", input) fmt.Printf("reversed: %q\n", rev) fmt.Printf("reversed again: %q\n", doubleRev) } This function will run a few `Reverse` operations, then print the output to the command line. This can be helpful for seeing the code in action, and potentially for debugging. 5. The `main` function uses the fmt package, so you will need to import it. The first lines of code should look like this: package main import "fmt" ### Run the code From the command line in the directory containing main.go, run the code. $ go run . original: "The quick brown fox jumped over the lazy dog" reversed: "god yzal eht revo depmuj xof nworb kciuq ehT" reversed again: "The quick brown fox jumped over the lazy dog" You can see the original string, the result of reversing it, then the result of reversing it again, which is equivalent to the original. Now that the code is running, it’s time to test it. Add a unit test --------------- In this step, you will write a basic unit test for the `Reverse` function. ### Write the code 1. Using your text editor, create a file called reverse\_test.go in the fuzz directory. 2. Paste the following code into reverse\_test.go. package main import ( "testing" ) func TestReverse(t *testing.T) { testcases := []struct { in, want string }{ {"Hello, world", "dlrow ,olleH"}, {" ", " "}, {"!12345", "54321!"}, } for _, tc := range testcases { rev := Reverse(tc.in) if rev != tc.want { t.Errorf("Reverse: %q, want %q", rev, tc.want) } } } This simple test will assert that the listed input strings will be correctly reversed. ### Run the code Run the unit test using `go test` $ go test PASS ok example/fuzz 0.013s Next, you will change the unit test into a fuzz test. Add a fuzz test --------------- The unit test has limitations, namely that each input must be added to the test by the developer. One benefit of fuzzing is that it comes up with inputs for your code, and may identify edge cases that the test cases you came up with didn’t reach. In this section you will convert the unit test to a fuzz test so that you can generate more inputs with less work! Note that you can keep unit tests, benchmarks, and fuzz tests in the same \*\_test.go file, but for this example you will convert the unit test to a fuzz test. ### Write the code In your text editor, replace the unit test in reverse\_test.go with the following fuzz test. func FuzzReverse(f *testing.F) { testcases := []string{"Hello, world", " ", "!12345"} for _, tc := range testcases { f.Add(tc) // Use f.Add to provide a seed corpus } f.Fuzz(func(t *testing.T, orig string) { rev := Reverse(orig) doubleRev := Reverse(rev) if orig != doubleRev { t.Errorf("Before: %q, after: %q", orig, doubleRev) } if utf8.ValidString(orig) && !utf8.ValidString(rev) { t.Errorf("Reverse produced invalid UTF-8 string %q", rev) } }) } Fuzzing has a few limitations as well. In your unit test, you could predict the expected output of the `Reverse` function, and verify that the actual output met those expectations. For example, in the test case `Reverse("Hello, world")` the unit test specifies the return as `"dlrow ,olleH"`. When fuzzing, you can’t predict the expected output, since you don’t have control over the inputs. However, there are a few properties of the `Reverse` function that you can verify in a fuzz test. The two properties being checked in this fuzz test are: 1. Reversing a string twice preserves the original value 2. The reversed string preserves its state as valid UTF-8. Note the syntax differences between the unit test and the fuzz test: * The function begins with FuzzXxx instead of TestXxx, and takes `*testing.F` instead of `*testing.T` * Where you would expect to see a `t.Run` execution, you instead see `f.Fuzz` which takes a fuzz target function whose parameters are `*testing.T` and the types to be fuzzed. The inputs from your unit test are provided as seed corpus inputs using `f.Add`. Ensure the new package, `unicode/utf8` has been imported. package main import ( "testing" "unicode/utf8" ) With the unit test converted to a fuzz test, it’s time to run the test again. ### Run the code 1. Run the fuzz test without fuzzing it to make sure the seed inputs pass. $ go test PASS ok example/fuzz 0.013s You can also run `go test -run=FuzzReverse` if you have other tests in that file, and you only wish to run the fuzz test. 2. Run `FuzzReverse` with fuzzing, to see if any randomly generated string inputs will cause a failure. This is executed using `go test` with a new flag, `-fuzz`, set to the parameter `Fuzz`. Copy the command below. $ go test -fuzz=Fuzz Another useful flag is `-fuzztime`, which restricts the time fuzzing takes. For example, specifying `-fuzztime 10s` in the test below would mean that, as long as no failures occurred earlier, the test would exit by default after 10 seconds had elapsed. See [this section](https://pkg.go.dev/cmd/go#hdr-Testing_flags) of the cmd/go documentation to see other testing flags. Now, run the command you just copied. $ go test -fuzz=Fuzz fuzz: elapsed: 0s, gathering baseline coverage: 0/3 completed fuzz: elapsed: 0s, gathering baseline coverage: 3/3 completed, now fuzzing with 8 workers fuzz: minimizing 38-byte failing input file... --- FAIL: FuzzReverse (0.01s) --- FAIL: FuzzReverse (0.00s) reverse_test.go:20: Reverse produced invalid UTF-8 string "\x9c\xdd" Failing input written to testdata/fuzz/FuzzReverse/af69258a12129d6cbba438df5d5f25ba0ec050461c116f777e77ea7c9a0d217a To re-run: go test -run=FuzzReverse/af69258a12129d6cbba438df5d5f25ba0ec050461c116f777e77ea7c9a0d217a FAIL exit status 1 FAIL example/fuzz 0.030s A failure occurred while fuzzing, and the input that caused the problem is written to a seed corpus file that will be run the next time `go test` is called, even without the `-fuzz` flag. To view the input that caused the failure, open the corpus file written to the testdata/fuzz/FuzzReverse directory in a text editor. Your seed corpus file may contain a different string, but the format will be the same. go test fuzz v1 string("泃") The first line of the corpus file indicates the encoding version. Each following line represents the value of each type making up the corpus entry. Since the fuzz target only takes 1 input, there is only 1 value after the version. 3. Run `go test` again without the `-fuzz` flag; the new failing seed corpus entry will be used: $ go test --- FAIL: FuzzReverse (0.00s) --- FAIL: FuzzReverse/af69258a12129d6cbba438df5d5f25ba0ec050461c116f777e77ea7c9a0d217a (0.00s) reverse_test.go:20: Reverse produced invalid string FAIL exit status 1 FAIL example/fuzz 0.016s Since our test has failed, it’s time to debug. Fix the invalid string error ---------------------------- In this section, you will debug the failure, and fix the bug. Feel free to spend some time thinking about this and trying to fix the issue yourself before moving on. ### Diagnose the error There are a few different ways you could debug this error. If you are using VS Code as your text editor, you can [set up your debugger](https://github.com/golang/vscode-go/blob/master/docs/debugging.md) to investigate. In this tutorial, we will log useful debugging info to your terminal. First, consider the docs for [`utf8.ValidString`](https://pkg.go.dev/unicode/utf8) . ValidString reports whether s consists entirely of valid UTF-8-encoded runes. The current `Reverse` function reverses the string byte-by-byte, and therein lies our problem. In order to preserve the UTF-8-encoded runes of the original string, we must instead reverse the string rune-by-rune. To examine why the input (in this case, the Chinese character `泃`) is causing `Reverse` to produce an invalid string when reversed, you can inspect the number of runes in the reversed string. #### Write the code In your text editor, replace the fuzz target within `FuzzReverse` with the following. f.Fuzz(func(t *testing.T, orig string) { rev := Reverse(orig) doubleRev := Reverse(rev) t.Logf("Number of runes: orig=%d, rev=%d, doubleRev=%d", utf8.RuneCountInString(orig), utf8.RuneCountInString(rev), utf8.RuneCountInString(doubleRev)) if orig != doubleRev { t.Errorf("Before: %q, after: %q", orig, doubleRev) } if utf8.ValidString(orig) && !utf8.ValidString(rev) { t.Errorf("Reverse produced invalid UTF-8 string %q", rev) } }) This `t.Logf` line will print to the command line if an error occurs, or if executing the test with `-v`, which can help you debug this particular issue. #### Run the code Run the test using go test $ go test --- FAIL: FuzzReverse (0.00s) --- FAIL: FuzzReverse/28f36ef487f23e6c7a81ebdaa9feffe2f2b02b4cddaa6252e87f69863046a5e0 (0.00s) reverse_test.go:16: Number of runes: orig=1, rev=3, doubleRev=1 reverse_test.go:21: Reverse produced invalid UTF-8 string "\x83\xb3\xe6" FAIL exit status 1 FAIL example/fuzz 0.598s The entire seed corpus used strings in which every character was a single byte. However, characters such as 泃 can require several bytes. Thus, reversing the string byte-by-byte will invalidate multi-byte characters. **Note:** If you’re curious about how Go deals with strings, read the blog post [Strings, bytes, runes and characters in Go](https://go.dev/blog/strings) for a deeper understanding. With a better understanding of the bug, correct the error in the `Reverse` function. ### Fix the error To correct the `Reverse` function, let’s traverse the string by runes, instead of by bytes. #### Write the code In your text editor, replace the existing Reverse() function with the following. func Reverse(s string) string { r := []rune(s) for i, j := 0, len(r)-1; i < len(r)/2; i, j = i+1, j-1 { r[i], r[j] = r[j], r[i] } return string(r) } The key difference is that `Reverse` is now iterating over each `rune` in the string, rather than each `byte`. Note that this is just an example, and does not handle [combining characters](https://en.wikipedia.org/wiki/Combining_character) correctly. #### Run the code 1. Run the test using `go test` $ go test PASS ok example/fuzz 0.016s The test now passes! 2. Fuzz it again with `go test -fuzz`, to see if there are any new bugs. $ go test -fuzz=Fuzz fuzz: elapsed: 0s, gathering baseline coverage: 0/37 completed fuzz: minimizing 506-byte failing input file... fuzz: elapsed: 0s, gathering baseline coverage: 5/37 completed --- FAIL: FuzzReverse (0.02s) --- FAIL: FuzzReverse (0.00s) reverse_test.go:33: Before: "\x91", after: "�" Failing input written to testdata/fuzz/FuzzReverse/1ffc28f7538e29d79fce69fef20ce5ea72648529a9ca10bea392bcff28cd015c To re-run: go test -run=FuzzReverse/1ffc28f7538e29d79fce69fef20ce5ea72648529a9ca10bea392bcff28cd015c FAIL exit status 1 FAIL example/fuzz 0.032s We can see that the string is different from the original after being reversed twice. This time the input itself is invalid unicode. How is this possible if we’re fuzzing with strings? Let’s debug again. Fix the double reverse error ---------------------------- In this section, you will debug the double reverse failure and fix the bug. Feel free to spend some time thinking about this and trying to fix the issue yourself before moving on. ### Diagnose the error Like before, there are several ways you could debug this failure. In this case, using a [debugger](https://github.com/golang/vscode-go/blob/master/docs/debugging.md) would be a great approach. In this tutorial, we will log useful debugging info in the `Reverse` function. Look closely at the reversed string to spot the error. In Go, [a string is a read only slice of bytes](https://go.dev/blog/strings) , and can contain bytes that aren’t valid UTF-8. The original string is a byte slice with one byte, `'\x91'`. When the input string is set to `[]rune`, Go encodes the byte slice to UTF-8, and replaces the byte with the UTF-8 character �. When we compare the replacement UTF-8 character to the input byte slice, they are clearly not equal. #### Write the code 1. In your text editor, replace the `Reverse` function with the following. func Reverse(s string) string { fmt.Printf("input: %q\n", s) r := []rune(s) fmt.Printf("runes: %q\n", r) for i, j := 0, len(r)-1; i < len(r)/2; i, j = i+1, j-1 { r[i], r[j] = r[j], r[i] } return string(r) } This will help us understand what is going wrong when converting the string to a slice of runes. #### Run the code This time, we only want to run the failing test in order to inspect the logs. To do this, we will use `go test -run`. To run a specific corpus entry within FuzzXxx/testdata, you can provide {FuzzTestName}/{filename} to `-run`. This can be helpful when debugging. In this case, set the `-run` flag equal to the exact hash of the failing test. Copy and paste the unique hash from your terminal; it will be different than the one below. $ go test -run=FuzzReverse/28f36ef487f23e6c7a81ebdaa9feffe2f2b02b4cddaa6252e87f69863046a5e0 input: "\x91" runes: ['�'] input: "�" runes: ['�'] --- FAIL: FuzzReverse (0.00s) --- FAIL: FuzzReverse/28f36ef487f23e6c7a81ebdaa9feffe2f2b02b4cddaa6252e87f69863046a5e0 (0.00s) reverse_test.go:16: Number of runes: orig=1, rev=1, doubleRev=1 reverse_test.go:18: Before: "\x91", after: "�" FAIL exit status 1 FAIL example/fuzz 0.145s Knowing that the input is invalid unicode, let’s fix the error in our `Reverse` function. ### Fix the error To fix this issue, let’s return an error if the input to `Reverse` isn’t valid UTF-8. #### Write the code 1. In your text editor, replace the existing `Reverse` function with the following. func Reverse(s string) (string, error) { if !utf8.ValidString(s) { return s, errors.New("input is not valid UTF-8") } r := []rune(s) for i, j := 0, len(r)-1; i < len(r)/2; i, j = i+1, j-1 { r[i], r[j] = r[j], r[i] } return string(r), nil } This change will return an error if the input string contains characters which are not valid UTF-8. 2. Since the Reverse function now returns an error, modify the `main` function to discard the extra error value. Replace the existing `main` function with the following. func main() { input := "The quick brown fox jumped over the lazy dog" rev, revErr := Reverse(input) doubleRev, doubleRevErr := Reverse(rev) fmt.Printf("original: %q\n", input) fmt.Printf("reversed: %q, err: %v\n", rev, revErr) fmt.Printf("reversed again: %q, err: %v\n", doubleRev, doubleRevErr) } These calls to `Reverse` should return a nil error, since the input string is valid UTF-8. 3. You will need to import the errors and the unicode/utf8 packages. The import statement in main.go should look like the following. import ( "errors" "fmt" "unicode/utf8" ) 4. Modify the reverse\_test.go file to check for errors and skip the test if errors are generated by returning. func FuzzReverse(f *testing.F) { testcases := []string {"Hello, world", " ", "!12345"} for _, tc := range testcases { f.Add(tc) // Use f.Add to provide a seed corpus } f.Fuzz(func(t *testing.T, orig string) { rev, err1 := Reverse(orig) if err1 != nil { return } doubleRev, err2 := Reverse(rev) if err2 != nil { return } if orig != doubleRev { t.Errorf("Before: %q, after: %q", orig, doubleRev) } if utf8.ValidString(orig) && !utf8.ValidString(rev) { t.Errorf("Reverse produced invalid UTF-8 string %q", rev) } }) } Rather than returning, you can also call `t.Skip()` to stop the execution of that fuzz input. #### Run the code 1. Run the test using go test $ go test PASS ok example/fuzz 0.019s 2. Fuzz it with `go test -fuzz=Fuzz`, then after a few seconds has passed, stop fuzzing with `ctrl-C`. The fuzz test will run until it encounters a failing input unless you pass the `-fuzztime` flag. The default is to run forever if no failures occur, and the process can be interrupted with `ctrl-C`. $ go test -fuzz=Fuzz fuzz: elapsed: 0s, gathering baseline coverage: 0/38 completed fuzz: elapsed: 0s, gathering baseline coverage: 38/38 completed, now fuzzing with 4 workers fuzz: elapsed: 3s, execs: 86342 (28778/sec), new interesting: 2 (total: 35) fuzz: elapsed: 6s, execs: 193490 (35714/sec), new interesting: 4 (total: 37) fuzz: elapsed: 9s, execs: 304390 (36961/sec), new interesting: 4 (total: 37) ... fuzz: elapsed: 3m45s, execs: 7246222 (32357/sec), new interesting: 8 (total: 41) ^Cfuzz: elapsed: 3m48s, execs: 7335316 (31648/sec), new interesting: 8 (total: 41) PASS ok example/fuzz 228.000s 3. Fuzz it with `go test -fuzz=Fuzz -fuzztime 30s` which will fuzz for 30 seconds before exiting if no failure was found. $ go test -fuzz=Fuzz -fuzztime 30s fuzz: elapsed: 0s, gathering baseline coverage: 0/5 completed fuzz: elapsed: 0s, gathering baseline coverage: 5/5 completed, now fuzzing with 4 workers fuzz: elapsed: 3s, execs: 80290 (26763/sec), new interesting: 12 (total: 12) fuzz: elapsed: 6s, execs: 210803 (43501/sec), new interesting: 14 (total: 14) fuzz: elapsed: 9s, execs: 292882 (27360/sec), new interesting: 14 (total: 14) fuzz: elapsed: 12s, execs: 371872 (26329/sec), new interesting: 14 (total: 14) fuzz: elapsed: 15s, execs: 517169 (48433/sec), new interesting: 15 (total: 15) fuzz: elapsed: 18s, execs: 663276 (48699/sec), new interesting: 15 (total: 15) fuzz: elapsed: 21s, execs: 771698 (36143/sec), new interesting: 15 (total: 15) fuzz: elapsed: 24s, execs: 924768 (50990/sec), new interesting: 16 (total: 16) fuzz: elapsed: 27s, execs: 1082025 (52427/sec), new interesting: 17 (total: 17) fuzz: elapsed: 30s, execs: 1172817 (30281/sec), new interesting: 17 (total: 17) fuzz: elapsed: 31s, execs: 1172817 (0/sec), new interesting: 17 (total: 17) PASS ok example/fuzz 31.025s Fuzzing passed! In addition to the `-fuzz` flag, several new flags have been added to `go test` and can be viewed in the [documentation](https://go.dev/security/fuzz/#custom-settings) . See [Go Fuzzing](https://go.dev/security/fuzz/#command-line-output) for more information on terms used in fuzzing output. For example, “new interesting” refers to inputs that expand the code coverage of the existing fuzz test corpus. The number of “new interesting” inputs can be expected to increase sharply as fuzzing begins, spike several times as new code paths are discovered, then taper off over time. Conclusion ---------- Nicely done! You’ve just introduced yourself to fuzzing in Go. The next step is to choose a function in your code that you’d like to fuzz, and try it out! If fuzzing finds a bug in your code, consider adding it to the [trophy case](https://go.dev/wiki/Fuzzing-trophy-case) . If you experience any problems or have an idea for a feature, [file an issue](https://go.dev/issue/new/?&labels=fuzz) . For discussion and general feedback about the feature, you can also participate in the [#fuzzing channel](https://gophers.slack.com/archives/CH5KV1AKE) in Gophers Slack. Check out the documentation at [go.dev/security/fuzz](https://go.dev/security/fuzz/#requirements) for further reading. Completed code -------------- — main.go — package main import ( "errors" "fmt" "unicode/utf8" ) func main() { input := "The quick brown fox jumped over the lazy dog" rev, revErr := Reverse(input) doubleRev, doubleRevErr := Reverse(rev) fmt.Printf("original: %q\n", input) fmt.Printf("reversed: %q, err: %v\n", rev, revErr) fmt.Printf("reversed again: %q, err: %v\n", doubleRev, doubleRevErr) } func Reverse(s string) (string, error) { if !utf8.ValidString(s) { return s, errors.New("input is not valid UTF-8") } r := []rune(s) for i, j := 0, len(r)-1; i < len(r)/2; i, j = i+1, j-1 { r[i], r[j] = r[j], r[i] } return string(r), nil } — reverse\_test.go — package main import ( "testing" "unicode/utf8" ) func FuzzReverse(f *testing.F) { testcases := []string{"Hello, world", " ", "!12345"} for _, tc := range testcases { f.Add(tc) // Use f.Add to provide a seed corpus } f.Fuzz(func(t *testing.T, orig string) { rev, err1 := Reverse(orig) if err1 != nil { return } doubleRev, err2 := Reverse(rev) if err2 != nil { return } if orig != doubleRev { t.Errorf("Before: %q, after: %q", orig, doubleRev) } if utf8.ValidString(orig) && !utf8.ValidString(rev) { t.Errorf("Reverse produced invalid UTF-8 string %q", rev) } }) } [Back to top](https://go.dev/doc/tutorial/fuzz#top) go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # C? Go? Cgo! - The Go Programming Language [The Go Blog](https://go.dev/blog/) ==================================== C? Go? Cgo! =========== Andrew Gerrand 17 March 2011 Introduction ------------ Cgo lets Go packages call C code. Given a Go source file written with some special features, cgo outputs Go and C files that can be combined into a single Go package. To lead with an example, here’s a Go package that provides two functions - `Random` and `Seed` - that wrap C’s `random` and `srandom` functions. package rand /* #include */ import "C" func Random() int { return int(C.random()) } func Seed(i int) { C.srandom(C.uint(i)) } Let’s look at what’s happening here, starting with the import statement. The `rand` package imports `"C"`, but you’ll find there’s no such package in the standard Go library. That’s because `C` is a “pseudo-package”, a special name interpreted by cgo as a reference to C’s name space. The `rand` package contains four references to the `C` package: the calls to `C.random` and `C.srandom`, the conversion `C.uint(i)`, and the `import` statement. The `Random` function calls the standard C library’s `random` function and returns the result. In C, `random` returns a value of the C type `long`, which cgo represents as the type `C.long`. It must be converted to a Go type before it can be used by Go code outside this package, using an ordinary Go type conversion: func Random() int { return int(C.random()) } Here’s an equivalent function that uses a temporary variable to illustrate the type conversion more explicitly: func Random() int { var r C.long = C.random() return int(r) } The `Seed` function does the reverse, in a way. It takes a regular Go `int`, converts it to the C `unsigned int` type, and passes it to the C function `srandom`. func Seed(i int) { C.srandom(C.uint(i)) } Note that cgo knows the `unsigned int` type as `C.uint`; see the [cgo documentation](https://go.dev/cmd/cgo) for a complete list of these numeric type names. The one detail of this example we haven’t examined yet is the comment above the `import` statement. /* #include */ import "C" Cgo recognizes this comment. Any lines starting with `#cgo` followed by a space character are removed; these become directives for cgo. The remaining lines are used as a header when compiling the C parts of the package. In this case those lines are just a single `#include` statement, but they can be almost any C code. The `#cgo` directives are used to provide flags for the compiler and linker when building the C parts of the package. There is a limitation: if your program uses any `//export` directives, then the C code in the comment may only include declarations (`extern int f();`), not definitions (`int f() { return 1; }`). You can use `//export` directives to make Go functions accessible to C code. The `#cgo` and `//export` directives are documented in the [cgo documentation](https://go.dev/cmd/cgo/) . Strings and things ------------------ Unlike Go, C doesn’t have an explicit string type. Strings in C are represented by a zero-terminated array of chars. Conversion between Go and C strings is done with the `C.CString`, `C.GoString`, and `C.GoStringN` functions. These conversions make a copy of the string data. This next example implements a `Print` function that writes a string to standard output using C’s `fputs` function from the `stdio` library: package print // #include // #include import "C" import "unsafe" func Print(s string) { cs := C.CString(s) C.fputs(cs, (*C.FILE)(C.stdout)) C.free(unsafe.Pointer(cs)) } Memory allocations made by C code are not known to Go’s memory manager. When you create a C string with `C.CString` (or any C memory allocation) you must remember to free the memory when you’re done with it by calling `C.free`. The call to `C.CString` returns a pointer to the start of the char array, so before the function exits we convert it to an [`unsafe.Pointer`](https://go.dev/pkg/unsafe/#Pointer) and release the memory allocation with `C.free`. A common idiom in cgo programs is to [`defer`](https://go.dev/doc/articles/defer_panic_recover.html) the free immediately after allocating (especially when the code that follows is more complex than a single function call), as in this rewrite of `Print`: func Print(s string) { cs := C.CString(s) defer C.free(unsafe.Pointer(cs)) C.fputs(cs, (*C.FILE)(C.stdout)) } Building cgo packages --------------------- To build cgo packages, just use [`go build`](https://go.dev/cmd/go/#hdr-Compile_packages_and_dependencies) or [`go install`](https://go.dev/cmd/go/#hdr-Compile_and_install_packages_and_dependencies) as usual. The go tool recognizes the special `"C"` import and automatically uses cgo for those files. More cgo resources ------------------ The [cgo command](https://go.dev/cmd/cgo/) documentation has more detail about the C pseudo-package and the build process. The [cgo examples](https://go.dev/misc/cgo/) in the Go tree demonstrate more advanced concepts. Finally, if you’re curious as to how all this works internally, take a look at the introductory comment of the runtime package’s [cgocall.go](https://go.dev/src/runtime/cgocall.go) . **Next article:** [Gobs of data](https://go.dev/blog/gob) **Previous article:** [Go becomes more stable](https://go.dev/blog/stable-releases) **[Blog Index](https://go.dev/blog/all) ** go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Error handling and Go - The Go Programming Language [The Go Blog](https://go.dev/blog/) ==================================== Error handling and Go ===================== Andrew Gerrand 12 July 2011 Introduction ------------ If you have written any Go code you have probably encountered the built-in `error` type. Go code uses `error` values to indicate an abnormal state. For example, the `os.Open` function returns a non-nil `error` value when it fails to open a file. func Open(name string) (file *File, err error) The following code uses `os.Open` to open a file. If an error occurs it calls `log.Fatal` to print the error message and stop. f, err := os.Open("filename.ext") if err != nil { log.Fatal(err) } // do something with the open *File f You can get a lot done in Go knowing just this about the `error` type, but in this article we’ll take a closer look at `error` and discuss some good practices for error handling in Go. The error type -------------- The `error` type is an interface type. An `error` variable represents any value that can describe itself as a string. Here is the interface’s declaration: type error interface { Error() string } The `error` type, as with all built in types, is [predeclared](https://go.dev/doc/go_spec.html#Predeclared_identifiers) in the [universe block](https://go.dev/doc/go_spec.html#Blocks) . The most commonly-used `error` implementation is the [errors](https://go.dev/pkg/errors/) package’s unexported `errorString` type. // errorString is a trivial implementation of error. type errorString struct { s string } func (e *errorString) Error() string { return e.s } You can construct one of these values with the `errors.New` function. It takes a string that it converts to an `errors.errorString` and returns as an `error` value. // New returns an error that formats as the given text. func New(text string) error { return &errorString{text} } Here’s how you might use `errors.New`: func Sqrt(f float64) (float64, error) { if f < 0 { return 0, errors.New("math: square root of negative number") } // implementation } A caller passing a negative argument to `Sqrt` receives a non-nil `error` value (whose concrete representation is an `errors.errorString` value). The caller can access the error string (“math: square root of…”) by calling the `error`’s `Error` method, or by just printing it: f, err := Sqrt(-1) if err != nil { fmt.Println(err) } The [fmt](https://go.dev/pkg/fmt/) package formats an `error` value by calling its `Error() string` method. It is the error implementation’s responsibility to summarize the context. The error returned by `os.Open` formats as “open /etc/passwd: permission denied,” not just “permission denied.” The error returned by our `Sqrt` is missing information about the invalid argument. To add that information, a useful function is the `fmt` package’s `Errorf`. It formats a string according to `Printf`’s rules and returns it as an `error` created by `errors.New`. if f < 0 { return 0, fmt.Errorf("math: square root of negative number %g", f) } In many cases `fmt.Errorf` is good enough, but since `error` is an interface, you can use arbitrary data structures as error values, to allow callers to inspect the details of the error. For instance, our hypothetical callers might want to recover the invalid argument passed to `Sqrt`. We can enable that by defining a new error implementation instead of using `errors.errorString`: type NegativeSqrtError float64 func (f NegativeSqrtError) Error() string { return fmt.Sprintf("math: square root of negative number %g", float64(f)) } A sophisticated caller can then use a [type assertion](https://go.dev/doc/go_spec.html#Type_assertions) to check for a `NegativeSqrtError` and handle it specially, while callers that just pass the error to `fmt.Println` or `log.Fatal` will see no change in behavior. As another example, the [json](https://go.dev/pkg/encoding/json/) package specifies a `SyntaxError` type that the `json.Decode` function returns when it encounters a syntax error parsing a JSON blob. type SyntaxError struct { msg string // description of error Offset int64 // error occurred after reading Offset bytes } func (e *SyntaxError) Error() string { return e.msg } The `Offset` field isn’t even shown in the default formatting of the error, but callers can use it to add file and line information to their error messages: if err := dec.Decode(&val); err != nil { if serr, ok := err.(*json.SyntaxError); ok { line, col := findLine(f, serr.Offset) return fmt.Errorf("%s:%d:%d: %v", f.Name(), line, col, err) } return err } (This is a slightly simplified version of some [actual code](https://github.com/camlistore/go4/blob/03efcb870d84809319ea509714dd6d19a1498483/jsonconfig/eval.go#L123-L135) from the [Camlistore](http://camlistore.org/) project.) The `error` interface requires only a `Error` method; specific error implementations might have additional methods. For instance, the [net](https://go.dev/pkg/net/) package returns errors of type `error`, following the usual convention, but some of the error implementations have additional methods defined by the `net.Error` interface: package net type Error interface { error Timeout() bool // Is the error a timeout? Temporary() bool // Is the error temporary? } Client code can test for a `net.Error` with a type assertion and then distinguish transient network errors from permanent ones. For instance, a web crawler might sleep and retry when it encounters a temporary error and give up otherwise. if nerr, ok := err.(net.Error); ok && nerr.Temporary() { time.Sleep(1e9) continue } if err != nil { log.Fatal(err) } Simplifying repetitive error handling ------------------------------------- In Go, error handling is important. The language’s design and conventions encourage you to explicitly check for errors where they occur (as distinct from the convention in other languages of throwing exceptions and sometimes catching them). In some cases this makes Go code verbose, but fortunately there are some techniques you can use to minimize repetitive error handling. Consider an [App Engine](https://cloud.google.com/appengine/docs/go/) application with an HTTP handler that retrieves a record from the datastore and formats it with a template. func init() { http.HandleFunc("/view", viewRecord) } func viewRecord(w http.ResponseWriter, r *http.Request) { c := appengine.NewContext(r) key := datastore.NewKey(c, "Record", r.FormValue("id"), 0, nil) record := new(Record) if err := datastore.Get(c, key, record); err != nil { http.Error(w, err.Error(), 500) return } if err := viewTemplate.Execute(w, record); err != nil { http.Error(w, err.Error(), 500) } } This function handles errors returned by the `datastore.Get` function and `viewTemplate`’s `Execute` method. In both cases, it presents a simple error message to the user with the HTTP status code 500 (“Internal Server Error”). This looks like a manageable amount of code, but add some more HTTP handlers and you quickly end up with many copies of identical error handling code. To reduce the repetition we can define our own HTTP `appHandler` type that includes an `error` return value: type appHandler func(http.ResponseWriter, *http.Request) error Then we can change our `viewRecord` function to return errors: func viewRecord(w http.ResponseWriter, r *http.Request) error { c := appengine.NewContext(r) key := datastore.NewKey(c, "Record", r.FormValue("id"), 0, nil) record := new(Record) if err := datastore.Get(c, key, record); err != nil { return err } return viewTemplate.Execute(w, record) } This is simpler than the original version, but the [http](https://go.dev/pkg/net/http/) package doesn’t understand functions that return `error`. To fix this we can implement the `http.Handler` interface’s `ServeHTTP` method on `appHandler`: func (fn appHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) { if err := fn(w, r); err != nil { http.Error(w, err.Error(), 500) } } The `ServeHTTP` method calls the `appHandler` function and displays the returned error (if any) to the user. Notice that the method’s receiver, `fn`, is a function. (Go can do that!) The method invokes the function by calling the receiver in the expression `fn(w, r)`. Now when registering `viewRecord` with the http package we use the `Handle` function (instead of `HandleFunc`) as `appHandler` is an `http.Handler` (not an `http.HandlerFunc`). func init() { http.Handle("/view", appHandler(viewRecord)) } With this basic error handling infrastructure in place, we can make it more user friendly. Rather than just displaying the error string, it would be better to give the user a simple error message with an appropriate HTTP status code, while logging the full error to the App Engine developer console for debugging purposes. To do this we create an `appError` struct containing an `error` and some other fields: type appError struct { Error error Message string Code int } Next we modify the appHandler type to return `*appError` values: type appHandler func(http.ResponseWriter, *http.Request) *appError (It’s usually a mistake to pass back the concrete type of an error rather than `error`, for reasons discussed in [the Go FAQ](https://go.dev/doc/go_faq.html#nil_error) , but it’s the right thing to do here because `ServeHTTP` is the only place that sees the value and uses its contents.) And make `appHandler`’s `ServeHTTP` method display the `appError`’s `Message` to the user with the correct HTTP status `Code` and log the full `Error` to the developer console: func (fn appHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) { if e := fn(w, r); e != nil { // e is *appError, not os.Error. c := appengine.NewContext(r) c.Errorf("%v", e.Error) http.Error(w, e.Message, e.Code) } } Finally, we update `viewRecord` to the new function signature and have it return more context when it encounters an error: func viewRecord(w http.ResponseWriter, r *http.Request) *appError { c := appengine.NewContext(r) key := datastore.NewKey(c, "Record", r.FormValue("id"), 0, nil) record := new(Record) if err := datastore.Get(c, key, record); err != nil { return &appError{err, "Record not found", 404} } if err := viewTemplate.Execute(w, record); err != nil { return &appError{err, "Can't display record", 500} } return nil } This version of `viewRecord` is the same length as the original, but now each of those lines has specific meaning and we are providing a friendlier user experience. It doesn’t end there; we can further improve the error handling in our application. Some ideas: * give the error handler a pretty HTML template, * make debugging easier by writing the stack trace to the HTTP response when the user is an administrator, * write a constructor function for `appError` that stores the stack trace for easier debugging, * recover from panics inside the `appHandler`, logging the error to the console as “Critical,” while telling the user “a serious error has occurred.” This is a nice touch to avoid exposing the user to inscrutable error messages caused by programming errors. See the [Defer, Panic, and Recover](https://go.dev/doc/articles/defer_panic_recover.html) article for more details. Conclusion ---------- Proper error handling is an essential requirement of good software. By employing the techniques described in this post you should be able to write more reliable and succinct Go code. **Next article:** [Go for App Engine is now generally available](https://go.dev/blog/appengine-ga) **Previous article:** [First Class Functions in Go](https://go.dev/blog/functions-codewalk) **[Blog Index](https://go.dev/blog/all) ** go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # talks/2013 - The Go Programming Language Go talks ======== talks/2013 ---------- #### Slide decks: [advconc.slide](https://go.dev/talks/2013/advconc.slide) : Advanced Go Concurrency Patterns [bestpractices.slide](https://go.dev/talks/2013/bestpractices.slide) : Twelve Go Best Practices [distsys.slide](https://go.dev/talks/2013/distsys.slide) : Go, for Distributed Systems [go-sreops.slide](https://go.dev/talks/2013/go-sreops.slide) : Go Language for Ops and Site Reliability Engineering [go1.1.slide](https://go.dev/talks/2013/go1.1.slide) : What's new in Go 1.1 [go4python.slide](https://go.dev/talks/2013/go4python.slide) : Go for Pythonistas [highperf.slide](https://go.dev/talks/2013/highperf.slide) : High Performance Apps with Go on App Engine [oscon-dl.slide](https://go.dev/talks/2013/oscon-dl.slide) : dl.google.com: Powered by Go #### Sub-directories: [advconc](https://go.dev/talks/2013/advconc) [bestpractices](https://go.dev/talks/2013/bestpractices) [distsys](https://go.dev/talks/2013/distsys) [go-sreops](https://go.dev/talks/2013/go-sreops) [go1.1](https://go.dev/talks/2013/go1.1) [go4python](https://go.dev/talks/2013/go4python) [highperf](https://go.dev/talks/2013/highperf) [oscon-dl](https://go.dev/talks/2013/oscon-dl) Opens in new window. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Generic interfaces - The Go Programming Language [The Go Blog](https://go.dev/blog/) ==================================== Generic interfaces ================== Axel Wagner 7 July 2025 There is an idea that is not obvious until you hear about it for the first time: as interfaces are types themselves, they too can have type parameters. This idea proves to be surprisingly powerful when it comes to expressing constraints on generic functions and types. In this post, we’ll demonstrate it, by discussing the use of interfaces with type parameters in a couple of common scenarios. A simple tree set ----------------- As a motivating example, assume we need a generic version of a [binary search tree](https://en.wikipedia.org/wiki/Binary_search_tree) . The elements stored in such a tree need to be ordered, so our type parameter needs a constraint that determines the ordering to use. A simple option is to use the [cmp.Ordered](https://go.dev/pkg/cmp#Ordered) constraint, introduced in Go 1.21. It restricts a type parameter to ordered types (strings and numbers) and allows methods of the type to use the built-in ordering operators. // The zero value of a Tree is a ready-to-use empty tree. type Tree[E cmp.Ordered] struct { root *node[E] } func (t *Tree[E]) Insert(element E) { t.root = t.root.insert(element) } type node[E cmp.Ordered] struct { value E left *node[E] right *node[E] } func (n *node[E]) insert(element E) *node[E] { if n == nil { return &node[E]{value: element} } switch { case element < n.value: n.left = n.left.insert(element) case element > n.value: n.right = n.right.insert(element) } return n } ([playground](https://go.dev/play/p/H7-n33X7P2h) ) However, this approach has the disadvantage that it only works on basic types for which `<` is defined; you cannot insert struct types, like [time.Time](https://go.dev/pkg/time#Time) . We can remedy that by requiring the user to provide a comparison function: // A FuncTree must be created with NewFuncTree. type FuncTree[E any] struct { root *funcNode[E] cmp func(E, E) int } func NewFuncTree[E any](cmp func(E, E) int) *FuncTree[E] { return &FuncTree[E]{cmp: cmp} } func (t *FuncTree[E]) Insert(element E) { t.root = t.root.insert(t.cmp, element) } type funcNode[E any] struct { value E left *funcNode[E] right *funcNode[E] } func (n *funcNode[E]) insert(cmp func(E, E) int, element E) *funcNode[E] { if n == nil { return &funcNode[E]{value: element} } sign := cmp(element, n.value) switch { case sign < 0: n.left = n.left.insert(cmp, element) case sign > 0: n.right = n.right.insert(cmp, element) } return n } ([playground](https://go.dev/play/p/tiEjuxCHtFF) ) This works, but it also comes with downsides. We can no longer use the zero value of our container type, because it needs to have an explicitly initialized comparison function. And the use of a function field makes it harder for the compiler to inline the comparison calls, which can introduce a significant runtime overhead. Using a method on the element type can solve these issues, because methods are directly associated with a type. A method does not have to be explicitly passed and the compiler can see the target of the call and may be able to inline it. But how can we express the constraint to require that element types provide the necessary method? Using the receiver in constraints --------------------------------- The first approach we might try is to define a plain old interface with a `Compare` method: type Comparer interface { Compare(Comparer) int } However, we quickly realize that this does not work well. To implement this interface, the method’s parameter must itself be `Comparer`. Not only does that mean that the implementation of this method must type-assert the parameter to its own type, it also requires that every type must explicitly refer to our package with the `Comparer` type by name (otherwise the method signatures would not be identical). That is not very orthogonal. A better approach is to make the `Comparer` interface itself generic: type Comparer[T any] interface { Compare(T) int } This `Comparer` now describes a whole family of interfaces, one for each type that `Comparer` may be instantiated with. A type that implements `Comparer[T]` declares “I can compare myself to a `T`”. For instance, `time.Time` naturally implements `Comparer[time.Time]` because [it has a matching `Compare` method](https://go.dev/pkg/time#Time.Compare) : // Implements Comparer[Time] func (t Time) Compare(u Time) int This is better, but not enough. What we really want is a constraint that says that a type parameter can be compared to _itself_: we want the constraint to be self-referential. The subtle insight is that the self-referential aspect does not have to be part of the interface definition itself; specifically, the constraint for `T` in the `Comparer` type is just `any`. Instead, it is a consequence of how we use `Comparer` as a constraint for the type parameter of `MethodTree`: // The zero value of a MethodTree is a ready-to-use empty tree. type MethodTree[E Comparer[E]] struct { root *methodNode[E] } func (t *MethodTree[E]) Insert(element E) { t.root = t.root.insert(element) } type methodNode[E Comparer[E]] struct { value E left *methodNode[E] right *methodNode[E] } func (n *methodNode[E]) insert(element E) *methodNode[E] { if n == nil { return &methodNode[E]{value: element} } sign := element.Compare(n.value) switch { case sign < 0: n.left = n.left.insert(element) case sign > 0: n.right = n.right.insert(element) } return n } ([playground](https://go.dev/play/p/LuhzYej_2SP) ) Because `time.Time` implements `Comparer[time.Time]` it is now a valid type argument for this container, and we can still use the zero value as an empty container: var t MethodTree[time.Time] t.Insert(time.Now()) For full flexibility, a library can provide all three API versions. If we want to minimize repetition, all versions could use a shared implementation. We could use the function version for that, as it is the most general: type node[E any] struct { value E left *node[E] right *node[E] } func (n *node[E]) insert(cmp func(E, E) int, element E) *node[E] { if n == nil { return &node[E]{value: element} } sign := cmp(element, n.value) switch { case sign < 0: n.left = n.left.insert(cmp, element) case sign > 0: n.right = n.right.insert(cmp, element) } return n } // Insert inserts element into the tree, if E implements cmp.Ordered. func (t *Tree[E]) Insert(element E) { t.root = t.root.insert(cmp.Compare[E], element) } // Insert inserts element into the tree, using the provided comparison function. func (t *FuncTree[E]) Insert(element E) { t.root = t.root.insert(t.cmp, element) } // Insert inserts element into the tree, if E implements Comparer[E]. func (t *MethodTree[E]) Insert(element E) { t.root = t.root.insert(E.Compare, element) } ([playground](https://go.dev/play/p/jzmoaH5eaIv) ) An important observation here is that the shared implementation (the function-based variant) is not constrained in any way. It must remain maximally flexible to serve as a common core. We also do not store the comparison function in a struct field. Instead, we pass it as a parameter because function arguments are easier for the compiler to analyze than struct fields. There is still some amount of boilerplate involved, of course. All the exported implementations need to replicate the full API with slightly different call patterns. But this part is straightforward to write and to read. Combining methods and type sets ------------------------------- We can use our new tree data structure to implement an ordered set, providing element lookup in logarithmic time. Let’s now imagine we need to make lookup run in constant time; we might try to do this by maintaining an ordinary Go map alongside the tree: type OrderedSet[E Comparer[E]] struct { tree MethodTree[E] // for efficient iteration in order elements map[E]bool // for (near) constant time lookup } func (s *OrderedSet[E]) Has(e E) bool { return s.elements[e] } func (s *OrderedSet[E]) Insert(e E) { if s.elements == nil { s.elements = make(map[E]bool) } if s.elements[e] { return } s.elements[e] = true s.tree.Insert(e) } func (s *OrderedSet[E]) All() iter.Seq[E] { return func(yield func(E) bool) { s.tree.root.all(yield) } } func (n *node[E]) all(yield func(E) bool) bool { return n == nil || (n.left.all(yield) && yield(n.value) && n.right.all(yield)) } ([playground](https://go.dev/play/p/TANUnnSnDqf) ) However, compiling this code will produce an error: > invalid map key type E (missing comparable constraint) The error message tells us that we need to further constrain our type parameter to be able to use it as a map key. The `comparable` constraint is a special predeclared constraint that is satisfied by all types for which the equality operators `==` and `!=` are defined. In Go, that is also the set of types which can be used as keys for the built-in `map` type. We have three options to add this constraint to our type parameter, all with different tradeoffs: 1. We can [embed](https://go.dev/ref/spec#Embedded_interfaces) `comparable` into our original `Comparer` definition ([playground](https://go.dev/play/p/g8NLjZCq97q) ): type Comparer[E any] interface { comparable Compare(E) int } This has the downside that it would also make our `Tree` types only usable with types that are `comparable`. In general, we do not want to unnecessarily restrict generic types. 2. We can add a new constraint definition ([playground](https://go.dev/play/p/Z2eg4X8xK5Z) ). type Comparer[E any] interface { Compare(E) int } type ComparableComparer[E any] interface { comparable Comparer[E] } This is tidy, but it introduces a new identifier (`ComparableComparer`) into our API, and naming is hard. 3. We can add the constraint inline into our more constrained type ([playground](https://go.dev/play/p/ZfggVma_jNc) ): type OrderedSet[E interface {\ comparable\ Comparer[E]\ }] struct { tree Tree[E] elements map[E]struct{} } This can become a bit hard to read, especially if it needs to happen often. It also makes it harder to reuse the constraint in other places. Which of these to use is a style choice and ultimately up to personal preference. (Not) constraining generic interfaces ------------------------------------- At this point it is worth discussing constraints on generic interfaces. You might want to define an interface for a generic container type. For example, say you have an algorithm that requires a set data structure. There are many different kinds of set implementations with different tradeoffs. Defining an interface for the set operations you require can add flexibility to your package, leaving the decision of what tradeoffs are right for the specific application to the user: type Set[E any] interface { Insert(E) Delete(E) Has(E) bool All() iter.Seq[E] } A natural question here is what the constraint on this interface should be. If possible, type parameters on generic interfaces should use `any` as a constraint, allowing arbitrary types. From our discussions above, the reasons should be clear: Different concrete implementations might require different constraints. All the `Tree` types we have examined above, as well as the `OrderedSet` type, can implement `Set` for their element types, even though these types have different constraints. The point of defining an interface is to leave the implementation up to the user. Since one cannot predict what kinds of constraints a user may want to impose on their implementation, try to leave any constraints (stronger than `any`) to concrete implementations, not the interfaces. Pointer receivers ----------------- Let us try to use the `Set` interface in an example. Consider a function that removes duplicate elements in a sequence: // Unique removes duplicate elements from the input sequence, yielding only // the first instance of any element. func Unique[E comparable](input iter.Seq[E]) iter.Seq[E] { return func(yield func(E) bool) { seen := make(map[E]bool) for v := range input { if seen[v] { continue } if !yield(v) { return } seen[v] = true } } } ([playground](https://go.dev/play/p/hsYoFjkU9kA) ) This uses a `map[E]bool` as a simple set of `E` elements. Consequently, it works only for types that are `comparable` and which therefore define the built-in equality operators. If we want to generalize this to arbitrary types, we need to replace that with a generic set: // Unique removes duplicate elements from the input sequence, yielding only // the first instance of any element. func Unique[E any](input iter.Seq[E]) iter.Seq[E] { return func(yield func(E) bool) { var seen Set[E] for v := range input { if seen.Has(v) { continue } if !yield(v) { return } seen.Insert(v) } } } ([playground](https://go.dev/play/p/FZYPNf56nnY) ) However, this does not work. `Set[E]` is an interface type, and the `seen` variable will be initialized to `nil`. We need to use a concrete implementation of the `Set[E]` interface. But as we have seen in this post, there is no general implementation of a set that works for `any` element type. We have to ask the user to provide a concrete implementation we can use, as an extra type parameter: // Unique removes duplicate elements from the input sequence, yielding only // the first instance of any element. func Unique[E any, S Set[E]](input iter.Seq[E]) iter.Seq[E] { return func(yield func(E) bool) { var seen S for v := range input { if seen.Has(v) { continue } if !yield(v) { return } seen.Insert(v) } } } ([playground](https://go.dev/play/p/kjkGy5cNz8T) ) However, if we instantiate this with our set implementation, we run into another problem: // OrderedSet[E] does not satisfy Set[E] (method All has pointer receiver) Unique[E, OrderedSet[E]](slices.Values(s)) // panic: invalid memory address or nil pointer dereference Unique[E, *OrderedSet[E]](slices.Values(s)) The first problem is clear from the error message: Our type constraint says that the type argument for `S` needs to implement the `Set[E]` interface. And as the methods on `OrderedSet` use a pointer receiver, the type argument also has to be the pointer type. When trying to do that, we run into the second problem. This stems from the fact that we declare a variable in the implementation: var seen S If `S` is `*OrderedSet[E]`, the variable is initialized with `nil`, as before. Calling `seen.Insert` panics. If we only have the pointer type, we cannot get a valid variable of the value type. And if we only have the value type, we cannot call pointer-methods on it. The consequence is that we need both the value _and_ the pointer type. So we have to introduce an additional type parameter `PS` with a new constraint `PtrToSet`: // PtrToSet is implemented by a pointer type implementing the Set[E] interface. type PtrToSet[S, E any] interface { *S Set[E] } // Unique removes duplicate elements from the input sequence, yielding only // the first instance of any element. func Unique[E, S any, PS PtrToSet[S, E]](input iter.Seq[E]) iter.Seq[E] { return func(yield func(E) bool) { // We convert to PS, as only that is constrained to have the methods. // The conversion is allowed, because the type set of PS only contains *S. seen := PS(new(S)) for v := range input { if seen.Has(v) { continue } if !yield(v) { return } seen.Insert(v) } } } ([playground](https://go.dev/play/p/Kp1jJRVjmYa) ) The trick here is the connection of the two type parameters in the function signature via the extra type parameter on the `PtrToSet` interface. `S` itself is unconstrained, but `PS` must have type `*S` and it must have the methods we need. So effectively, we are restricting `S` to have some methods, but those methods need to use a pointer receiver. While the definition of a function with this kind of constraint requires an additional type parameter, importantly this does not complicate code using it: as long as this extra type parameter is at the end of the type parameter list, it [can be inferred](https://go.dev/blog/type-inference) : // The third type argument is inferred to be *OrderedSet[int] Unique[int, OrderedSet[int]](slices.Values(s)) This is a general pattern, and worth remembering: for when you encounter it in someone else’s work, or when you want to use it in your own. func SomeFunction[T any, PT interface{ *T; SomeMethods }]() If you have two type parameters, where one is constrained to be a pointer to the other, the constraint ensures that the relevant methods use a pointer receiver. Should you constrain to pointer receivers? ------------------------------------------ At this point, you might feel pretty overwhelmed. This is rather complicated and it seems unreasonable to expect every Go programmer to understand what is going on in this function signature. We also had to introduce yet more names into our API. When people cautioned against adding generics to Go in the first place, this is one of the things they were worried about. So if you find yourself entangled in these problems, it is worth taking a step back. We can often avoid this complexity by thinking about our problem in a different way. In this example, we built a function that takes an `iter.Seq[E]` and returns an `iter.Seq[E]` with the unique elements. But to do the deduplication, we needed to collect the unique elements into a set. And as this requires us to allocate the space for the entire result, we do not really benefit from representing the result as a stream. If we rethink this problem, we can avoid the extra type parameter altogether by using `Set[E]` as a regular interface value: // InsertAll adds all unique elements from seq into set. func InsertAll[E any](set Set[E], seq iter.Seq[E]) { for v := range seq { set.Insert(v) } } ([playground](https://go.dev/play/p/woZcHodAgaa) ) By using `Set` as a plain interface type, it is clear that the caller has to provide a valid value of their concrete implementation. This is a very common pattern. And if they need an `iter.Seq[E]`, they can simply call `All()` on the `set` to obtain one. This complicates things for callers slightly, but it has another advantage over the constraint to pointer receivers: remember that we started with a `map[E]bool` as a simple set type. It is easy to implement the `Set[E]` interface on that basis: type HashSet[E comparable] map[E]bool func (s HashSet[E]) Insert(v E) { s[v] = true } func (s HashSet[E]) Delete(v E) { delete(s, v) } func (s HashSet[E]) Has(v E) bool { return s[v] } func (s HashSet[E]) All() iter.Seq[E] { return maps.Keys(s) } ([playground](https://go.dev/play/p/KPPpWa7M93d) ) This implementation does not use pointer receivers. So while this is perfectly valid, it would not be usable with the complicated constraint to pointer receivers. But it works fine with our `InsertAll` version. As with many constraints, enforcing that methods use a pointer receiver might actually be overly restrictive for many practical use cases. Conclusion ---------- I hope this illustrates some of the patterns and trade-offs that type parameters on interfaces enable. It is a powerful tool, but it also comes with a cost. The primary take-aways are: 1. Use generic interfaces to express constraints on the receiver by using them self-referentially. 2. Use them to create constrained relationships between different type parameters. 3. Use them to abstract over different implementations with different kinds of constraints. 4. When you find yourself in a situation where you need to constrain to pointer receivers, consider whether you can refactor your code to avoid the extra complexity. See [“Should you constrain to pointer receivers?”](https://go.dev/blog/generic-interfaces#should-you-constrain-to-pointer-receivers) . As always, do not over-engineer things: a less flexible but simpler and more readable solution may ultimately be the wiser choice. **Next article:** [The FIPS 140-3 Go Cryptographic Module](https://go.dev/blog/fips140) **Previous article:** [\[ On | No \] syntactic support for error handling](https://go.dev/blog/error-syntax) **[Blog Index](https://go.dev/blog/all) ** go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Introducing the Go Race Detector - The Go Programming Language [The Go Blog](https://go.dev/blog/) ==================================== Introducing the Go Race Detector ================================ Dmitry Vyukov and Andrew Gerrand 26 June 2013 Introduction ------------ [Race conditions](http://en.wikipedia.org/wiki/Race_condition)  are among the most insidious and elusive programming errors. They typically cause erratic and mysterious failures, often long after the code has been deployed to production. While Go’s concurrency mechanisms make it easy to write clean concurrent code, they don’t prevent race conditions. Care, diligence, and testing are required. And tools can help. We’re happy to announce that Go 1.1 includes a [race detector](https://go.dev/doc/articles/race_detector.html) , a new tool for finding race conditions in Go code. It is currently available for Linux, OS X, and Windows systems with 64-bit x86 processors. The race detector is based on the C/C++ [ThreadSanitizer runtime library](https://github.com/google/sanitizers) , which has been used to detect many errors in Google’s internal code base and in [Chromium](http://www.chromium.org/) . The technology was integrated with Go in September 2012; since then it has detected [42 races](https://github.com/golang/go/issues?utf8=%E2%9C%93&q=ThreadSanitizer) in the standard library. It is now part of our continuous build process, where it continues to catch race conditions as they arise. How it works ------------ The race detector is integrated with the go tool chain. When the `-race` command-line flag is set, the compiler instruments all memory accesses with code that records when and how the memory was accessed, while the runtime library watches for unsynchronized accesses to shared variables. When such “racy” behavior is detected, a warning is printed. (See [this article](https://github.com/google/sanitizers/wiki/ThreadSanitizerAlgorithm) for the details of the algorithm.) Because of its design, the race detector can detect race conditions only when they are actually triggered by running code, which means it’s important to run race-enabled binaries under realistic workloads. However, race-enabled binaries can use ten times the CPU and memory, so it is impractical to enable the race detector all the time. One way out of this dilemma is to run some tests with the race detector enabled. Load tests and integration tests are good candidates, since they tend to exercise concurrent parts of the code. Another approach using production workloads is to deploy a single race-enabled instance within a pool of running servers. Using the race detector ----------------------- The race detector is fully integrated with the Go tool chain. To build your code with the race detector enabled, just add the `-race` flag to the command line: $ go test -race mypkg    // test the package $ go run -race mysrc.go  // compile and run the program $ go build -race mycmd   // build the command $ go install -race mypkg // install the package To try out the race detector for yourself, copy this example program into `racy.go`: package main import "fmt" func main() { done := make(chan bool) m := make(map[string]string) m["name"] = "world" go func() { m["name"] = "data race" done <- true }() fmt.Println("Hello,", m["name"]) <-done } Then run it with the race detector enabled: $ go run -race racy.go Examples -------- Here are two examples of real issues caught by the race detector. ### Example 1: Timer.Reset The first example is a simplified version of an actual bug found by the race detector. It uses a timer to print a message after a random duration between 0 and 1 second. It does so repeatedly for five seconds. It uses [`time.AfterFunc`](https://go.dev/pkg/time/#AfterFunc) to create a [`Timer`](https://go.dev/pkg/time/#Timer) for the first message and then uses the [`Reset`](https://go.dev/pkg/time/#Timer.Reset) method to schedule the next message, re-using the `Timer` each time. package main import ( "fmt" "math/rand" "time" ) 10  func main() { 11   start := time.Now() 12   var t \*time.Timer 13   t = time.AfterFunc(randomDuration(), func() { 14   fmt.Println(time.Now().Sub(start)) 15   t.Reset(randomDuration()) 16   }) 17   time.Sleep(5 \* time.Second) 18  } 19   20  func randomDuration() time.Duration { 21   return time.Duration(rand.Int63n(1e9)) 22  } 23   This looks like reasonable code, but under certain circumstances it fails in a surprising way: panic: runtime error: invalid memory address or nil pointer dereference [signal 0xb code=0x1 addr=0x8 pc=0x41e38a] goroutine 4 [running]: time.stopTimer(0x8, 0x12fe6b35d9472d96) src/pkg/runtime/ztime_linux_amd64.c:35 +0x25 time.(*Timer).Reset(0x0, 0x4e5904f, 0x1) src/pkg/time/sleep.go:81 +0x42 main.func·001() race.go:14 +0xe3 created by time.goFunc src/pkg/time/sleep.go:122 +0x48 What’s going on here? Running the program with the race detector enabled is more illuminating: ================== WARNING: DATA RACE Read by goroutine 5: main.func·001() race.go:16 +0x169 Previous write by goroutine 1: main.main() race.go:14 +0x174 Goroutine 5 (running) created at: time.goFunc() src/pkg/time/sleep.go:122 +0x56 timerproc() src/pkg/runtime/ztime_linux_amd64.c:181 +0x189 ================== The race detector shows the problem: an unsynchronized read and write of the variable `t` from different goroutines. If the initial timer duration is very small, the timer function may fire before the main goroutine has assigned a value to `t` and so the call to `t.Reset` is made with a nil `t`. To fix the race condition we change the code to read and write the variable `t` only from the main goroutine: package main import ( "fmt" "math/rand" "time" ) 10  func main() { 11   start := time.Now() 12   reset := make(chan bool) 13   var t \*time.Timer 14   t = time.AfterFunc(randomDuration(), func() { 15   fmt.Println(time.Now().Sub(start)) 16   reset <- true 17   }) 18   for time.Since(start) < 5\*time.Second { 19   <-reset 20   t.Reset(randomDuration()) 21   } 22  } 23   func randomDuration() time.Duration { return time.Duration(rand.Int63n(1e9)) } Here the main goroutine is wholly responsible for setting and resetting the `Timer` `t` and a new reset channel communicates the need to reset the timer in a thread-safe way. A simpler but less efficient approach is to [avoid reusing timers](https://go.dev/play/p/kuWTrY0pS4) . ### Example 2: ioutil.Discard The second example is more subtle. The `ioutil` package’s [`Discard`](https://go.dev/pkg/io/ioutil/#Discard) object implements [`io.Writer`](https://go.dev/pkg/io/#Writer) , but discards all the data written to it. Think of it like `/dev/null`: a place to send data that you need to read but don’t want to store. It is commonly used with [`io.Copy`](https://go.dev/pkg/io/#Copy) to drain a reader, like this: io.Copy(ioutil.Discard, reader) Back in July 2011 the Go team noticed that using `Discard` in this way was inefficient: the `Copy` function allocates an internal 32 kB buffer each time it is called, but when used with `Discard` the buffer is unnecessary since we’re just throwing the read data away. We thought that this idiomatic use of `Copy` and `Discard` should not be so costly. The fix was simple. If the given `Writer` implements a `ReadFrom` method, a `Copy` call like this: io.Copy(writer, reader) is delegated to this potentially more efficient call: writer.ReadFrom(reader) We [added a ReadFrom method](https://go.dev/cl/4817041) to Discard’s underlying type, which has an internal buffer that is shared between all its users. We knew this was theoretically a race condition, but since all writes to the buffer should be thrown away we didn’t think it was important. When the race detector was implemented it immediately [flagged this code](https://go.dev/issue/3970) as racy. Again, we considered that the code might be problematic, but decided that the race condition wasn’t “real”. To avoid the “false positive” in our build we implemented [a non-racy version](https://go.dev/cl/6624059) that is enabled only when the race detector is running. But a few months later [Brad](https://bradfitz.com/) encountered a [frustrating and strange bug](https://go.dev/issue/4589) . After a few days of debugging, he narrowed it down to a real race condition caused by `ioutil.Discard`. Here is the known-racy code in `io/ioutil`, where `Discard` is a `devNull` that shares a single buffer between all of its users. var blackHole \[4096\]byte // shared buffer func (devNull) ReadFrom(r io.Reader) (n int64, err error) { readSize := 0 for { readSize, err = r.Read(blackHole\[:\]) n += int64(readSize) if err != nil { if err == io.EOF { return n, nil } return } } } Brad’s program includes a `trackDigestReader` type, which wraps an `io.Reader` and records the hash digest of what it reads. type trackDigestReader struct { r io.Reader h hash.Hash } func (t trackDigestReader) Read(p []byte) (n int, err error) { n, err = t.r.Read(p) t.h.Write(p[:n]) return } For example, it could be used to compute the SHA-1 hash of a file while reading it: tdr := trackDigestReader{r: file, h: sha1.New()} io.Copy(writer, tdr) fmt.Printf("File hash: %x", tdr.h.Sum(nil)) In some cases there would be nowhere to write the data—but still a need to hash the file—and so `Discard` would be used: io.Copy(ioutil.Discard, tdr) But in this case the `blackHole` buffer isn’t just a black hole; it is a legitimate place to store the data between reading it from the source `io.Reader` and writing it to the `hash.Hash`. With multiple goroutines hashing files simultaneously, each sharing the same `blackHole` buffer, the race condition manifested itself by corrupting the data between reading and hashing. No errors or panics occurred, but the hashes were wrong. Nasty! func (t trackDigestReader) Read(p []byte) (n int, err error) { // the buffer p is blackHole n, err = t.r.Read(p) // p may be corrupted by another goroutine here, // between the Read above and the Write below t.h.Write(p[:n]) return } The bug was finally [fixed](https://go.dev/cl/7011047) by giving a unique buffer to each use of `ioutil.Discard`, eliminating the race condition on the shared buffer. Conclusions ----------- The race detector is a powerful tool for checking the correctness of concurrent programs. It will not issue false positives, so take its warnings seriously. But it is only as good as your tests; you must make sure they thoroughly exercise the concurrent properties of your code so that the race detector can do its job. What are you waiting for? Run `"go test -race"` on your code today! **Next article:** [The first Go program](https://go.dev/blog/first-go-program) **Previous article:** [Go and the Google Cloud Platform](https://go.dev/blog/io2013-talks-cloud) **[Blog Index](https://go.dev/blog/all) ** go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Go Fuzzing - The Go Programming Language 1. [Documentation](https://go.dev/doc/) 2. [Security](https://go.dev/doc/security/) 3. [Go Fuzzing](https://go.dev/doc/security/fuzz/) Go Fuzzing ========== Go supports fuzzing in its standard toolchain beginning in Go 1.18. Native Go fuzz tests are [supported by OSS-Fuzz](https://google.github.io/oss-fuzz/getting-started/new-project-guide/go-lang/#native-go-fuzzing-support) . **Try out the [tutorial for fuzzing with Go](https://go.dev/doc/tutorial/fuzz) .** Overview -------- Fuzzing is a type of automated testing which continuously manipulates inputs to a program to find bugs. Go fuzzing uses coverage guidance to intelligently walk through the code being fuzzed to find and report failures to the user. Since it can reach edge cases which humans often miss, fuzz testing can be particularly valuable for finding security exploits and vulnerabilities. Below is an example of a [fuzz test](https://go.dev/doc/security/fuzz/#glos-fuzz-test) , highlighting its main components. ![Example code showing the overall fuzz test, with a fuzz target within\ it. Before the fuzz target is a corpus addition with f.Add, and the parameters\ of the fuzz target are highlighted as the fuzzing arguments.](https://go.dev/security/fuzz/example-dark.png) ![Example code showing the overall fuzz test, with a fuzz target within\ it. Before the fuzz target is a corpus addition with f.Add, and the parameters\ of the fuzz target are highlighted as the fuzzing arguments.](https://go.dev/security/fuzz/example.png) Writing fuzz tests ------------------ ### Requirements Below are rules that fuzz tests must follow. * A fuzz test must be a function named like `FuzzXxx`, which accepts only a `*testing.F`, and has no return value. * Fuzz tests must be in \*\_test.go files to run. * A [fuzz target](https://go.dev/doc/security/fuzz/#glos-fuzz-target) must be a method call to `[(*testing.F).Fuzz](https://pkg.go.dev/testing#F.Fuzz) ` which accepts a `*testing.T` as the first parameter, followed by the fuzzing arguments. There is no return value. * There must be exactly one fuzz target per fuzz test. * All [seed corpus](https://go.dev/doc/security/fuzz/#glos-seed-corpus) entries must have types which are identical to the [fuzzing arguments](https://go.dev/doc/security/fuzz/#glos-fuzzing-arguments) , in the same order. This is true for calls to `[(*testing.F).Add](https://pkg.go.dev/testing#F.Add) ` and any corpus files in the testdata/fuzz directory of the fuzz test. * The fuzzing arguments can only be the following types: * `string`, `[]byte` * `int`, `int8`, `int16`, `int32`/`rune`, `int64` * `uint`, `uint8`/`byte`, `uint16`, `uint32`, `uint64` * `float32`, `float64` * `bool` ### Suggestions Below are suggestions that will help you get the most out of fuzzing. * Fuzz targets should be fast and deterministic so the fuzzing engine can work efficiently, and new failures and code coverage can be easily reproduced. * Since the fuzz target is invoked in parallel across multiple workers and in nondeterministic order, the state of a fuzz target should not persist past the end of each call, and the behavior of a fuzz target should not depend on global state. Running fuzz tests ------------------ There are two modes of running your fuzz test: as a unit test (default `go test`), or with fuzzing (`go test -fuzz=FuzzTestName`). Fuzz tests are run much like a unit test by default. Each [seed corpus entry](https://go.dev/doc/security/fuzz/#glos-seed-corpus) will be tested against the fuzz target, reporting any failures before exiting. To enable fuzzing, run `go test` with the `-fuzz` flag, providing a regex matching a single fuzz test. By default, all other tests in that package will run before fuzzing begins. This is to ensure that fuzzing won’t report any issues that would already be caught by an existing test. Note that it is up to you to decide how long to run fuzzing. It is very possible that an execution of fuzzing could run indefinitely if it doesn’t find any errors. There will be support to run these fuzz tests continuously using tools like OSS-Fuzz in the future, see [Issue #50192](https://go.dev/issue/50192) . **Note:** Fuzzing should be run on a platform that supports coverage instrumentation (currently AMD64 and ARM64) so that the corpus can meaningfully grow as it runs, and more code can be covered while fuzzing. ### Command line output While fuzzing is in progress, the [fuzzing engine](https://go.dev/doc/security/fuzz/#glos-fuzzing-engine) generates new inputs and runs them against the provided fuzz target. By default, it continues to run until a [failing input](https://go.dev/doc/security/fuzz/#glos-failing-input) is found, or the user cancels the process (e.g. with Ctrl^C). The output will look something like this: ~ go test -fuzz FuzzFoo fuzz: elapsed: 0s, gathering baseline coverage: 0/192 completed fuzz: elapsed: 0s, gathering baseline coverage: 192/192 completed, now fuzzing with 8 workers fuzz: elapsed: 3s, execs: 325017 (108336/sec), new interesting: 11 (total: 202) fuzz: elapsed: 6s, execs: 680218 (118402/sec), new interesting: 12 (total: 203) fuzz: elapsed: 9s, execs: 1039901 (119895/sec), new interesting: 19 (total: 210) fuzz: elapsed: 12s, execs: 1386684 (115594/sec), new interesting: 21 (total: 212) PASS ok foo 12.692s The first lines indicate that the “baseline coverage” is gathered before fuzzing begins. To gather baseline coverage, the fuzzing engine executes both the [seed corpus](https://go.dev/doc/security/fuzz/#glos-seed-corpus) and the [generated corpus](https://go.dev/doc/security/fuzz/#glos-generated-corpus) , to ensure that no errors occurred and to understand the code coverage the existing corpus already provides. The lines following provide insight into the active fuzzing execution: * elapsed: the amount of time that has elapsed since the process began * execs: the total number of inputs that have been run against the fuzz target (with an average execs/sec since the last log line) * new interesting: the total number of “interesting” inputs that have been added to the generated corpus during this fuzzing execution (with the total size of the entire corpus) For an input to be “interesting”, it must expand the code coverage beyond what the existing generated corpus can reach. It’s typical for the number of new interesting inputs to grow quickly at the start and eventually slow down, with occasional bursts as new branches are discovered. You should expect to see the “new interesting” number taper off over time as the inputs in the corpus begin to cover more lines of the code, with occasional bursts if the fuzzing engine finds a new code path. ### Failing input A failure may occur while fuzzing for several reasons: * A panic occurred in the code or the test. * The fuzz target called `t.Fail`, either directly or through methods such as `t.Error` or `t.Fatal`. * A non-recoverable error occurred, such as an `os.Exit` or stack overflow. * The fuzz target took too long to complete. Currently, the timeout for an execution of a fuzz target is 1 second. This may fail due to a deadlock or infinite loop, or from intended behavior in the code. This is one reason why it is [suggested that your fuzz target be fast](https://go.dev/doc/security/fuzz/#suggestions) . If an error occurs, the fuzzing engine will attempt to minimize the input to the smallest possible and most human readable value which will still produce an error. To configure this, see the [custom settings](https://go.dev/doc/security/fuzz/#custom-settings) section. Once minimization is complete, the error message will be logged, and the output will end with something like this: Failing input written to testdata/fuzz/FuzzFoo/a878c3134fe0404d44eb1e662e5d8d4a24beb05c3d68354903670ff65513ff49 To re-run: go test -run=FuzzFoo/a878c3134fe0404d44eb1e662e5d8d4a24beb05c3d68354903670ff65513ff49 FAIL exit status 1 FAIL foo 0.839s The fuzzing engine wrote this [failing input](https://go.dev/doc/security/fuzz/#glos-failing-input) to the seed corpus for that fuzz test, and it will now be run by default with `go test`, serving as a regression test once the bug has been fixed. The next step for you will be to diagnose the problem, fix the bug, verify the fix by re-running `go test`, and submit the patch with the new testdata file acting as your regression test. ### Custom settings The default go command settings should work for most use cases of fuzzing. So typically, an execution of fuzzing on the command line should look like this: $ go test -fuzz={FuzzTestName} However, the `go` command does provide a few settings when running fuzzing. These are documented in the [`cmd/go` package docs](https://pkg.go.dev/cmd/go) . To highlight a few: * `-fuzztime`: the total time or number of iterations that the fuzz target will be executed before exiting, default indefinitely. * `-fuzzminimizetime`: the time or number of iterations that the fuzz target will be executed during each minimization attempt, default 60sec. You can completely disable minimization by setting `-fuzzminimizetime 0` when fuzzing. * `-parallel`: the number of fuzzing processes running at once, default `$GOMAXPROCS`. Currently, setting -cpu during fuzzing has no effect. Corpus file format ------------------ Corpus files are encoded in a special format. This is the same format for both the [seed corpus](https://go.dev/doc/security/fuzz/#glos-seed-corpus) , and the [generated corpus](https://go.dev/doc/security/fuzz/#glos-generated-corpus) . Below is an example of a corpus file: go test fuzz v1 []byte("hello\\xbd\\xb2=\\xbc ⌘") int64(572293) The first line is used to inform the fuzzing engine of the file’s encoding version. Although no future versions of the encoding format are currently planned, the design must support this possibility. Each of the lines following are the values that make up the corpus entry, and can be copied directly into Go code if desired. In the example above, we have a `[]byte` followed by an `int64`. These types must match the fuzzing arguments exactly, in that order. A fuzz target for these types would look like this: f.Fuzz(func(*testing.T, []byte, int64) {}) The easiest way to specify your own seed corpus values is to use the `(*testing.F).Add` method. In the example above, that would look like this: f.Add([]byte("hello\\xbd\\xb2=\\xbc ⌘"), int64(572293)) However, you may have large binary files that you’d prefer not to copy as code into your test, and instead remain as individual seed corpus entries in the testdata/fuzz/{FuzzTestName} directory. The [`file2fuzz`](https://pkg.go.dev/golang.org/x/tools/cmd/file2fuzz) tool at golang.org/x/tools/cmd/file2fuzz can be used to convert these binary files to corpus files encoded for `[]byte`. To use this tool: $ go install golang.org/x/tools/cmd/file2fuzz@latest $ file2fuzz -h Resources --------- * **Tutorial**: * Try out the [tutorial for fuzzing with Go](https://go.dev/doc/tutorial/fuzz) for a deep dive into the new concepts. * For a shorter, introductory tutorial of fuzzing with Go, please see [the blog post](https://go.dev/blog/fuzz-beta) . * **Documentation**: * The [`testing`](https://pkg.go.dev//testing#hdr-Fuzzing) package docs describes the `testing.F` type which is used when writing fuzz tests. * The [`cmd/go`](https://pkg.go.dev/cmd/go) package docs describe the flags associated with fuzzing. * **Technical details**: * [Design draft](https://go.dev/s/draft-fuzzing-design) * [Proposal](https://go.dev/issue/44551) Glossary -------- **corpus entry:** An input in the corpus which can be used while fuzzing. This can be a specially-formatted file, or a call to `[(*testing.F).Add](https://pkg.go.dev/testing#F.Add) `. **coverage guidance:** A method of fuzzing which uses expansions in code coverage to determine which corpus entries are worth keeping for future use. **failing input:** A failing input is a corpus entry that will cause an error or panic when run against the [fuzz target](https://go.dev/doc/security/fuzz/#glos-fuzz-target) . **fuzz target:** The function of the fuzz test which is executed for corpus entries and generated values while fuzzing. It is provided to the fuzz test by passing the function to `[(*testing.F).Fuzz](https://pkg.go.dev/testing#F.Fuzz) `. **fuzz test:** A function in a test file of the form `func FuzzXxx(*testing.F)` which can be used for fuzzing. **fuzzing:** A type of automated testing which continuously manipulates inputs to a program to find issues such as bugs or [vulnerabilities](https://go.dev/doc/security/fuzz/#glos-vulnerability) to which the code may be susceptible. **fuzzing arguments:** The types which will be passed to the fuzz target, and mutated by the [mutator](https://go.dev/doc/security/fuzz/#glos-mutator) . **fuzzing engine:** A tool that manages fuzzing, including maintaining the corpus, invoking the mutator, identifying new coverage, and reporting failures. **generated corpus:** A corpus which is maintained by the fuzzing engine over time while fuzzing to keep track of progress. It is stored in `$GOCACHE`/fuzz. These entries are only used while fuzzing. **mutator:** A tool used while fuzzing which randomly manipulates corpus entries before passing them to a fuzz target. **package:** A collection of source files in the same directory that are compiled together. See the [Packages section](https://go.dev/ref/spec#Packages) in the Go Language Specification. **seed corpus:** A user-provided corpus for a fuzz test which can be used to guide the fuzzing engine. It is composed of the corpus entries provided by f.Add calls within the fuzz test, and the files in the testdata/fuzz/{FuzzTestName} directory within the package. These entries are run by default with `go test`, whether fuzzing or not. **test file:** A file of the format xxx\_test.go that may contain tests, benchmarks, examples and fuzz tests. **vulnerability:** A security-sensitive weakness in code which can be exploited by an attacker. Feedback -------- If you experience any problems or have an idea for a feature, please [file an issue](https://go.dev/issue/new?&labels=fuzz) . For discussion and general feedback about the feature, you can also participate in the [#fuzzing channel](https://gophers.slack.com/archives/CH5KV1AKE) in Gophers Slack. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # A new experimental Go API for JSON - The Go Programming Language [The Go Blog](https://go.dev/blog/) ==================================== A new experimental Go API for JSON ================================== Joe Tsai, Daniel Martí, Johan Brandhorst-Satzkorn, Roger Peppe, Chris Hines, and Damien Neil 9 September 2025 Introduction ------------ [JavaScript Object Notation (JSON)](https://datatracker.ietf.org/doc/html/rfc8259) is a simple data interchange format. Almost 15 years ago, we wrote about [support for JSON in Go](https://go.dev/blog/json) , which introduced the ability to serialize and deserialize Go types to and from JSON data. Since then, JSON has become the most popular data format used on the Internet. It is widely read and written by Go programs, and encoding/json now ranks as the 5th most imported Go package. Over time, packages evolve with the needs of their users, and `encoding/json` is no exception. This blog post is about Go 1.25’s new experimental `encoding/json/v2` and `encoding/json/jsontext` packages, which bring long-awaited improvements and fixes. This post argues for a new major API version, provides an overview of the new packages, and explains how you can make use of it. The experimental packages are not visible by default and may undergo future API changes. Problems with `encoding/json` ----------------------------- Overall, `encoding/json` has held up well. The idea of marshaling and unmarshaling arbitrary Go types with some default representation in JSON, combined with the ability to customize the representation, has proven to be highly flexible. However, in the years since its introduction, various users have identified numerous shortcomings. ### Behavior flaws There are various behavioral flaws in `encoding/json`: * **Imprecise handling of JSON syntax**: Over the years, JSON has seen increased standardization in order for programs to properly communicate. Generally, decoders have become stricter at rejecting ambiguous inputs, to reduce the chance that two implementations will have different (successful) interpretations of a particular JSON value. * `encoding/json` currently accepts invalid UTF-8, whereas the latest Internet Standard (RFC 8259) requires valid UTF-8. The default behavior should report an error in the presence of invalid UTF-8, instead of introducing silent data corruption, which may cause problems downstream. * `encoding/json` currently accepts objects with duplicate member names. RFC 8259 does not specify how to handle duplicate names, so an implementation is free to choose an arbitrary value, merge the values, discard the values, or report an error. The presence of a duplicate name results in a JSON value without a universally agreed upon meaning. This could be [exploited by attackers in security applications](https://www.youtube.com/watch?v=avilmOcHKHE&t=1057s) and has been exploited before (as in [CVE-2017-12635](https://nvd.nist.gov/vuln/detail/CVE-2017-12635) ). The default behavior should err on the side of safety and reject duplicate names. * **Leaking nilness of slices and maps**: JSON is often used to communicate with programs using JSON implementations that do not allow `null` to be unmarshaled into a data type expected to be a JSON array or object. Since `encoding/json` marshals a nil slice or map as a JSON `null`, this may lead to errors when unmarshaling by other implementations. [A survey](https://go.dev/issue/63397#discussioncomment-7201222) indicated that most Go users prefer that nil slices and maps are marshaled as an empty JSON array or object by default. * **Case-insensitive unmarshaling**: When unmarshaling, a JSON object member name is resolved to a Go struct field name using a case-insensitive match. This is a surprising default, a potential security vulnerability, and a performance limitation. * **Inconsistent calling of methods**: Due to an implementation detail, `MarshalJSON` methods declared on a pointer receiver are [inconsistently called by `encoding/json`](https://go.dev/issue/22967) . While regarded as a bug, this cannot be fixed as too many applications depend on the current behavior. ### API deficiencies The API of `encoding/json` can be tricky or restrictive: * It is difficult to correctly unmarshal from an `io.Reader`. Users often write `json.NewDecoder(r).Decode(v)`, which fails to reject trailing junk at the end of the input. * Options can be set on the `Encoder` and `Decoder` types, but cannot be used with the `Marshal` and `Unmarshal` functions. Similarly, types implementing the `Marshaler` and `Unmarshaler` interfaces cannot make use of the options and there is no way to plumb options down the call stack. For example, the `Decoder.DisallowUnknownFields` option loses its effect when calling a custom `UnmarshalJSON` method. * The `Compact`, `Indent`, and `HTMLEscape` functions write to a `bytes.Buffer` instead of something more flexible like a `[]byte` or `io.Writer`. This limits the usability of those functions. ### Performance limitations Setting aside internal implementation details, the public API commits it to certain performance limitations: * **MarshalJSON**: The `MarshalJSON` interface method forces the implementation to allocate the returned `[]byte`. Also, the semantics require that `encoding/json` verify that the result is valid JSON and also to reformat it to match the specified indentation. * **UnmarshalJSON**: The `UnmarshalJSON` interface method requires that a complete JSON value be provided (without any trailing data). This forces `encoding/json` to parse the JSON value to be unmarshaled in its entirety to determine where it ends before it can call `UnmarshalJSON`. Afterwards, the `UnmarshalJSON` method itself must parse the provided JSON value again. * **Lack of streaming**: Even though the `Encoder` and `Decoder` types operate on an `io.Writer` or `io.Reader`, they buffer the entire JSON value in memory. The `Decoder.Token` method for reading individual tokens is allocation-heavy and there is no corresponding API for writing tokens. Furthermore, if the implementation of a `MarshalJSON` or `UnmarshalJSON` method recursively calls the `Marshal` or `Unmarshal` function, then the performance becomes quadratic. Trying to fix `encoding/json` directly -------------------------------------- Introducing a new, incompatible major version of a package is a heavy consideration. If possible, we should try to fix the existing package. While it is relatively easy to add new features, it is difficult to change existing features. Unfortunately, these problems are inherent consequences of the existing API, making them practically impossible to fix within the [Go 1 compatibility promise](https://go.dev/doc/go1compat) . We could in principle declare separate names, such as `MarshalV2` or `UnmarshalV2`, but that is tantamount to creating a parallel namespace within the same package. This leads us to `encoding/json/v2` (henceforth called `v2`), where we can make these changes within a separate `v2` namespace in contrast to `encoding/json` (henceforth called `v1`). Planning for `encoding/json/v2` ------------------------------- The planning for a new major version of `encoding/json` spanned years. In late 2020, spurred on by the inability to fix issues in the current package, Daniel Martí (one of the maintainers of `encoding/json`) first drafted his thoughts on [what a hypothetical `v2` package should look like](https://docs.google.com/document/d/1WQGoM44HLinH4NGBEv5drGlw5_RNW-GP7DdGEpm7Y3o) . Separately, after previous work on the [Go API for Protocol Buffers](https://go.dev/blog/protobuf-apiv2) , Joe Tsai was disapppointed that [the `protojson` package](https://go.dev/pkg/google.golang.org/protobuf/encoding/protojson) needed to use a custom JSON implementation because `encoding/json` was neither capable of adhering to the stricter JSON standard that the Protocol Buffer specification required, nor of efficiently serializing JSON in a streaming manner. Believing a brighter future for JSON was both beneficial and achievable, Daniel and Joe joined forces to brainstorm on `v2` and [started to build a prototype](https://github.com/go-json-experiment/json) (with the initial code being a polished version of the JSON serialization logic from the Go protobuf module). Over time, a few others (Roger Peppe, Chris Hines, Johan Brandhorst-Satzkorn, and Damien Neil) joined the effort by providing design review, code review, and regression testing. Many of the early discussions are publicly available in our [recorded meetings](https://www.youtube.com/playlist?list=PLZgrQPcV8W8EChkaAvv-3NUu6PYmnGG3b) and [meeting notes](https://docs.google.com/document/d/1rovrOTd-wTawGMPPlPuKhwXaYBg9VszTXR9AQQL5LfI) . This work has been public since the beginning, and we increasingly involved the wider Go community, first with a [GopherCon talk](https://www.youtube.com/watch?v=avilmOcHKHE) and [discussion posted in late 2023](https://go.dev/issue/63397) , [formal proposal posted in early 2025](https://go.dev/issue/71497) , and most recently [adopting `encoding/json/v2` as a Go experiment](https://go.dev/issue/71845) (available in Go 1.25) for wider-scale testing by all Go users. The `v2` effort has been going on for 5 years, incorporating feedback from many contributors and also gaining valuable empirical experience from use in production settings. It’s worth noting that it’s largely been developed and promoted by people not employed by Google, demonstrating that the Go project is a collaborative endeavor with a thriving global community dedicated to improving the Go ecosystem. Building on `encoding/json/jsontext` ------------------------------------ Before discussing the `v2` API, we first introduce the experimental [`encoding/json/jsontext`](https://go.dev/pkg/encoding/json/jsontext) package that lays the foundation for future improvements to JSON in Go. JSON serialization in Go can be broken down into two primary components: * _syntactic functionality_ that is concerned with processing JSON based on its grammar, and * _semantic functionality_ that defines the relationship between JSON values and Go values. We use the terms “encode” and “decode” to describe syntactic functionality and the terms “marshal” and “unmarshal” to describe semantic functionality. We aim to provide a clear distinction between functionality that is purely concerned with encoding versus that of marshaling. ![](https://go.dev/blog/jsonv2-exp/api.png) This diagram provides an overview of this separation. Purple blocks represent types, while blue blocks represent functions or methods. The direction of the arrows approximately represents the flow of data. The bottom half of the diagram, implemented by the `jsontext` package, contains functionality that is only concerned with syntax, while the upper half, implemented by the `json/v2` package, contains functionality that assigns semantic meaning to syntactic data handled by the bottom half. The basic API of `jsontext` is the following: package jsontext type Encoder struct { ... } func NewEncoder(io.Writer, ...Options) *Encoder func (*Encoder) WriteValue(Value) error func (*Encoder) WriteToken(Token) error type Decoder struct { ... } func NewDecoder(io.Reader, ...Options) *Decoder func (*Decoder) ReadValue() (Value, error) func (*Decoder) ReadToken() (Token, error) type Kind byte type Value []byte func (Value) Kind() Kind type Token struct { ... } func (Token) Kind() Kind The `jsontext` package provides functionality for interacting with JSON at a syntactic level and derives its name from [RFC 8259, section 2](https://datatracker.ietf.org/doc/html/rfc8259#section-2) where the grammar for JSON data is literally called `JSON-text`. Since it only interacts with JSON at a syntactic level, it does not depend on Go reflection. The [`Encoder`](https://go.dev/pkg/encoding/json/jsontext#Encoder) and [`Decoder`](https://go.dev/pkg/encoding/json/jsontext#Decoder) provide support for encoding and decoding JSON values and tokens. The constructors [accept variadic options](https://go.dev/pkg/encoding/json/jsontext#Options) that affect the particular behavior of encoding and decoding. Unlike the `Encoder` and `Decoder` types declared in `v1`, the new types in `jsontext` avoid muddling the distinction between syntax and semantics and operate in a truly streaming manner. A JSON value is a complete unit of data and is represented in Go as [a named `[]byte`](https://go.dev/pkg/encoding/json/jsontext#Value) . It is identical to [`RawMessage`](https://go.dev/pkg/encoding/json#RawMessage) in `v1`. A JSON value is syntactically composed of one or more JSON tokens. A JSON token is represented in Go as the [opaque `Token` type](https://go.dev/pkg/encoding/json/jsontext#Token) with constructors and accessor methods. It is analogous to [`Token`](https://go.dev/pkg/encoding/json#Token) in `v1` but is designed represent arbitrary JSON tokens without allocation. To resolve the fundamental performance problems with the `MarshalJSON` and `UnmarshalJSON` interface methods, we need an efficient way of encoding and decoding JSON as a streaming sequence of tokens and values. In `v2`, we introduce the `MarshalJSONTo` and `UnmarshalJSONFrom` interface methods that operate on an `Encoder` or `Decoder`, allowing the methods’ implementations to process JSON in a purely streaming manner. Thus, the `json` package need not be responsible for validating or formatting a JSON value returned by `MarshalJSON`, nor would it need to be responsible for determining the boundaries of a JSON value provided to `UnmarshalJSON`. These responsibilities belong to the `Encoder` and `Decoder`. Introducing `encoding/json/v2` ------------------------------ Building on the `jsontext` package, we now introduce the experimental [`encoding/json/v2`](https://go.dev/pkg/encoding/json/v2) package. It is designed to fix the aforementioned problems, while remaining familiar to users of the `v1` package. Our goal is that usages of `v1` will operate _mostly_ the same if directly migrated to `v2`. In this article, we will primarily cover the high-level API of `v2`. For examples on how to use it, we encourage readers to study [the examples in the `v2` package](https://go.dev/pkg/encoding/json/v2#pkg-examples) or read [Anton Zhiyanov’s blog covering the topic](https://antonz.org/go-json-v2/) . The basic API of `v2` is the following: package json func Marshal(in any, opts ...Options) (out []byte, err error) func MarshalWrite(out io.Writer, in any, opts ...Options) error func MarshalEncode(out *jsontext.Encoder, in any, opts ...Options) error func Unmarshal(in []byte, out any, opts ...Options) error func UnmarshalRead(in io.Reader, out any, opts ...Options) error func UnmarshalDecode(in *jsontext.Decoder, out any, opts ...Options) error The [`Marshal`](https://go.dev/pkg/encoding/json/v2#Marshal) and [`Unmarshal`](https://go.dev/pkg/encoding/json/v2#Unmarshal) functions have a signature similar to `v1`, but accept options to configure their behavior. The [`MarshalWrite`](https://go.dev/pkg/encoding/json/v2#MarshalWrite) and [`UnmarshalRead`](https://go.dev/pkg/encoding/json/v2#UnmarshalRead) functions directly operate on an `io.Writer` or `io.Reader`, avoiding the need to temporarily construct an `Encoder` or `Decoder` just to write or read from such types. The [`MarshalEncode`](https://go.dev/pkg/encoding/json/v2#MarshalEncode) and [`UnmarshalDecode`](https://go.dev/pkg/encoding/json/v2#UnmarshalDecode) functions operate on a `jsontext.Encoder` and `jsontext.Decoder` and is actually the underlying implementation of the previously mentioned functions. Unlike `v1`, options are a first-class argument to each of the marshal and unmarshal functions, greatly extending the flexibility and configurability of `v2`. There are [several options available](https://go.dev/pkg/encoding/json/v2#Options) in `v2` which are not covered by this article. ### Type-specified customization Similar to `v1`, `v2` allows types to define their own JSON representation by satisfying particular interfaces. type Marshaler interface { MarshalJSON() ([]byte, error) } type MarshalerTo interface { MarshalJSONTo(*jsontext.Encoder) error } type Unmarshaler interface { UnmarshalJSON([]byte) error } type UnmarshalerFrom interface { UnmarshalJSONFrom(*jsontext.Decoder) error } The [`Marshaler`](https://go.dev/pkg/encoding/json/v2#Marshaler) and [`Unmarshaler`](https://go.dev/pkg/encoding/json/v2#Unmarshaler) interfaces are identical to those in `v1`. The new [`MarshalerTo`](https://go.dev/pkg/encoding/json/v2#MarshalerTo) and [`UnmarshalerFrom`](https://go.dev/pkg/encoding/json/v2#UnmarshalerFrom) interfaces allow a type to represent itself as JSON using a `jsontext.Encoder` or `jsontext.Decoder`. This allows options to be forwarded down the call stack, since options can be retrieved via the `Options` accessor method on the `Encoder` or `Decoder`. See [the `OrderedObject` example](https://go.dev/pkg/encoding/json/v2#example-package-OrderedObject) for how to implement a custom type that maintains the ordering of JSON object members. ### Caller-specified customization In `v2`, the caller of `Marshal` and `Unmarshal` can also specify a custom JSON representation for any arbitrary type, where caller-specified functions take precedence over type-defined methods or the default representation for a particular type. func WithMarshalers(*Marshalers) Options type Marshalers struct { ... } func MarshalFunc[T any](fn func(T) ([]byte, error)) *Marshalers func MarshalToFunc[T any](fn func(*jsontext.Encoder, T) error) *Marshalers func WithUnmarshalers(*Unmarshalers) Options type Unmarshalers struct { ... } func UnmarshalFunc[T any](fn func([]byte, T) error) *Unmarshalers func UnmarshalFromFunc[T any](fn func(*jsontext.Decoder, T) error) *Unmarshalers [`MarshalFunc`](https://go.dev/pkg/encoding/json/v2#MarshalFunc) and [`MarshalToFunc`](https://go.dev/pkg/encoding/json/v2#MarshalToFunc) construct a custom marshaler that can be passed to a `Marshal` call using `WithMarshalers` to override the marshaling of particular types. Similarly, [`UnmarshalFunc`](https://go.dev/pkg/encoding/json/v2#UnmarshalFunc) and [`UnmarshalFromFunc`](https://go.dev/pkg/encoding/json/v2#UnmarshalFromFunc) support similar customization for `Unmarshal`. [The `ProtoJSON` example](https://go.dev/pkg/encoding/json/v2#example-package-ProtoJSON) demonstrates how this feature allows serialization of all [`proto.Message`](https://go.dev/pkg/google.golang.org/protobuf/proto#Message) types to be handled by the [`protojson`](https://go.dev/pkg/google.golang.org/protobuf/encoding/protojson) package. ### Behavior differences While `v2` aims to behave _mostly_ the same as `v1`, its behavior has changed [in some ways](https://go.dev/pkg/github.com/go-json-experiment/json/v1#hdr-Migrating_to_v2) to address problems in `v1`, most notably: * `v2` reports an error in the presence of invalid UTF-8. * `v2` reports an error if a JSON object contains a duplicate name. * `v2` marshals a nil Go slice or Go map as an empty JSON array or JSON object, respectively. * `v2` unmarshals a JSON object into a Go struct using a case-sensitive match from the JSON member name to the Go field name. * `v2` redefines the `omitempty` tag option to omit a field if it would have encoded as an “empty” JSON value (which are `null`, `""`, `[]`, and `{}`). * `v2` reports an error when trying to serialize a `time.Duration`, which currently has [no default representation](https://go.dev/issue/71631) , but provides options to let the caller decide. For most behavior changes, there is a struct tag option or caller-specified option that can configure the behavior to operate under `v1` or `v2` semantics, or even other caller-determined behavior. See [“Migrating to v2”](https://go.dev/pkg/github.com/go-json-experiment/json/v1#hdr-Migrating_to_v2) for more information. ### Performance optimizations The `Marshal` performance of `v2` is roughly at parity with `v1`. Sometimes it is slightly faster, but other times it is slightly slower. The `Unmarshal` performance of `v2` is significantly faster than `v1`, with benchmarks demonstrating improvements of up to 10x. In order to obtain greater performance gains, existing implementations of [`Marshaler`](https://go.dev/pkg/encoding/json/v2#Marshaler) and [`Unmarshaler`](https://go.dev/pkg/encoding/json/v2#Unmarshaler) should migrate to also implement [`MarshalerTo`](https://go.dev/pkg/encoding/json/v2#MarshalerTo) and [`UnmarshalerFrom`](https://go.dev/pkg/encoding/json/v2#UnmarshalerFrom) , so that they can benefit from processing JSON in a purely streaming manner. For example, recursive parsing of OpenAPI specifications in `UnmarshalJSON` methods significantly hurt performance in a particular service of Kubernetes (see [kubernetes/kube-openapi#315](https://github.com/kubernetes/kube-openapi/issues/315) ), while switching to `UnmarshalJSONFrom` improved performance by orders of magnitude. For more information, see the [`go-json-experiment/jsonbench`](https://github.com/go-json-experiment/jsonbench) repository. Retroactively improving `encoding/json` --------------------------------------- We want to avoid two separate JSON implementations in the Go standard library, so it is critical that, under the hood, `v1` is implemented in terms of `v2`. There are several benefits to this approach: 1. **Gradual migration**: The `Marshal` and `Unmarshal` functions in `v1` or `v2` represent a set of default behaviors that operate according to `v1` or `v2` semantics. Options can be specified that configure `Marshal` or `Unmarshal` to operate with entirely `v1`, mostly `v1` with a some `v2`, a mix of `v1` or `v2`, mostly `v2` with some `v1`, or entirely `v2` semantics. This allows for gradual migration between the default behaviors of the two versions. 2. **Feature inheritance**: As backward-compatible features are added to `v2`, they will inherently be made available in `v1`. For example, `v2` adds support for several new struct tag options such as `inline` or `format` and also support for the `MarshalJSONTo` and `UnmarshalJSONFrom` interface methods, which are both more performant and flexible. When `v1` is implemented in terms of `v2`, it will inherit support for these features. 3. **Reduced maintenance**: Maintenance of a widely used package demands significant effort. By having `v1` and `v2` use the same implementation, the maintenance burden is reduced. In general, a single change will fix bugs, improve performance, or add functionality to both versions. There is no need to backport a `v2` change with an equivalent `v1` change. While select parts of `v1` may be deprecated over time (supposing `v2` graduates from being an experiment), the package as a whole will never be deprecated. Migrating to `v2` will be encouraged, but not required. The Go project will not drop support for `v1`. Experimenting with `jsonv2` --------------------------- The newer API in the `encoding/json/jsontext` and `encoding/json/v2` packages are not visible by default. To use them, build your code with `GOEXPERIMENT=jsonv2` set in your environment or with the `goexperiment.jsonv2` build tag. The nature of an experiment is that the API is unstable and may change in the future. Though the API is unstable, the implementation is of a high quality and has been successfully used in production by several major projects. The fact that `v1` is implemented in terms of `v2` means that the underlying implementation of `v1` is completely different when building under the `jsonv2` experiment. Without changing any code, you should be able to run your tests under `jsonv2` and theoretically nothing new should fail: GOEXPERIMENT=jsonv2 go test ./... The re-implementation of `v1` in terms of `v2` aims to provide identical behavior within the bounds of the [Go 1 compatibility promise](https://go.dev/doc/go1compat) , though some differences might be observable such as the exact wording of error messages. We encourage you to run your tests under `jsonv2` and report any regressions [on the issue tracker](https://go.dev/issues) . Becoming an experiment in Go 1.25 is a significant milestone on the road to formally adopting `encoding/json/jsontext` and `encoding/json/v2` into the standard library. However, the purpose of the `jsonv2` experiment is to gain broader experience. Your feedback will determine our next steps, and the outcome of this experiment, which may result in anything from abandonment of the effort, to adoption as stable packages of Go 1.26. Please share your experience on [go.dev/issue/71497](https://go.dev/issue/71497) , and help determine the future of Go. **Next article:** [It's survey time! How has Go has been working out for you?](https://go.dev/blog/survey2025-announce) **Previous article:** [Testing Time (and other asynchronicities)](https://go.dev/blog/testing-time) **[Blog Index](https://go.dev/blog/all) ** go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Go 1.3 Release Notes - The Go Programming Language Go 1.3 Release Notes ==================== Introduction to Go 1.3 ---------------------- The latest Go release, version 1.3, arrives six months after 1.2, and contains no language changes. It focuses primarily on implementation work, providing precise garbage collection, a major refactoring of the compiler toolchain that results in faster builds, especially for large projects, significant performance improvements across the board, and support for DragonFly BSD, Solaris, Plan 9 and Google’s Native Client architecture (NaCl). It also has an important refinement to the memory model regarding synchronization. As always, Go 1.3 keeps the [promise of compatibility](https://go.dev/doc/go1compat.html) , and almost everything will continue to compile and run without change when moved to 1.3. Changes to the supported operating systems and architectures ------------------------------------------------------------ ### Removal of support for Windows 2000 Microsoft stopped supporting Windows 2000 in 2010. Since it has [implementation difficulties](https://codereview.appspot.com/74790043) regarding exception handling (signals in Unix terminology), as of Go 1.3 it is not supported by Go either. ### Support for DragonFly BSD Go 1.3 now includes experimental support for DragonFly BSD on the `amd64` (64-bit x86) and `386` (32-bit x86) architectures. It uses DragonFly BSD 3.6 or above. ### Support for FreeBSD It was not announced at the time, but since the release of Go 1.2, support for Go on FreeBSD requires FreeBSD 8 or above. As of Go 1.3, support for Go on FreeBSD requires that the kernel be compiled with the `COMPAT_FREEBSD32` flag configured. In concert with the switch to EABI syscalls for ARM platforms, Go 1.3 will run only on FreeBSD 10. The x86 platforms, 386 and amd64, are unaffected. ### Support for Native Client Support for the Native Client virtual machine architecture has returned to Go with the 1.3 release. It runs on the 32-bit Intel architectures (`GOARCH=386`) and also on 64-bit Intel, but using 32-bit pointers (`GOARCH=amd64p32`). There is not yet support for Native Client on ARM. Note that this is Native Client (NaCl), not Portable Native Client (PNaCl). Details about Native Client are [here](https://developers.google.com/native-client/dev/) ; how to set up the Go version is described [here](https://go.dev/wiki/NativeClient) . ### Support for NetBSD As of Go 1.3, support for Go on NetBSD requires NetBSD 6.0 or above. ### Support for OpenBSD As of Go 1.3, support for Go on OpenBSD requires OpenBSD 5.5 or above. ### Support for Plan 9 Go 1.3 now includes experimental support for Plan 9 on the `386` (32-bit x86) architecture. It requires the `Tsemacquire` syscall, which has been in Plan 9 since June, 2012. ### Support for Solaris Go 1.3 now includes experimental support for Solaris on the `amd64` (64-bit x86) architecture. It requires illumos, Solaris 11 or above. Changes to the memory model --------------------------- The Go 1.3 memory model [adds a new rule](https://codereview.appspot.com/75130045) concerning sending and receiving on buffered channels, to make explicit that a buffered channel can be used as a simple semaphore, using a send into the channel to acquire and a receive from the channel to release. This is not a language change, just a clarification about an expected property of communication. Changes to the implementations and tools ---------------------------------------- ### Stack Go 1.3 has changed the implementation of goroutine stacks away from the old, “segmented” model to a contiguous model. When a goroutine needs more stack than is available, its stack is transferred to a larger single block of memory. The overhead of this transfer operation amortizes well and eliminates the old “hot spot” problem when a calculation repeatedly steps across a segment boundary. Details including performance numbers are in this [design document](https://go.dev/s/contigstacks) . ### Changes to the garbage collector For a while now, the garbage collector has been _precise_ when examining values in the heap; the Go 1.3 release adds equivalent precision to values on the stack. This means that a non-pointer Go value such as an integer will never be mistaken for a pointer and prevent unused memory from being reclaimed. Starting with Go 1.3, the runtime assumes that values with pointer type contain pointers and other values do not. This assumption is fundamental to the precise behavior of both stack expansion and garbage collection. Programs that use [package unsafe](https://go.dev/pkg/unsafe/) to store integers in pointer-typed values are illegal and will crash if the runtime detects the behavior. Programs that use [package unsafe](https://go.dev/pkg/unsafe/) to store pointers in integer-typed values are also illegal but more difficult to diagnose during execution. Because the pointers are hidden from the runtime, a stack expansion or garbage collection may reclaim the memory they point at, creating [dangling pointers](https://en.wikipedia.org/wiki/Dangling_pointer) . _Updating_: Code that uses `unsafe.Pointer` to convert an integer-typed value held in memory into a pointer is illegal and must be rewritten. Such code can be identified by `go vet`. ### Map iteration Iterations over small maps no longer happen in a consistent order. Go 1 defines that “[The iteration order over maps is not specified and is not guaranteed to be the same from one iteration to the next.](https://go.dev/ref/spec#For_statements) ” To keep code from depending on map iteration order, Go 1.0 started each map iteration at a random index in the map. A new map implementation introduced in Go 1.1 neglected to randomize iteration for maps with eight or fewer entries, although the iteration order can still vary from system to system. This has allowed people to write Go 1.1 and Go 1.2 programs that depend on small map iteration order and therefore only work reliably on certain systems. Go 1.3 reintroduces random iteration for small maps in order to flush out these bugs. _Updating_: If code assumes a fixed iteration order for small maps, it will break and must be rewritten not to make that assumption. Because only small maps are affected, the problem arises most often in tests. ### The linker As part of the general [overhaul](https://go.dev/s/go13linker) to the Go linker, the compilers and linkers have been refactored. The linker is still a C program, but now the instruction selection phase that was part of the linker has been moved to the compiler through the creation of a new library called `liblink`. By doing instruction selection only once, when the package is first compiled, this can speed up compilation of large projects significantly. _Updating_: Although this is a major internal change, it should have no effect on programs. ### Status of gccgo GCC release 4.9 will contain the Go 1.2 (not 1.3) version of gccgo. The release schedules for the GCC and Go projects do not coincide, which means that 1.3 will be available in the development branch but that the next GCC release, 4.10, will likely have the Go 1.4 version of gccgo. ### Changes to the go command The [`cmd/go`](https://go.dev/cmd/go/) command has several new features. The [`go run`](https://go.dev/cmd/go/) and [`go test`](https://go.dev/cmd/go/) subcommands support a new `-exec` option to specify an alternate way to run the resulting binary. Its immediate purpose is to support NaCl. The test coverage support of the [`go test`](https://go.dev/cmd/go/) subcommand now automatically sets the coverage mode to `-atomic` when the race detector is enabled, to eliminate false reports about unsafe access to coverage counters. The [`go test`](https://go.dev/cmd/go/) subcommand now always builds the package, even if it has no test files. Previously, it would do nothing if no test files were present. The [`go build`](https://go.dev/cmd/go/) subcommand supports a new `-i` option to install dependencies of the specified target, but not the target itself. Cross compiling with [`cgo`](https://go.dev/cmd/cgo/) enabled is now supported. The CC\_FOR\_TARGET and CXX\_FOR\_TARGET environment variables are used when running all.bash to specify the cross compilers for C and C++ code, respectively. Finally, the go command now supports packages that import Objective-C files (suffixed `.m`) through cgo. ### Changes to cgo The [`cmd/cgo`](https://go.dev/cmd/cgo/) command, which processes `import "C"` declarations in Go packages, has corrected a serious bug that may cause some packages to stop compiling. Previously, all pointers to incomplete struct types translated to the Go type `*[0]byte`, with the effect that the Go compiler could not diagnose passing one kind of struct pointer to a function expecting another. Go 1.3 corrects this mistake by translating each different incomplete struct to a different named type. Given the C declaration `typedef struct S T` for an incomplete `struct S`, some Go code used this bug to refer to the types `C.struct_S` and `C.T` interchangeably. Cgo now explicitly allows this use, even for completed struct types. However, some Go code also used this bug to pass (for example) a `*C.FILE` from one package to another. This is not legal and no longer works: in general Go packages should avoid exposing C types and names in their APIs. _Updating_: Code confusing pointers to incomplete types or passing them across package boundaries will no longer compile and must be rewritten. If the conversion is correct and must be preserved, use an explicit conversion via [`unsafe.Pointer`](https://go.dev/pkg/unsafe/#Pointer) . ### SWIG 3.0 required for programs that use SWIG For Go programs that use SWIG, SWIG version 3.0 is now required. The [`cmd/go`](https://go.dev/cmd/go) command will now link the SWIG generated object files directly into the binary, rather than building and linking with a shared library. ### Command-line flag parsing In the gc toolchain, the assemblers now use the same command-line flag parsing rules as the Go flag package, a departure from the traditional Unix flag parsing. This may affect scripts that invoke the tool directly. For example, `go tool 6a -SDfoo` must now be written `go tool 6a -S -D foo`. (The same change was made to the compilers and linkers in [Go 1.1](https://go.dev/doc/go1.1#gc_flag) .) ### Changes to godoc When invoked with the `-analysis` flag, [godoc](https://godoc.org/golang.org/x/tools/cmd/godoc) now performs sophisticated static analysis of the code it indexes. The results of analysis are presented in both the source view and the package documentation view, and include the call graph of each package and the relationships between definitions and references, types and their methods, interfaces and their implementations, send and receive operations on channels, functions and their callers, and call sites and their callees. ### Miscellany The program `misc/benchcmp` that compares performance across benchmarking runs has been rewritten. Once a shell and awk script in the main repository, it is now a Go program in the `go.tools` repo. Documentation is [here](https://godoc.org/golang.org/x/tools/cmd/benchcmp) . For the few of us that build Go distributions, the tool `misc/dist` has been moved and renamed; it now lives in `misc/makerelease`, still in the main repository. Performance ----------- The performance of Go binaries for this release has improved in many cases due to changes in the runtime and garbage collection, plus some changes to libraries. Significant instances include: * The runtime handles defers more efficiently, reducing the memory footprint by about two kilobytes per goroutine that calls defer. * The garbage collector has been sped up, using a concurrent sweep algorithm, better parallelization, and larger pages. The cumulative effect can be a 50-70% reduction in collector pause time. * The race detector (see [this guide](https://go.dev/doc/articles/race_detector.html) ) is now about 40% faster. * The regular expression package [`regexp`](https://go.dev/pkg/regexp/) is now significantly faster for certain simple expressions due to the implementation of a second, one-pass execution engine. The choice of which engine to use is automatic; the details are hidden from the user. Also, the runtime now includes in stack dumps how long a goroutine has been blocked, which can be useful information when debugging deadlocks or performance issues. Changes to the standard library ------------------------------- ### New packages A new package [`debug/plan9obj`](https://go.dev/pkg/debug/plan9obj/) was added to the standard library. It implements access to Plan 9 [a.out](https://9p.io/magic/man2html/6/a.out) object files. ### Major changes to the library A previous bug in [`crypto/tls`](https://go.dev/pkg/crypto/tls/) made it possible to skip verification in TLS inadvertently. In Go 1.3, the bug is fixed: one must specify either ServerName or InsecureSkipVerify, and if ServerName is specified it is enforced. This may break existing code that incorrectly depended on insecure behavior. There is an important new type added to the standard library: [`sync.Pool`](https://go.dev/pkg/sync/#Pool) . It provides an efficient mechanism for implementing certain types of caches whose memory can be reclaimed automatically by the system. The [`testing`](https://go.dev/pkg/testing/) package’s benchmarking helper, [`B`](https://go.dev/pkg/testing/#B) , now has a [`RunParallel`](https://go.dev/pkg/testing/#B.RunParallel) method to make it easier to run benchmarks that exercise multiple CPUs. _Updating_: The crypto/tls fix may break existing code, but such code was erroneous and should be updated. ### Minor changes to the library The following list summarizes a number of minor changes to the library, mostly additions. See the relevant package documentation for more information about each change. * In the [`crypto/tls`](https://go.dev/pkg/crypto/tls/) package, a new [`DialWithDialer`](https://go.dev/pkg/crypto/tls/#DialWithDialer) function lets one establish a TLS connection using an existing dialer, making it easier to control dial options such as timeouts. The package also now reports the TLS version used by the connection in the [`ConnectionState`](https://go.dev/pkg/crypto/tls/#ConnectionState) struct. * The [`CreateCertificate`](https://go.dev/pkg/crypto/x509/#CreateCertificate) function of the [`crypto/tls`](https://go.dev/pkg/crypto/tls/) package now supports parsing (and elsewhere, serialization) of PKCS #10 certificate signature requests. * The formatted print functions of the `fmt` package now define `%F` as a synonym for `%f` when printing floating-point values. * The [`math/big`](https://go.dev/pkg/math/big/) package’s [`Int`](https://go.dev/pkg/math/big/#Int) and [`Rat`](https://go.dev/pkg/math/big/#Rat) types now implement [`encoding.TextMarshaler`](https://go.dev/pkg/encoding/#TextMarshaler) and [`encoding.TextUnmarshaler`](https://go.dev/pkg/encoding/#TextUnmarshaler) . * The complex power function, [`Pow`](https://go.dev/pkg/math/cmplx/#Pow) , now specifies the behavior when the first argument is zero. It was undefined before. The details are in the [documentation for the function](https://go.dev/pkg/math/cmplx/#Pow) . * The [`net/http`](https://go.dev/pkg/net/http/) package now exposes the properties of a TLS connection used to make a client request in the new [`Response.TLS`](https://go.dev/pkg/net/http/#Response) field. * The [`net/http`](https://go.dev/pkg/net/http/) package now allows setting an optional server error logger with [`Server.ErrorLog`](https://go.dev/pkg/net/http/#Server) . The default is still that all errors go to stderr. * The [`net/http`](https://go.dev/pkg/net/http/) package now supports disabling HTTP keep-alive connections on the server with [`Server.SetKeepAlivesEnabled`](https://go.dev/pkg/net/http/#Server.SetKeepAlivesEnabled) . The default continues to be that the server does keep-alive (reuses connections for multiple requests) by default. Only resource-constrained servers or those in the process of graceful shutdown will want to disable them. * The [`net/http`](https://go.dev/pkg/net/http/) package adds an optional [`Transport.TLSHandshakeTimeout`](https://go.dev/pkg/net/http/#Transport) setting to cap the amount of time HTTP client requests will wait for TLS handshakes to complete. It’s now also set by default on [`DefaultTransport`](https://go.dev/pkg/net/http#DefaultTransport) . * The [`net/http`](https://go.dev/pkg/net/http/) package’s [`DefaultTransport`](https://go.dev/pkg/net/http/#DefaultTransport) , used by the HTTP client code, now enables [TCP keep-alives](https://en.wikipedia.org/wiki/Keepalive#TCP_keepalive) by default. Other [`Transport`](https://go.dev/pkg/net/http/#Transport) values with a nil `Dial` field continue to function the same as before: no TCP keep-alives are used. * The [`net/http`](https://go.dev/pkg/net/http/) package now enables [TCP keep-alives](https://en.wikipedia.org/wiki/Keepalive#TCP_keepalive) for incoming server requests when [`ListenAndServe`](https://go.dev/pkg/net/http/#ListenAndServe) or [`ListenAndServeTLS`](https://go.dev/pkg/net/http/#ListenAndServeTLS) are used. When a server is started otherwise, TCP keep-alives are not enabled. * The [`net/http`](https://go.dev/pkg/net/http/) package now provides an optional [`Server.ConnState`](https://go.dev/pkg/net/http/#Server) callback to hook various phases of a server connection’s lifecycle (see [`ConnState`](https://go.dev/pkg/net/http/#ConnState) ). This can be used to implement rate limiting or graceful shutdown. * The [`net/http`](https://go.dev/pkg/net/http/) package’s HTTP client now has an optional [`Client.Timeout`](https://go.dev/pkg/net/http/#Client) field to specify an end-to-end timeout on requests made using the client. * The [`net/http`](https://go.dev/pkg/net/http/) package’s [`Request.ParseMultipartForm`](https://go.dev/pkg/net/http/#Request.ParseMultipartForm) method will now return an error if the body’s `Content-Type` is not `multipart/form-data`. Prior to Go 1.3 it would silently fail and return `nil`. Code that relies on the previous behavior should be updated. * In the [`net`](https://go.dev/pkg/net/) package, the [`Dialer`](https://go.dev/pkg/net/#Dialer) struct now has a `KeepAlive` option to specify a keep-alive period for the connection. * The [`net/http`](https://go.dev/pkg/net/http/) package’s [`Transport`](https://go.dev/pkg/net/http/#Transport) now closes [`Request.Body`](https://go.dev/pkg/net/http/#Request) consistently, even on error. * The [`os/exec`](https://go.dev/pkg/os/exec/) package now implements what the documentation has always said with regard to relative paths for the binary. In particular, it only calls [`LookPath`](https://go.dev/pkg/os/exec/#LookPath) when the binary’s file name contains no path separators. * The [`SetMapIndex`](https://go.dev/pkg/reflect/#Value.SetMapIndex) function in the [`reflect`](https://go.dev/pkg/reflect/) package no longer panics when deleting from a `nil` map. * If the main goroutine calls [`runtime.Goexit`](https://go.dev/pkg/runtime/#Goexit) and all other goroutines finish execution, the program now always crashes, reporting a detected deadlock. Earlier versions of Go handled this situation inconsistently: most instances were reported as deadlocks, but some trivial cases exited cleanly instead. * The runtime/debug package now has a new function [`debug.WriteHeapDump`](https://go.dev/pkg/runtime/debug/#WriteHeapDump) that writes out a description of the heap. * The [`CanBackquote`](https://go.dev/pkg/strconv/#CanBackquote) function in the [`strconv`](https://go.dev/pkg/strconv/) package now considers the `DEL` character, `U+007F`, to be non-printing. * The [`syscall`](https://go.dev/pkg/syscall/) package now provides [`SendmsgN`](https://go.dev/pkg/syscall/#SendmsgN) as an alternate version of [`Sendmsg`](https://go.dev/pkg/syscall/#Sendmsg) that returns the number of bytes written. * On Windows, the [`syscall`](https://go.dev/pkg/syscall/) package now supports the cdecl calling convention through the addition of a new function [`NewCallbackCDecl`](https://go.dev/pkg/syscall/#NewCallbackCDecl) alongside the existing function [`NewCallback`](https://go.dev/pkg/syscall/#NewCallback) . * The [`testing`](https://go.dev/pkg/testing/) package now diagnoses tests that call `panic(nil)`, which are almost always erroneous. Also, tests now write profiles (if invoked with profiling flags) even on failure. * The [`unicode`](https://go.dev/pkg/unicode/) package and associated support throughout the system has been upgraded from Unicode 6.2.0 to [Unicode 6.3.0](https://www.unicode.org/versions/Unicode6.3.0/) . go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Go Security Policy - The Go Programming Language 1. [Documentation](https://go.dev/doc/) 2. [Security](https://go.dev/doc/security/) 3. [Go Security Policy](https://go.dev/doc/security/policy) Go Security Policy ================== Overview -------- This document explains the Go Security team’s process for handling issues reported and what to expect in return. Reporting a Security Bug ------------------------ All security bugs in the Go distribution should be reported by email to [security@golang.org](mailto:security@golang.org) . This mail is delivered to the Go Security team. To ensure your report is not marked as spam, **please include the word “vulnerability”** anywhere in your email. Please use a descriptive subject line for your report email. Your email will be acknowledged within 7 days, and you’ll be kept up to date with the progress until resolution. Your issue will be fixed or made public within 90 days. If you have not received a reply to your email within 7 days, please follow up with the Go Security team again at [security@golang.org](mailto:security@golang.org) . Please make sure the word **vulnerability** is in your email. If after 3 more days you have still not received an acknowledgement of your report, it is possible that your email might have been marked as spam. In that case, please [file an issue here](https://g.co/vulnz) . Select _“I want to report a technical security or an abuse risk related bug in a Google product (SQLi, XSS, etc.)”_, and list _“Go”_ as the affected product. Tracks ------ Depending on the nature of your issue, it will be categorized by the Go Security team as an issue in the PUBLIC, PRIVATE, or URGENT track. All security issues will be issued CVE numbers. The Go Security team does not assign traditional fine-grained severity labels (e.g CRITICAL, HIGH, MEDIUM, LOW) to security issues because severity depends highly on how a user is using the affected API or functionality. For example, the impact of a resource exhaustion issue in the `encoding/json` parser depends on what is being parsed. If the user is parsing trusted JSON files from their local filesystem, the impact is likely to be low. If the user is parsing untrusted arbitrary JSON from an HTTP request body, the impact may be much higher. That said, the following issue tracks do signal how severe and/or wide-reaching the Security team believes an issue to be. For example, an issue with medium to significant impact for many users is a PRIVATE track issue in this policy, and an issue with negligible to minor impact, or which affects only a small subset of users, is a PUBLIC track issue. ### PUBLIC Issues in the PUBLIC track affect niche configurations, have very limited impact, or are already widely known. PUBLIC track issues are labeled with [`Proposal-Security`](https://github.com/golang/go/labels/Proposal-Security) , discussed through the [Go proposal review process](https://go.googlesource.com/proposal/+/master/README.md#proposal-review) **fixed in public**, and get backported to the next scheduled [minor releases](https://go.dev/wiki/MinorReleases) (which occur ~monthly). The release announcement includes details of these issues, but there is no pre-announcement. Examples of past PUBLIC issues include: * [#44916](https://go.dev/issue/44916) : archive/zip: can panic when calling Reader.Open * [#44913](https://go.dev/issue/44913) : encoding/xml: infinite loop when using xml.NewTokenDecoder with a custom TokenReader * [#43786](https://go.dev/issue/43786) : crypto/elliptic: incorrect operations on the P-224 curve * [#40928](https://go.dev/issue/40928) : net/http/cgi,net/http/fcgi: Cross-Site Scripting (XSS) when Content-Type is not specified * [#40618](https://go.dev/issue/40618) : encoding/binary: ReadUvarint and ReadVarint can read an unlimited number of bytes from invalid inputs * [#36834](https://go.dev/issue/36834) : crypto/x509: certificate validation bypass on Windows 10 ### PRIVATE Issues in the PRIVATE track are violations of committed security properties. PRIVATE track issues are **fixed in the next scheduled [minor releases](https://go.dev/wiki/MinorReleases) **, and are kept private until then. Three to seven days before the release, a pre-announcement is sent to golang-announce, announcing the presence of one or more security fixes in the upcoming releases, and whether the issues affect the standard library, the toolchain, or both, as well as reserved CVE IDs for each of the fixes. For issues that are present in a [major version release candidate](https://go.dev/s/release) , we follow the same process, including fixes in the next scheduled release candidate. Some examples of past PRIVATE issues include: * [#53416](https://go.dev/issue/53416) : path/filepath: stack exhaustion in Glob * [#53616](https://go.dev/issue/53616) : go/parser: stack exhaustion in all Parse\* functions * [#54658](https://go.dev/issue/54658) : net/http: handle server errors after sending GOAWAY * [#56284](https://go.dev/issue/56284) : syscall, os/exec: unsanitized NUL in environment variables ### URGENT URGENT track issues are a threat to the Go ecosystem’s integrity, or are being actively exploited in the wild leading to severe damage. There are no recent examples, but they would include remote code execution in net/http, or practical key recovery in crypto/tls. URGENT track issues are fixed in private, and **trigger an immediate dedicated security release**, possibly with no pre-announcement. Flagging Existing Issues as Security-related -------------------------------------------- If you believe that an [existing issue](https://go.dev/issue) is security-related, we ask that you send an email to [security@golang.org](mailto:security@golang.org) . The email should include the issue ID and a short description of why it should be handled according to this security policy. Disclosure Process ------------------ The Go project uses the following disclosure process: 1. Once the security report is received it is assigned a primary handler. This person coordinates the fix and release process. 2. The issue is confirmed and a list of affected software is determined. 3. Code is audited to find any potential similar problems. 4. If it is determined, in consultation with the submitter, that a CVE number is required, the primary handler will obtain one. 5. Fixes are prepared for the two most recent major releases and the head/master revision. Fixes are prepared for the two most recent major releases and merged to head/master. 6. On the date that the fixes are applied, announcements are sent to [golang-announce](https://groups.google.com/group/golang-announce) , [golang-dev](https://groups.google.com/group/golang-dev) , and [golang-nuts](https://groups.google.com/group/golang-nuts) . This process can take some time, especially when coordination is required with maintainers of other projects. Every effort will be made to handle the bug in as timely a manner as possible, however it’s important that we follow the process described above to ensure that disclosures are handled consistently. For security issues that include the assignment of a CVE number, the issue is listed publicly under the [“Golang” product on the CVEDetails website](https://www.cvedetails.com/vulnerability-list/vendor_id-14185/Golang.html) as well as the [National Vulnerability Disclosure site](https://web.nvd.nist.gov/view/vuln/search) . Receiving Security Updates -------------------------- The best way to receive security announcements is to subscribe to the [golang-announce](https://groups.google.com/forum/#!forum/golang-announce) mailing list. Any messages pertaining to a security issue will be prefixed with `[security]`. Comments on This Policy ----------------------- If you have any suggestions to improve this policy, please [file an issue](https://go.dev/issue/new) for discussion. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Installing Go from source - The Go Programming Language 1. [Documentation](https://go.dev/doc/) 2. [Download and install](https://go.dev/doc/install) 3. [Installing Go from source](https://go.dev/doc/install/source) Installing Go from source ========================= This topic describes how to build and run Go from source code. To install with an installer, see [Download and install](https://go.dev/doc/install) . Introduction ------------ Go is an open source project, distributed under a [BSD-style license](https://go.dev/LICENSE) . This document explains how to check out the sources, build them on your own machine, and run them. Most users don't need to do this, and will instead install from precompiled binary packages as described in [Download and install](https://go.dev/doc/install) , a much simpler process. If you want to help develop what goes into those precompiled packages, though, read on. There are two official Go compiler toolchains. This document focuses on the `gc` Go compiler and tools. For information on how to work on `gccgo`, a more traditional compiler using the GCC back end, see [Setting up and using gccgo](https://go.dev/doc/install/gccgo) . The Go compilers support the following instruction sets: `amd64`, `386` The `x86` instruction set, 64- and 32-bit. `arm64`, `arm` The `ARM` instruction set, 64-bit (`AArch64`) and 32-bit. `loong64` The 64-bit LoongArch instruction set. `mips64`, `mips64le`, `mips`, `mipsle` The `MIPS` instruction set, big- and little-endian, 64- and 32-bit. `ppc64`, `ppc64le` The 64-bit PowerPC instruction set, big- and little-endian. `riscv64` The 64-bit RISC-V instruction set. `s390x` The IBM z/Architecture. `wasm` [WebAssembly](https://webassembly.org/) . The compilers can target the AIX, Android, DragonFly BSD, FreeBSD, Illumos, Linux, macOS/iOS (Darwin), NetBSD, OpenBSD, Plan 9, Solaris, and Windows operating systems (although not all operating systems support all architectures). A list of ports which are considered "first class" is available at the [first class ports](https://go.dev/wiki/PortingPolicy#first-class-ports) wiki page. The full set of supported combinations is listed in the discussion of [environment variables](https://go.dev/doc/install/source#environment) below. See the Go Wiki MinimumRequirements page for the [overall system requirements](https://go.dev/wiki/MinimumRequirements) . Install Go compiler binaries for bootstrap ------------------------------------------ The Go toolchain is written in Go. To build it, you need a Go compiler installed. The scripts that do the initial build of the tools look for a "go" command in `$PATH`, so as long as you have Go installed in your system and configured in your `$PATH`, you are ready to build Go from source. Or if you prefer you can set `$GOROOT_BOOTSTRAP` to the root of a Go installation to use to build the new Go toolchain; `$GOROOT_BOOTSTRAP/bin/go` should be the go command to use. The minimum version of Go required depends on the target version of Go: * Go <= 1.4: a C toolchain. * 1.5 <= Go <= 1.19: a Go 1.4 compiler. * 1.20 <= Go <= 1.21: a Go 1.17 compiler. * 1.22 <= Go <= 1.23: a Go 1.20 compiler. * Going forward, Go version 1.N will require a Go 1.M compiler, where M is N-2 rounded down to an even number. Example: Go 1.24 and 1.25 require Go 1.22. There are four possible ways to obtain a bootstrap toolchain: * Download a recent binary release of Go. * Cross-compile a toolchain using a system with a working Go installation. * Use gccgo. * Compile a toolchain from Go 1.4, the last Go release with a compiler written in C. These approaches are detailed below. ### Bootstrap toolchain from binary release To use a binary release as a bootstrap toolchain, see [the downloads page](https://go.dev/dl/) or use any other packaged Go distribution meeting the minimum version requirements. ### Bootstrap toolchain from cross-compiled source To cross-compile a bootstrap toolchain from source, which is necessary on systems Go 1.4 did not target (for example, `linux/ppc64le`), install Go on a different system and run [bootstrap.bash](https://go.dev/src/bootstrap.bash) . When run as (for example) $ GOOS=linux GOARCH=ppc64 ./bootstrap.bash `bootstrap.bash` cross-compiles a toolchain for that `GOOS/GOARCH` combination, leaving the resulting tree in `../../go-${GOOS}-${GOARCH}-bootstrap`. That tree can be copied to a machine of the given target type and used as `GOROOT_BOOTSTRAP` to bootstrap a local build. ### Bootstrap toolchain using gccgo To use gccgo as the bootstrap toolchain, you need to arrange for `$GOROOT_BOOTSTRAP/bin/go` to be the go tool that comes as part of gccgo 5. For example on Ubuntu Vivid: $ sudo apt-get install gccgo-5 $ sudo update-alternatives --set go /usr/bin/go-5 $ GOROOT\_BOOTSTRAP=/usr ./make.bash ### Bootstrap toolchain from C source code To build a bootstrap toolchain from C source code, use either the git branch `release-branch.go1.4` or [go1.4-bootstrap-20171003.tar.gz](https://dl.google.com/go/go1.4-bootstrap-20171003.tar.gz) , which contains the Go 1.4 source code plus accumulated fixes to keep the tools running on newer operating systems. (Go 1.4 was the last distribution in which the toolchain was written in C.) After unpacking the Go 1.4 source, `cd` to the `src` subdirectory, set `CGO_ENABLED=0` in the environment, and run `make.bash` (or, on Windows, `make.bat`). Once the Go 1.4 source has been unpacked into your GOROOT\_BOOTSTRAP directory, you must keep this git clone instance checked out to branch `release-branch.go1.4`. Specifically, do not attempt to reuse this git clone in the later step named "Fetch the repository." The go1.4 bootstrap toolchain **must be able** to properly traverse the go1.4 sources that it assumes are present under this repository root. Note that Go 1.4 does not run on all systems that later versions of Go do. In particular, Go 1.4 does not support current versions of macOS. On such systems, the bootstrap toolchain must be obtained using one of the other methods. Install Git, if needed ---------------------- To perform the next step you must have Git installed. (Check that you have a `git` command before proceeding.) If you do not have a working Git installation, follow the instructions on the [Git downloads](https://git-scm.com/downloads) page. (Optional) Install a C compiler ------------------------------- To build a Go installation with `[cgo](https://go.dev/cmd/cgo) ` support, which permits Go programs to import C libraries, a C compiler such as `gcc` or `clang` must be installed first. Do this using whatever installation method is standard on the system. To build without `cgo`, set the environment variable `CGO_ENABLED=0` before running `all.bash` or `make.bash`. Fetch the repository -------------------- Change to the directory where you intend to install Go, and make sure the `goroot` directory does not exist. Then clone the repository and check out the latest release tag or release branch (`go1.22.0`, or `release-branch.go1.22`, for example): $ git clone https://go.googlesource.com/go goroot $ cd goroot $ git checkout __ Where `` is the version string of the release. Go will be installed in the directory where it is checked out. For example, if Go is checked out in `$HOME/goroot`, executables will be installed in `$HOME/goroot/bin`. The directory may have any name, but note that if Go is checked out in `$HOME/go`, it will conflict with the default location of `$GOPATH`. See [`GOPATH`](https://go.dev/doc/install/source#gopath) below. Reminder: If you opted to also compile the bootstrap binaries from source (in an earlier section), you still need to `git clone` again at this point (to checkout the latest ``), because you must keep your go1.4 repository distinct. (Optional) Switch to the master branch -------------------------------------- If you intend to modify the go source code, and [contribute your changes](https://go.dev/doc/contribute.html) to the project, then move your repository off the release tag, and onto the master (development) branch. Otherwise, skip this step. $ git checkout master Install Go ---------- To build the Go distribution, run $ cd src $ ./make.bash (To build under Windows use `make.bat`.) If all goes well, it will finish by printing output like: \--- Installed Go for linux/amd64 in /home/you/go. Installed commands in /home/you/go/bin. \*\*\* You need to add /home/you/go/bin to your $PATH. \*\*\* where the details on the last few lines reflect the operating system, architecture, and root directory used during the install. For more information about ways to control the build, see the discussion of [environment variables](https://go.dev/doc/install/source#environment) below. You can also run `all.bash` (or `all.bat`) to run important tests for Go, which can take more time than simply building Go. Testing your installation ------------------------- Check that Go is installed correctly by building a simple program. Create a file named `hello.go` and put the following program in it: package main import "fmt" func main() { fmt.Printf("hello, world\\n") } Then run it with the `go` tool: $ go run hello.go hello, world If you see the "hello, world" message then Go is installed correctly. Set up your work environment ---------------------------- You're almost done. You just need to do a little more setup. [How to Write Go Code Learn how to set up and use the Go tools](https://go.dev/doc/code.html) The [How to Write Go Code](https://go.dev/doc/code.html) document provides **essential setup instructions** for using the Go tools. Install additional tools ------------------------ The source code for several Go tools (including [gopls](https://pkg.go.dev/golang.org/x/tools/gopls) ) is kept in [the golang.org/x/tools repository](https://golang.org/x/tools) . To install one of the tools (`gopls` in this case): $ go install golang.org/x/tools/gopls@latest Community resources ------------------- The usual community resources listed on the [help page](https://go.dev/help/) have active developers that can help you with problems with your installation or your development work. For those who wish to keep up to date, there is another mailing list, [golang-checkins](https://groups.google.com/group/golang-checkins) , that receives a message summarizing each checkin to the Go repository. Bugs can be reported using the [Go issue tracker](https://go.dev/issue/new) . Keeping up with releases ------------------------ New releases are announced on the [golang-announce](https://groups.google.com/group/golang-announce) mailing list. Each announcement mentions the latest release tag, for instance, `go1.9`. To update an existing tree to the latest release, you can run: $ cd go/src $ git fetch $ git checkout __ $ ./all.bash Where `` is the version string of the release. Optional environment variables ------------------------------ The Go compilation environment can be customized by environment variables. _None is required by the build_, but you may wish to set some to override the defaults. * `$GOROOT` The root of the Go tree, often `$HOME/go1.X`. Its value is built into the tree when it is compiled, and defaults to the parent of the directory where `all.bash` was run. There is no need to set this unless you want to switch between multiple local copies of the repository. * `$GOROOT_FINAL` The value assumed by installed binaries and scripts when `$GOROOT` is not set explicitly. It defaults to the value of `$GOROOT`. If you want to build the Go tree in one location but move it elsewhere after the build, set `$GOROOT_FINAL` to the eventual location. * `$GOPATH` The directory where Go projects outside the Go distribution are typically checked out. For example, `golang.org/x/tools` might be checked out to `$GOPATH/src/golang.org/x/tools`. Executables outside the Go distribution are installed in `$GOPATH/bin` (or `$GOBIN`, if set). Modules are downloaded and cached in `$GOPATH/pkg/mod`. The default location of `$GOPATH` is `$HOME/go`, and it's not usually necessary to set `GOPATH` explicitly. However, if you have checked out the Go distribution to `$HOME/go`, you must set `GOPATH` to another location to avoid conflicts. * `$GOBIN` The directory where executables outside the Go distribution are installed using the [go command](https://go.dev/cmd/go) . For example, `go install golang.org/x/tools/gopls@latest` downloads, builds, and installs `$GOBIN/gopls`. By default, `$GOBIN` is `$GOPATH/bin` (or `$HOME/go/bin` if `GOPATH` is not set). After installing, you will want to add this directory to your `$PATH` so you can use installed tools. Note that the Go distribution's executables are installed in `$GOROOT/bin` (for executables invoked by people) or `$GOTOOLDIR` (for executables invoked by the go command; defaults to `$GOROOT/pkg/$GOOS_$GOARCH`) instead of `$GOBIN`. * `$GOOS` and `$GOARCH` The name of the target operating system and compilation architecture. These default to the values of `$GOHOSTOS` and `$GOHOSTARCH` respectively (described below). Choices for `$GOOS` are `android`, `darwin`, `dragonfly`, `freebsd`, `illumos`, `ios`, `js`, `linux`, `netbsd`, `openbsd`, `plan9`, `solaris`, `wasip1`, and `windows`. Choices for `$GOARCH` are `amd64` (64-bit x86, the most mature port), `386` (32-bit x86), `arm` (32-bit ARM), `arm64` (64-bit ARM), `ppc64le` (PowerPC 64-bit, little-endian), `ppc64` (PowerPC 64-bit, big-endian), `mips64le` (MIPS 64-bit, little-endian), `mips64` (MIPS 64-bit, big-endian), `mipsle` (MIPS 32-bit, little-endian), `mips` (MIPS 32-bit, big-endian), `s390x` (IBM System z 64-bit, big-endian), and `wasm` (WebAssembly 32-bit). The valid combinations of `$GOOS` and `$GOARCH` are: | | `$GOOS` | `$GOARCH` | | --- | --- | --- | | | `aix` | `ppc64` | | | `android` | `386` | | | `android` | `amd64` | | | `android` | `arm` | | | `android` | `arm64` | | | `darwin` | `amd64` | | | `darwin` | `arm64` | | | `dragonfly` | `amd64` | | | `freebsd` | `386` | | | `freebsd` | `amd64` | | | `freebsd` | `arm` | | | `illumos` | `amd64` | | | `ios` | `arm64` | | | `js` | `wasm` | | | `linux` | `386` | | | `linux` | `amd64` | | | `linux` | `arm` | | | `linux` | `arm64` | | | `linux` | `loong64` | | | `linux` | `mips` | | | `linux` | `mipsle` | | | `linux` | `mips64` | | | `linux` | `mips64le` | | | `linux` | `ppc64` | | | `linux` | `ppc64le` | | | `linux` | `riscv64` | | | `linux` | `s390x` | | | `netbsd` | `386` | | | `netbsd` | `amd64` | | | `netbsd` | `arm` | | | `openbsd` | `386` | | | `openbsd` | `amd64` | | | `openbsd` | `arm` | | | `openbsd` | `arm64` | | | `plan9` | `386` | | | `plan9` | `amd64` | | | `plan9` | `arm` | | | `solaris` | `amd64` | | | `wasip1` | `wasm` | | | `windows` | `386` | | | `windows` | `amd64` | | | `windows` | `arm` | | | `windows` | `arm64` | * `$GOHOSTOS` and `$GOHOSTARCH` The name of the host operating system and compilation architecture. These default to the local system's operating system and architecture. Valid choices are the same as for `$GOOS` and `$GOARCH`, listed above. The specified values must be compatible with the local system. For example, you should not set `$GOHOSTARCH` to `arm` on an x86 system. * `$GO386` (for `386` only, defaults to `sse2`) This variable controls how gc implements floating point computations. * `GO386=softfloat`: use software floating point operations; should support all x86 chips (Pentium MMX or later). * `GO386=sse2`: use SSE2 for floating point operations; has better performance but only available on Pentium 4/Opteron/Athlon 64 or later. * `$GOARM` (for `arm` only; default is auto-detected if building on the target processor, 7 if not) This sets the ARM floating point co-processor architecture version the run-time should target. If you are compiling on the target system, its value will be auto-detected. * `GOARM=5`: use software floating point; when CPU doesn't have VFP co-processor * `GOARM=6`: use VFPv1 only; default if cross compiling; usually ARM11 or better cores (VFPv2 or better is also supported) * `GOARM=7`: use VFPv3; usually Cortex-A cores If in doubt, leave this variable unset, and adjust it if required when you first run the Go executable. The [GoARM](https://go.dev/wiki/GoArm) page on the [Go community wiki](https://go.dev/wiki) contains further details regarding Go's ARM support. * `$GOAMD64` (for `amd64` only; default is `v1`) This sets the microarchitecture level for which to compile. Valid values are `v1` (default), `v2`, `v3`, `v4`. See [the Go wiki MinimumRequirements page](https://go.dev/wiki/MinimumRequirements#amd64) for more information. * `$GOMIPS` (for `mips` and `mipsle` only) `$GOMIPS64` (for `mips64` and `mips64le` only) These variables set whether to use floating point instructions. Set to "`hardfloat`" to use floating point instructions; this is the default. Set to "`softfloat`" to use soft floating point. * `$GOPPC64` (for `ppc64` and `ppc64le` only) This variable sets the processor level (i.e. Instruction Set Architecture version) for which the compiler will target. The default is `power8`. * `GOPPC64=power8`: generate ISA v2.07 instructions * `GOPPC64=power9`: generate ISA v3.00 instructions * `$GORISCV64` (for `riscv64` only) This variable sets the RISC-V user-mode application profile for which to compile. The default is `rva20u64`. * `GORISCV64=rva20u64`: only use RISC-V extensions that are mandatory in the [RVA20U64](https://github.com/riscv/riscv-profiles/blob/main/src/profiles.adoc#51-rva20u64-profile) profile * `GORISCV64=rva22u64`: only use RISC-V extensions that are mandatory in the [RVA22U64](https://github.com/riscv/riscv-profiles/blob/main/src/profiles.adoc#rva22u64-profile) profile * `GORISCV64=rva23u64`: only use RISC-V extensions that are mandatory in the [RVA23U64](https://github.com/riscv/riscv-profiles/blob/main/src/rva23-profile.adoc#rva23u64-profile) profile * `$GOWASM` (for `wasm` only) This variable is a comma separated list of [experimental WebAssembly features](https://github.com/WebAssembly/proposals) that the compiled WebAssembly binary is allowed to use. The default is to use no experimental features. * `GOWASM=satconv`: generate [saturating (non-trapping) float-to-int conversions](https://github.com/WebAssembly/nontrapping-float-to-int-conversions/blob/master/proposals/nontrapping-float-to-int-conversion/Overview.md) * `GOWASM=signext`: generate [sign-extension operators](https://github.com/WebAssembly/sign-extension-ops/blob/master/proposals/sign-extension-ops/Overview.md) Note that `$GOARCH` and `$GOOS` identify the _target_ environment, not the environment you are running on. In effect, you are always cross-compiling. By architecture, we mean the kind of binaries that the target environment can run: an x86-64 system running a 32-bit-only operating system must set `GOARCH` to `386`, not `amd64`. If you choose to override the defaults, set these variables in your shell profile (`$HOME/.bashrc`, `$HOME/.profile`, or equivalent). The settings might look something like this: export GOARCH=amd64 export GOOS=linux although, to reiterate, none of these variables needs to be set to build, install, and develop the Go tree. go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Gopls: The language server for Go - The Go Programming Language Gopls: The language server for Go ================================= `gopls` (pronounced “Go please”) is the official [language server](https://langserver.org/) for Go, developed by the Go team. It provides a wide variety of [IDE features](https://go.dev/gopls/features/) to any [LSP](https://microsoft.github.io/language-server-protocol/) \-compatible editor. You should not need to interact with `gopls` directly–it will be automatically integrated into your editor. The specific features and settings vary slightly by editor, so we recommend that you proceed to the [documentation for your editor](https://go.dev/gopls/#editors) below. Also, the gopls documentation for each feature describes whether it is supported in each client editor. This documentation ([https://go.dev/gopls](https://go.dev/gopls) ) describes the most recent release of gopls. To preview documentation for the release under development, visit [https://tip.golang.org/gopls](https://tip.golang.org/gopls) . Features -------- Gopls supports a wide range of standard LSP features for navigation, completion, diagnostics, analysis, and refactoring, and a number of additional features not found in other language servers. See the [Index of features](https://go.dev/gopls/features/) for complete documentation on what Gopls can do for you. Editors ------- To get started with `gopls`, install an LSP plugin in your editor of choice. * [Acme](https://github.com/fhs/acme-lsp/blob/master/README.md) * [Atom](https://github.com/MordFustang21/ide-gopls/blob/master/README.md) * [Emacs](https://go.dev/gopls/editor/emacs.md) * [Helix](https://go.dev/gopls/editor/helix.md) * [Lapce](https://github.com/lapce-community/lapce-go/blob/master/README.md) * [Sublime Text](https://go.dev/gopls/editor/sublime.md) * [VS Code](https://github.com/golang/vscode-go/blob/master/README.md) * [Vim or Neovim](https://go.dev/gopls/editor/vim.md) * [Zed](https://go.dev/gopls/editor/zed.md) If you use `gopls` with an editor that is not on this list, please send us a CL [updating this documentation](https://go.dev/gopls/contributing.md) . Installation ------------ To install the latest stable release of `gopls`, run the following command: go install golang.org/x/tools/gopls@latest Some editors, such as VS Code, will handle this step for you, and ensure that Gopls is updated when a new stable version is released. After updating, you may need to restart running Gopls processes to observe the effect. Each client has its own way to restart the server. (On a UNIX machine, you can use the command `killall gopls`.) Learn more in the [advanced installation instructions](https://go.dev/gopls/advanced.md#installing-unreleased-versions) . Releases -------- Gopls [releases](https://go.dev/gopls/release/) follow [semantic versioning](http://semver.org/) , with major changes and new features introduced only in new minor versions (i.e. versions of the form `v*.N.0` for some N). Subsequent patch releases contain only cherry-picked fixes or superficial updates. In order to align with the [Go release timeline](https://github.com/golang/go/wiki/Go-Release-Cycle#timeline) , we aim to release a new minor version of Gopls approximately every three months, with patch releases approximately every month, according to the following table: | Month | Version(s) | | --- | --- | | Jan | `v*..0` | | Jan-Mar | `v*..*` | | Apr | `v*..0` | | Apr-Jun | `v*..*` | | Jul | `v*..0` | | Jul-Sep | `v*..*` | | Oct | `v*..0` | | Oct-Dec | `v*..*` | For more background on this policy, see [https://go.dev/issue/55267](https://go.dev/issue/55267) . Setting up your workspace ------------------------- `gopls` supports both Go module, multi-module and GOPATH modes. See the [workspace documentation](https://go.dev/gopls/workspace.md) for information on supported workspace layouts. Configuration ------------- You can configure `gopls` to change your editor experience or view additional debugging information. Configuration options will be made available by your editor, so see your [editor’s instructions](https://go.dev/gopls/#editors) for specific details. A full list of `gopls` settings can be found in the [settings documentation](https://go.dev/gopls/settings.md) . ### Environment variables `gopls` inherits your editor’s environment, so be aware of any environment variables you configure. Some editors, such as VS Code, allow users to selectively override the values of some environment variables. Support policy -------------- Gopls is maintained by engineers on the [Go tools team](https://github.com/orgs/golang/teams/tools-team/members) , who actively monitor the [Go](https://github.com/golang/go/issues?q=is%3Aissue+is%3Aopen+label%3Agopls) and [VS Code Go](https://github.com/golang/vscode-go/issues) issue trackers. ### Supported Go versions `gopls` follows the [Go Release Policy](https://go.dev/doc/devel/release#policy) , meaning that it officially supports only the two most recent major Go releases. When using gopls, there are three versions to be aware of: 1. The _gopls build go version_: the version of Go used to build gopls. 2. The _go command version_: the version of the go list command executed by gopls to load information about your workspace. 3. The _language version_: the version in the go directive of the current file’s enclosing go.mod file, which determines the file’s Go language semantics. Starting with the release of Go 1.23.0 and gopls@v0.17.0 in August 2024, we will only support the most recent Go version as the _gopls build go version_. However, due to the [forward compatibility](https://go.dev/blog/toolchain) support added in Go 1.21, as long as Go 1.21 or later are used to install gopls, any necessary toolchain upgrade will be handled automatically, just like any other dependency. Additionally, starting with gopls@v0.17.0, the _go command version_ will narrow from 4 versions to 3. This is more consistent with the Go Release Policy. Gopls supports **all** Go versions as its _language version_, by providing compiler errors based on the language version and filtering available standard library symbols based on the standard library APIs available at that Go version. Maintaining support for building gopls with legacy versions of Go caused [significant friction](https://go.dev/issue/50825) for gopls maintainers and held back other improvements. If you are unable to install a supported version of Go on your system, you can still install an older version of gopls. The following table shows the final gopls version that supports a given Go version. Go releases more recent than those in the table can be used with any version of gopls. | Go Version | Final gopls version with support (without warnings) | | --- | --- | | Go 1.12 | [gopls@v0.7.5](https://github.com/golang/tools/releases/tag/gopls%2Fv0.7.5) | | Go 1.15 | [gopls@v0.9.5](https://github.com/golang/tools/releases/tag/gopls%2Fv0.9.5) | | Go 1.17 | [gopls@v0.11.0](https://github.com/golang/tools/releases/tag/gopls%2Fv0.11.0) | | Go 1.18 | [gopls@v0.14.2](https://github.com/golang/tools/releases/tag/gopls%2Fv0.14.2) | | Go 1.20 | [gopls@v0.15.3](https://github.com/golang/tools/releases/tag/gopls%2Fv0.15.3) | ### Supported build systems `gopls` currently only supports the `go` command, so if you are using a different build system, `gopls` will not work well. Bazel is not officially supported, but may be made to work with an appropriately configured [go/packages driver](https://pkg.go.dev/golang.org/x/tools/go/packages#hdr-The_driver_protocol) . See [bazelbuild/rules\_go#512](https://github.com/bazelbuild/rules_go/issues/512) for more information. You can follow [these instructions](https://github.com/bazelbuild/rules_go/wiki/Editor-setup) to configure your `gopls` to work with Bazel. ### Troubleshooting If you are having issues with `gopls`, please follow the steps described in the [troubleshooting guide](https://go.dev/gopls/troubleshooting.md) . Additional information ---------------------- * [Command-line interface](https://go.dev/gopls/command-line.md) * [Advanced topics](https://go.dev/gopls/advanced.md) * [Open issues](https://github.com/golang/go/issues?q=is%3Aissue+is%3Aopen+label%3Agopls) * [Contributing to `gopls`](https://go.dev/gopls/contributing.md) * * * _The source files for this documentation can be found beneath [golang.org/x/tools/gopls/doc](https://cs.opensource.google/go/x/tools/+/master:gopls/doc/) ._ go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. [Learn more.](https://policies.google.com/technologies/cookies) Okay --- # Go 1.13 Release Notes - The Go Programming Language Go 1.13 Release Notes ===================== Introduction to Go 1.13 ----------------------- The latest Go release, version 1.13, arrives six months after [Go 1.12](https://go.dev/doc/go1.12) . Most of its changes are in the implementation of the toolchain, runtime, and libraries. As always, the release maintains the Go 1 [promise of compatibility](https://go.dev/doc/go1compat.html) . We expect almost all Go programs to continue to compile and run as before. As of Go 1.13, the go command by default downloads and authenticates modules using the Go module mirror and Go checksum database run by Google. See [https://proxy.golang.org/privacy](https://proxy.golang.org/privacy) for privacy information about these services and the [go command documentation](https://go.dev/cmd/go/#hdr-Module_downloading_and_verification) for configuration details including how to disable the use of these servers or use different ones. If you depend on non-public modules, see the [documentation for configuring your environment](https://go.dev/cmd/go/#hdr-Module_configuration_for_non_public_modules) . Changes to the language ----------------------- Per the [number literal proposal](https://github.com/golang/proposal/blob/master/design/19308-number-literals.md) , Go 1.13 supports a more uniform and modernized set of number literal prefixes. * [Binary integer literals](https://go.dev/ref/spec#Integer_literals) : The prefix `0b` or `0B` indicates a binary integer literal such as `0b1011`. * [Octal integer literals](https://go.dev/ref/spec#Integer_literals) : The prefix `0o` or `0O` indicates an octal integer literal such as `0o660`. The existing octal notation indicated by a leading `0` followed by octal digits remains valid. * [Hexadecimal floating point literals](https://go.dev/ref/spec#Floating-point_literals) : The prefix `0x` or `0X` may now be used to express the mantissa of a floating-point number in hexadecimal format such as `0x1.0p-1021`. A hexadecimal floating-point number must always have an exponent, written as the letter `p` or `P` followed by an exponent in decimal. The exponent scales the mantissa by 2 to the power of the exponent. * [Imaginary literals](https://go.dev/ref/spec#Imaginary_literals) : The imaginary suffix `i` may now be used with any (binary, decimal, hexadecimal) integer or floating-point literal. * Digit separators: The digits of any number literal may now be separated (grouped) using underscores, such as in `1_000_000`, `0b_1010_0110`, or `3.1415_9265`. An underscore may appear between any two digits or the literal prefix and the first digit. Per the [signed shift counts proposal](https://github.com/golang/proposal/blob/master/design/19113-signed-shift-counts.md) Go 1.13 removes the restriction that a [shift count](https://go.dev/ref/spec#Operators) must be unsigned. This change eliminates the need for many artificial `uint` conversions, solely introduced to satisfy this (now removed) restriction of the `<<` and `>>` operators. These language changes were implemented by changes to the compiler, and corresponding internal changes to the library packages [`go/scanner`](https://go.dev/doc/go1.13#go/scanner) and [`text/scanner`](https://go.dev/doc/go1.13#text/scanner) (number literals), and [`go/types`](https://go.dev/doc/go1.13#go/types) (signed shift counts). If your code uses modules and your `go.mod` files specifies a language version, be sure it is set to at least `1.13` to get access to these language changes. You can do this by editing the `go.mod` file directly, or you can run `go mod edit -go=1.13`. Ports ----- Go 1.13 is the last release that will run on Native Client (NaCl). For `GOARCH=wasm`, the new environment variable `GOWASM` takes a comma-separated list of experimental features that the binary gets compiled with. The valid values are documented [here](https://go.dev/cmd/go/#hdr-Environment_variables) . ### AIX AIX on PPC64 (`aix/ppc64`) now supports cgo, external linking, and the `c-archive` and `pie` build modes. ### Android Go programs are now compatible with Android 10. ### Darwin As [announced](https://go.dev/doc/go1.12#darwin) in the Go 1.12 release notes, Go 1.13 now requires macOS 10.11 El Capitan or later; support for previous versions has been discontinued. ### FreeBSD As [announced](https://go.dev/doc/go1.12#freebsd) in the Go 1.12 release notes, Go 1.13 now requires FreeBSD 11.2 or later; support for previous versions has been discontinued. FreeBSD 12.0 or later requires a kernel with the `COMPAT_FREEBSD11` option set (this is the default). ### Illumos Go now supports Illumos with `GOOS=illumos`. The `illumos` build tag implies the `solaris` build tag. ### Windows The Windows version specified by internally-linked Windows binaries is now Windows 7 rather than NT 4.0. This was already the minimum required version for Go, but can affect the behavior of system calls that have a backwards-compatibility mode. These will now behave as documented. Externally-linked binaries (any program using cgo) have always specified a more recent Windows version. Tools ----- ### Modules #### Environment variables The [`GO111MODULE`](https://go.dev/cmd/go/#hdr-Module_support) environment variable continues to default to `auto`, but the `auto` setting now activates the module-aware mode of the `go` command whenever the current working directory contains, or is below a directory containing, a `go.mod` file — even if the current directory is within `GOPATH/src`. This change simplifies the migration of existing code within `GOPATH/src` and the ongoing maintenance of module-aware packages alongside non-module-aware importers. The new [`GOPRIVATE`](https://go.dev/cmd/go/#hdr-Module_configuration_for_non_public_modules) environment variable indicates module paths that are not publicly available. It serves as the default value for the lower-level `GONOPROXY` and `GONOSUMDB` variables, which provide finer-grained control over which modules are fetched via proxy and verified using the checksum database. The [`GOPROXY` environment variable](https://go.dev/cmd/go/#hdr-Module_downloading_and_verification) may now be set to a comma-separated list of proxy URLs or the special token `direct`, and its [default value](https://go.dev/doc/go1.13#introduction) is now `https://proxy.golang.org,direct`. When resolving a package path to its containing module, the `go` command will try all candidate module paths on each proxy in the list in succession. An unreachable proxy or HTTP status code other than 404 or 410 terminates the search without consulting the remaining proxies. The new [`GOSUMDB`](https://go.dev/cmd/go/#hdr-Module_authentication_failures) environment variable identifies the name, and optionally the public key and server URL, of the database to consult for checksums of modules that are not yet listed in the main module’s `go.sum` file. If `GOSUMDB` does not include an explicit URL, the URL is chosen by probing the `GOPROXY` URLs for an endpoint indicating support for the checksum database, falling back to a direct connection to the named database if it is not supported by any proxy. If `GOSUMDB` is set to `off`, the checksum database is not consulted and only the existing checksums in the `go.sum` file are verified. Users who cannot reach the default proxy and checksum database (for example, due to a firewalled or sandboxed configuration) may disable their use by setting `GOPROXY` to `direct`, and/or `GOSUMDB` to `off`. [`go` `env` `-w`](https://go.dev/doc/go1.13#go-env-w) can be used to set the default values for these variables independent of platform: go env -w GOPROXY=direct go env -w GOSUMDB=off #### `go` `get` In module-aware mode, [`go` `get`](https://go.dev/cmd/go/#hdr-Add_dependencies_to_current_module_and_install_them) with the `-u` flag now updates a smaller set of modules that is more consistent with the set of packages updated by `go` `get` `-u` in GOPATH mode. `go` `get` `-u` continues to update the modules and packages named on the command line, but additionally updates only the modules containing the packages _imported by_ the named packages, rather than the transitive module requirements of the modules containing the named packages. Note in particular that `go` `get` `-u` (without additional arguments) now updates only the transitive imports of the package in the current directory. To instead update all of the packages transitively imported by the main module (including test dependencies), use `go` `get` `-u` `all`. As a result of the above changes to `go` `get` `-u`, the `go` `get` subcommand no longer supports the `-m` flag, which caused `go` `get` to stop before loading packages. The `-d` flag remains supported, and continues to cause `go` `get` to stop after downloading the source code needed to build dependencies of the named packages. By default, `go` `get` `-u` in module mode upgrades only non-test dependencies, as in GOPATH mode. It now also accepts the `-t` flag, which (as in GOPATH mode) causes `go` `get` to include the packages imported by _tests of_ the packages named on the command line. In module-aware mode, the `go` `get` subcommand now supports the version suffix `@patch`. The `@patch` suffix indicates that the named module, or module containing the named package, should be updated to the highest patch release with the same major and minor versions as the version found in the build list. If a module passed as an argument to `go` `get` without a version suffix is already required at a newer version than the latest released version, it will remain at the newer version. This is consistent with the behavior of the `-u` flag for module dependencies. This prevents unexpected downgrades from pre-release versions. The new version suffix `@upgrade` explicitly requests this behavior. `@latest` explicitly requests the latest version regardless of the current version. #### Version validation When extracting a module from a version control system, the `go` command now performs additional validation on the requested version string. The `+incompatible` version annotation bypasses the requirement of [semantic import versioning](https://go.dev/cmd/go/#hdr-Module_compatibility_and_semantic_versioning) for repositories that predate the introduction of modules. The `go` command now verifies that such a version does not include an explicit `go.mod` file. The `go` command now verifies the mapping between [pseudo-versions](https://go.dev/cmd/go/#hdr-Pseudo_versions) and version-control metadata. Specifically: * The version prefix must be of the form `vX.0.0`, or derived from a tag on an ancestor of the named revision, or derived from a tag that includes [build metadata](https://semver.org/#spec-item-10) on the named revision itself. * The date string must match the UTC timestamp of the revision. * The short name of the revision must use the same number of characters as what the `go` command would generate. (For SHA-1 hashes as used by `git`, a 12-digit prefix.) If a `require` directive in the [main module](https://go.dev/cmd/go/#hdr-The_main_module_and_the_build_list) uses an invalid pseudo-version, it can usually be corrected by redacting the version to just the commit hash and re-running a `go` command, such as `go` `list` `-m` `all` or `go` `mod` `tidy`. For example, require github.com/docker/docker v1.14.0-0.20190319215453-e7b5f7dbe98c can be redacted to require github.com/docker/docker e7b5f7dbe98c which currently resolves to require github.com/docker/docker v0.7.3-0.20190319215453-e7b5f7dbe98c If one of the transitive dependencies of the main module requires an invalid version or pseudo-version, the invalid version can be replaced with a valid one using a [`replace` directive](https://go.dev/cmd/go/#hdr-The_go_mod_file) in the `go.mod` file of the main module. If the replacement is a commit hash, it will be resolved to the appropriate pseudo-version as above. For example, replace github.com/docker/docker v1.14.0-0.20190319215453-e7b5f7dbe98c => github.com/docker/docker e7b5f7dbe98c currently resolves to replace github.com/docker/docker v1.14.0-0.20190319215453-e7b5f7dbe98c => github.com/docker/docker v0.7.3-0.20190319215453-e7b5f7dbe98c ### Go command The [`go` `env`](https://go.dev/cmd/go/#hdr-Environment_variables) command now accepts a `-w` flag to set the per-user default value of an environment variable recognized by the `go` command, and a corresponding `-u` flag to unset a previously-set default. Defaults set via `go` `env` `-w` are stored in the `go/env` file within [`os.UserConfigDir()`](https://go.dev/pkg/os/#UserConfigDir) . The [`go` `version`](https://go.dev/cmd/go/#hdr-Print_Go_version) command now accepts arguments naming executables and directories. When invoked on an executable, `go` `version` prints the version of Go used to build the executable. If the `-m` flag is used, `go` `version` prints the executable’s embedded module version information, if available. When invoked on a directory, `go` `version` prints information about executables contained in the directory and its subdirectories. The new [`go` `build` flag](https://go.dev/cmd/go/#hdr-Compile_packages_and_dependencies) `-trimpath` removes all file system paths from the compiled executable, to improve build reproducibility. If the `-o` flag passed to `go` `build` refers to an existing directory, `go` `build` will now write executable files within that directory for `main` packages matching its package arguments. The `go` `build` flag `-tags` now takes a comma-separated list of build tags, to allow for multiple tags in [`GOFLAGS`](https://go.dev/cmd/go/#hdr-Environment_variables) . The space-separated form is deprecated but still recognized and will be maintained. [`go` `generate`](https://go.dev/cmd/go/#hdr-Generate_Go_files_by_processing_source) now sets the `generate` build tag so that files may be searched for directives but ignored during build. As [announced](https://go.dev/doc/go1.12#binary-only) in the Go 1.12 release notes, binary-only packages are no longer supported. Building a binary-only package (marked with a `//go:binary-only-package` comment) now results in an error. ### Compiler toolchain The compiler has a new implementation of escape analysis that is more precise. For most Go code should be an improvement (in other words, more Go variables and expressions allocated on the stack instead of heap). However, this increased precision may also break invalid code that happened to work before (for example, code that violates the [`unsafe.Pointer` safety rules](https://go.dev/pkg/unsafe/#Pointer) ). If you notice any regressions that appear related, the old escape analysis pass can be re-enabled with `go` `build` `-gcflags=all=-newescape=false`. The option to use the old escape analysis will be removed in a future release. The compiler no longer emits floating point or complex constants to `go_asm.h` files. These have always been emitted in a form that could not be used as numeric constant in assembly code. ### Assembler The assembler now supports many of the atomic instructions introduced in ARM v8.1. ### gofmt `gofmt` (and with that `go fmt`) now canonicalizes number literal prefixes and exponents to use lower-case letters, but leaves hexadecimal digits alone. This improves readability when using the new octal prefix (`0O` becomes `0o`), and the rewrite is applied consistently. `gofmt` now also removes unnecessary leading zeroes from a decimal integer imaginary literal. (For backwards-compatibility, an integer imaginary literal starting with `0` is considered a decimal, not an octal number. Removing superfluous leading zeroes avoids potential confusion.) For instance, `0B1010`, `0XabcDEF`, `0O660`, `1.2E3`, and `01i` become `0b1010`, `0xabcDEF`, `0o660`, `1.2e3`, and `1i` after applying `gofmt`. ### `godoc` and `go` `doc` The `godoc` webserver is no longer included in the main binary distribution. To run the `godoc` webserver locally, manually install it first: go get golang.org/x/tools/cmd/godoc godoc The [`go` `doc`](https://go.dev/cmd/go/#hdr-Show_documentation_for_package_or_symbol) command now always includes the package clause in its output, except for commands. This replaces the previous behavior where a heuristic was used, causing the package clause to be omitted under certain conditions. Runtime ------- Out of range panic messages now include the index that was out of bounds and the length (or capacity) of the slice. For example, `s[3]` on a slice of length 1 will panic with “runtime error: index out of range \[3\] with length 1”. This release improves performance of most uses of `defer` by 30%. The runtime is now more aggressive at returning memory to the operating system to make it available to co-tenant applications. Previously, the runtime could retain memory for five or more minutes following a spike in the heap size. It will now begin returning it promptly after the heap shrinks. However, on many OSes, including Linux, the OS itself reclaims memory lazily, so process RSS will not decrease until the system is under memory pressure. Standard library ---------------- ### TLS 1.3 As announced in Go 1.12, Go 1.13 enables support for TLS 1.3 in the `crypto/tls` package by default. It can be disabled by adding the value `tls13=0` to the `GODEBUG` environment variable. The opt-out will be removed in Go 1.14. See [the Go 1.12 release notes](https://go.dev/doc/go1.12#tls_1_3) for important compatibility information. ### [crypto/ed25519](https://go.dev/pkg/crypto/ed25519/) The new [`crypto/ed25519`](https://go.dev/pkg/crypto/ed25519/) package implements the Ed25519 signature scheme. This functionality was previously provided by the [`golang.org/x/crypto/ed25519`](https://godoc.org/golang.org/x/crypto/ed25519) package, which becomes a wrapper for `crypto/ed25519` when used with Go 1.13+. ### Error wrapping Go 1.13 contains support for error wrapping, as first proposed in the [Error Values proposal](https://go.googlesource.com/proposal/+/master/design/29934-error-values.md) and discussed on [the associated issue](https://go.dev/issue/29934) . An error `e` can _wrap_ another error `w` by providing an `Unwrap` method that returns `w`. Both `e` and `w` are available to programs, allowing `e` to provide additional context to `w` or to reinterpret it while still allowing programs to make decisions based on `w`. To support wrapping, [`fmt.Errorf`](https://go.dev/doc/go1.13#fmt) now has a `%w` verb for creating wrapped errors, and three new functions in the [`errors`](https://go.dev/doc/go1.13#errors) package ( [`errors.Unwrap`](https://go.dev/pkg/errors/#Unwrap) , [`errors.Is`](https://go.dev/pkg/errors/#Is) and [`errors.As`](https://go.dev/pkg/errors/#As) ) simplify unwrapping and inspecting wrapped errors. For more information, read the [`errors` package documentation](https://go.dev/pkg/errors/) , or see the [Error Value FAQ](https://go.dev/wiki/ErrorValueFAQ) . There will soon be a blog post as well. ### Minor changes to the library As always, there are various minor changes and updates to the library, made with the Go 1 [promise of compatibility](https://go.dev/doc/go1compat) in mind. #### [bytes](https://go.dev/pkg/bytes/) The new [`ToValidUTF8`](https://go.dev/pkg/bytes/#ToValidUTF8) function returns a copy of a given byte slice with each run of invalid UTF-8 byte sequences replaced by a given slice. #### [context](https://go.dev/pkg/context/) The formatting of contexts returned by [`WithValue`](https://go.dev/pkg/context/#WithValue) no longer depends on `fmt` and will not stringify in the same way. Code that depends on the exact previous stringification might be affected. #### [crypto/tls](https://go.dev/pkg/crypto/tls/) Support for SSL version 3.0 (SSLv3) [is now deprecated and will be removed in Go 1.14](https://go.dev/issue/32716) . Note that SSLv3 is the [cryptographically broken](https://tools.ietf.org/html/rfc7568) protocol predating TLS. SSLv3 was always disabled by default, other than in Go 1.12, when it was mistakenly enabled by default server-side. It is now again disabled by default. (SSLv3 was never supported client-side.) Ed25519 certificates are now supported in TLS versions 1.2 and 1.3. #### [crypto/x509](https://go.dev/pkg/crypto/x509/) Ed25519 keys are now supported in certificates and certificate requests according to [RFC 8410](https://www.rfc-editor.org/info/rfc8410) , as well as by the [`ParsePKCS8PrivateKey`](https://go.dev/pkg/crypto/x509/#ParsePKCS8PrivateKey) , [`MarshalPKCS8PrivateKey`](https://go.dev/pkg/crypto/x509/#MarshalPKCS8PrivateKey) , and [`ParsePKIXPublicKey`](https://go.dev/pkg/crypto/x509/#ParsePKIXPublicKey) functions. The paths searched for system roots now include `/etc/ssl/cert.pem` to support the default location in Alpine Linux 3.7+. #### [database/sql](https://go.dev/pkg/database/sql/) The new [`NullTime`](https://go.dev/pkg/database/sql/#NullTime) type represents a `time.Time` that may be null. The new [`NullInt32`](https://go.dev/pkg/database/sql/#NullInt32) type represents an `int32` that may be null. #### [debug/dwarf](https://go.dev/pkg/debug/dwarf/) The [`Data.Type`](https://go.dev/pkg/debug/dwarf/#Data.Type) method no longer panics if it encounters an unknown DWARF tag in the type graph. Instead, it represents that component of the type with an [`UnsupportedType`](https://go.dev/pkg/debug/dwarf/#UnsupportedType) object. #### [errors](https://go.dev/pkg/errors/) The new function [`As`](https://go.dev/pkg/errors/#As) finds the first error in a given error’s chain (sequence of wrapped errors) that matches a given target’s type, and if so, sets the target to that error value. The new function [`Is`](https://go.dev/pkg/errors/#Is) reports whether a given error value matches an error in another’s chain. The new function [`Unwrap`](https://go.dev/pkg/errors/#Unwrap) returns the result of calling `Unwrap` on a given error, if one exists. #### [fmt](https://go.dev/pkg/fmt/) The printing verbs `%x` and `%X` now format floating-point and complex numbers in hexadecimal notation, in lower-case and upper-case respectively. The new printing verb `%O` formats integers in base 8, emitting the `0o` prefix. The scanner now accepts hexadecimal floating-point values, digit-separating underscores and leading `0b` and `0o` prefixes. See the [Changes to the language](https://go.dev/doc/go1.13#language) for details. The [`Errorf`](https://go.dev/pkg/fmt/#Errorf) function has a new verb, `%w`, whose operand must be an error. The error returned from `Errorf` will have an `Unwrap` method which returns the operand of `%w`. #### [go/scanner](https://go.dev/pkg/go/scanner/) The scanner has been updated to recognize the new Go number literals, specifically binary literals with `0b`/`0B` prefix, octal literals with `0o`/`0O` prefix, and floating-point numbers with hexadecimal mantissa. The imaginary suffix `i` may now be used with any number literal, and underscores may be used as digit separators for grouping. See the [Changes to the language](https://go.dev/doc/go1.13#language) for details. #### [go/types](https://go.dev/pkg/go/types/) The type-checker has been updated to follow the new rules for integer shifts. See the [Changes to the language](https://go.dev/doc/go1.13#language) for details. #### [html/template](https://go.dev/pkg/html/template/) When using a `