Profpatsch/nix/buildGo

View full contents ↓

buildGo(7)

buildGo - build Go libraries and programs by invoking the compiler directly

buildGo is a Nix build system for Go that calls go(1)'s underlying tools (go tool compile, go tool link) directly, rather than delegating to go build and its module resolution. Composition of packages and programs therefore stays inside Nix, in the style of Bazel's rules_go. Every dependency is an explicit Nix value; there is no go.mod resolution, no network access and no vendor directory at build time.

buildGo is a function, not a package: it is imported by //nix/packages.nix and reached as depot.nix.buildGo. It produces no bin/ output of its own and cannot be built or installed.

Any go.mod in the tree exists only for editor tooling and go test; it has no effect on the Nix build. Dependency versions come from go-deps.nix files, which follow the versions the go.work workspace resolves; see Versions can drift under CAVEATS.

program({ name, srcs, deps ? [], x_defs ? {}, embeds ? [] })

Build an executable.

name

Name of the program and of the resulting executable. Required.

srcs

List of paths to source files. Required. All sources are compiled into package main.

deps

List of Go libraries to link against.

x_defs

Attribute set of linker variable definitions, passed as -X flags. Used to stamp values such as a version into the binary at link time.

embeds

List of files and directories the sources' //go:embed directives draw from. See EMBEDDING FILES.

package({ name, srcs, deps ? [], path ? name, sfiles ? [], embeds ? [] })

Build an importable library. The result carries both the compiled archive and the sources, as downstream consumers need both.

name

Name of the library. Required.

srcs

List of paths to source files. Required.

embeds

As for program().

deps

List of Go libraries to link against.

path

Go import path of the result. Defaults to name.

sfiles

List of Go assembly (*.s) files, assembled and packed into the archive. Required by several packages under golang.org/x.

external({ path, src, deps ? [], tags ? [] })

Build a Go repository that was not written for buildGo, laid out in the standard Go tooling style.

path

Go import path of the repository, e.g. github.com/emersion/go-imap. Required.

src

Path to the source directory, typically a fetchFromGitHub() result. Required.

deps

List of other external() results this repository imports.

tags

Build tags to set while analysing the repository. See BUILD TAGS.

external() analyses the repository and returns a nested attribute set mirroring its directory structure. Each directory that contains a Go package gains a gopkg attribute holding the compiled library. A dependency must always be referred to by an attribute that has a gopkg, which is not necessarily the root. Three shapes occur:

Single-package repository

gopkg sits at the root, so the result is used directly:

deps = [ goDeps.creack-pty ];

Repository with subpackages

Navigate to the ones actually imported:

deps = [ goDeps.golang-x-net.html ];

Repository with no root package

golang.org/x/sys has code only in subdirectories:

deps = [ goDeps.golang-x-sys.unix ];

Passing an attribute without a gopkg is an error, and the resulting message enumerates the valid paths within that dependency.

Dependencies between packages inside one repository are resolved automatically from the import statements. Only imports of other repositories need to be listed in deps, where they are matched by their goImportPath.

tags sets the build tags used when the analyser decides which files a package consists of. It selects sources, not compiler behaviour: a file excluded by a tag is never handed to go tool compile at all.

The case that motivates it is assembly. Several cryptographic repositories ship hand-written amd64 assembly next to a pure-Go implementation of the same routines, chosen by the purego tag. buildGo stages each package into a flat directory and copies only *.go and *.s files, so an assembly file that #includes a header fails to assemble:

fp_amd64.s:6: #include: open .../pkg/include/fp_amd64.h:
    no such file or directory

Selecting the pure-Go sources avoids the problem at its root rather than patching around it:

circl = depot.nix.buildGo.external {
  path = "github.com/cloudflare/circl";
  src = pkgs.fetchFromGitHub { /* ... */ };
  tags = [ "purego" ];
};

Copying the headers alongside the assembly would not be enough in general, because such headers are routinely included by relative path across package boundaries – circl/dh/x25519 includes ../../math/fp25519/fp_amd64.h – which presumes the original repository tree that the flat staging has already flattened away.

The cost is performance: the pure-Go path is slower than the assembly it replaces. For work measured in a handful of operations per request this is not observable, but a tag should not be reached for merely to silence a build error in a hot path.

buildGo supports //go:embed, but only for files named in embeds:

depot.nix.buildGo.program {
  name = "myprog";
  srcs = [ ./main.go ./serve.go ];
  embeds = [ ./templates ./static/style.css ];
}

Each entry is a file or a directory inside the package directory, which is the directory of the first entry of srcs. Its name as a //go:embed pattern sees it is its path relative to that directory: ./templates above is templates, so //go:embed templates/*.html finds it. A directory is taken whole and the patterns choose from it, so listing ./templates once covers every template added later. An entry outside the package directory is an evaluation error.

Two mistakes that used to build fine now fail the build:

The compiler cannot glob the filesystem, so go build normally resolves patterns and hands the result to go tool compile as a -embedcfg JSON file. buildGo does the same in a separate derivation: it stages srcs and embeds into one directory under their package-relative names, reads the patterns with go list, and resolves each one with mkembedcfg, which follows go build's rules: a pattern is a path.Match() glob, a matched directory contributes every regular file below it, names beginning with '.' or '_' are skipped unless the pattern has the all: prefix, and nested modules are not entered. The compile step reads the embedded files from that derivation and nothing else. A package with no embeds gets no such derivation at all.

external() lists embeds itself: its analyser records what each package's patterns match in the fetched source tree.

Earlier versions took the directory of the first source, dirOf (head srcs), and put it into the embed derivation. For path-literal sources that is a path to the working directory, and putting a path into a derivation copies it into the store at evaluation time - the whole directory, including build output and anything else untracked, for every package, whether it embedded anything or not. For one package that was 235MB, hashed on every evaluation of the system that used it, and every change to an unrelated file in the directory rebuilt the package.

The same versions documented the opposite: that path-literal sources left the directory as /nix/store and embeds silently empty, and that sources had to be staged into a directory by hand and passed as strings. The first half was never true; the staging worked, but only by accident of the copy. Listing embeds makes what is embedded explicit and copies nothing it does not name.

A program with one local library:

let
  lib = depot.nix.buildGo.package {
    name = "somelib";
    srcs = [ ./lib/foo.go ./lib/bar.go ];
  };
in depot.nix.buildGo.program {
  name = "my-program";
  srcs = [ ./main.go ];
  deps = [ lib ];
}

External dependencies are conventionally pinned in a go-deps.nix beside the package, or taken from the shared one:

{ depot, pkgs, ... }:
{
  creack-pty = depot.nix.buildGo.external {
    path = "github.com/creack/pty";
    src = pkgs.fetchFromGitHub {
      owner = "creack";
      repo = "pty";
      rev = "edfbf75025b0ba4ee17c19f52d9b600fad80a787";
      sha256 = "0yy4zhfb7vrrbwd13rcw0zzcq0ami3zv3hp0x7g7il6mrbadcf25";
    };
  };
}

Rather than reproducing every pattern here, the following files in this repository are worth reading as worked examples:

//nix/buildGo/example

A minimal
**package**()
plus
**program**(),
including
*x_defs*.

//users/Profpatsch/link-check/default.nix

The smallest realistic case: one source file, one external subpackage
dependency.

//users/Profpatsch/go-deps.nix

The shared dependency pin set, and the idiom of navigating to subpackages.

//users/Profpatsch/mailweb/default.nix

*embeds*
with two directories of assets.

//users/Profpatsch/asciinema-server/default.nix

*embeds*
listing single files, where a directory holds more than the patterns should
take.

//users/Profpatsch/source-forge/default.nix

A program with a separate
*man*
output, via
**symlinkJoin**().

//users/Profpatsch/maildir-varlink/default.nix

Wrapping the result with
**wrapProgram**()
to put runtime dependencies on
`PATH`.

go(1), go-deps-check(1)

//README.md for the default.nix versus packages.nix rules that govern how a package is named and found.

nix-1p, a brief introduction to the Nix language.

buildGo originates from the TVL depot, where it was written by Vincent Ambo, and is licensed Apache 2.0. It has since gained //go:embed support, assembly support and static linking.

buildGo has no test support. *_test.go files are simply left out of srcs, where they are inert. Run them with the ordinary Go tooling instead, which the go.work workspace at the repository root makes possible:

$ go test ./users/Profpatsch/mailweb/

Consequently a package can build under Nix while its tests do not compile.

The Nix build ignores go.mod entirely, so the version pinned in go-deps.nix is what is actually compiled. When the two disagree, editor tooling and go test see one version and the built artefact another, and the build itself gives no diagnostic.

The rule that keeps them together is that go.work decides and go-deps.nix follows. A workspace resolves exactly one version per module across all of its modules, which is the same shape as a set of Nix pins, so each pin is simply that version. To add or upgrade a dependency:

$ go -C users/Profpatsch/foo get example.com/bar@v1.3.0
$ go work sync
$ go-deps-check

go work sync writes the workspace's choice back into every module's go.mod, so each module still resolves on its own, as pkgs.buildGoModule() sees it; commit those go.mod and go.sum changes on their own. go-deps-check(1) then lists every pin that differs. Bump those, update their hashes, and rebuild what uses them - a new version may import packages its deps do not list yet. A buildGoModule() package's vendorHash changes with it.

A project outside go.work keeps pins of its own and is not checked.

The standard library is built with CGO_ENABLED=0 so that everything links statically and no libc or dynamic loader becomes a runtime dependency. Packages that require cgo cannot be built with buildGo; use pkgs.buildGoModule() for those.

Manual pages and other documentation are installed by hand, typically by wrapping the result in symlinkJoin() with an added man output.