Profpatsch/nix/buildGo

View full contents ↓

buildGo(7)

buildGo - build Go libraries and programs by invoking the compiler directly

buildGo is a Nix build system for Go that calls go(1)'s underlying tools (go tool compile, go tool link) directly, rather than delegating to go build and its module resolution. Composition of packages and programs therefore stays inside Nix, in the style of Bazel's rules_go. Every dependency is an explicit Nix value; there is no go.mod resolution, no network access and no vendor directory at build time.

buildGo is a function, not a package: it is imported by //nix/packages.nix and reached as depot.nix.buildGo. It produces no bin/ output of its own and cannot be built or installed.

Any go.mod in the tree exists only for editor tooling and go test; it has no effect on the Nix build. Dependency versions come from go-deps.nix files, and the two can disagree without any warning – see CAVEATS.

program({ name, srcs, deps ? [], x_defs ? {} })

Build an executable.

name

Name of the program and of the resulting executable. Required.

srcs

List of paths to source files. Required. All sources are compiled into package main.

deps

List of Go libraries to link against.

x_defs

Attribute set of linker variable definitions, passed as -X flags. Used to stamp values such as a version into the binary at link time.

package({ name, srcs, deps ? [], path ? name, sfiles ? [] })

Build an importable library. The result carries both the compiled archive and the sources, as downstream consumers need both.

name

Name of the library. Required.

srcs

List of paths to source files. Required.

deps

List of Go libraries to link against.

path

Go import path of the result. Defaults to name.

sfiles

List of Go assembly (*.s) files, assembled and packed into the archive. Required by several packages under golang.org/x.

external({ path, src, deps ? [], tags ? [] })

Build a Go repository that was not written for buildGo, laid out in the standard Go tooling style.

path

Go import path of the repository, e.g. github.com/emersion/go-imap. Required.

src

Path to the source directory, typically a fetchFromGitHub() result. Required.

deps

List of other external() results this repository imports.

tags

Build tags to set while analysing the repository. See BUILD TAGS.

external() analyses the repository and returns a nested attribute set mirroring its directory structure. Each directory that contains a Go package gains a gopkg attribute holding the compiled library. A dependency must always be referred to by an attribute that has a gopkg, which is not necessarily the root. Three shapes occur:

Single-package repository

gopkg sits at the root, so the result is used directly:

deps = [ goDeps.creack-pty ];

Repository with subpackages

Navigate to the ones actually imported:

deps = [ goDeps.golang-x-net.html ];

Repository with no root package

golang.org/x/sys has code only in subdirectories:

deps = [ goDeps.golang-x-sys.unix ];

Passing an attribute without a gopkg is an error, and the resulting message enumerates the valid paths within that dependency.

Dependencies between packages inside one repository are resolved automatically from the import statements. Only imports of other repositories need to be listed in deps, where they are matched by their goImportPath.

tags sets the build tags used when the analyser decides which files a package consists of. It selects sources, not compiler behaviour: a file excluded by a tag is never handed to go tool compile at all.

The case that motivates it is assembly. Several cryptographic repositories ship hand-written amd64 assembly next to a pure-Go implementation of the same routines, chosen by the purego tag. buildGo stages each package into a flat directory and copies only *.go and *.s files, so an assembly file that #includes a header fails to assemble:

fp_amd64.s:6: #include: open .../pkg/include/fp_amd64.h:
    no such file or directory

Selecting the pure-Go sources avoids the problem at its root rather than patching around it:

circl = depot.nix.buildGo.external {
  path = "github.com/cloudflare/circl";
  src = pkgs.fetchFromGitHub { /* ... */ };
  tags = [ "purego" ];
};

Copying the headers alongside the assembly would not be enough in general, because such headers are routinely included by relative path across package boundaries – circl/dh/x25519 includes ../../math/fp25519/fp_amd64.h – which presumes the original repository tree that the flat staging has already flattened away.

The cost is performance: the pure-Go path is slower than the assembly it replaces. For work measured in a handful of operations per request this is not observable, but a tag should not be reached for merely to silence a build error in a hot path.

buildGo supports //go:embed. The compiler cannot glob the filesystem itself, so buildGo runs go list over the source directory to discover which files each pattern matches, and converts the result into the -embedcfg JSON the compiler expects.

This is where the one genuinely surprising constraint of buildGo lives.

The source directory handed to go list is computed as the directory of the first entry of srcs. When sources are given as ordinary path literals:

srcs = [ ./main.go ./serve.go ];

each path is copied into the store individually, becoming its own /nix/store/<hash>-main.go. Their common directory is therefore /nix/store itself, which contains no embeddable files. go list finds nothing, the embed configuration comes out empty, and the embedded variables are silently empty at runtime. The build does not fail.

The remedy is to stage the sources and the files they embed into a single directory first, and pass strings pointing into it rather than path literals:

let
  src = pkgs.runCommandLocal "myprog-src" {} ''
    mkdir -p $out/templates
    cp ${./main.go} $out/main.go
    cp ${./templates}/*.html $out/templates/
  '';
in depot.nix.buildGo.program {
  name = "myprog";
  srcs = [ "${src}/main.go" ];
}

A path literal (./main.go) is copied to the store on its own; a string ("${src}/main.go") names a file inside an existing store directory and keeps its siblings reachable. The distinction is invisible at the call site and matters only here.

Note that the staging derivation copies a fixed set of files. Adding a template with a new extension means widening the cp glob as well as the //go:embed pattern; forgetting the former leaves the file out of the build entirely.

Patterns are matched with filepath.Match(), so templates/*.html and plain paths both work, and a pattern naming a directory matches the files directly below it.

A pattern that matches nothing falls back to embedding every file go list reported, rather than failing. A typo in a pattern thus yields a working build with the wrong contents.

A program with one local library:

let
  lib = depot.nix.buildGo.package {
    name = "somelib";
    srcs = [ ./lib/foo.go ./lib/bar.go ];
  };
in depot.nix.buildGo.program {
  name = "my-program";
  srcs = [ ./main.go ];
  deps = [ lib ];
}

External dependencies are conventionally pinned in a go-deps.nix beside the package, or taken from the shared one:

{ depot, pkgs, ... }:
{
  creack-pty = depot.nix.buildGo.external {
    path = "github.com/creack/pty";
    src = pkgs.fetchFromGitHub {
      owner = "creack";
      repo = "pty";
      rev = "edfbf75025b0ba4ee17c19f52d9b600fad80a787";
      sha256 = "0yy4zhfb7vrrbwd13rcw0zzcq0ami3zv3hp0x7g7il6mrbadcf25";
    };
  };
}

Rather than reproducing every pattern here, the following files in this repository are worth reading as worked examples:

//nix/buildGo/example

A minimal package() plus program(), including x_defs.

//users/Profpatsch/link-check/default.nix

The smallest realistic case: one source file, one external subpackage dependency.

//users/Profpatsch/go-deps.nix

The shared dependency pin set, and the idiom of navigating to subpackages.

//users/Profpatsch/mailweb/default.nix

Embed staging with a subdirectory of assets, as described above.

//users/Profpatsch/source-forge/default.nix

A program with a separate man output, via symlinkJoin().

//users/Profpatsch/maildir-varlink/default.nix

Wrapping the result with wrapProgram() to put runtime dependencies on PATH.

go(1)

//README.md for the default.nix versus packages.nix rules that govern how a package is named and found.

nix-1p, a brief introduction to the Nix language.

buildGo originates from the TVL depot, where it was written by Vincent Ambo, and is licensed Apache 2.0. It has since gained //go:embed support, assembly support and static linking.

buildGo has no test support. *_test.go files are simply left out of srcs, where they are inert. Run them with the ordinary Go tooling instead, which the go.work workspace at the repository root makes possible:

$ go test ./users/Profpatsch/mailweb/

Consequently a package can build under Nix while its tests do not compile.

The Nix build ignores go.mod entirely, so the version pinned in go-deps.nix is what is actually compiled. When the two disagree, editor tooling and go test see one version and the built artefact another, with no diagnostic.

The standard library is built with CGO_ENABLED=0 so that everything links statically and no libc or dynamic loader becomes a runtime dependency. Packages that require cgo cannot be built with buildGo; use pkgs.buildGoModule() for those.

Manual pages and other documentation are installed by hand, typically by wrapping the result in symlinkJoin() with an added man output.