Revising Go packaging guidelines (2024)
export CGO_CPPFLAGS="${CPPFLAGS}" export CGO_CFLAGS="${CFLAGS}" export CGO_CXXFLAGS="${CXXFLAGS}" export CGO_LDFLAGS="${LDFLAGS}" export GOFLAGS="-buildmode=pie -trimpath -ldflags=-linkmode=external -mod=readonly -modcacherw"
# or alternatively you can define some of these flags from the CLI
hello, the Arch Linux Go packaging guidelines[0] currently suggest this for building software written in Go: options
go build \ -trimpath \ -buildmode=pie \ -mod=readonly \ -modcacherw \ -ldflags "-linkmode external -extldflags \"${LDFLAGS}\"" \ .
[0]: https://wiki.archlinux.org/title/Go_package_guidelines There's also section "2.2.2 Supporting debug packages" which contradicts the previous section. ## Suggestion I'd like to propose changing this to something like:
prepare() { cd ${pkgname}-${pkgver} go mod download }
build() { cd ${pkgname}-${pkgver} go build . }
package() { cd ${pkgname}-${pkgver} install -Dm0755 -t "${pkgdir}/usr/bin/" "${pkgname}" }
Plus any extra flags (for example for debug packages to work, if any are needed at all - there's a note about this towards the end of this essay). Instead of suggesting two (three) different approaches, the packaging guidelines should endorse argv over environment variables (how to set them with environment variables can still be documented in a later section of the guideline). The reason I'm suggesting this: ## Passing flags over argv Setting these options through environment variables is for cases when `go` isn't executed directly by our PKGBUILD, but instead by some indirection like a Makefile. Makefiles are more popular in Go projects than they are in e.g. Rust projects, but presumably because for many years (10+) there was no analogous to `env!("CARGO_PKG_VERSION")` to embed a version string into the binary for --version output. Since almost exactly two years ago, in Go 1.18 released March 2022, there's now `runtime/debug.ReadBuildInfo` which gives you a reference to an embedded struct that contains a version string[1]. [1]: https://pkg.go.dev/runtime/debug#ReadBuildInfo Hopefully more Go projects are going to favor this in the future over custom build scripts and passing `-X main.Version=...` with output from `git describe --tags`. If this catches on, people might think of `go build` more like they currently think of `cargo build` and I think the Go package guideline should encourage this. Not just for Go but for software in general, using `git describe` should be considered an anti-pattern. In the way the Arch Linux build system currently works, only the checked out files are authenticated and pinned by sha256sums= and friends, but `git describe` explicitly works on unauthenticated git objects in `.git/`. I've published a writeup on how to take advantage of this on the reproducible builds list in September 2023[2]. [2]: https://lists.reproducible-builds.org/pipermail/rb-general/2023-September/00... ## Using the Go native Linker Binaries built by the Go compiler have been reproducible for a long time, however Go binaries built by Arch Linux usually aren't. This has led to some confusion, with bug reports to upstreams of Go software because it wasn't obvious that the package not being reproducible in this case isn't a problem in upstream's source code, but the way Arch Linux compiles it. Arch Linux generally prefers binaries with hardening flags enabled[3] and therefore explicitly opts into a non-default linker (Cgo), but it seems to be the unpopular choice. Cgo seems to have less support than normal Go on Linux (evidently, since Cgo has been known to have reproducible build issues for years now). Other Linux distributions seem to stick to normal Go too, Nix has Cgo off by default[4] and searching through nixpkgs for "buildGoModule" yields about 2k results, searching for "CGO_ENABLED" has 105 results, with more than half being "CGO_ENABLED=0" though. Alpine Linux only sets GOFLAGS globally[5] (which Arch Linux currently doesn't), searching for `rg -l --multiline '\tgo.*build'` in aports matches 356 files, `rg -l CGO_ENABLED=0` matches 23 files and `rg -l CGO_ENABLED=1` matches 18 files. [3]: https://github.com/jvoisin/compiler-flags-distro [4]: https://github.com/NixOS/nixpkgs/blob/b081342f1c16e4cbe4f40f139bbdda1475ea30... [5]: https://git.alpinelinux.org/abuild/tree/default.conf?h=3.12.0#n5 The global GOFLAGS as specified in Alpine Linux are:
export GOFLAGS="-buildmode=pie -modcacherw -trimpath -buildvcs=false"
Which we could consider setting globally too, like we do with RUSTFLAGS. Figuring out what Debian does was somewhat challenging, their integration with the go build system is called dh-golang[6], they seem to set CGO_ENABLED=1 only when cross-compiling, with go >= 1.13 (released 2019) they seem to build with `go install -trimpath ...` and no further GOFLAGS environment variable. [6]: https://salsa.debian.org/go-team/packages/dh-golang/-/blob/debian/sid/lib/De... I understand the concern of Go possibly being more prone to ROP-style exploits when 1) built with the native linker and 2) used with e.g. a vulnerable version of libgit2, however, as of 2024, there have been barely any memory-corruption based exploits for Go software. ## Motivation - Most of our Go software is currently not reproducible due to Cgo, including core/libcap, which is the last unreproducible package in docker.io/library/archlinux - The barrier for packaging Go in Arch Linux is currently somewhat high (compared to e.g. packaging Rust), the guideline requires too much interpretation and could be improved - Quirks that are only needed for old Go projects (like 2.1.1) should be listed towards the end instead of being the first code block in the guideline --- cheers, kpcyrd
On Tue, Mar 19, 2024 at 12:17:11AM +0100, kpcyrd wrote:
hello,
Yo, Thanks for writing this.
## Suggestion
I'd like to propose changing this to something like:
prepare() { cd ${pkgname}-${pkgver} go mod download }
build() { cd ${pkgname}-${pkgver} go build . }
package() { cd ${pkgname}-${pkgver} install -Dm0755 -t "${pkgdir}/usr/bin/" "${pkgname}" }
Plus any extra flags (for example for debug packages to work, if any are needed at all - there's a note about this towards the end of this essay). Instead of suggesting two (three) different approaches, the packaging guidelines should endorse argv over environment variables (how to set them with environment variables can still be documented in a later section of the guideline).
While it would be cool if `go build` would do what you expect, it doesn't. Go does not have some similar to `build.rs` that describes the targets so you need to figure out where the binaries resides before you can build anything. You could *try* to do `go build ./...` but you might end up with binaries you don't want to build. In more complicated examples you would need to use the `Makefile` as you do in other projects because they are doing to much in their build system for the raw `go build` calls to work. tl;dr: It needs to be more complicated because upstream still pretends that `go build` is a buildsystem.
## Passing flags over argv
Setting these options through environment variables is for cases when `go` isn't executed directly by our PKGBUILD, but instead by some indirection like a Makefile. Makefiles are more popular in Go projects than they are in e.g. Rust projects, but presumably because for many years (10+) there was no analogous to `env!("CARGO_PKG_VERSION")` to embed a version string into the binary for --version output.
Since almost exactly two years ago, in Go 1.18 released March 2022, there's now `runtime/debug.ReadBuildInfo` which gives you a reference to an embedded struct that contains a version string[1].
[1]: https://pkg.go.dev/runtime/debug#ReadBuildInfo
Hopefully more Go projects are going to favor this in the future over custom build scripts and passing `-X main.Version=...` with output from `git describe --tags`. If this catches on, people might think of `go build` more like they currently think of `cargo build` and I think the Go package guideline should encourage this.
This assumes we are pulling from git, *and* don't use `-buildvcs=false`. However I don't see this likely for all projects and Makefiles do a lot more then *just* being a wrapper for `-X` flags. So while nice in theory, I don't think it would be workable in practise and people would end up reverse engineering Makefiles if they want to drop it. It's fine to use it if it exists.
## Using the Go native Linker
Binaries built by the Go compiler have been reproducible for a long time, however Go binaries built by Arch Linux usually aren't. This has led to some confusion, with bug reports to upstreams of Go software because it wasn't obvious that the package not being reproducible in this case isn't a problem in upstream's source code, but the way Arch Linux compiles it.
Arch Linux generally prefers binaries with hardening flags enabled[3] and therefore explicitly opts into a non-default linker (Cgo), but it seems to be the unpopular choice. Cgo seems to have less support than normal Go on Linux (evidently, since Cgo has been known to have reproducible build issues for years now). Other Linux distributions seem to stick to normal Go too, Nix has Cgo off by default[4] and searching through nixpkgs for "buildGoModule" yields about 2k results, searching for "CGO_ENABLED" has 105 results, with more than half being "CGO_ENABLED=0" though. Alpine Linux only sets GOFLAGS globally[5] (which Arch Linux currently doesn't), searching for `rg -l --multiline '\tgo.*build'` in aports matches 356 files, `rg -l CGO_ENABLED=0` matches 23 files and `rg -l CGO_ENABLED=1` matches 18 files.
Generally I agree that we should remove `CGO_ENABLED` and rather set it for unsupported plattform. It will however break existing porting efforts but it should be solved somewhere else regardless. ie riscv/arm/$someport set this in their buildflags.
The global GOFLAGS as specified in Alpine Linux are:
export GOFLAGS="-buildmode=pie -modcacherw -trimpath -buildvcs=false"
Which we could consider setting globally too, like we do with RUSTFLAGS.
The issue here is because `GOFLAGS` was standardized late in the development a lot of the `Makefiles` people write for Go overwrites the entire variable and is re-purposed to carry CLI flags which isn't compatible. So while I think we should distribute proper Goflags here it's not going to solve it for all cases as people still would need to deal with bad build systems. However we still need a couple of more flags: export GOFLAGS="-buildmode=pie -modcacherw -buildvcs=false -ldflags=-Bgobuildid -ldflags=-compressdwarf=false" Which would cover all our current usage of debug packages and reproducible builds and some bare minimum hardening with `-buildmode=pie`.
Figuring out what Debian does was somewhat challenging, their integration with the go build system is called dh-golang[6], they seem to set CGO_ENABLED=1 only when cross-compiling, with go >= 1.13 (released 2019) they seem to build with `go install -trimpath ...` and no further GOFLAGS environment variable.
Debian also packages everything locally so they will avoid having network calls during `go install`, however for all purposes it's equivalent to `go build`. -- Morten Linderud PGP: 9C02FF419FECBE16
participants (2)
-
kpcyrd
-
Morten Linderud