gomod

• Specifying modules to process
• Using fetched dependencies
• gomod flags
• Vendoring
• Understanding reported dependencies
• Go 1.21+
• Full example walkthrough

Specifying modules ¹ to process

Hermeto can be run as follows

hermeto fetch-deps \
  --source ./my-repo \
  --output ./hermeto-output \
  '<JSON input>'

where 'JSON input' is

{
  // "gomod" tells Hermeto to process a go module
  "type": "gomod",
  // path to the module (relative to the --source directory)
  // defaults to "."
  "path": "."
}

The main argument accepts alternative forms of input, see Example: Pre-fetch dependencies for details.

Using fetched dependencies

See the Example for a complete walkthrough of Hermeto usage.

Hermeto downloads the required modules into the deps/gomod/ subpath of the output directory (hermeto-output/deps/gomod). Further down the file tree, at hermeto-output/deps/gomod/pkg/mod, is a directory formatted as the Go module cache

hermeto-output/deps/gomod/pkg/mod
└── cache
    └── download
        ├── github.com
        │   └── ...
        └── golang.org
            └── ...

To use this module cache during your build, set the GOMODCACHE environment variable. Hermeto generates GOMODCACHE along with other expected environment variables for you. See Example: Generate environment variables for more details.

For more information on Go's environment variables

go help environment

Note that the deps/gomod/ layout described above does not apply when using vendoring. With vendoring enabled, deps/gomod/ will be an empty directory. Instead, dependencies will be inside the vendor subdirectory of your module.

my-repo
└── vendor
    ├── github.com
    │   └── ...
    ├── golang.org
    │   └── ...
    └── modules.txt

Go will use the vendored dependencies automatically, but it's not a bad idea to set the environment variables generated by Hermeto anyway.

gomod flags

The hermeto fetch-deps command accepts the following gomod-related flags

--cgo-disable

Makes Hermeto internally disable cgo while processing your Go modules. Typically, you would want to use this flag if your modules do use C code and Hermeto is failing to process them. Hermeto will not attempt to disable cgo in your build (nor should you disable it yourself if you rely on C).

Disabling cgo should not prevent Hermeto from fetching your Go dependencies as usual. Note that Hermeto will not make any attempts to fetch missing C libraries. If required, you would need to get them through other means.

Deprecated flags

• --gomod-vendor (deprecated in v0.11.0)
• --gomod-vendor-check (deprecated in v0.11.0)
• --force-gomod-tidy (deprecated in v0.18.0)

All of them are deprecated and will have no effect when set. They are only kept for backwards compatibility reasons and will be removed in future releases.

Vendoring

Go supports vendoring to store the source code of all dependencies in the vendor/ directory alongside your module. Before go 1.17, go mod vendor used to download fewer dependencies than go mod download. Starting with 1.17, that is no longer true.

We generally discourage vendoring, but Hermeto does support processing repositories that contain vendored content. In this case, instead of a regular prefetching of dependencies, Hermeto will only validate if the contents of the vendor directory are consistent with what go mod vendor would produce.

Understanding reported dependencies

Hermeto reports two (arguably three) different types of dependencies in the generated SBOM for your Go modules

• gomod dependencies (Go modules)

• go-package dependencies (Go packages)

• from the downloaded modules

• from the standard library

gomod vs go-package

Best explained by the Go modules documentation

A module is a collection of packages that are released, versioned, and distributed together.

Your Go code imports individual packages, which come from modules. You might import a single package from a module that provides many, but Go (and Hermeto) has to download the whole module anyway. Effectively, modules are the smallest "unit of distribution." Go does have the ability to list the individual packages that your project imports. Hermeto makes use of this ability to report both the downloaded modules and the required packages.

The list of go-package dependencies reported by Hermeto is the full set of packages (transitively) required by your project.

⚠ If any of your module dependencies has a missing checksum in go.sum, the list may be incomplete.

The list of gomod dependencies is the set of modules that Hermeto downloaded to satisfy the go-package dependencies.

Note that versioning applies to modules, not packages. When reporting the versions of Go packages, Hermeto uses the version of the module that provides the package.

How to match a package to a module?

Borrowing from the Go modules documentation again

For example, the module "golang.org/x/net" contains a package in the directory "html". That package’s path is "golang.org/x/net/html"

The name of a package starts with the name of the module that provides it.

In the source tree, what are modules? What are packages?

To simplify a little

• Does the directory have a go.mod file? It's a module (provides packages).
• Does the directory have any *.go files? It's a package (is importable).
• Does it have both? It's both a module and a package.

stdlib dependencies

Go is able to list even the standard library packages that your project imports. Hermeto exposes these as go-package dependencies, with caveats. Hermeto uses some version of Go to list the dependencies. This may or may not be the same version that you will use to build your project. We do not presume that the versions would be the same, hence why:

• the reported stdlib packages may be slightly inaccurate (e.g. new packages in new Go versions)
• the versions of stdlib packages are not reported

What identifies stdlib dependencies in the go-package list?

• does not have a version

• the name does not start with a hostname

• io/fs - standard library

• golang.org/x/net - external

Missing checksums

Go stores the checksums of all your dependency modules in the go.sum file. Go typically manages this file entirely on its own, but if any of your dependencies do end up missing, it can cause issues for Hermeto and for Go itself.

For Hermeto, a missing checksum means that the offending module gets downloaded without checksum verification (or with partial checksum verification - Hermeto does consult the Go checksum database). Due to go list behavior, it also means that the go-package dependency listing may be incomplete².

For Go, a missing checksum will cause the go build or go run commands to fail.

Please make sure to keep your go.sum file up to date, perhaps by incorporating the go mod tidy command in your dev workflow.

Go 1.21+ (since v0.5.0)

Starting with Go 1.21, Go changed the meaning of the go 1.X directive in that it now specifies the minimum required version of Go rather than a suggested version as it originally did. The format of the version string in the go directive now also includes the micro release and if you don't include the micro release in your go.qmod file yourself (i.e. you only specify the language release) Go will try to correct it automatically inside the file. Last but not least, Go 1.21 also introduced a new keyword [toolchain][] to the go.mod file. What this all means in practice for end users is that you may not be able to process your go.mod file with an older version of Go (and hence older hermeto) as you could in the past for various reasons. Many projects bump their required Go toolchain's micro release as soon as it becomes available upstream (i.e. not waiting for distributions to bundle them properly). This caused problems in version v0.5.0 because the container image's version simply may not have been high enough to process a given project's go.mod file. Therefore, version v0.7.0 introduced a mechanism to always rely on the origin 0th release of a toolchain (e.g. 1.21.0) and use the GOTOOLCHAIN=auto setting to instruct Go to fetch any toolchain as specified by the go.mod file automatically, hence allowing us to keep up with frequent micro version bumps. Note that such a language version would still need to be officially marked as supported by hermeto, i.e. we'd not allow Go to fetch e.g. a 1.22 toolchain if the maximum supported Go version by hermeto were 1.21!

Example

Let's show Hermeto usage by building the glorious fzf CLI tool hermetically. To follow along, clone the repository to your local disk.

git clone https://github.com/junegunn/fzf --branch=0.34.0

Pre-fetch dependencies

In order to pre-fetch the dependencies, we will pass the source and output directories as well as the path for the gomod package manager to be able to find the go.mod file.

See the gomod documentation for more details about running Hermeto for pre-fetching gomod dependencies.

hermeto fetch-deps \
  --source ./fzf \
  --output ./hermeto-output \
  '{"path": ".", "type": "gomod"}'

Generate environment variables

Next, we need to generate the environment file so that the go build command can find the cached dependencies

hermeto generate-env ./hermeto-output -o ./hermeto.env --for-output-dir /tmp/hermeto-output

We can see the variables needed by the compiler

$ cat hermeto.env
export GOCACHE=/tmp/hermeto-output/deps/gomod
export GOMODCACHE=/tmp/hermeto-output/deps/gomod/pkg/mod
export GOPATH=/tmp/hermeto-output/deps/gomod

Inject project files

While the gomod package manager does not currently need to modify any content in the source directory to inject the dependencies, the inject-files command should be run to ensure that the operation is performed if this step becomes a requirement in the future.

hermeto inject-files ./hermeto-output --for-output-dir /tmp/hermeto-output

Write the Dockerfile

As mentioned in the steps above, the only change that needs to be made in the Dockerfile or Dockerfile is to source the environment file before building the binary.

FROM golang:1.19.2-alpine3.16 AS build

COPY ./fzf /src/fzf
WORKDIR /src/fzf

RUN source /tmp/hermeto.env && \
    go build -o /fzf

FROM registry.access.redhat.com/ubi9/ubi-minimal:9.0.0

COPY --from=build /fzf /usr/bin/fzf

CMD ls | fzf

Build the container

Finally, we can build and test the container to ensure that we have successfully built the binary.

podman build . \
  --volume "$(realpath ./hermeto-output)":/tmp/hermeto-output:Z \
  --volume "$(realpath ./hermeto.env)":/tmp/hermeto.env:Z \
  --network none \
  --tag fzf

# test that it worked
podman run --rm -ti fzf

---

1. You may have noticed a slight naming issue. You use the main argument, also called PKG, to specify a module to process. Even worse, Go has packages as well (see gomod vs go-package). What gives? As far as we know, most languages/package managers use the opposite naming. For example, in Python, modules are *.py files, packages are collections of modules. In npm, modules are directories/files you can require(), packages are the top-level directories with package.json. In Hermeto, we stick to the more common naming. ↩

2. When a module does not have a checksum in go.sum, the go list command returns only basic information and an error for the packages from said module. Go doesn't return any information about the dependencies of the affected packages. This can cause Hermeto to miss the transitive package dependencies of packages from checksum-less modules. ↩