pip

• Specifying packages to process
• requirements.txt
• Project metadata
• Distribution formats
• Using fetched dependencies
• Working with Rust-based dependencies
• Troubleshooting
• Full example walkthrough

Specifying packages to process

The "pip packages" that Hermeto can process are root directories of Python projects. They should have

• One or more requirements files (unless the project has no dependencies)

• See what Hermeto requires

• A file defining the project metadata

• See what Hermeto supports

Then Hermeto can be run as follows

hermeto fetch-deps \
  --source ./my-repo \
  --output ./hermeto-output \
  '<JSON input>'

where 'JSON input' is

{
  "type": "pip",
  // path to the package (relative to the --source directory)
  // defaults to "."
  "path": ".",
  // specify requirements files (relative to the package path)
  // defaults to ["requirements.txt"] or [] if the file does not exist
  "requirements_files": ["requirements.txt", "requirements-extra.txt"],
  // specify *build* requirements files
  // defaults to ["requirements-build.txt"] or [] if the file does not exist
  "requirements_build_files": ["requirements-build.txt"],
  // binary filter object to prefetch specific wheels
  // defaults to None - only sdists are prefetched by default
  "binary": {
    "packages": "tensorflow",
    "arch": "x86_64",
    "os": "linux",
    "py_version": 312
  }
}

or more simply by just invoking hermeto fetch-deps pip.

For more information on using build requirements and binary filter object, see Distribution Formats section.

The main argument accepts alternative forms of input, see Example: Pre-fetch dependencies.

requirements.txt

Hermeto downloads dependencies explicitly declared in lockfiles. For pip, the closest thing to a lockfile would be a "fully resolved" requirements.txt - must contain all the transitive dependencies, must pin them to exact versions.

A good way to generate requirements.txt is via pip-compile. Note that pip-compile supports reading dependencies directly from project files (e.g. pyproject.toml, setup.cfg, setup.py) or from "requirements.in" input files.

Example: pyproject.toml

[project]
name = "my_package"
version = "0.1.0"
dependencies = [
    "requests",
    "dockerfile-parse @ https://github.com/containerbuildsystem/dockerfile-parse/archive/refs/tags/2.0.0.tar.gz"
]

pip-compile pyproject.toml --generate-hashes

Example: requirements.in

# requirements.in
requests
dockerfile-parse @ https://github.com/containerbuildsystem/dockerfile-parse/archive/refs/tags/2.0.0.tar.gz

pip-compile requirements.in --generate-hashes

Result: requirements.txt

#
# This file is autogenerated by pip-compile with Python 3.10
# by the following command:
#
#    pip-compile --generate-hashes pyproject.toml
#
certifi==2022.12.7 \
    --hash=sha256:35824b4c3a97115964b408844d64aa14db1cc518f6562e8d7261699d1350a9e3 \
    --hash=sha256:4ad3232f5e926d6718ec31cfc1fcadfde020920e278684144551c91769c7bc18
    # via requests
charset-normalizer==3.0.1 \
    --hash=sha256:00d3ffdaafe92a5dc603cb9bd5111aaa36dfa187c8285c543be562e61b755f6b \
    --hash=sha256:024e606be3ed92216e2b6952ed859d86b4cfa52cd5bc5f050e7dc28f9b43ec42 \
    --hash=sha256:0298eafff88c99982a4cf66ba2efa1128e4ddaca0b05eec4c456bbc7db691d8d \
    --hash=sha256:02a51034802cbf38db3f89c66fb5d2ec57e6fe7ef2f4a44d070a593c3688667b \
    --hash=sha256:083c8d17153ecb403e5e1eb76a7ef4babfc2c48d58899c98fcaa04833e7a2f9a \
    --hash=sha256:0a11e971ed097d24c534c037d298ad32c6ce81a45736d31e0ff0ad37ab437d59 \
    --hash=sha256:0bf2dae5291758b6f84cf923bfaa285632816007db0330002fa1de38bfcb7154 \
    --hash=sha256:0c0a590235ccd933d9892c627dec5bc7511ce6ad6c1011fdf5b11363022746c1 \
    --hash=sha256:0f438ae3532723fb6ead77e7c604be7c8374094ef4ee2c5e03a3a17f1fca256c \
    --hash=sha256:109487860ef6a328f3eec66f2bf78b0b72400280d8f8ea05f69c51644ba6521a \
    --hash=sha256:11b53acf2411c3b09e6af37e4b9005cba376c872503c8f28218c7243582df45d \
    --hash=sha256:12db3b2c533c23ab812c2b25934f60383361f8a376ae272665f8e48b88e8e1c6 \
    --hash=sha256:14e76c0f23218b8f46c4d87018ca2e441535aed3632ca134b10239dfb6dadd6b \
    --hash=sha256:16a8663d6e281208d78806dbe14ee9903715361cf81f6d4309944e4d1e59ac5b \
    --hash=sha256:292d5e8ba896bbfd6334b096e34bffb56161c81408d6d036a7dfa6929cff8783 \
    --hash=sha256:2c03cc56021a4bd59be889c2b9257dae13bf55041a3372d3295416f86b295fb5 \
    --hash=sha256:2e396d70bc4ef5325b72b593a72c8979999aa52fb8bcf03f701c1b03e1166918 \
    --hash=sha256:2edb64ee7bf1ed524a1da60cdcd2e1f6e2b4f66ef7c077680739f1641f62f555 \
    --hash=sha256:31a9ddf4718d10ae04d9b18801bd776693487cbb57d74cc3458a7673f6f34639 \
    --hash=sha256:356541bf4381fa35856dafa6a965916e54bed415ad8a24ee6de6e37deccf2786 \
    --hash=sha256:358a7c4cb8ba9b46c453b1dd8d9e431452d5249072e4f56cfda3149f6ab1405e \
    --hash=sha256:37f8febc8ec50c14f3ec9637505f28e58d4f66752207ea177c1d67df25da5aed \
    --hash=sha256:39049da0ffb96c8cbb65cbf5c5f3ca3168990adf3551bd1dee10c48fce8ae820 \
    --hash=sha256:39cf9ed17fe3b1bc81f33c9ceb6ce67683ee7526e65fde1447c772afc54a1bb8 \
    --hash=sha256:3ae1de54a77dc0d6d5fcf623290af4266412a7c4be0b1ff7444394f03f5c54e3 \
    --hash=sha256:3b590df687e3c5ee0deef9fc8c547d81986d9a1b56073d82de008744452d6541 \
    --hash=sha256:3e45867f1f2ab0711d60c6c71746ac53537f1684baa699f4f668d4c6f6ce8e14 \
    --hash=sha256:3fc1c4a2ffd64890aebdb3f97e1278b0cc72579a08ca4de8cd2c04799a3a22be \
    --hash=sha256:4457ea6774b5611f4bed5eaa5df55f70abde42364d498c5134b7ef4c6958e20e \
    --hash=sha256:44ba614de5361b3e5278e1241fda3dc1838deed864b50a10d7ce92983797fa76 \
    --hash=sha256:4a8fcf28c05c1f6d7e177a9a46a1c52798bfe2ad80681d275b10dcf317deaf0b \
    --hash=sha256:4b0d02d7102dd0f997580b51edc4cebcf2ab6397a7edf89f1c73b586c614272c \
    --hash=sha256:502218f52498a36d6bf5ea77081844017bf7982cdbe521ad85e64cabee1b608b \
    --hash=sha256:503e65837c71b875ecdd733877d852adbc465bd82c768a067badd953bf1bc5a3 \
    --hash=sha256:5995f0164fa7df59db4746112fec3f49c461dd6b31b841873443bdb077c13cfc \
    --hash=sha256:59e5686dd847347e55dffcc191a96622f016bc0ad89105e24c14e0d6305acbc6 \
    --hash=sha256:601f36512f9e28f029d9481bdaf8e89e5148ac5d89cffd3b05cd533eeb423b59 \
    --hash=sha256:608862a7bf6957f2333fc54ab4399e405baad0163dc9f8d99cb236816db169d4 \
    --hash=sha256:62595ab75873d50d57323a91dd03e6966eb79c41fa834b7a1661ed043b2d404d \
    --hash=sha256:70990b9c51340e4044cfc394a81f614f3f90d41397104d226f21e66de668730d \
    --hash=sha256:71140351489970dfe5e60fc621ada3e0f41104a5eddaca47a7acb3c1b851d6d3 \
    --hash=sha256:72966d1b297c741541ca8cf1223ff262a6febe52481af742036a0b296e35fa5a \
    --hash=sha256:74292fc76c905c0ef095fe11e188a32ebd03bc38f3f3e9bcb85e4e6db177b7ea \
    --hash=sha256:761e8904c07ad053d285670f36dd94e1b6ab7f16ce62b9805c475b7aa1cffde6 \
    --hash=sha256:772b87914ff1152b92a197ef4ea40efe27a378606c39446ded52c8f80f79702e \
    --hash=sha256:79909e27e8e4fcc9db4addea88aa63f6423ebb171db091fb4373e3312cb6d603 \
    --hash=sha256:7e189e2e1d3ed2f4aebabd2d5b0f931e883676e51c7624826e0a4e5fe8a0bf24 \
    --hash=sha256:7eb33a30d75562222b64f569c642ff3dc6689e09adda43a082208397f016c39a \
    --hash=sha256:81d6741ab457d14fdedc215516665050f3822d3e56508921cc7239f8c8e66a58 \
    --hash=sha256:8499ca8f4502af841f68135133d8258f7b32a53a1d594aa98cc52013fff55678 \
    --hash=sha256:84c3990934bae40ea69a82034912ffe5a62c60bbf6ec5bc9691419641d7d5c9a \
    --hash=sha256:87701167f2a5c930b403e9756fab1d31d4d4da52856143b609e30a1ce7160f3c \
    --hash=sha256:88600c72ef7587fe1708fd242b385b6ed4b8904976d5da0893e31df8b3480cb6 \
    --hash=sha256:8ac7b6a045b814cf0c47f3623d21ebd88b3e8cf216a14790b455ea7ff0135d18 \
    --hash=sha256:8b8af03d2e37866d023ad0ddea594edefc31e827fee64f8de5611a1dbc373174 \
    --hash=sha256:8c7fe7afa480e3e82eed58e0ca89f751cd14d767638e2550c77a92a9e749c317 \
    --hash=sha256:8eade758719add78ec36dc13201483f8e9b5d940329285edcd5f70c0a9edbd7f \
    --hash=sha256:911d8a40b2bef5b8bbae2e36a0b103f142ac53557ab421dc16ac4aafee6f53dc \
    --hash=sha256:93ad6d87ac18e2a90b0fe89df7c65263b9a99a0eb98f0a3d2e079f12a0735837 \
    --hash=sha256:95dea361dd73757c6f1c0a1480ac499952c16ac83f7f5f4f84f0658a01b8ef41 \
    --hash=sha256:9ab77acb98eba3fd2a85cd160851816bfce6871d944d885febf012713f06659c \
    --hash=sha256:9cb3032517f1627cc012dbc80a8ec976ae76d93ea2b5feaa9d2a5b8882597579 \
    --hash=sha256:9cf4e8ad252f7c38dd1f676b46514f92dc0ebeb0db5552f5f403509705e24753 \
    --hash=sha256:9d9153257a3f70d5f69edf2325357251ed20f772b12e593f3b3377b5f78e7ef8 \
    --hash=sha256:a152f5f33d64a6be73f1d30c9cc82dfc73cec6477ec268e7c6e4c7d23c2d2291 \
    --hash=sha256:a16418ecf1329f71df119e8a65f3aa68004a3f9383821edcb20f0702934d8087 \
    --hash=sha256:a60332922359f920193b1d4826953c507a877b523b2395ad7bc716ddd386d866 \
    --hash=sha256:a8d0fc946c784ff7f7c3742310cc8a57c5c6dc31631269876a88b809dbeff3d3 \
    --hash=sha256:ab5de034a886f616a5668aa5d098af2b5385ed70142090e2a31bcbd0af0fdb3d \
    --hash=sha256:c22d3fe05ce11d3671297dc8973267daa0f938b93ec716e12e0f6dee81591dc1 \
    --hash=sha256:c2ac1b08635a8cd4e0cbeaf6f5e922085908d48eb05d44c5ae9eabab148512ca \
    --hash=sha256:c512accbd6ff0270939b9ac214b84fb5ada5f0409c44298361b2f5e13f9aed9e \
    --hash=sha256:c75ffc45f25324e68ab238cb4b5c0a38cd1c3d7f1fb1f72b5541de469e2247db \
    --hash=sha256:c95a03c79bbe30eec3ec2b7f076074f4281526724c8685a42872974ef4d36b72 \
    --hash=sha256:cadaeaba78750d58d3cc6ac4d1fd867da6fc73c88156b7a3212a3cd4819d679d \
    --hash=sha256:cd6056167405314a4dc3c173943f11249fa0f1b204f8b51ed4bde1a9cd1834dc \
    --hash=sha256:db72b07027db150f468fbada4d85b3b2729a3db39178abf5c543b784c1254539 \
    --hash=sha256:df2c707231459e8a4028eabcd3cfc827befd635b3ef72eada84ab13b52e1574d \
    --hash=sha256:e62164b50f84e20601c1ff8eb55620d2ad25fb81b59e3cd776a1902527a788af \
    --hash=sha256:e696f0dd336161fca9adbb846875d40752e6eba585843c768935ba5c9960722b \
    --hash=sha256:eaa379fcd227ca235d04152ca6704c7cb55564116f8bc52545ff357628e10602 \
    --hash=sha256:ebea339af930f8ca5d7a699b921106c6e29c617fe9606fa7baa043c1cdae326f \
    --hash=sha256:f4c39b0e3eac288fedc2b43055cfc2ca7a60362d0e5e87a637beac5d801ef478 \
    --hash=sha256:f5057856d21e7586765171eac8b9fc3f7d44ef39425f85dbcccb13b3ebea806c \
    --hash=sha256:f6f45710b4459401609ebebdbcfb34515da4fc2aa886f95107f556ac69a9147e \
    --hash=sha256:f97e83fa6c25693c7a35de154681fcc257c1c41b38beb0304b9c4d2d9e164479 \
    --hash=sha256:f9d0c5c045a3ca9bedfc35dca8526798eb91a07aa7a2c0fee134c6c6f321cbd7 \
    --hash=sha256:ff6f3db31555657f3163b15a6b7c6938d08df7adbfc9dd13d9d19edad678f1e8
    # via requests
dockerfile-parse @ https://github.com/containerbuildsystem/dockerfile-parse/archive/refs/tags/2.0.0.tar.gz \
    --hash=sha256:36e4469abb0d96b0e3cd656284d5016e8a674cd57b8ebe5af64786fe63b8184d
    # via my-package (pyproject.toml)
idna==3.4 \
    --hash=sha256:814f528e8dead7d329833b91c5faa87d60bf71824cd12a7530b5526063d02cb4 \
    --hash=sha256:90b77e79eaa3eba6de819a0c442c0b4ceefc341a7a2ab77d7562bf49f425c5c2
    # via requests
requests==2.28.2 \
    --hash=sha256:64299f4909223da747622c030b781c0d7811e359c37124b4bd368fb8c6518baa \
    --hash=sha256:98b1b2782e3c6c4904938b84c0eb932721069dfdb9134313beff7c83c2df24bf
    # via my-package (pyproject.toml)
urllib3==1.26.14 \
    --hash=sha256:076907bf8fd355cde77728471316625a4d2f7e713c125f51953bb5b3eecf4f72 \
    --hash=sha256:75edcdc2f7d85b137124a6c3c9fc3933cdeaa12ecb9a6a959f22797a0feca7e1
    # via requests

Hashes

Using hashes is strongly recommended.

If using pip-compile, use the --generate-hashes option.

External dependencies

For dependencies coming from somewhere other than PyPI, Hermeto supports a subset of the PEP 440 direct references.

https urls

dockerfile-parse @ https://github.com/containerbuildsystem/dockerfile-parse/archive/refs/tags/2.0.0.tar.gz \
    --hash=sha256:36e4469abb0d96b0e3cd656284d5016e8a674cd57b8ebe5af64786fe63b8184d

For https dependencies, Hermeto requires exactly one --hash option as protection from remote tampering.

Note that if at least one dependency in your requirements file uses --hash, pip requires hashes for all dependencies. Use pip-compile --generate-hashes to generate compliant requirements files.

Hermeto does not support PEP 440 hashes in the url fragment, only --hash options.

git urls

dockerfile-parse @ git+https://github.com/containerbuildsystem/dockerfile-parse@b6230230987950cfb16d8858c6f9a9642f4d0952

Git dependencies are incompatible with pip's hash checking. Please use an HTTPS URL instead, if possible

- dockerfile-parse @ git+https://github.com/containerbuildsystem/dockerfile-parse@b6230230987950cfb16d8858c6f9a9642f4d0952
+ dockerfile-parse @ https://github.com/containerbuildsystem/dockerfile-parse/archive/refs/tags/2.0.0.tar.gz \
+     --hash=sha256:36e4469abb0d96b0e3cd656284d5016e8a674cd57b8ebe5af64786fe63b8184d

If you do need to use a git url, Hermeto requires that it specifies a full commit hash.

Hermeto does not support PEP 440 commit hashes in the url fragment (the # part), only directly after @.

NOTE

It's impossible to craft a requirements.txt file that would download dependencies from both https urls and git urls. Hermeto requires hashes for https. Using one --hash makes pip require hashes for everything. Pip does not support hashes for git dependencies. Please use https urls instead.

Supported options

Requirements files support some pip install options - refer to the Pip docs.

Hermeto supports a small subset of them, ignores those that are not relevant for prefetching, and raises an error for those that are relevant but aren't supported.

Global

--index-url

Supported since v0.8.0.

Make Hermeto download packages from the specified Python Package Index server.

NOTE

Applies to all the packages (and only the packages) from the file which contains the --index-url option. If file A contains --index-url and file B does not, Hermeto will download the packages declared in B from the default index server (https://pypi.org/simple).

Do not include credentials in the index url. If needed, provide authentication via a .netrc file.

--require-hashes

Enables hash-checking mode. Typically redundant, since the presence of any --hash option enables hash-checking mode as well.

--trusted-host

Disables HTTPS validation for a host. Don't use this for production builds.

Per-requirement

--hash

Specifies the expected hashes for package archives. See also the hashes section.

Project metadata

Hermeto looks for the name and version of your project in the following project files

• pyproject.toml
• setup.cfg
• setup.py

If Hermeto fails to resolve the project name, it will generate a name based on the git repository origin url (and package subpath if the package is not in the repository root). If Hermeto fails to resolve the version, it will omit the version.

pyproject.toml PEP 621 metadata

Supported cases

[project]
name = "my_package"
version = "0.1.0"

Unsupported cases

[project]
name = "my_package"
dynamic = ["version"]

setup.cfg declarative config

Supported cases

[metadata]
name = my_package
version = 0.1.0

[metadata]
name = my_package
version = file: VERSION
# taken from ./VERSION
# example content:
#   0.1.0

[metadata]
name = my_package
version = attr: my_package.VERSION
# taken from my_package/__init__.py or my_package.py
# example content:
#   VERSION = "0.1.0"

Unsupported cases

• missing version
• some forms of version = attr: (those requiring executing the module)

setup.py

Using setup.py is discouraged.

Supported cases

setup(name="my_package", version="0.1.0", ...)

# basic variable usage is supported
NAME = "my_package"
VERSION = "0.1.0"

if __name__ == "__main__":
    # setup() call can be anywhere in the file
    setup(name=NAME, version=VERSION, ...)

Distribution formats

Python packages typically distribute both the binary format (called wheel) and the source format (called sdist).

Wheels are much more convenient; they are the pre-built format, installing from a wheel amounts to unzipping the wheel and copying the files to the right place.

Sdists are more difficult to install. Pip must first build a wheel from the sdist using a PEP 517 build system. To do that, pip has to install the build system and its dependencies (defined via PEP 518).

Building with wheels

Building with wheels is helpful when you do not want or cannot build from source. Some projects do not even distribute as sdists. For example, tensorflow (as of version 2.11.0) distributes wheels only.

You can use the binary filter object to prefetch the wheels you need to reduce the total download size and ensure compatibility with your target platform.

Hermeto supports three binary fetching strategies, analogous to pip's --no-binary, --prefer-binary, and --only-binary options:

1. No Binaries (default)
2. Configuration: binary field unspecified

3. Behavior: Hermeto operates in source-only mode. No binary artifacts will be prefetched for any packages. This is the default behavior.

4. Prefer binaries

5. Configuration: binary field specified with packages filter set to :all:

6. Behavior: Hermeto will attempt to prefetch compatible binaries for all dependencies where possible. If no matching binary is available, it will fall back to prefetching sdists instead.

7. Only binaries for specific packages

8. Configuration: binary field specified with packages filter set to specific package names
9. Behavior: For packages in the packages filter, Hermeto will attempt to prefetch only binaries. If no matching binary can be found for one of these packages, the operation will fail. Sdists for these packages will not be prefetched. For packages not in the packages filter, Hermeto falls back to the "no binary" mode above, i.e. only sdists are prefetched.

Filter Options

The binary object accepts the following filter options:

• packages: Comma-separated list of package names to limit the filtering scope. Default: :all:.

• arch: Architecture filter. Accepts comma-separated values. Default: "x86_64".

• os: Operating system filter. Accepts comma-separated values. Default: "linux".

• py_version: Python version filter. Accepts a single integer consisting of the major and minor versions combined (e.g., 312 for Python 3.12). Default: None.

• py_impl: Python implementation filter. Accepts comma-separated values. Default: "cp".

• abi: ABI filter. Accepts comma-separated values. Default: :all:.

• platform: Regex pattern to match against platform tags. Default: None.

For more information, see the platform compatibility tags specification.

Filter Logic

• :all: or None are equivalent and mean that no filter is applied
• multiple values within a single filter field are combined with OR logic
• multiple filter fields are combined with AND logic
• arch and os are mutually exclusive with platform

Examples

Prefetch wheels with default filters:

{
  "type": "pip",
  "binary": {}
}

Prefetch wheels for Python 3.12 on Linux aarch64:

{
  "type": "pip",
  "binary": {
    "os": "linux",
    "arch": "aarch64",
    "py_version": 312,
    "py_impl": "cp,pp"
  }
}

Prefetch wheels for specific packages and platforms:

{
  "type": "pip",
  "binary": {
    "packages": "numpy,pandas",
    "platform": "^(any|musllinux.*)$"
  }
}

Building from source

Building wheels from sdists takes a long time, but building from source gives you an important guarantee which using pre-built wheels does not: what you installed matches the source code. This can be especially important for Python packages implemented in C or other compiled languages.

requirements-build.txt

To allow building from source in a network-isolated environment, Hermeto must download all the PEP 517 build dependencies before the build starts.

Hermeto requires a fully resolved requirements-build.txt to do this. The file follows the same rules as requirements.txt, but contains build dependencies rather than runtime dependencies.

NOTE

This file must contain all the transitive build dependencies of each of your transitive runtime dependencies (you are installing dependencies from source).

We recommend the pybuild-deps package to generate the requirements-build.txt file. It will automatically generate build requirements from your requirements.txt file.

Adding a requirements-build.txt should not require changes in your build process. Pip should install the build dependencies automatically as needed, you don't have to install them explicitly. The purpose of requirements-build.txt is to make Hermeto fetch the build dependencies and provide them to pip for offline installation.

Using fetched dependencies

See the Example for a complete walkthrough of Hermeto usage.

Hermeto downloads the Python dependencies into the deps/pip/ subpath of the output directory. The directory is a flat list of the downloaded distributions of your runtime and build dependencies.

hermeto-output/deps/pip
├── certifi-2022.12.7.tar.gz
├── ...
├── pdm-pep517-1.0.6.tar.gz
├── requests-2.28.2.tar.gz
├── ...
└── wheel-0.38.4.tar.gz

To make pip use the downloaded archives, use the --find-links and --no-index options. The --find-links option tells pip to look for dependency archives in a directory, --no-index prevents pip from preferring PyPI over the local directory. Pip also accepts environment variables; Hermeto generates PIP_FIND_LINKS and PIP_NO_INDEX for you. See Example: Generate environment variables for more details.

Using external dependencies

It gets a bit trickier with external dependencies. Pip does not respect the --find-links option for dependencies specified via URLs. Instead, Hermeto's inject-files subcommand rewrites your requirements.txt file(s) in-place to replace the URLs with file paths in the local file system (it is also the case that an environment variable for telling Cargo where to look for prefetched deps is set by inject-files).

$ hermeto inject-files ./hermeto-output --for-output-dir /tmp/hermeto-output
2023-01-26 16:41:09,990 INFO Overwriting /tmp/test/foo-project/requirements.txt

We can look at the git diff to see what the package remapping looks like. As an example,

- dockerfile-parse @ https://github.com/.../2.0.0.tar.gz \
+ dockerfile-parse @ file:///absolute-path/hermeto-output/deps/pip/.../dockerfile-parse-...tar.gz

External dependencies are stored a bit further down the deps/pip tree to avoid mixing them with PyPI dependencies. The path and filename is an implementation detail.

hermeto-output/deps/pip
├── ...
├── external-dockerfile-parse
│   └── dockerfile-parse-external-sha256-36e4469abb0d96b0e3cd656284d5016e8a674cd57b8ebe5af64786fe63b8184d.tar.gz
└── ...

Working with Rust-based dependencies

Hermeto provides a way to prepare hermetic build of a Python package which depends on Rust-based packages. This process cannot be fully automated on Hermeto's side, thus it would require minor intervention from users.

Building such project requires that all build dependencies are listed in requirements-build.txt. To achieve this pybuild-deps could be used. The following command will take care of generating the list

pybuild-deps compile --generate-hashes -o requirements-build.txt requirements.txt

Note that this step requires internet connection.

Once requirements-build.txt is populated fetch could be done as usual. Hermeto will fetch all run time and build time dependencies for both Python and Rust parts.

Note that a system which is to be used for building these extensions must have rustc, cargo, and all necessary C libraries installed.

With these preparations running a pip installation as usual should be sufficient to build and install a Rust-based extension.

Note that sometimes Rust-based extensions can break a build. This could happen when such dependency is distributed with a Cargo.lock not matching Cargo.toml (while rare this seems to happen due to peculiarities of the release process for some packages). In this case a package will be rejected with a note about lock file mismatch and an additional report from Cargo about inability to load package lock file due to a mismatch. There is no good solution for this problem on Hermeto's side and the best course of action is to reach out to maintainers of this extension and notify them about the mismatch. Switching to a binary distribution of the package would also resolve this problem at the price of not building it from sources (see Building with wheels for additional context).

hermeto --mode permissive fetch-deps pip

(note that the parameter is global, not a fetch-deps-specific one). This would instruct Cargo PM which handles Rust dependencies to attempt to regenerate Cargo.lock. This is problematic as well since it makes builds unpredictable in the sense that two different builds of the same package can result in different versions of dependencies used because the lock file was regenerated at build time instead of being locked by maintainers. Please be aware that this is still a workaround and that the proper solution is to engage with a package maintainers and ensure that they supply correct Cargo.lock.

Troubleshooting

Common issues you may face when fetching dependencies or when installing the fetched dependencies.

First, please make sure that your project meets Hermeto's requirements (this document) and that you are using Hermeto as intended (for reference, see the Example for a complete walkthrough).

Miscellaneous errors while building from source

Have you read Building from source?

Even if you have all the build dependencies available, installing from source can come with unforeseen complications. Pip's --no-binary flag can help debug faster.

# on your machine
virtualenv venv && source venv/bin/activate
# or in a container
podman run --rm -ti -v "$PWD:$PWD:z" -w "$PWD" ubi8/python-39 bash

pip install --no-binary :all: -r requirements.txt

Notably, older versions of pip and setuptools have a fair share of bugs related to PEP 517 handling. A good first course of action can be to upgrade pip and setuptools and try again.

Other pip install options such as --use-pep517 may also be of interest.

Need to install newer pip

Problem: you've found out that some build errors are caused by bugs in an older pip version. But the base image for your container build comes with pip==<old> and you cannot upgrade during the build because you're building with network isolation.

Solution: make Hermeto fetch a newer pip for you. Then you can upgrade pip from the prefetched archive.

# add to requirements-build.txt or use a separate file
pip==22.3.1 --hash=...

RUN source /tmp/hermeto.env && \
    pip install -U pip && \
    pip install .

You can use a similar approach to upgrade setuptools or other build dependencies before installing your app. Build dependencies other than pip should be part or requirements-build.txt already.

Failing to compile a dependency

Building dependencies written in C typically requires gcc, CPython headers and other development libraries. Hermeto does not fetch these, getting them into the build is up to you. The best case scenario, if you're building a container, is that the base image already contains everything you need. For example, the ubi8/python-39 image contains most of the typical development libraries.

To find out what non-Python dependencies you need, try to pip install --no-binary :all: in a clean environment (e.g. a container) as shown above. The error messages you get should hopefully point you to the required dependencies.

For dependencies compiled from other languages, such as Rust, we don't know of any good solutions for offline installation. If you do manage to make it work, please let us know.

Dependency does not distribute sources

Some projects do not distribute sdists to PyPI. For example, tensorflow (as of version 2.11.0) distributes only wheels.

Possible workarounds:

• Enable pre-fetching wheels using binary filter object in JSON input.
• Find the git repository for the project, get the source tarball for a release.

In requirements.txt, specify the dependency via an https url.

- tensorflow==2.11.0
+ tensorflow @ https://github.com/tensorflow/tensorflow/archive/refs/tags/v2.11.0.tar.gz \
+     --hash=sha256:99c732b92b1b37fc243a559e02f9aef5671771e272758aa4aec7f34dc92dac48

Example

Let's build a basic pip project.

Get the repo if you want to try for yourself

git clone https://github.com/hermetoproject/doc-examples.git --branch=pip-basic

then cd into the doc-examples directory.

Pre-fetch dependencies

As mentioned above in Specifying packages to process, Hermeto pre-fetches using the fetch-deps command.

Sources can be fetched with

hermeto fetch-deps pip

Note that path is not specified explicitly and defaults to ., and a short-hand notation for package manager is used.

It's possible to add parameters in JSON format to point Hermeto at a specific source directory, and/or additional requirements files, if needed, e.g.

hermeto fetch-deps --source ./foo-project'{
  "type": "pip",
  "requirements_files": ["requirements.txt"],
  "requirements_build_files": ["requirements-build.txt"]
}'

Generate environment variables

Next, we need to generate the environment file so that the pip install command can find the cached dependencies

hermeto generate-env ./hermeto-output -o ./hermeto.env --for-output-dir /tmp/hermeto-output

We can see the variables needed by the package manager

$ cat hermeto.env
export PIP_FIND_LINKS=/tmp/hermeto-output/deps/pip
export PIP_NO_INDEX=true

Build the application image

In order to support the network isolated build, we need to remember to source the environment file in the step that executes pip install.

Dockerfile

FROM registry.access.redhat.com/ubi9/python-312

COPY . /src/simple-color-output
WORKDIR /src/simple-color-output

# Need to source the hermeto.env file to set the environment variables
# (in the same RUN instruction as the pip commands)
RUN source /tmp/hermeto.env \
    && python3 -m pip install -U pip \
    && python3 -m pip install --use-pep517 -r requirements.txt \
    && python3 -m pip install --use-pep517 .

CMD ["python3", "-m", "simple_color_output"]

After mounting the required Hermeto data, we can then build the image!

podman build . \
  --volume "$(realpath ./hermeto-output)":/tmp/hermeto-output:Z \
  --volume "$(realpath ./hermeto.env)":/tmp/hermeto.env:Z \
  --network none \
  --tag simple_color_output

We can then run the image to see the output

podman run --rm simple_color_output:latest

which should output

Hello colorful World!

Note that the repo also contains a build.sh script which will execute all of these steps for you.