The same stack somewhere else

What you will be able to do

  1. Get an optimised software stack onto a machine the operator does not administer, by three different routes, and say which route fits which machine.

  2. Say how EESSI decides which build to give the host it lands on, and override that decision.

  3. Say what a site adds on top without writing to /cvmfs.

Before this chapter

Reading a stack off the wire: CernVM-FS dissected a repository. The chapter employs that repository on machines lacking their own software stack, such as a laptop, a continuous-integration runner, a cloud instance, and a cluster whose stack is managed elsewhere. The tested claim originates from EESSI: “The host OS still provides a couple of things, like drivers for network and GPU, support for shared filesystems like GPFS and Lustre, a resource manager like Slurm, and so on.” All of those components are delivered via HTTP.

Three layers, and what each one does not do

EESSI is three layers, and the division is the interesting part. The filesystem layer is CernVM-FS, Reading a stack off the wire: CernVM-FS. It delivers bytes and verifies them, and it knows nothing about software. The compatibility layer is Gentoo Prefix: a set of Gentoo packages installed into a non-standard prefix with Portage. It provides the libc, the shell, the coreutils and the linker that everything above is built against, so the host distribution stops mattering. The prefix path is $EESSI_PREFIX/compat/$EESSI_OS_TYPE/$EESSI_CPU_FAMILY, for example /cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64. The software layer is EasyBuild output: everything from A toolchain is a hierarchy onwards, built per microarchitecture, with Lmod modulefiles. The layer boundary that matters for a reader on a laptop is the second one. A stack built against your distribution’s libc would need your distribution. A stack built against a prefix that travels with it needs a kernel and not much else.

Route one: mount it

The whole of a native install, on any machine you have root on

# install the CernVM-FS client and the EESSI configuration package
# (cvmfs-config-eessi ships the public key that verifies the repository)

$ sudo bash -c 'echo "CVMFS_CLIENT_PROFILE=single" > /etc/cvmfs/default.local'
$ sudo bash -c 'echo "CVMFS_QUOTA_LIMIT=10000" >> /etc/cvmfs/default.local'
$ sudo cvmfs_config setup
$ ls /cvmfs/software.eessi.io

Recorded: EESSI native-installation instructions, https://www.eessi.io/docs/getting_access/native_installation/, read 2026-09-10

Two configuration lines affect a single machine. The setting CVMFS_CLIENT_PROFILE=single is shipped for “laptops or fewer than 5 machines” and prevents the client from expecting a site proxy. Setting CVMFS_QUOTA_LIMIT=10000 raises the cache from the default 4000 megabytes. The stack is then one module use away.

module use /cvmfs/software.eessi.io/init/modules
module load EESSI/2023.06

Or, if Lmod is older than 8.6, unset MODULEPATH before sourcing the Lmod init rather than the bash‑only one.

unset MODULEPATH
source /cvmfs/software.eessi.io/versions/2023.06/init/lmod/bash

The init/bash path is irreversible and bash‑only. Do not use it as the alternative that leaves MODULEPATH untouched.

Route two: no root

Two mechanisms exist for two different reasons. cvmfsexec mounts a repository without privileges, using user namespaces, and CernVM-FS documents it as the supported way to do that. It is the route for a shared machine where the account is not an administrator and cannot become one. A container is the other, and EESSI publishes the script eessi_container.sh, which runs Apptainer or Singularity with the repository mounted inside. This is also the route their own build bot takes, which is worth noticing: the thing that builds the stack runs in a container mounting the stack.

Route three: a runner that exists for ninety seconds

Continuous integration presents a conflict between “install a software stack” and “the job is already over”, and EESSI provides a solution; the runner mounts, resolves required modules, and downloads only the files touched by those modules. On GitHub Actions the documented route is not a hand‑rolled mount. It is the published action:.

- uses: eessi/github-action-eessi@v3
  with:
    eessi_stack_version: '2025.06'

The action relies on cvmfs-contrib/github-action-cvmfs. An operator‑owned runner may still select route one or route two, and the cache results from that choice. A fresh runner starts with an empty cache, causing the initial job to fetch each required object.

How it chooses what to serve

This component determines whether the stack is fast or merely present. The initialisation script requests an ordered list of targets from a detector and selects the first one that exists on disk.

all_cpupaths=$(${EESSI_INIT_DIR_PATH}/eessi_archdetect.sh -a cpupath)
IFS=: read -r -a archs <<< "${all_cpupaths}"
for arch in "${archs[@]}"; do
  if [ -d "${EESSI_PREFIX}/software/${EESSI_OS_TYPE}/${arch}" ]; then
    export EESSI_SOFTWARE_SUBDIR="${arch}"
    show_msg "archdetect says ${EESSI_SOFTWARE_SUBDIR}"
    break
  fi
done

eessi_archdetect.sh -a cpupath produces a colon‑separated list ordered from the most specific to the most generic. On an Ice Lake host, according to EESSI’s own documentation:

x86_64/intel/icelake:x86_64/intel/cascadelake:x86_64/intel/skylake_avx512:x86_64/intel/haswell:x86_64/generic

A host whose exact microarchitecture lacks a build receives the best subset of its instruction set that does have a build, with x86_64/generic serving as the floor. The entire fallback policy consists of four lines of shell rather than a resolver. Two variables are often confused but are distinct. EESSI_CPU_FAMILY corresponds directly to the output of uname -m, such as x86_64, aarch64, or riscv64. The microarchitecture is represented by EESSI_SOFTWARE_SUBDIR, which the loop above sets.

EB-Elsewhere-2 — Which target did you get

A user reports that a job using an EESSI module is three times slower on one node type than on another, with the same module name loaded.

  1. What is the first variable you read, and on which machine?

  2. Two mechanisms could produce this. Name both.

  3. What would you have to do to confirm which one, and what would you have to avoid concluding from a login-node measurement?

Solution

  1. $EESSI_SOFTWARE_SUBDIR, on the slow node, inside the job. Not $EESSI_CPU_FAMILY, which is only uname -m and will be the same on both.

  2. Either that node’s microarchitecture has no native build and archdetect fell back to a more generic one, which is the fallback policy working as designed; or an override is set in the job’s environment, naming a target that is not the best available. The second is not validated, so nothing warns you.

  3. Read the ordered list with eessi_archdetect.sh -a cpupath on that node and compare the first entry against the directory that exists, then check the environment for the override variables. What a login node cannot tell you is which target a compute node resolves detection reads that node’s own /proc/cpuinfo. Same lesson as Four things that actually happened, one layer up.

Reading a stack off the wire: CernVM-FS concluded with the mechanism: a symlink whose stored target is a variable reference, expanded by the client. EESSI’s is host_injections, and the live repository stores it as $(EESSI_HOST_INJECTIONS:-/opt/eessi)=. Point it to a writable location, and a site obtains a tree that behaves as if it were inside the repository:.

What a site adds without writing to /cvmfs

Three uses exist. A stack built elsewhere cannot know them. GPU drivers. The driver belongs to the host kernel and cannot ship in a repository. EESSI’s script link_nvidia_host_libraries.sh populates the injection tree with links to the host’s driver libraries. The documentation states: “You should re-run this script every time you update the NVIDIA GPU drivers on the host system, as it may expose libraries that are new to your driver version.” A full CUDA installation. The CUDA and cuDNN in the repository have been stripped down to contain only the runtime libraries. Compiling CUDA code therefore requires additional components. install_cuda_and_libraries.sh installs those components into the injection tree through EasyBuild. Site software and site MPI. The initialisation script derives the site tree from the stack path by string substitution.

sudo bash -c "echo 'EESSI_HOST_INJECTIONS=/shared_fs/path/to/host/injections/' \
  > /etc/cvmfs/domain.d/eessi.io.local"
sudo cvmfs_config reload

Thus /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2 has a site twin at /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen2, and this path appears on MODULEPATH together with the mounted one. It corresponds to the three lists described in Build your own, without asking anyone, where the site’s list resides behind a symlink managed by the site. Per-node-type differences are supported, which explains why the mechanism uses a client-side variable instead of a repository path: two GPU node types with drivers in separate locations receive distinct EESSI_HOST_INJECTIONS values in their respective CernVM-FS configurations.

site_software_path="${EESSI_SOFTWARE_PATH/versions/host_injections}"
export EESSI_SITE_MODULEPATH="${site_software_path}/modules/all"

What it is, and what it is not

Two limits to state plainly this needs them.

The stack is built for a microarchitecture, and the fallback means a host can silently get a more generic build than it deserves. Reading $EESSI_SOFTWARE_SUBDIR after initialisation is the check, and it is one line.

And the host still provides the driver, the network stack, the filesystem client and the scheduler. An MPI from the repository talking to a site fabric is exactly the case A site stack next to EESSI, not inside it is for, and it is the one place where “mount it and go” is not the whole story. The 2025.06 tree also carries lfoss/2025b (the linear-algebra foss); the table above lists the documented common toolchains, not every top-level module.

EB-Elsewhere-1 — Four machines

For each machine, name the route, the two configuration decisions you would make, and the one check you would run before believing the result.

  1. Your laptop, root available, 40 GB free.

  2. A university cluster login node where you have no privileges and the administrators are slow.

  3. A GitHub Actions runner, for a project whose test suite needs a recent compiler.

  4. A GPU node whose driver was updated last night.

Solution

  1. Route one, native. Set CVMFS_CLIENT_PROFILE=single and raise CVMFS_QUOTA_LIMIT; 40 GB allows a generous cache and the default 4000 MB will thrash. Check $EESSI_SOFTWARE_SUBDIR after initialisation fallback list matters.

  2. cvmfsexec, or a container with eessi_container.sh. The decisions are where the cache goes, since your home directory may be quota’d and on a network filesystem, and which version you pin. Check that the modules you need exist for your target rather than in general, with module avail after initialisation.

  3. Either, depending on privileges; the runner is ephemeral so the decisions are the cache location and pinning the EESSI version so the pipeline does not change under you. Check the first job’s wall time against the second’s: if they are the same, the cache is not being reused and every run is paying full price.

  4. Route one, plus host injections, and this is the case that needs the fourth step. The driver update is exactly when link_nvidia_host_libraries.sh has to be re-run, per EESSI’s own documentation. The check is to run something CUDA on that node, not on a login node, which is Four things that actually happened’s whole subject.

What to remember

  • Three routes onto a machine: a native client with root, cvmfsexec or a container without it, and the same two on an ephemeral runner where the cache is the whole of the install.

  • The compatibility layer is why the host distribution stops mattering.

  • eessi_archdetect.sh -a cpupath returns an ordered list and the first target that exists on disk wins, so read $EESSI_SOFTWARE_SUBDIR rather than assuming.

  • Overrides are not validated: a target that does not exist gives an empty module avail.

  • A site adds software and drivers through a variant symlink, without writing to /cvmfs and without invalidating a signature.