The same stack somewhere else

two sites mount the same Stratum-0 and see the same module tree

The same stack somewhere else is the same mount, not a rebuild.

the same mount at another site

The same stack somewhere else is the same CernVM-FS tree.

one CVMFS tree mounted on two hosts

The stack is the same bytes. The host is not. Compatibility is a client fact, not a rebuild.

What you will be able to do

  1. Get an optimised software stack onto a machine the operator does not administer, by three different routes, and say which route fits which machine.

  2. Say how EESSI decides which build to give the host it runs on, and override that decision.

  3. Say what a site adds on top without writing to /cvmfs.

Before this page

Reading a stack off the wire: CernVM-FS dissected a repository. The page uses that repository on machines without a software stack of their own. The examples are a laptop, a continuous-integration runner, a cloud instance, and a cluster whose stack someone else manages. The tested claim originates from EESSI: “The host OS still provides a couple of things, like drivers for network and GPU, support for shared filesystems like GPFS and Lustre, a resource manager like Slurm, and so on.” Those host pieces stay on the host. CernVM-FS carries the three layers.

Three layers, and what each one does not do

EESSI has three layers. The division is the interesting part.

The filesystem layer is CernVM-FS. Reading a stack off the wire: CernVM-FS. It moves bytes. It verifies them. It knows nothing about software.

The compatibility layer is Gentoo Prefix. It is a set of Gentoo packages installed into a non-standard prefix with Portage. It provides the libc. It provides the shell. It provides the coreutils. It provides the linker that everything above is built against. The host distribution stops mattering.

The prefix path is $EESSI_PREFIX/compat/$EESSI_OS_TYPE/$EESSI_CPU_FAMILY. For example /cvmfs/software.eessi.io/versions/2023.06/compat/linux/x86_64.

The software layer leans on this layer through one EasyBuild setting. EESSI-extend passes --filter-deps with the packages the compatibility layer already provides. For 2026.06 the list starts binutils,bzip2,DBus,flex,gettext and includes ncurses, XZ and zlib. A filtered dependency is never built and never loaded. The build takes the compatibility layer’s copy instead. That makes a build inside EESSI a weak test of an easyconfig’s dependency list. A missing binutils or zlib goes unnoticed, because EESSI would have filtered it anyway. The same easyconfig then fails on a site whose own as or libz is older. Build dependencies are not dependencies walks through one such failure.

The software layer is EasyBuild output. It includes everything from A toolchain is a hierarchy onwards. It is built per microarchitecture. It uses Lmod modulefiles.

The layer boundary that matters for a reader on a laptop is the second one.

A stack built against your distribution’s libc would need your distribution.

A stack built against a prefix that travels with it needs a kernel. It needs not much else.

Route one: mount it

The whole of a native install, on any machine you have root on

# install the CernVM-FS client and the EESSI configuration package
# (cvmfs-config-eessi ships the public key that verifies the repository)

$ sudo bash -c 'echo "CVMFS_CLIENT_PROFILE=single" > /etc/cvmfs/default.local'
$ sudo bash -c 'echo "CVMFS_QUOTA_LIMIT=10000" >> /etc/cvmfs/default.local'
$ sudo cvmfs_config setup
$ ls /cvmfs/software.eessi.io

Recorded: EESSI native-installation instructions, https://www.eessi.io/docs/getting_access/native_installation/, read 2026-09-10

Two configuration lines affect a single machine. The setting CVMFS_CLIENT_PROFILE=single is meant for “laptops or fewer than 5 machines”. The setting CVMFS_CLIENT_PROFILE=single prevents the client from expecting a site proxy. Setting CVMFS_QUOTA_LIMIT=10000 raises the cache from the default 4000 megabytes. The stack is then one module use away.

module use /cvmfs/software.eessi.io/init/modules
module load EESSI/2025.06

unset MODULEPATH
source /cvmfs/software.eessi.io/versions/2025.06/init/lmod/bash

Route two: no root

Two mechanisms exist for two different reasons. cvmfsexec mounts a repository without privileges, using user namespaces. CernVM-FS documents it as the supported way to do that. The route applies to a shared machine where the account is not an administrator. The account cannot become an administrator. A container provides the alternative mechanism. EESSI publishes the script eessi_container.sh, which runs Apptainer or Singularity with the repository mounted inside. The route is also used by their own build bot, which is worth noticing. The component that builds the stack runs in a container mounting the stack.

Route three: a runner that exists for ninety seconds

Continuous integration presents a conflict between “install a software stack” and “the job is already over”. EESSI provides a solution. The runner mounts, resolves required modules, and downloads only the files touched by those modules. On GitHub Actions the documented route is not a hand‑rolled mount. It is this published action:

- uses: eessi/github-action-eessi@v3
 with:
 eessi_stack_version: '2025.06'

The action relies on cvmfs-contrib/github-action-cvmfs. An operator‑owned runner may still select route one or route two, and the cache reflects that choice. A fresh runner starts with an empty cache. The initial job fetches each required object.

How it chooses what to serve

The component assesses if the stack is fast or merely present. The initialisation script requests an ordered list of targets from a detector. The script selects the first target that exists on disk.

all_cpupaths=$(${EESSI_INIT_DIR_PATH}/eessi_archdetect.sh -a cpupath)
IFS=: read -r -a archs <<< "${all_cpupaths}"
for arch in "${archs[@]}"; do
 if [ -d "${EESSI_PREFIX}/software/${EESSI_OS_TYPE}/${arch}" ]; then
 export EESSI_SOFTWARE_SUBDIR="${arch}"
 show_msg "archdetect says ${EESSI_SOFTWARE_SUBDIR}"
 break
 fi
done

eessi_archdetect.sh -a cpupath produces a colon‑separated list ordered from the most specific to the most generic. On an Ice Lake host, EESSI documentation reports the following.

x86_64/intel/icelake:x86_64/intel/cascadelake:x86_64/intel/skylake_avx512:x86_64/intel/haswell:x86_64/generic

Hosts lacking an exact microarchitecture build receive the best subset of the instruction set that has a build, with x86_64/generic as the floor. The fallback policy comprises four lines of shell instead of a resolver. Two variables are frequently confused. They are distinct. EESSI_CPU_FAMILY maps directly to the output of uname -m. Examples include x86_64, aarch64, or riscv64. EESSI_SOFTWARE_SUBDIR represents the microarchitecture. It is set by the loop above.

EB-Elsewhere-2 — Which target did you get

A user reports that a job using an EESSI module is three times slower on one node type than on another, with the same module name loaded.

  1. What is the first variable you read, and on which machine?

  2. Two mechanisms could produce this. Name both.

  3. What would you have to do to confirm which one, and what would you have to avoid concluding from a login-node measurement?

Solution

  1. $EESSI_SOFTWARE_SUBDIR, on the slow node, inside the job. Not $EESSI_CPU_FAMILY, which is only uname -m and will be the same on both.

  2. Either that node’s microarchitecture has no native build and archdetect fell back to a more generic one, which is the fallback policy working as designed; or an override is set in the job’s environment, naming a target that is not the best available. The second is not validated, so nothing warns you.

  3. Read the ordered list with eessi_archdetect.sh -a cpupath on that node and compare the first entry against the directory that exists, then check the environment for the override variables. What a login node cannot tell you is which target a compute node resolves. Detection reads that node’s own /proc/cpuinfo. Same lesson as Four things that actually happened, one layer up.

Reading a stack off the wire, CernVM-FS uses a symlink whose stored target is a variable reference. The client expands the variable reference. EESSI provides host_injections. The live repository stores it as $(EESSI_HOST_INJECTIONS:-/opt/eessi). Point it to a writable location. A site obtains a tree that behaves as if it were inside the repository.

What a site adds without writing to /cvmfs

Three uses are defined. A stack built elsewhere cannot know the uses. GPU drivers. The driver belongs to the host kernel and cannot travel in a repository. On 2025.06 the script writes NVIDIA driver links into compat/.../lib/nvidia via EESSI_NVIDIA_OVERRIDE_DEFAULT. It does not populate host_injections for those drivers. The documentation states: “Re‑run this script each time the NVIDIA GPU drivers on the host system are updated.” It may expose libraries that are new to the driver version. A full CUDA installation. The CUDA and cuDNN packages in the repository contain only runtime libraries. Compiling CUDA code requires additional components. The script install_cuda_and_libraries.sh installs those components into the injection tree via EasyBuild. Site software and site MPI. The initialisation script derives the site tree from the stack path using string substitution.

sudo bash -c "echo 'EESSI_HOST_INJECTIONS=/shared_fs/path/to/host/injections/' \
 > /etc/cvmfs/domain.d/eessi.io.local"
sudo cvmfs_config reload

The path /cvmfs/software.eessi.io/versions/2023.06/software/linux/x86_64/amd/zen2 has a site twin at /cvmfs/software.eessi.io/host_injections/2023.06/software/linux/x86_64/amd/zen2.

The path appears on MODULEPATH together with the mounted one.

The path corresponds to the three lists described in Build your own, without asking anyone.

The site list resides behind a symlink managed by the site.

Per-node-type differences are supported.

The mechanism uses a client-side variable instead of a repository path.

Two GPU node types with drivers in separate locations receive distinct EESSI_NVIDIA_OVERRIDE_DEFAULT values in 2025.06. EESSI_HOST_INJECTIONS still aims the site software tree, not the NVIDIA compat symlink.

site_software_path="${EESSI_SOFTWARE_PATH/versions/host_injections}"
export EESSI_SITE_MODULEPATH="${site_software_path}/modules/all"

What it is, and what it is not

Two limits must be stated plainly. These limits are needed. The stack is built for a microarchitecture. The fallback allows a host to receive a more generic build than it deserves. Reading $EESSI_SOFTWARE_SUBDIR after initialisation is the check. The check consists of one line. The host still provides the driver, the network stack, the filesystem client and the scheduler. An MPI from the repository communicating with a site fabric matches the case A site stack next to EESSI, not inside it. It is the sole location where “mount it and go” does not cover the entire scenario. The 2025.06 tree also carries lfoss/2025b (LLVM foss). The versions table in the sources box lists the documented common toolchains, not every top-level module.

EB-Elsewhere-1 — Four machines

For each machine, name the route, the two configuration decisions you would make, and the one check you would run before believing the result.

  1. Your laptop, root available, 40 GB free.

  2. A university cluster login node where you have no privileges and the administrators are slow.

  3. A GitHub Actions runner, for a project whose test suite needs a recent compiler.

  4. A GPU node whose driver was updated last night.

Solution

  1. Route one, native. Set CVMFS_CLIENT_PROFILE=single and raise CVMFS_QUOTA_LIMIT; 40 GB allows a generous cache and the default 4000 MB will thrash. Check $EESSI_SOFTWARE_SUBDIR after initialisation. The fallback list matters.

  2. cvmfsexec, or a container with eessi_container.sh. The decisions are where the cache goes, since your home directory may be quota’d and on a network filesystem, and which version you pin. Check that the modules you need exist for your target rather than in general, with module avail after initialisation.

  3. Either, depending on privileges; the runner is ephemeral so the decisions are the cache location and pinning the EESSI version so the pipeline does not change under you. Check the first job’s wall time against the second’s: if they are the same, the cache is not being reused and every run is paying full price.

  4. Route one, plus EESSI_NVIDIA_OVERRIDE_DEFAULT on 2025.06 (or EESSI_HOST_INJECTIONS on 2023.06). Re-run link_nvidia_host_libraries.sh after the driver update. The check is CUDA on that node, not on a login node.

What to remember

  • Three routes onto a machine: a native client with root, cvmfsexec or a container without it, and the same two on an ephemeral runner where the cache is the whole of the install.

  • The compatibility layer is why the host distribution stops mattering.

  • eessi_archdetect.sh -a cpupath returns an ordered list and the first target that exists on disk wins, so read $EESSI_SOFTWARE_SUBDIR rather than assuming.

  • Overrides are not validated: a target that does not exist gives an empty module avail.

  • A site adds software and drivers through a variant symlink, without writing to /cvmfs and without invalidating a signature.