A site stack next to EESSI, not inside it

Questions

  • What does EESSI_SITE_SOFTWARE_PREFIX change, and when must it be set?

  • When is host_injections the right tree, and when is a separate prefix the one the webinar recommends?

  • What still has to be true of a site OpenMPI for the EESSI stack above it to keep working?

What you will be able to do

  1. Install one package on top of EESSI into an operator-chosen prefix with EESSI-extend, and say why that prefix contains a microarchitecture.

  2. Tell Approach 1 (shared filesystem) from Approach 2 (site CVMFS plus the bot), and say which extra moving parts the second one adds.

  3. Point NVIDIA drivers, a full CUDA SDK, and an ABI-compatible site MPI at the paths EESSI actually searches.

Before this chapter

EESSI constitutes a shared stack. A site may still possess software that EESSI does not ship: a closed‑source licence, software not yet on the tree, and software tuned for a specific fabric. The webinar by Caspar van Leeuwen and Bob Dröge on 1 June 2026 addresses that situation. The official host_injections page contains four bullets and a symlink. The webinar presents two approaches, two live clusters, and the variables that cause a login session to see both trees.

Why a site builds at all

Three reasons are presented on the webinar’s motivation slide, and they are not the same job. Licensed software cannot go in software.eessi.io. It may also be limited to a subset of the site’s users. Software that is not on the tree yet is a time problem. Waiting for the bot on every architecture is the right path into EESSI (chapters 30 and 31). It is the wrong path for a group that needs the binary this week. Custom installations are the fabric, the licence server, the site MPI. Those cannot be expressed as an easyconfig that EESSI would accept. They are facts about the machine.

Approach 1: EESSI-extend into a shared filesystem

The inexpensive approach. It requires no additional CernVM-FS, no bot instance, and no object store. EESSI-extend already knows how to write into an architecture-specific prefix and add that prefix to MODULEPATH. The webinar includes a Hábrók demo at the University of Groningen using AlmaLinux 9.8 on zen3 with an unprivileged installer account.

export EESSI_SITE_INSTALL=1
export EESSI_SITE_SOFTWARE_PREFIX=/software
module load EESSI/2025.06
module load EESSI-extend
echo $EESSI_SITE_SOFTWARE_PATH
# /software/versions/2025.06/software/linux/x86_64/amd/zen3
eb --show-config   # installpath is that path
eb cowsay-3.04.eb
eb attr-2.5.2-GCCcore-14.3.0.eb
module avail
# ---- /software/versions/2025.06/software/linux/x86_64/amd/zen3/modules/all ----
#    attr/2.5.2-GCCcore-14.3.0    cowsay/3.04

Set EESSI_SITE_SOFTWARE_PREFIX before the EESSI module loads. The module computes EESSI_SITE_SOFTWARE_PATH from it. Setting the prefix later has no effect. That matches the load-time rule for EESSI, somebody else’s stack’s EESSI_USER_INSTALL.

Both EESSI_SITE_INSTALL and EESSI_CVMFS_INSTALL are exclusive of EESSI_USER_INSTALL and EESSI_PROJECT_INSTALL. User and project can be set together; site and CVMFS cannot.

The webinar repeats the chapter‑11 warning, applied to a site prefix: EESSI-extend performs native optimisation and installs into a prefix named for the host architecture. The Hábrók zen3 modules remain invisible on a zen4 node until the same eb is executed there.

There are two locations where the prefix can reside.

Option 1, the variant symlink. /cvmfs/software.eessi.io/host_injections stores the target $(EESSI_HOST_INJECTIONS:-/opt/eessi). A site CernVM-FS config of

# /etc/cvmfs/domain.d/eessi.io.local
EESSI_HOST_INJECTIONS=/shared_fs/path/to/host/injections/

followed by cvmfs_config reload makes that tree behave as if it were inside the repository, without writing to the repository and without invalidating a signature. Different node types may point the symlink at different directories: two GPU partitions with drivers in different places need two targets.

Option 2, the one the webinar marks recommended: a prefix the site owns, EESSI_SITE_SOFTWARE_PREFIX=/software or /cvmfs/software.caspar.nl, which is not the variant symlink. The EESSI module still prepends the derived EESSI_SITE_SOFTWARE_PATH to MODULEPATH. The bytes live wherever the site put them.

The cons of Approach 1 are the ones the webinar lists: the procedure is manual, and a shared filesystem is a worse startup path than CernVM-FS. For a handful of packages on one cluster it is enough.

A container is available when the host is not a safe build environment. eessi_container.sh -b $EESSI_SITE_SOFTWARE_PREFIX then the same four exports and module loads inside.

Approach 2: the bot, pointed at a site repository

The webinar’s second half is The bot is three processes and a comment aimed at a different CernVM-FS name. The goal, quoted: Build using dependencies from EESSI. In a CVMFS repository of our own. With the EESSI build bot. Using the same build scripts from software-layer-scripts. What that adds, from the “what we need” slide: a site Stratum-0 (and, in production, Stratum-1s and proxies), one bot instance per cluster that should build natively, an S3‑compatible bucket, a GitHub repository whose easystack path is

easystacks/<site_cvmfs_repo>/<EESSI_VERSION>/eessi-<EESSI_VERSION>-eb-<eb_version>-<anything>.yml

Something on the Stratum-0 ingests tarballs. The demo repository is software.caspar.nl. The demo pull request adds Biopython-1.86-gfbf-2025b.eb. After ingest:

export EESSI_SITE_SOFTWARE_PREFIX=/cvmfs/software.caspar.nl
source /cvmfs/software.eessi.io/versions/2025.06/init/lmod/bash
module load Biopython/1.86-gfbf-2025b
python -c 'import Bio.motifs; print(Bio.motifs.__path__)'
# .../software.caspar.nl/versions/2025.06/software/linux/x86_64/amd/zen2/...
which python
# /cvmfs/software.eessi.io/versions/2025.06/.../Python/3.13.5-...

The site module is the EESSI interpreter. That constitutes the entire purpose. A site ingest does not require EESSI/staging_bundles. Caspar’s ingest_tarballs.sh lists the bucket, downloads the contents, optionally verifies a signature, calls filesystem-layer/scripts/ingest-tarball.sh, regenerates nested catalogs, and remounts. Nested catalogs follow the EESSI .cvmfsdirtab pattern. One catalog per installation. The Stratum-0 cannot employ the automounter. software.eessi.io is mounted from fstab when the ingest script requires EESSI’s Lmod to rebuild caches. Rebuilds on a site repository use the same rebuilds/ directory as An easystack is the list the bot will build, and the scripts include the --rebuild option. Removal constitutes a Stratum-0 transaction and an Lmod cache rebuild, not a bot comment.

Drivers, CUDA, and an ABI-compatible MPI

A site adds three items into the injection tree, even if the remaining site stack resides in the prefix recommended by Approach 1. NVIDIA drivers, runtime. The driver is bound to the host kernel and cannot be shipped in the repository. For EESSI 2025.06 and later the compatibility-layer linker searches

/cvmfs/software.eessi.io/versions/<ver>/compat/<os>/<arch>/lib/nvidia

It is itself a variant symlink. The CernVM-FS config that is targeted is EESSI_NVIDIA_OVERRIDE_DEFAULT, not EESSI_HOST_INJECTIONS.

# /etc/cvmfs/default.local  (or domain.d/eessi.io.local)
EESSI_NVIDIA_OVERRIDE_DEFAULT=/opt/eessi/nvidia

cvmfs_config reload software.eessi.io, then

/cvmfs/software.eessi.io/versions/${EESSI_VERSION}/scripts/gpu_support/nvidia/link_nvidia_host_libraries.sh

Re-run after each driver update. The per-version override EESSI_202506_NVIDIA_OVERRIDE applies when 2023.06 and 2025.06 must not share a symlink set. In 2023.06 the same script continues to write under host_injections, and EESSI_HOST_INJECTIONS designates that location. Mixing the two conventions causes a 2025.06 node to obtain a linker search path that points to an empty directory. The official GPU page states that “ROCm support is not yet available”. The live 2025.06 catalogs already include gfx* trees. The page represents the missing component. A full CUDA SDK, compile time. The CUDA and cuDNN packages in the repository are runtime‑only; the EULA prohibits redistribution of nvcc. install_cuda_and_libraries.sh installs the missing components into the injection tree via EasyBuild, restoring symlinks such as

.../software/CUDA/12.1.1/bin/nvcc ->
  /cvmfs/software.eessi.io/host_injections/2023.06/software/.../CUDA/12.1.1/bin/nvcc

The installer account is not root. EasyBuild does not run as root unless EASYBUILD_ALLOW_USE_AS_ROOT_AND_ACCEPT_CONSEQUENCES is set, and the GPU page recommends changing ownership of the injection directory instead. Site MPI. EESSI’s OpenMPI does not know a site fabric. An ABI‑compatible replacement goes where the RPATH of the EESSI binaries already looks. The Slingshot-11 write-up puts OpenMPI 5.x here, for 2023.06 on Grace:

/cvmfs/software.eessi.io/host_injections/2023.06/software/linux/aarch64/nvidia/grace/rpath_overrides/OpenMPI/system/lib

The EESSI GROMACS and the EESSI OSU benchmarks retain their module names. The libraries they resolve at runtime are those provided by the site. No package above MPI is rebuilt. This is true only if the replacement is ABI‑compatible with the EESSI OpenMPI that the rest of the tree was linked against. Lmod hooks constitute the other half of fabric tuning: environment variables at load time rather than a replaced .so. EESSI’s SitePackage.lua searches for two additional files under host_injections:

$EESSI_CVMFS_REPO/host_injections/$EESSI_VERSION/.lmod/SitePackage.lua
$EESSI_CVMFS_REPO/host_injections/$EESSI_VERSION/software/$EESSI_OS_TYPE/$EESSI_SOFTWARE_SUBDIR/.lmod/SitePackage.lua

The initial version is architecture-independent. The subsequent version lacks architecture independence. Version 2023.06 provides Lmod 8.7.30, which cannot register two load hooks; the site file merges eessi_load_hook with the site function into a single registration. Lmod 8.7.35 and newer are able to register both.

What a login session has to export

Two patterns were presented in the webinar’s last section. When using EESSI’s Lmod, a file in /etc/profile.d sets EESSI_SITE_SOFTWARE_PREFIX and sources /cvmfs/software.eessi.io/versions/2025.06/init/lmod/bash. When using the site’s Lmod, Hábrók’s 01-local_lmod.sh:

export EESSI_SITE_SOFTWARE_PREFIX=/software
export MODULEPATH=/cvmfs/software.eessi.io/init/modules
export LMOD_CACHED_LOADS=yes
export LMOD_SHORT_TIME=86400

The z01_StdEnv.sh sets LMOD_SYSTEM_DEFAULT_MODULES=EESSI/2025.06, causing a login to load EESSI and to see /software/versions/2025.06/.../modules/all next to the CernVM-FS tree. The webinar names EESSI_MODULE_STICKY and EESSI_MODULE_FAMILY_NAME as the other two variables. These variables make the EESSI module sticky and enable coexistence with a second, incompatible site stack. Licensed software that only some users may run is identified as a poor fit for a site CVMFS repository. The setting CVMFS_CLAIM_OWNERSHIP=no preserves ownership but does not constitute a security boundary, as a container can override it. The webinar proposes keeping those installations on the shared filesystem of Approach 1.

EB-Site-1 — Which prefix, which variable

Four requests. For each, name the approach, the variable that has to be set before any module load, and the one check that says the result is where it should be.

  1. A single extra package, this week, on one zen3 partition, no CernVM-FS admin access.

  2. NVIDIA driver libraries on a 2025.06 node after last night’s driver update.

  3. OpenMPI rebuilt against the site Slingshot, consumed by the EESSI GROMACS already on the tree.

  4. Twenty packages, three microarchitectures, a Git history of what was built, same bot comments as EESSI itself.

Solution

  1. Approach 1. EESSI_SITE_INSTALL=1 and EESSI_SITE_SOFTWARE_PREFIX pointing at a writable shared directory, set before module load EESSI. Check $EESSI_SITE_SOFTWARE_PATH ends in .../x86_64/amd/zen3 and that eb --show-config reports that path as installpath.

  2. Not a prefix variable. EESSI_NVIDIA_OVERRIDE_DEFAULT in the CernVM-FS config, cvmfs_config reload, then re-run link_nvidia_host_libraries.sh. Check deviceQuery from CUDA-Samples on that node. 2023.06 would have used EESSI_HOST_INJECTIONS instead.

  3. The replacement libraries go in host_injections/<ver>/software/linux/<software_subdir>/rpath_overrides/OpenMPI/system/lib (the live Slingshot-11 tree is .../linux/aarch64/nvidia/grace/rpath_overrides/OpenMPI/system/lib). The check is ldd on an EESSI MPI binary, on the compute node, showing the site .so, plus an OSU run. ABI compatibility is the constraint; a rebuild of GROMACS is the thing this path exists to avoid.

  4. Approach 2. A site CernVM-FS repo, a bot instance, easystacks under easystacks/<that.repo>/<ver>/, EESSI_SITE_SOFTWARE_PREFIX set to the repo root on the clients. Check a loaded site module’s path is under the site repo while its Python (or MPI, or libc) still resolves under software.eessi.io.

EB-Site-2 — 2023.06 or 2025.06

A site config contains only this line, in /etc/cvmfs/domain.d/eessi.io.local:

EESSI_HOST_INJECTIONS=/shared/eessi-inject

link_nvidia_host_libraries.sh was executed last year targeting 2023.06. Users have now switched to EESSI/2025.06. CUDA applications fail to find libcuda.so.

  1. Why did this work on 2023.06 and fail on 2025.06?

  2. What is added to the CernVM-FS config, and what is re-run?

  3. Two GPU partitions with drivers in different directories: how many injection trees, and where is the difference expressed?

Solution

  1. 2023.06’s linker search path for NVIDIA is under host_injections, which EESSI_HOST_INJECTIONS aims. 2025.06 searches /cvmfs/software.eessi.io/versions/2025.06/compat/.../lib/nvidia, a different variant symlink, aimed by EESSI_NVIDIA_OVERRIDE_DEFAULT. The old variable still aims the old tree. The new linker is not looking there.

  2. EESSI_NVIDIA_OVERRIDE_DEFAULT=/shared/eessi-inject/nvidia (or a per-version EESSI_202506_NVIDIA_OVERRIDE), cvmfs_config reload software.eessi.io, then the 2025.06 link_nvidia_host_libraries.sh. Last year’s symlinks are not reused.

  3. One tree per partition. The difference is the CernVM-FS config on those nodes, each with its own EESSI_HOST_INJECTIONS / EESSI_NVIDIA_OVERRIDE_DEFAULT. The repository content is the same; the client-side target is not.

What to remember

  • EESSI_SITE_SOFTWARE_PREFIX is set before the EESSI module loads. EESSI-extend then writes an architecture-specific prefix and puts it on MODULEPATH.

  • Approach 1 is that prefix on a shared filesystem. Approach 2 is the same bot and the same scripts, aimed at a site CernVM-FS repository. The webinar recommends the owned prefix over stuffing everything into the host_injections symlink.

  • 2025.06 aims NVIDIA driver symlinks with EESSI_NVIDIA_OVERRIDE_DEFAULT. 2023.06 used EESSI_HOST_INJECTIONS. They are not interchangeable.

  • A site MPI that is ABI-compatible with EESSI’s OpenMPI goes in rpath_overrides/OpenMPI/system/lib. The modules above MPI do not get rebuilt.