A site stack next to EESSI, not inside it

site stack joins MODULEPATH beside EESSI, not inside the mount

A site stack is next to EESSI. Injecting into the mount is a different, wrong, object.

site stack beside EESSI

Host injections sit next to EESSI. They do not write into the mount.

site MPI injected beside EESSI, not copied into it

A host injection is next to EESSI. It does not rewrite the CVMFS tree.

Questions

  • What does EESSI_SITE_SOFTWARE_PREFIX change, and when must it be set?

  • When is host_injections the right tree, and when is a separate prefix the one the webinar recommends?

  • What still has to be true of a site OpenMPI for the EESSI stack above it to keep working?

What you will be able to do

  1. Install one package on top of EESSI into an operator-chosen prefix with EESSI-extend, and say why that prefix contains a microarchitecture.

  2. Tell Approach 1 (shared filesystem) from Approach 2 (site CVMFS plus the bot), and say which extra moving parts the second one adds.

  3. Point NVIDIA drivers, a full CUDA SDK, and an ABI-compatible site MPI at the paths EESSI actually searches.

Before this page

EESSI provides a shared software stack. A site may have software EESSI does not provide. Examples are a closed‑source licence, software not yet on the tree, and software tuned for a specific fabric. A webinar on 1 June 2026 by Caspar van Leeuwen and Bob Dröge addresses that situation. The official host_injections page lists four bullets and a symlink. The webinar describes two approaches, two live clusters, and the variables that enable a login session to see both trees.

Why a site builds at all

Three reasons appear on the webinar’s motivation slide. They are not the same job. Licensed software cannot go in software.eessi.io. It may also be limited to a subset of the site’s users. Software that is not on the tree yet is a time problem. Waiting for the bot on every architecture is the right path into EESSI (pages 30 and 31). It is the wrong path for a group that needs the binary this week. Custom installations are the fabric, the licence server, the site MPI. Those cannot be expressed as an easyconfig that EESSI would accept. They are facts about the machine.

Approach 1: EESSI-extend into a shared filesystem

The inexpensive approach. It requires no additional CernVM-FS, no bot instance, and no object store. EESSI-extend already knows how to write into an architecture-specific prefix and add that prefix to MODULEPATH. The webinar includes a Hábrók demo at the University of Groningen using AlmaLinux 9.8 on zen3 with an unprivileged installer account.

export EESSI_SITE_INSTALL=1
export EESSI_SITE_SOFTWARE_PREFIX=/software
module load EESSI/2025.06
module load EESSI-extend
echo $EESSI_SITE_SOFTWARE_PATH
# /software/versions/2025.06/software/linux/x86_64/amd/zen3
eb --show-config # installpath is that path
eb cowsay-3.04.eb
eb attr-2.5.2-GCCcore-14.3.0.eb
module avail
# ---- /software/versions/2025.06/software/linux/x86_64/amd/zen3/modules/all ----
# attr/2.5.2-GCCcore-14.3.0 cowsay/3.04

Set EESSI_SITE_SOFTWARE_PREFIX before the EESSI module loads. The module computes EESSI_SITE_SOFTWARE_PATH from it. Setting the prefix later has no effect. That matches the load-time rule for EESSI, somebody else’s stack’s EESSI_USER_INSTALL.

Both EESSI_SITE_INSTALL and EESSI_CVMFS_INSTALL are exclusive of EESSI_USER_INSTALL and EESSI_PROJECT_INSTALL. User and project can be set together; site and CVMFS cannot.

The webinar repeats the page‑11 warning, applied to a site prefix: EESSI-extend performs native optimisation and installs into a prefix named for the host architecture. The Hábrók zen3 modules remain invisible on a zen4 node until the same eb is executed there.

There are two locations where the prefix can reside.

Option 1, the variant symlink. /cvmfs/software.eessi.io/host_injections stores the target $(EESSI_HOST_INJECTIONS:-/opt/eessi). A site CernVM-FS config of

# /etc/cvmfs/domain.d/eessi.io.local
EESSI_HOST_INJECTIONS=/shared_fs/path/to/host/injections/

Running cvmfs_config reload makes that tree behave as if it were inside the repository. It does not write to the repository. It does not invalidate a signature. Different node types may point the symlink at different directories. Two GPU partitions with drivers in different places need two EESSI_NVIDIA_OVERRIDE_DEFAULT targets in 2025.06, not two EESSI_HOST_INJECTIONS values. Option 2 is the one the webinar marks recommended. It uses a prefix the site owns, EESSI_SITE_SOFTWARE_PREFIX=/software or /cvmfs/software.caspar.nl. This prefix is not the variant symlink. The EESSI module still prepends the derived EESSI_SITE_SOFTWARE_PATH to MODULEPATH. The bytes live wherever the site puts them. The cons of Approach 1 are the ones the webinar lists. The procedure is manual. A shared filesystem is a worse startup path than CernVM-FS. For a few packages on one cluster it is enough. A container is available when the host is not a safe build environment. Run eessi_container.sh -b $EESSI_SITE_SOFTWARE_PREFIX to get the same four exports and module loads inside.

Approach 2: the bot, pointed at a site repository

The webinar’s second half covers the bot as three processes and a comment aimed at a different CernVM-FS name. The quoted goal is to build using dependencies from EESSI. The build occurs in a CVMFS repository of our own. The build uses the EESSI build bot. The same build scripts from software-layer-scripts are used. The “what we need” slide adds a site Stratum-0 and, in production, Stratum-1 servers and proxies. One bot instance per cluster builds natively. An S3‑compatible bucket is required. A GitHub repository whose easystack path is this file:

easystacks/<site_cvmfs_repo>/<EESSI_VERSION>/eessi-<EESSI_VERSION>-eb-<eb_version>-<anything>.yml

The Stratum-0 component ingests tarballs. The demo repository is software.caspar.nl. The demo pull request adds Biopython-1.86-gfbf-2025b.eb. After ingest:

export EESSI_SITE_SOFTWARE_PREFIX=/cvmfs/software.caspar.nl
source /cvmfs/software.eessi.io/versions/2025.06/init/lmod/bash
module load Biopython/1.86-gfbf-2025b
python -c 'import Bio.motifs; print(Bio.motifs.__path__)'
# .../software.caspar.nl/versions/2025.06/software/linux/x86_64/amd/zen2/...
which python
# /cvmfs/software.eessi.io/versions/2025.06/.../Python/3.13.5-...

The site module is the EESSI interpreter. It fulfills the entire purpose. A site ingest does not require EESSI/staging_bundles. ingest_tarballs.sh lists the bucket, downloads the contents, optionally verifies a signature, calls filesystem-layer/scripts/ingest-tarball.sh, regenerates nested catalogs, and remounts. Nested catalogs follow the EESSI .cvmfsdirtab pattern. Each installation contains one catalog. The Stratum-0 cannot employ the automounter. software.eessi.io is mounted from fstab when the ingest script requires EESSI’s Lmod to rebuild caches. Rebuilds on a site repository use the same rebuilds/ directory. An easystack is the list the bot will build. The scripts include the --rebuild option. Removal is a Stratum-0 transaction and an Lmod cache rebuild; a bot comment cannot do it.

Drivers, CUDA and an ABI-compatible MPI

Three items are added into the injection tree by a site, even if the remaining site stack resides in the prefix recommended by Approach 1. The runtime includes NVIDIA drivers. The driver is bound to the host kernel. It cannot be distributed in the repository. For EESSI 2025.06 and later the compatibility‑layer linker searches.

/cvmfs/software.eessi.io/versions/<ver>/compat/<os>/<arch>/lib/nvidia

It is a variant symlink itself. The CernVM-FS config targeted is EESSI_NVIDIA_OVERRIDE_DEFAULT, not EESSI_HOST_INJECTIONS.

# /etc/cvmfs/default.local (or domain.d/eessi.io.local)
EESSI_NVIDIA_OVERRIDE_DEFAULT=/opt/eessi/nvidia

cvmfs_config reload software.eessi.io, then

/cvmfs/software.eessi.io/versions/${EESSI_VERSION}/scripts/gpu_support/nvidia/link_nvidia_host_libraries.sh

Re-run after each driver update. The per-version override EESSI_202506_NVIDIA_OVERRIDE applies when 2023.06 and 2025.06 must not share a symlink set. In 2023.06 the same script continues to write under host_injections, and EESSI_HOST_INJECTIONS designates that location. Mixing the two conventions causes a 2025.06 node to obtain a linker search path that points to an empty directory. The official GPU page states that “ROCm support is not yet available”. The live 2025.06 catalogs already include gfx* trees. The page represents the missing component. A full CUDA SDK, compile time. The CUDA and cuDNN packages in the repository are runtime‑only; the EULA prohibits redistribution of nvcc. install_cuda_and_libraries.sh installs the missing components into the injection tree via EasyBuild, restoring symlinks such as

.../software/CUDA/12.1.1/bin/nvcc ->
 /cvmfs/software.eessi.io/host_injections/2023.06/software/.../CUDA/12.1.1/bin/nvcc

The installer account is not root. EasyBuild runs as non-root unless EASYBUILD_ALLOW_USE_AS_ROOT_AND_ACCEPT_CONSEQUENCES is set. The GPU page recommends changing ownership of the injection directory. Site MPI. EESSI’s OpenMPI does not know a site fabric. An ABI‑compatible replacement is placed where the RPATH of the EESSI binaries already points. The Slingshot-11 write-up places OpenMPI 5.x for 2023.06 on Grace:

/cvmfs/software.eessi.io/host_injections/2023.06/software/linux/aarch64/nvidia/grace/rpath_overrides/OpenMPI/system/lib

The EESSI GROMACS and the EESSI OSU benchmarks retain their module names. The libraries they resolve at runtime are those provided by the site. No package above MPI is rebuilt. That is true only if the replacement is ABI‑compatible with the EESSI OpenMPI that the rest of the tree was linked against. Lmod hooks constitute the other half of fabric tuning: environment variables at load time rather than a replaced .so. EESSI’s SitePackage.lua searches for two additional files under host_injections:.

$EESSI_CVMFS_REPO/host_injections/$EESSI_VERSION/.lmod/SitePackage.lua
$EESSI_CVMFS_REPO/host_injections/$EESSI_VERSION/software/$EESSI_OS_TYPE/$EESSI_SOFTWARE_SUBDIR/.lmod/SitePackage.lua

The initial version is architecture-independent. The subsequent version lacks architecture independence. Version 2023.06 provides Lmod 8.7.30. Lmod 8.7.30 cannot register two load hooks. The site file merges eessi_load_hook with the site function into a single registration. Lmod 8.7.35 and newer are able to register both.

What a login session has to export

The webinar’s last section presented two patterns. The first pattern uses EESSI’s Lmod. A file in /etc/profile.d sets EESSI_SITE_SOFTWARE_PREFIX. The same file sources /cvmfs/software.eessi.io/versions/2025.06/init/lmod/bash. The second pattern uses the site’s Lmod. Hábrók’s 01-local_lmod.sh is employed.

export EESSI_SITE_SOFTWARE_PREFIX=/software
export MODULEPATH=/cvmfs/software.eessi.io/init/modules
export LMOD_CACHED_LOADS=yes
export LMOD_SHORT_TIME=86400

The z01_StdEnv.sh sets LMOD_SYSTEM_DEFAULT_MODULES=EESSI/2025.06. A login loads EESSI and shows /software/versions/2025.06/.../modules/all next to the CernVM-FS tree. The webinar names EESSI_MODULE_STICKY and EESSI_MODULE_FAMILY_NAME as the other two variables. These variables make the EESSI module sticky. They enable coexistence with a second, incompatible site stack. Licensed software that only some users may run is identified as a poor fit for a site CVMFS repository. The setting CVMFS_CLAIM_OWNERSHIP=no preserves ownership. A container can override it, so ownership is not a security boundary. The webinar proposes keeping those installations on the shared filesystem of Approach 1.

EB-Site-1 — Which prefix, which variable

Four requests. For each, name the approach, the variable that has to be set before any module load, and the one check that says the result is in the site prefix.

  1. A single extra package, this week, on one zen3 partition, no CernVM-FS admin access.

  2. NVIDIA driver libraries on a 2025.06 node after last night’s driver update.

  3. OpenMPI rebuilt against the site Slingshot, consumed by the EESSI GROMACS already on the tree.

  4. Twenty packages, three microarchitectures, a Git history of what was built, same bot comments as EESSI itself.

Solution

  1. Approach 1. EESSI_SITE_INSTALL=1 and EESSI_SITE_SOFTWARE_PREFIX pointing at a writable shared directory, set before module load EESSI. Check $EESSI_SITE_SOFTWARE_PATH ends in .../x86_64/amd/zen3 and that eb --show-config reports that path as installpath.

  2. Not a prefix variable. EESSI_NVIDIA_OVERRIDE_DEFAULT in the CernVM-FS config, cvmfs_config reload, then re-run link_nvidia_host_libraries.sh. Check deviceQuery from CUDA-Samples on that node. 2023.06 would have used EESSI_HOST_INJECTIONS instead.

  3. The replacement libraries go in host_injections/<ver>/software/linux/<software_subdir>/rpath_overrides/OpenMPI/system/lib (the live Slingshot-11 tree is .../linux/aarch64/nvidia/grace/rpath_overrides/OpenMPI/system/lib). The check is ldd on an EESSI MPI binary, on the compute node, showing the site .so, plus an OSU run. ABI compatibility is the constraint; a rebuild of GROMACS is the thing this path exists to avoid.

  4. Approach 2. A site CernVM-FS repo, a bot instance, easystacks under easystacks/<that.repo>/<ver>/, EESSI_SITE_SOFTWARE_PREFIX set to the repo root on the clients. Check a loaded site module’s path is under the site repo while its Python (or MPI, or libc) still resolves under software.eessi.io.

EB-Site-2 — 2023.06 or 2025.06

A site config contains only this line, in /etc/cvmfs/domain.d/eessi.io.local:

EESSI_HOST_INJECTIONS=/shared/eessi-inject

link_nvidia_host_libraries.sh executed last year targeting 2023.06. Users switched to EESSI/2025.06. CUDA applications fail to locate libcuda.so. Identify the cause of the success on 2023.06 and the failure on 2025.06. Identify the addition to the CernVM-FS config. Identify the component that is re‑run. Determine the number of injection trees for two GPU partitions with drivers in different directories. Determine the location where the difference is expressed.

Solution

  1. 2023.06’s linker search path for NVIDIA is under host_injections, which EESSI_HOST_INJECTIONS aims. 2025.06 searches /cvmfs/software.eessi.io/versions/2025.06/compat/.../lib/nvidia, a different variant symlink, aimed by EESSI_NVIDIA_OVERRIDE_DEFAULT. The old variable still aims the old tree. The new linker is not looking there.

  2. EESSI_NVIDIA_OVERRIDE_DEFAULT=/shared/eessi-inject/nvidia (or a per-version EESSI_202506_NVIDIA_OVERRIDE), cvmfs_config reload software.eessi.io, then the 2025.06 link_nvidia_host_libraries.sh. Last year’s symlinks are not reused.

  3. Two EESSI_NVIDIA_OVERRIDE_DEFAULT targets, one per GPU partition, then cvmfs_config reload and link_nvidia_host_libraries.sh on each. 2025.06 drivers live under compat/.../lib/nvidia. They do not use a second EESSI_HOST_INJECTIONS tree.

What to remember

  • EESSI_SITE_SOFTWARE_PREFIX is set before the EESSI module loads. EESSI-extend then writes an architecture-specific prefix and puts it on MODULEPATH.

  • Approach 1 is that prefix on a shared filesystem. Approach 2 is the same bot and the same scripts, aimed at a site CernVM-FS repository. The webinar recommends the owned prefix over stuffing everything into the host_injections symlink.

  • 2025.06 aims NVIDIA driver symlinks with EESSI_NVIDIA_OVERRIDE_DEFAULT. 2023.06 used EESSI_HOST_INJECTIONS. They are not interchangeable.

  • A site MPI that is ABI-compatible with EESSI’s OpenMPI goes in rpath_overrides/OpenMPI/system/lib. The modules above MPI do not get rebuilt.