EESSI, somebody else’s stack

Questions

  • What is EESSI, if it is not an install of EasyBuild?

  • Which init script configures Lmod, and which one is irreversible?

  • When does a plan bootstrap GCC instead of using the stack?

What you will be able to do

  1. Initialise EESSI and install on top of it into an operator-owned prefix.

  2. Decide whether your toolchain generation is available before planning anything.

  3. Say what a green build on your own host establishes about the other CPU targets.

A directory exists on many clusters and on many laptops that was not built by anyone at the site. MultiXscale funds a large portion of EESSI’s workforce as part of the EuroHPC Centre of Excellence (CECAM × EESSI). CASTIEL 2 is the tutorial portal rather than the build farm. The EFP Federated Software Catalogue (CSC.fi with Ghent) is a 2026 catalogue layer above. Compute Canada acts as the design parent of the three-layer split (Boissonneault et al. [2019]). Dröge et al. [2023] evaluate why a generic binary does not constitute the stack: GROMACS 2020.4 on Cascade Lake achieves 1.18 ns/day when compiled for SSE 4.1 and 1.86 ns/day when compiled for AVX‑512, using unchanged source. That is a 57 % gap. The production tree and the bot were created after that paper.

The stack is a filesystem

ls /cvmfs/software.eessi.io/versions/

These are versions of a complete scientific software stack: compilers, MPI, the usual suspects, tuned per microarchitecture, delivered on a filesystem that arrived over HTTP. EESSI is not installed. It is mounted, or the session runs on a machine where it has already been mounted. Pick one and initialise it:

source /cvmfs/software.eessi.io/versions/2025.06/init/lmod/bash

It produces a compatibility layer, a module tree, and a collection of toolchains, without requiring an install step or any maintainer approval. Two warnings should be placed here, before any other content, since each consumes actual time. The first concerns the ls command. List the versions directory and verify that a version is valid before using it:.

ls /cvmfs/software.eessi.io/versions/<ver>/init/lmod/bash

A listed version may be an empty placeholder lacking any init/.

Speculatively descending into such a placeholder can wedge the filesystem client for all host users.

The lookup neither resolves nor raises an error, and readers accumulate in an uninterruptible state beyond any timeout reach.

List first.

Confirm.

Then use.

The second concerns the generation question, which determines whether the remainder of this chapter applies.

Check the generation before planning anything

The compatibility layer lags behind the newest EasyBuild toolchain generation. A version of EESSI includes a specific set of top‑level toolchains, while an easyconfig names a particular generation, and these two facts must agree. If they do not agree, EESSI reports the mismatch, and an opt‑in is available.

export EESSI_SITE_TOP_LEVEL_TOOLCHAINS_2025_06='[{"name": "GCCcore", "version": "15.2.0"}]'

A setting causes EasyBuild plan the build. Read the plan before the script runs. If the generation is absent from the compatibility layer, every dependency is skipped. The run then begins by bootstrapping the compiler itself and downloads a couple of hundred megabytes of GCC. Avoid that. Bootstrapping an entire generation into a user prefix duplicates what EESSI’s own build bots do centrally. It also creates a long detour to a stack that cannot be shared. A generation gap means “not yet on this EESSI version”, which is a finding rather than a failure.

Installing on top, with EESSI-extend

EESSI provides a module that configures EasyBuild for installing onto the stack. The build bots use the same module, so this path matches what the bot will do:

source /cvmfs/software.eessi.io/versions/2025.06/init/lmod/bash

export EESSI_USER_INSTALL=$HOME/eessi/2025.06
mkdir -p $EESSI_USER_INSTALL

module load EESSI-extend
eb --show-config
eb <easyconfig>.eb --robot

Build your own, without asking anyone’s --show-config earns its place again. Run it and read installpath: it comes back qualified by microarchitecture, something like .../software/linux/x86_64/amd/zen5.

Stop and look at that path. It is What “installed” means’s per-architecture tree, and here the architecture is in the path rather than hidden behind a symlink. Same arrangement, made visible. A binary built here is for one microarchitecture and says so.

There are four prefix variables, and they differ in who can read the result:

Variable

What it means

EESSI_USER_INSTALL

private to the operator

EESSI_PROJECT_INSTALL

shared with a group

EESSI_SITE_INSTALL

site-wide, on top of EESSI

EESSI_CVMFS_INSTALL

into the distributed repository itself

Set the prefix before loading the module. EESSI-extend reads these at load time; setting them afterwards changes nothing.

EESSI_CVMFS_INSTALL and EESSI_SITE_INSTALL are exclusive of the others. EESSI_USER_INSTALL and EESSI_PROJECT_INSTALL may be set together: both prefixes appear on MODULEPATH, and new installs go to the user prefix.

EB-EESSI-1 — Every dependency says SKIPPED

You have set one prefix variable, loaded EESSI-extend on the 2025.06 tree, and run a plan for an easyconfig on foss-2026.1. Every dependency comes back [SKIPPED], and the run opens by fetching a couple of hundred megabytes of GCC.

  1. What has happened?

  2. You could let it run. Give two reasons not to.

  3. What is the honest thing to report, and to whom?

Solution

The compatibility layer for EESSI 2025.06 does not carry foss-2026.1. (It does carry foss-2025b; that pin would not have produced this plan.) Nothing above the compiler exists to be reused, so the plan is to build the generation itself, starting with the compiler.

Two reasons not to. It duplicates work the EESSI build bots do centrally for every supported CPU target, where yours would cover one. And what you end up with is a whole generation inside a user prefix that nobody else can use and that you now maintain.

The honest report is a generation gap: this recipe needs foss-2026.1, this EESSI version’s layer stops short of it, so the ceiling here is a build on an ordinary target rather than an EESSI-verified one. That goes to whoever asked for the software, and the software-layer entry waits for an EESSI version whose layer carries the generation.

When the filesystem client is the problem

The symptoms manifest in a specific way. A host’s client can report healthy while being unusable. The probe reports fine and the server answers a plain HTTP request in milliseconds. Every directory listing into a nested path then blocks forever, and poisons the shell it ran in. Two solutions exist, ordered by preference. If the account has polkit rights on the host, which an ordinary user often does, restarting the mount clears the stuck readers. The operation should be performed from a terminal that did not run the hanging command. If the account lacks polkit rights, a container can be used. The container mounts the distributed filesystem itself, requires only unprivileged user namespaces, and is independent of the host client state.

git clone --depth 1 https://github.com/EESSI/software-layer-scripts.git
export APPTAINER_CACHEDIR=$PWD/storage/apptainer-cache
./software-layer-scripts/eessi_container.sh \
  --mode exec --access ro --storage $PWD/storage \
  --extra-bind-paths "$PWD/ecs:/ecs:ro" \
  -- /bin/bash /ecs/run-extend.sh

Four details that each cost an attempt:

  • the script lives in software-layer-scripts.

  • It moved, and software-layer no longer carries it.

  • --mode run warns and behaves as exec.

  • Pass exec.

  • the script that is run must be inside a bound path.

  • One that resolves outside the bind mount fails as though the file were absent.

  • set APPTAINER_CACHEDIR.

  • With the older SINGULARITY_ name also set, the older one wins and the newer is dropped with no message.

Diagnosing the host client further rarely pays. On one host, root-catalog fetches succeeded while nested traversal stalled. A second server, disabled timeouts and a wiped cache each changed nothing, and a container on the same host and network traversed everything instantly.

Building by hand on top of it

Sometimes building without EasyBuild is necessary. Load the buildenv module of the toolchain instead of improvising.

module load buildenv/default-foss-2025b
module load <dependency modules>
make
readelf -d <binary>
ldd <binary>

buildenv configures $CC and $CFLAGS as EasyBuild does. It also installs wrappers that inject RPATH. The two commands following make constitute What “installed” means’s third check, executed manually. Confirm the binary records a runtime path. Confirm nothing resolves outside the stack.

Contributing back

An installation in a local prefix helps that operator. Getting it into EESSI proper is a different job, and the order matters. The six steps below are the postcard. An easystack is the list the bot will build is the YAML. The bot is three processes and a comment is the comment that starts the build and the ingest that makes a module. A site stack next to EESSI, not inside it is the site tree that never goes into software.eessi.io.

  1. Get the easyconfig into EasyBuild, or open a pull request for it.

  2. Build it in the EESSI build environment with EESSI-extend.

  3. Add an entry to the easystack file for that EESSI version and toolchain generation.

  4. Open a pull request against the software layer, targeting main.

  5. A builder posts one bot: build for:arch…= per supported CPU target.

  6. On green, bot:deploy uploads the tarballs; a Stratum-0 ingest is what publishes them.

Two conditions before the work starts.

The easyconfig comes first. A software-layer entry references an easyconfig rather than restating one. So deployment waits on the EasyBuild pull request being merged, even though the two can be open at once.

And a green build on your host is not the claim. The bot builds x86-64 down to generic and up through Zen and Sapphire Rapids, plus aarch64 from generic through Neoverse and Grace. Software that works on your microarchitecture and not the others is not ready, however green it looks locally. That is the same distinction as What “installed” means’s, with a larger set of trees.

The three that bite

All remaining content in this chapter describes procedures. These three steps consume real time, and they must be performed in the given order.

List before descending. A version directory may serve as an unpopulated placeholder. Speculative listing into such a directory wedges the filesystem client for every user on the host. Confirm that an init/ exists before proceeding.

Check the generation before planning. The compatibility layer lags behind the newest toolchain generation. A gap should be reported as a finding rather than used to bootstrap a compiler.

One prefix variable, set before the module load. The EESSI-extend reads it at load time. A second or late variable causes a failure that appears as the module ignoring the request.

After that, keep two aspects in view. The installpath returns a microarchitecture‑qualified path, which corresponds to What “installed” means’s per‑architecture tree with the architecture visible in the path. The all‑CPU‑target claim belongs to the bot rather than to the host.

Next: the engine on which these samples run, and the one idea in it worth taking even if never used.