Four things that actually happened

What you will be able to do

  1. Recognise each of these four failure shapes when it happens.

  2. Say, for each, which check would have caught it and why it did not.

  3. Say what a login node can and cannot report about a cluster.

Each case in this book is real and was presented one at a time. The mechanism illustrated. They are shown as a historical sequence, ordered by diagnosis, and include parts that do not serve as lessons. Reading them together demonstrates that the four cases failed in four distinct ways, and each case was green in some aspect.

The module that was right and the tree that was wrong

NVHPC/25.11-CUDA-12.9.1 declares CUDA 12.9.1, depends on CUDA/12.9.1, and on two of four architectures its unversioned math_libs and comm_libs entries point into the CUDA 13.0 tree provided by the same cuda_multi tarball. A binary linking cuSPARSE through that toolchain obtains libcusparse.so.12 from the 13.0 tree, which requires libnvJitLink.so.13, available only in the 13.0 redistributable and absent from any runtime path:

The whole user-visible symptom

vasp_std: error while loading shared libraries: libnvJitLink.so.13:
cannot open shared object file: No such file or directory

Recorded: SURF Snellius, GPU node, reported 2026-09, VASP6 6.5.1 on NVHPC-25.11-CUDA-12.9.1-ACC

It was not a version-detection fault, and the fix everyone reaches for first would have changed nothing. EBNVHPCCUDAVER read 12.9 in the generated modulefiles on all four architectures. The easyblock had resolved the active CUDA version correctly and patched the installer’s own DESIREDCUDA with it. The installer linked the unversioned entries at the newest CUDA in the tarball regardless. Reaching for a version-detection parameter would have set, more emphatically, a value that was already correct.

It was one batch, not a partial rebuild. The four installs are stamped 16:14, 16:18, 16:25 and 17:59 on 2026-01-27. Same day, same easyconfig, same run. Two of them came out one way and two the other.

And the split is not the one a first guess would pick. AMD-ZEN2 and AMD-ZEN4 were correct; AMD-ZEN4-H100 and INTEL-AVX512 were not. INTEL-AVX512 has no GPU in it. Any theory of the form “the GPU builds got the newer CUDA” is contradicted by the data, which is why A versionsuffix is part of the name gives the table rather than a summary.

The fix set the symlinks in postinstallcmds, which runs before the sanity check, so the check that had been failing could pass honestly.

The login node that lied, twice

This part required a day. /sw/arch is a symlink for each node architecture:

What a login node resolves

$ readlink -f /sw/arch
/gpfs/admin/_hpc/sw/arch/AMD-ZEN2

Recorded: SURF Snellius, int4 interactive node, 2026-09-10

The login nodes run AMD‑ZEN2, one of the two architectures that were correct. Thus every check from a login node reported a working module, while the check on a GPU node reported a broken module with the same name. The problem was declared fixed on that basis. It was not.

The second mistake, subtler and worth spelling out, follows. Pointing directly at the H100 tree with an absolute /gpfs/.../AMD-ZEN4-H100/... path does not rescue the test…= entries, and those resolve against the node underfoot. So ldd on the H100 binary, run from a login node, resolves cuSPARSE out of the ZEN2 tree and reports zero missing libraries.

A correct‑looking ldd on the wrong node is not weak evidence. It is evidence for a different question.

The rule that came out of it: anything about library resolution on a heterogeneous system has to be measured on the architecture it will run on. A login node is not a stand‑in. What “installed” means states that as a caution; eb-stack ports a recipe and bumps one’s claim ladder turns it into a refusal.

The check that was switched off

Although the toolchain was broken, the VASP easyconfig built on it carried:

skipsteps = ['sanitycheck']

The reasoning was that the sanity check reported a failure the package could not fix. Each half of that was true. The check was reporting libnvJitLink.so.13, the defect was in the toolchain module, and no change to the VASP easyconfig could have repaired it.

The conclusion was still wrong. The next two facts are easy to state and were not obvious at the time.

Switching the check off did not make the binary start. It moved the discovery from a build host to a user, which is What “installed” means’s whole argument.

And the check was carrying information: the toolchain module is broken. That is a thing to escalate. A failing check on a package that cannot be fixed is a report about something else, and deleting the report does not make the something else go away.

There was also a misconception in the way, and it is common enough to name. On a GPU package the RPATH check looks like it fails on libcuda.so.1, which lives with the driver and exists only on GPU nodes. It does not: EasyBuild exempts that library by default, along with libcuda.so, libnvidia-ml.so and libnvidia-ml.so.1. Anything else it reports is a real finding.

The parameter that was right about the old source

start_dir = 'qmcpack' remained correct while the recipe built from a pinned development commit. The 4.4.0 release bump switched the source from a checkout to a tarball, which extracts to qmcpack-4.4.0/ with the sources at the top level. The checksum changed and attracted attention, since a checksum mismatch aborts the build and reports the computed value. start_dir stayed unchanged, and no component requested a modification. The outcome matches the transcript in An easyconfig is Python: correct URL, verified checksum, tar xzf exit 0, successful patching, loaded toolchain and all build dependencies, followed by a nonexistent path. All expected checks had already succeeded. Two lessons emerged that were absent in earlier cases. A dry run fails to detect the issue. eb -x does not extract anything, so a start_dir that specifies a directory absent from the archive appears valid until the build extracts the archive. The verification run prior to pushing was inherently unable to locate the fault. The claim that “the flags are identical” constituted an incorrect comparison. Although the flags matched, the source block had changed, and start_dir depends on the source block. Comparing only unchanged parts can overlook a parameter whose meaning relies on a changed component.

What the four have in common

Each one was green somewhere. The NVHPC module was correct on two architectures and in its own name. The login-node check passed. The VASP build with the sanity check disabled succeeded. The QMCPACK bump had a verified checksum. The useful question is never “did it pass”. It is what did it pass, and where. That is what eb-stack ports a recipe and bumps one’s ladder is bookkeeping for, and all four of these are why somebody bothered to write it down.

EB-History-1 — Which of the four is this

A user reports that a module works on the login node and fails in their job. You have not yet looked at anything.

  1. Name two of the four cases above that could produce that report, and say what distinguishes them.

  2. What is the first command you run, and on which machine?

  3. Your check on the login node comes back clean. What have you learned?

Solution

The first case and the second, and they are different faults with the same shape. The first is a genuinely per-architecture install: the tree under /sw/arch differs, and the module is broken on one architecture and fine on another. The second is a measurement error: the install may be uniformly broken and your check resolved against the wrong tree.

They are distinguished by comparing the two trees directly, from a node of each architecture, rather than by comparing the module names.

The first command is readlink -f /sw/arch, and you run it on the node the user’s job ran on. Until you know which tree you are looking at, no other result means anything.

A clean check on the login node has taught you that the login node’s architecture is fine. Given that the report came from a different architecture, that is not an answer to the question asked, and it is exactly the inference that cost a day the first time.

What to remember

  • Every one of the four was green somewhere, so the question is never “did it pass” but what it passed and where.

  • A symlink such as /sw/arch resolves against the node underfoot on, which makes an ldd on a login node evidence about the login node.

  • A failing check on a package that cannot be fixed is a report about something else; deleting the report does not fix the something else.

  • A parameter whose meaning depends on the shape of the source has to be re-read whenever the source changes.