Four things that actually happened

Four things that actually happened. Each one is a page already written, on a real log.

Four things that actually happened. Each one is already a page.

Four things that happened are four machines, not one story. Each has its own module view.

What you will be able to do

1. Recognise each of these four failure shapes when it happens.

2. Say, for each, which check would have caught it and why it did not.

3. Say what a login node can and cannot report about a cluster.

Each case in this book is real. Each case was presented one at a time. The mechanism is illustrated. They are shown as a historical sequence. They are ordered by diagnosis. They include parts that are not lessons. Reading them together shows that the four cases failed in four distinct ways. Each case was green in some aspect.

The module that was right and the tree that was wrong

NVHPC/25.11-CUDA-12.9.1 declares CUDA 12.9.1. It depends on CUDA/12.9.1. On two of four architectures its unversioned math_libs and comm_libs entries point into the CUDA 13.0 tree provided by the same cuda_multi tarball. A binary linking cuSPARSE through that toolchain obtains libcusparse.so.12 from the 13.0 tree. It requires libnvJitLink.so.13. libnvJitLink.so.13 is available only in the 13.0 redistributable. libnvJitLink.so.13 is absent from any runtime path.

The whole user-visible symptom

vasp_std: error while loading shared libraries: libnvJitLink.so.13:
cannot open shared object file: No such file or directory

Recorded: SURF Snellius, GPU node, reported 2026-09, VASP6 6.5.1 on NVHPC-25.11-CUDA-12.9.1-ACC

It was not a version-detection fault. The fix everyone reaches for first would have changed nothing. EBNVHPCCUDAVER read 12.9 in the generated modulefiles on all four architectures. The easyblock resolved the active CUDA version correctly and patched the installer’s own DESIREDCUDA with it. The installer linked the unversioned entries at the newest CUDA in the tarball regardless. Reaching for a version-detection parameter would have set a value that was already correct.

It was one batch, with every install rebuilt. The four installs carry the times 16:14, 16:18, 16:25 and 17:59 on 2026-01-27. Same day, same easyconfig, same run. Two of them came out one way and two the other.

The split is not the one a first guess would pick. AMD-ZEN2 and AMD-ZEN4 were correct. AMD-ZEN4-H100 and INTEL-AVX512 were not. INTEL-AVX512 has no GPU in it. Any theory of the form “the GPU builds got the newer CUDA” is contradicted by the data. A versionsuffix is part of the name gives the table rather than a summary.

The fix set the symlinks in postinstallcmds. It runs before the sanity check. The check that had been failing could pass on its merits.

The login node that lied, twice

The task took one day. /sw/arch is a symlink for each node architecture.

What a login node resolves

$ readlink -f /sw/arch
/sw/arch/AMD-ZEN2

Recorded: SURF Snellius, 2026-09-10

Login nodes run AMD‑ZEN2, one of the two correct architectures. Every check from a login node reported a working module, while a check on a GPU node reported a broken module with the same name. The problem was declared fixed on that basis. It was not fixed.

The absolute path that still lied

The second mistake, subtler and worth spelling out, follows. Pointing directly at the H100 tree with an absolute /gpfs/.../AMD-ZEN4-H100/... path does not rescue the test. Modulepath entries still resolve against the node underfoot. ldd on the H100 binary, run from a login node, resolves cuSPARSE out of the ZEN2 tree and reports zero missing libraries. A correct‑looking ldd on the wrong node is not weak evidence. It is evidence for a different question. The rule that came out of it: anything about library resolution on a heterogeneous system has to be measured on the architecture it will run on. A login node is not a stand‑in. What “installed” means states that as a caution. eb-stack ports a recipe and bumps one’s claim ladder turns it into a refusal.

The check that was switched off

The VASP easyconfig built on the broken toolchain carried:

skipsteps = ['sanitycheck']

The reasoning indicated that the sanity check reported a failure the package could not fix. Each half of that was true. The check was reporting libnvJitLink.so.13. The defect was in the toolchain module. No change to the VASP easyconfig could have repaired it. The conclusion was still wrong. The next two facts are easy to state. They were not obvious at the time. Switching the check off did not make the binary start. It moved the discovery from a build machine to a user. That is What “installed” means’s whole argument. The check was carrying information: the toolchain module is broken. That requires escalation. A failing check on a package that cannot be fixed is a report about something else. Deleting the report does not make the something else go away. A misconception also existed in the approach. It is common enough to be named. On a GPU package the RPATH check looks like it fails on libcuda.so.1. That library lives with the driver and exists only on GPU nodes. EasyBuild exempts that library by default. EasyBuild also exempts libcuda.so, libnvidia-ml.so and libnvidia-ml.so.1. Anything else it reports is a real finding.

The parameter that was right about the old source

start_dir = 'qmcpack' remained correct during the build from a pinned development commit. The 4.4.0 release bump switched the source from a checkout to a tarball. The tarball extracts to qmcpack-4.4.0/ with sources at the top level. The checksum changed and attracted attention. A checksum mismatch aborts the build and reports the computed value. start_dir stayed unchanged and no component requested a modification. The outcome matches the transcript in An easyconfig is Python: correct URL, verified checksum, tar xzf exit 0, successful patching, loaded toolchain and all build dependencies, followed by a nonexistent path. All expected checks had already succeeded. Two lessons emerged that were absent in earlier cases. A dry run fails to detect the issue. eb -x does not extract anything. A start_dir that specifies a directory absent from the archive appears valid until the build extracts the archive. The verification run prior to pushing was unable to locate the fault. The claim that “the flags are identical” constituted an incorrect comparison. The flags matched. The source block had changed. start_dir depends on the source block. Comparing only unchanged parts can overlook a parameter whose meaning relies on a changed component.

What the four have in common

Each item achieved a green status somewhere. The NVHPC module was correct on two architectures and in its own name. The login-node check passed. The VASP build succeeded with the sanity check disabled. The QMCPACK bump included a verified checksum. The useful question never asks “did it pass”. The focus is what did it pass, and where. That is what eb-stack ports a recipe and bumps one’s ladder records, and all four items explain why it was documented.

EB-History-1 — Which of the four is this

A user reports that a module works on the login node and fails in their job. You have not yet looked at anything.

1. Name two of the four cases above that could produce that report, and say what distinguishes them.

2. What is the first command you run, and on which machine?

3. Your check on the login node comes back clean. What have you learned?

Solution

The first case and the second, and they are different faults with the same shape. The first is a genuinely per-architecture install: the tree under /sw/arch differs, and the module is broken on one architecture and fine on another. The second is a measurement error: the install may be uniformly broken and your check resolved against the wrong tree.

They are distinguished by comparing the two trees directly, from a node of each architecture, rather than by comparing the module names.

The first command is readlink -f /sw/arch, and you run it on the node the user’s job ran on. Until you know which tree you are looking at, no other result means anything.

A clean check on the login node has taught you that the login node’s architecture is fine. Given that the report came from a different architecture, that is not an answer to the question asked, and it is exactly the inference that cost a day the first time.

What to remember

• Every one of the four was green somewhere, so the question is never “did it pass” but what it passed and where.

• A symlink such as /sw/arch resolves against the node underfoot, which makes an ldd on a login node evidence about the login node.

• A failing check on a package that cannot be fixed is a report about something else; deleting the report does not fix the something else.

• A parameter whose meaning depends on the shape of the source has to be re-read whenever the source changes.