Your first week on somebody’s stack

Avail, load, then extend the stack. Do not fork it.

Read the site tree, then add one package. Do not replace the stack.

The first week is a path through somebody else’s stack, not a rewrite of it.

What you will be able to do

1. Take a “bump this package” ticket from the request to a built module, in the order that finds the problems early.

2. Say what your site’s pipeline adds on top of eb, and find those parts in a repository not seen before.

3. Diagnose a failure that a build host reports as green.

Before this page

• Reading a failed build three tasks end in a failed build.

• A pin is for one generation and A versionsuffix is part of the name, for versions and suffixes.

• Read access to whatever repository your site keeps its easyconfigs in.

What is between the operator and eb

Before the first task, find these in the site repository: the easyconfig tree, the patches, the stack list, the build wrapper, and the site config. Each site stores them under different names. A newcomer who finds them within the first hour stops guessing. Most site repositories contain one directory per initial, matching upstream’s layout, with patches beside the easyconfig that needs them. One file per stack generation lists what gets built, in order, with one entry per line, often including per-entry flags. A human edits that file to schedule a build. The easyconfig alone schedules nothing. A component reads that list and calls eb once per entry, per architecture, using the site’s configuration: install prefix, --optarch, CUDA compute capabilities, module naming, hooks. The entire pipeline is read once. All differences between the site and this book’s transcripts originate there. The hooks are covered in A hook changes a file you are not reading. They explain why a recipe can state one thing while the build does another. Locate the directory. Determine which hook applies to the package about to be touched. Starter easyconfigs exist, one per easyblock shape. Copy the site’s template rather than upstream’s example. The first contribution will resemble everyone else’s.

referenceFive questions to answer before your first build

Write the answers down; they are needed all week.

Question	Where to look
Which generation am I building into, and which is in production?	the build lists, and the install prefix each pipeline uses
Which architectures does one entry become?	the pipeline’s per-architecture configuration
What does a build inherit that is not in the easyconfig?	eb --show-config, then the hooks directory
How do I schedule one package rather than the whole list?	the pipeline’s test or single-entry mode
Where do the logs of a failed build end up, and for how long?	the pipeline’s output directory, and its retention

The last one decides whether a failure tomorrow can be diagnosed. On a site that discards build directories nightly, Reading a failed build’s advice about config.log has a deadline.

Task one: the bump that was verified four ways and still failed

The request states that this package is pinned to a development commit. Release 4.4.0 includes the package. The four recipes are moved onto the release. Before any build, four recipes are copied and pointed at the release archive. Four lines are added to the current generation’s build list. eb-stack recipe lint: clean. Dependency resolution: clean. The SHA256 of the release archive matches the one written into the recipes. A diff of build flags, dependencies and sanity checks against the previous version’s recipes shows those four recipes identical. All four builds fail before compiling anything.

Specified start dir /scratch/build/.../qmcpack-4.4.0/qmcpack does not exist

The error originated from start_dir being carried across. The behavior remained correct when the recipe built from a git commit. EasyBuild archives a clone under the repository’s name and places the sources in qmcpack/. The release tarball extracts to qmcpack-4.4.0/ with CMakeLists.txt at the top. Reading a failed build’s descent rule then resolves a relative start_dir under that. The verifying diff in step 6 deliberately excluded the source block. The six-check pipeline passed. start_dir is not in that block. It is derived from that block. Six checks passed correctly. None detected the fault. The general form: a version bump is a change of source, and every parameter that describes the shape of the source must be re-read. Examples include start_dir, patch strip levels, sanity_check_paths, and any value holding a directory name.

Before you look it up

The version has been bumped, a checksum injected, and the build fails in the configure step with a path that does not exist. Nothing has yet opened the log.

Which two commands run first, and which one of them a dry run could have replaced?

What happens

eb --last-log and then tar tzf on the new archive.

Neither is replaceable by a dry run, and that is the point: eb -x never extracts anything, so a parameter that names a directory the archive does not contain looks fine under it. The archive listing is what is in the tarball.

If --force or a rebuild was the first move, that is the reflex worth losing: nothing about this failure is stale state.

Task two: the check that was switched off over the defect it found

The request arrived twenty days late. “the GPU build of this code will not start.”

vasp_std: error while loading shared libraries: libnvJitLink.so.13:
cannot open shared object file: No such file or directory

The whole sequence is the lesson. A production run installs the package on a vendor toolchain. After two hours and sixteen minutes, the RPATH sanity check fails: Library libnvJitLink.so.13 not found. skipsteps = ['sanitycheck'] goes into the recipe, with a comment naming the newer CUDA tree inside the vendor toolchain. The check reports a defect this package cannot fix. A second recipe for another version of the same package on the same toolchain is added and inherits the skipsteps. Twenty days pass. A user loads the module and gets the loader error above. skipsteps comes out of both recipes. A root cause is published and then retracted, having been checked from a login node. The root cause is re‑established on the hardware. The toolchain recipe is then fixed: postinstallcmds relink math_libs and comm_libs at the CUDA version the module declares. A temporary --rebuild goes on every build-list line that used the broken toolchain. It comes off after a check on each architecture. Three separate mistakes are in that list and only one of them is the symlink. Switching the check off moved the discovery to a user. The check was reporting a real defect and no change to this package could repair it. The checks were true. The conclusion was not. The check was carrying information about the toolchain. Deleting the report left the toolchain broken and the package in front of users. Testing, and the four things it can mean. The workaround spread while the defect was hidden. A skipped check costs more than one package: the next recipe copied the workaround along with everything else. The retraction is the part to learn from. ldd on the GPU binary, run from a login node, reports zero missing libraries. Pointing at the other architecture’s tree by its full path does not rescue the test either. The RPATH inside the binary still resolves against the node the command runs on. Where a first‑timer goes wrong: they publish the clean check. A clean check is persuasive. A clean check is cheap. On a heterogeneous machine a clean check is evidence about the node it ran on and nothing else. The general form: a claim about library resolution has to be measured on the architecture that will run the binary. The first command in that diagnosis is readlink -f on the per‑node path, on the node the user’s job ran on. A rebuild flag added to fix a batch is temporary. A temporary rebuild flag needs a named owner and a date. Without a name and date the flag becomes permanent and nobody remembers what it was for.

EB-Week-2 — The commit you would write

You have just fixed the toolchain from task two: the unversioned library directories now point at the CUDA version the module declares, set in postinstallcmds.

Write the commit message. Specifically decide:

1. What has to be in it for the person who reads it in a year.

2. Whether the rebuilt packages need anything, and how you would say so.

3. What you would say about the twenty days, and to whom.

4. Which of these belongs in the commit and which belongs somewhere else: the loader error, the architectures affected, how you measured it, the ticket, and a sanity check that had been switched off.

Solution

1. What was wrong, in one line a search will find: the unversioned math_libs and comm_libs entries pointed into the newer CUDA tree the same tarball carries. Then why the fix lives on that line: postinstallcmds runs before the sanity check, so the check that had been failing can pass honestly.

2. Yes: anything built against the broken toolchain linked the wrong library and has to be rebuilt. Say which packages, and say that the rebuild flag is temporary and who removes it. A temporary flag with no named owner becomes permanent.

3. That the check had been switched off, that it was reporting this defect, and that the twenty days are the cost of that decision rather than of the defect. It goes to whoever will make the same call next month, which means the commit message and the team’s notes, not the ticket. The user gets a working module and a date, not an autopsy.

4. In the commit: the error string the next person will search for, the architectures the bug affected, and how it was measured, in one line. Library resolution is only as good as the node it was made on. Not in the commit: the ticket reference on its own, which ages out, and the retracted diagnosis, which belongs in the notes.

Task three: a user says a module is broken and the report cannot be reproduced

The ticket said 24 regressions. The pipeline printed zero. The pipeline still billed a full hour.

A pipeline’s summary can be empty for two different reasons

A bump that succeeded on three architecture classes failed on the fourth. The archive was the same. A pipeline that summarises a run typically fills EasyBuild log: by grepping for the line eb prints at the end. That line is Results of the build can be found in the log file(s). eb prints that line on failed builds as well. An empty log field does not indicate “no information”. It indicates that eb never printed the line. The failure is before the easyconfig was processed, during the fetch phase. A field such as Last successful build: never is not “no data” when the line is new. It is the summariser’s default. Examine the pipeline’s summariser to determine which fields can be empty for reasons other than absence.

referenceFour habits that shorten the week

1. State where a result came from. Every claim about a build carries a machine and an architecture, in the ticket and in the commit message. This book’s transcripts do it so a later reader can rerun the command.

2. Read the archive, not the diff. A version bump is a change of source.

3. Do not switch a check off to make a build pass. A failing check on a package that cannot be fixed is a report about something else. Testing, and the four things it can mean.

4. Write the diagnosis down where the next person will look. The install already carries the easyconfig, the log and reprod/; what it does not carry is why that choice was made.

EB-Week-1 — Your actual first ticket

A user writes: “Could you install version 6.5.1? We need the GPU build, and the version you have is too old for our workflow.”

Write the plan before you type anything. Specifically:

1. The first four things you find out, and from where.

2. Which of the three tasks above this is, and which parts of the other two it also contains.

3. The two failures you expect, and the check that would catch each one before a user does.

4. What the reply will carry, and what it will omit until you have measured it.

Solution

1. Four facts, from four places. Does upstream have 6.5.1 (eb --search). Which toolchain the GPU build needs, and whether this generation has it. What the site’s current version was built with, from the archived easyconfig in the install directory. Whether the licence covers this group. The licence question blocks the rest and has nothing to do with EasyBuild.

2. Mostly task two, with task one inside it: a version change and a toolchain choice. If the toolchain is new to the site it is also a stack decision, and that is somebody else’s call as much as yours.

3. A source-shape failure, caught by listing the archive and re-reading every parameter that names a directory. And an architecture-specific runtime failure, caught only by running the binary on the target hardware; the build host passing tells you about the build host.

4. The reply says which version, which toolchain, which architectures it was built for, and on which architecture it ran. It does not say “it works” on the strength of a login-node check, and it does not promise a date before the toolchain question is settled.

What to remember

• A site’s pipeline, build lists and hooks are what stand between an easyconfig and a module; find all three before your first build.

• A version bump is a change of source: re-read every parameter that describes the shape of the archive.

• A dry run cannot see a fault that depends on the contents of an archive.

• On a heterogeneous system, measure library resolution on the architecture that will run the binary, not on the one underfoot.

• Every claim about a build carries the machine and the architecture it was measured on, or it is not a claim.