Your first week on somebody’s stack

What you will be able to do

1. Take a “bump this package” ticket from the request to a built module, in the order that finds the problems early.

2. Say what your site’s pipeline adds on top of eb, and find those parts in a repository not seen before.

3. Diagnose a failure that a build host reports as green.

Before this chapter

• Reading a failed build three tasks end in a failed build.

• A pin is for one generation and A versionsuffix is part of the name, for versions and suffixes.

• Read access to whatever repository your site keeps its easyconfigs in.

All preceding material is a mechanism. This chapter constitutes the job.

Three authentic tasks, typical of those assigned in a first month, are presented. Each is expressed as a followable sequence, with the error point for a beginner marked.

What is between the operator and eb

Nothing in this chapter concerns EasyBuild’s defaults. Before the first task, locate these five items in the site’s repository. Each site contains them under different names, and a newcomer who locates them within the first hour stops guessing.

One directory per initial exists in most site repositories, matching upstream’s layout, with patches beside the easyconfig that needs them.

A file per stack generation names what gets built, in order, with one entry per line, often including per-entry flags. A human edits that file to schedule a build; the easyconfig alone schedules nothing.

A component reads that list and calls eb once per entry, per architecture, using the site’s configuration: install prefix, --optarch, CUDA compute capabilities, module naming, hooks. The entire pipeline should be read once. Every difference between the site and this book’s transcripts originates there.

The hooks are the subject of A hook changes a file you are not reading, and they explain why a recipe can state one thing while the build does another. Locate the directory and determine which hook applies to the package about to be touched.

Starter easyconfigs exist, one per easyblock shape. Copy the site’s template rather than upstream’s example, and the first contribution will resemble everyone else’s.

referenceFive questions to answer before your first build

Write the answers down; they are needed all week.

Question	Where to look
Which generation am I building into, and which is in production?	the build lists, and the install prefix each pipeline uses
Which architectures does one entry become?	the pipeline’s per-architecture configuration
What does a build inherit that is not in the easyconfig?	eb --show-config, then the hooks directory
How do I schedule one package rather than the whole list?	the pipeline’s test or single-entry mode
Where do the logs of a failed build end up, and for how long?	the pipeline’s output directory, and its retention

The last one decides whether a failure tomorrow can be diagnosed. On a site that discards build directories nightly, Reading a failed build’s advice about config.log has a deadline.

Task one: the bump that was verified four ways and still failed

The request: this package is pinned to a development commit. Release 4.4.0 has it. Move the four recipes onto the release. What was done, in order, before anything was built: Four recipes copied and pointed at the release archive. Four lines added to the current generation’s build list. eb-stack recipe lint: clean. Dependency resolution: clean. The SHA256 of the release archive, compared against the one written into the recipes: identical. A diff of build flags, dependencies and sanity checks against the previous version’s recipes: identical. All four builds then failed, before compiling anything.

Specified start dir /scratch/build/.../qmcpack-4.4.0/qmcpack does not exist

The error stemmed from start_dir being carried across. The behavior remained correct while the recipe built from a git commit, since EasyBuild archives a clone under the repository’s name, placing the sources in qmcpack/. The release tarball extracts to qmcpack-4.4.0/ with CMakeLists.txt at the top, and Reading a failed build’s descent rule then resolves a relative start_dir under that. The method failure provides the more useful half. The verifying diff in step 6 deliberately excluded the source block. start_dir is not in that block, and it is derived from it. six checks passed, each correctly, and none detected the fault. The general form: a version bump is a change of source, and every parameter that describes the shape of the source must be re-read. Examples include start_dir, patch strip levels, sanity_check_paths, and any value holding a directory name.

Before you look it up

The version has been bumped, a checksum injected, and the build fails in the configure step with a path that does not exist. Nothing has yet opened the log.

Which two commands run first, and which one of them a dry run could have replaced?

What happens

eb --last-log and then tar tzf on the new archive.

Neither is replaceable by a dry run, and that is the point: eb -x never extracts anything, so a parameter that names a directory the archive does not contain looks fine under it. The archive listing is the only thing that settles it.

If --force or a rebuild was the first move, that is the reflex worth losing: nothing about this failure is stale state.

Task two: the check that was switched off over the defect it found

The request arrived twenty days late: “the GPU build of this code will not start.”

vasp_std: error while loading shared libraries: libnvJitLink.so.13:
cannot open shared object file: No such file or directory

The whole sequence is the lesson.

A production run installs the package on a vendor toolchain.

After two hours and sixteen minutes, the RPATH sanity check fails: Library libnvJitLink.so.13 not found.

skipsteps = ['sanitycheck'] goes into the recipe, with a comment naming the newer CUDA tree inside the vendor toolchain.

The reasoning: the check reports a defect this package cannot fix.

A second recipe, for another version of the same package on the same toolchain, is added and inherits the skipsteps.

Twenty days pass.

A user loads the module and gets the loader error above.

skipsteps comes out of both recipes.

A root cause is published, and then retracted, having been checked from a login node.

The root cause is re-established, on the hardware, and the toolchain’s own recipe is fixed: postinstallcmds relinking the unversioned math_libs and comm_libs entries at the CUDA version the module declares.

A temporary --rebuild goes on the build-list lines for everything built against the broken toolchain, and comes off once the results are checked on a node of each architecture class.

Three separate mistakes are in that list, and only one of them is the symlink.

Switching the check off moved the discovery to a user.

Step 2’s reasoning was true on both counts: the check was reporting a real defect, and no change to this package could repair it.

The conclusion did not follow.

The check was carrying information about the toolchain, and deleting the report left the toolchain broken and the package shipping.

Testing, and the four things it can mean.

The workaround spread while the defect was hidden.

Step 3 is what a skipped check costs beyond one package: the next recipe copied the workaround along with everything else.

The retraction is the part to learn from.

Step 6 happened.

So ldd on the GPU binary, run from a login node, reports zero missing libraries.

Pointing at the other architecture’s tree by its full path does not rescue the test either: the RPATH inside the binary still resolves against the node the command runs on.

Where a first-timer goes wrong: they publish step 6.

A clean check is persuasive, it is cheap, and on a heterogeneous machine it is evidence about the node it ran on and nothing else.

The general form: a claim about library resolution has to be measured on the architecture that will run the binary.

The first command in that diagnosis is readlink -f on the per-node path, on the node the user’s job ran on.

And the thing to take from step 8: a rebuild flag added to fix a batch is temporary, so it needs a named owner and a date, or it becomes permanent and nobody remembers what it was for.

EB-Week-2 — The commit you would write

You have just fixed the toolchain from task two: the unversioned library directories now point at the CUDA version the module declares, set in postinstallcmds.

Write the commit message. Specifically decide:

1. What has to be in it for the person who reads it in a year.

2. Whether the rebuilt packages need anything, and how you would say so.

3. What you would say about the twenty days, and to whom.

4. Which of these belongs in the commit and which belongs somewhere else: the loader error, the architectures affected, how you measured it, the ticket, and the fact that a sanity check had been switched off.

Solution

1. What was wrong, in one line a search will find: the unversioned math_libs and comm_libs entries pointed into the newer CUDA tree the same tarball carries. Then why the fix is where it is: postinstallcmds runs before the sanity check, so the check that had been failing can pass honestly.

2. Yes: anything built against the broken toolchain linked the wrong library and has to be rebuilt. Say which packages, and say that the rebuild flag is temporary and who removes it. A temporary flag with no named owner becomes permanent.

3. That the check had been switched off, that it was reporting this defect, and that the twenty days are the cost of that decision rather than of the defect. It goes to whoever will make the same call next month, which means the commit message and the team’s notes, not the ticket. The user gets a working module and a date, not an autopsy.

4. In the commit: the error string will search for; the architectures affected of the bug; how you measured it, in one line library resolution is only as good as the node it was made on. Not in the commit: the ticket reference on its own, which ages out, and the detail of the earlier retracted diagnosis, which belongs in the notes where somebody can read the whole sequence.

Task three: a user says a module is broken and the report cannot be reproduced

The request: “this module works for me and fails for them”, which is the most common ticket in this job and has the most incorrect answers. The sequence is a bisection over four differing factors: Which architecture. Resolve the per-node path on their node and on yours. Different trees mean different installs, and the module name reports nothing. Which module. Have them send module list rather than the name they think they loaded. A different generation on MODULEPATH is the second most common answer. Which environment. eb --show-config on your side, their shell’s environment on theirs. A site hook or a user’s own MODULEPATH is A hook changes a file you are not reading. Which node. A GPU library, a filesystem client, a fabric: all host-provided, none in the module. The same stack somewhere else’s last section is this list. Where a first-timer goes wrong: they answer the ticket after step 3. Three of the four checks pass, the module looks identical, and the temptation is to reply that it works. The fourth is where the difference tends to be, and a reply that closes a ticket names the node and the architecture it was tested on.

A pipeline’s summary can be empty for two different reasons

One more trap arises from a bump that succeeded on three architecture classes and failed on one, using the same prebuilt archive. A pipeline that summarises a run typically fills a field such as EasyBuild log: by grepping its own output for the line that eb prints at the end, Results of the build can be found in the log file(s). eb prints that line on failed builds as well. Thus an empty log field does not indicate “no information”; it indicates that eb never printed the line. The failure is before the easyconfig was processed, during the fetch phase. A field such as Last successful build: reporting never provides no signal for a newly added line. Examine the pipeline’s summariser to determine which fields can be empty for reasons other than absence.

referenceFour habits that shorten the week

1. State where a result came from. Every claim about a build carries a machine and an architecture, in the ticket and in the commit message. This book’s transcripts do it for the same reason.

2. Read the archive, not the diff. A version bump is a change of source.

3. Do not switch a check off to make a build pass. A failing check on a package that cannot be fixed is a report about something else. Testing, and the four things it can mean.

4. Write the diagnosis down where the next person will look. The install already carries the easyconfig, the log and reprod/; what it does not carry is why that choice was made.

EB-Week-1 — Your actual first ticket

A user writes: “Could you install version 6.5.1? We need the GPU build, and the version you have is too old for our workflow.”

Write the plan before you type anything. Specifically:

1. The first four things you find out, and from where.

2. Which of the three tasks above this is, and which parts of the other two it also contains.

3. The two failures you expect, and the check that would catch each one before a user does.

4. What the reply will carry, and what it will omit until you have measured it.

Solution

1. Whether upstream has an easyconfig for 6.5.1 at all (eb --search); which toolchain the GPU build needs and whether your site has it at this generation; what the site’s current version was built with, from the archived easyconfig in its install directory; and whether the licence covers this group, which is the question that blocks everything else and has nothing to do with EasyBuild.

2. Mostly task two, with task one inside it: a version change and a toolchain choice. If the toolchain is new to the site it is also a stack decision, and that is somebody else’s call as much as yours.

3. A source-shape failure, caught by listing the archive and re-reading every parameter that names a directory. And an architecture-specific runtime failure, caught only by running the binary on the target hardware; the build host passing tells you about the build host.

4. The reply says which version, which toolchain, which architectures it was built for, and on which architecture it was actually run. It does not say “it works” on the strength of a login-node check, and it does not promise a date before the toolchain question is settled.

What to remember

• A site’s pipeline, build lists and hooks are what stand between an easyconfig and a module; find all three before your first build.

• A version bump is a change of source: re-read every parameter that describes the shape of the archive.

• A dry run cannot see a fault that depends on the contents of an archive.

• On a heterogeneous system, measure library resolution on the architecture that will run the binary, not on the one underfoot.

• Every claim about a build carries the machine and the architecture it was measured on, or it is not a claim.