What to remember¶
What to remember is produced from the pages. The lines are claims, not slogans.¶
What to remember is a claim plus the page that would falsify it.¶
What to remember is a claim with a page behind it.¶
The digest for each page is produced in reading order from the pages. Wilson [2016] is the Carpentries design this page follows: a claim plus the page that would falsify it. It has two purposes. Before a page, it indicates whether the page’s purpose is already known. After finishing the book, it provides a page to reread instead of the entire book.
Tutorial: read an easyconfig the way EasyBuild does
The file is Python. EasyBuild evaluates it in a prepared namespace.
A missing version stops the parse. A missing sources line is legal and useless.
A missing easyblock line is still an easyblock: the name is transformed.
A toolchain is the top of a hierarchy. That hierarchy is the set of legal dependencies.
An easyconfig is a Python file evaluated in a namespace EasyBuild prepared.
Unquoted names are variables EasyBuild supplied, not free identifiers.
If no software-specific easyblock exists, the framework raises MISSING_EASYBLOCK.
A toolchain is a list with an order, not a label.
get_toolchain_hierarchy names the legal dependency toolchains.
_verify_toolchain runs in prepare, before any unpack.
A version beside a dependency is a pin for one generation, not a minimum.
Spack @1.14.6 is the closed range @1.14.6:1.14.6. Or-newer is @1.14.6:.
The same number on another generation is a different module.
A filename is a query against the robot path.
EasyBuild 5: -D does not imply -r. The plan is -Dr.
resolve_dependencies discovers the graph while it walks; it does not sort a finished graph.
A versionsuffix is part of the name
A versionsuffix is part of the module name, not a binding mechanism.
The dependency does the binding. The suffix reports it once at parse time.
The RPATH check already exempts the NVIDIA driver libraries.
A completed build prints a line and writes a modulefile. Neither proves the software runs.
sanity_check_paths is a count of files, not a test of behaviour.
The gap between those two claims is where user-facing bugs live.
A checksum is a claim about bytes
A checksum names bytes, not a version and not an architecture.
A dict maps resolved filenames to checksums. The keys are names, not amd64.
A dict with two filename keys is two claims: each archive is checked against its own digest.
The file is not the install. The same easyconfig is written into as many prefixes as there are (toolchain, arch) pairs.
exts_list is more installs inside one file, not one install with extras.
A generation is the set of those pairs that move together.
A hook changes a file you are not reading
A hook runs in the site's EasyBuild process, outside the easyconfig being read.
The robot path and hooks are the two ways a site changes behaviour without editing upstream files.
A hook that is not in the log is still what ran.
Build your own, without asking anyone
Three lists decide what EasyBuild will find: robot path, source path, install path.
A local prefix does not need a site admin. It does need those three lists set once.
--dry-run first. Then build.
EESSI is a stack to mount rather than to install.
List /cvmfs/software.eessi.io/versions before descending: an empty placeholder can wedge the client.
init/lmod/bash configures Lmod. init/bash is the irreversible client init.
eb-stack ports a recipe and bumps one
A port reads conda-forge or Spack and writes an easyconfig. package plan --format conda-forge or --format spack.
A bump reads an easyconfig and writes the same software on a new toolchain generation. Application version, sources and patches stay; every pin that lives on the toolchain moves.
A Spack eigen@3.4: is a lower bound. The robot either finds a pin this generation ships or the range stays a residual.
Nothing in inspect, plan or bump compiles the package. campaign run calls eb.
The step in the error message rules out every step before it.
Toolchain mismatch is prepare, via _verify_toolchain, not ready.
fetch reads templated sources. The name that failed may not be the text in the file.
EasyBuild does not build in the shell's environment.
detect_loaded_modules defaults to warn. ModulesTool.load is called with purge False.
module purge is not the isolation mechanism, and it is documented as broken on Cray.
Build dependencies are not dependencies
builddependencies are not in the module. dependencies are.
Python can appear in both lists. The runtime one wins after the build is gone.
Confirm the claim by reading the install rather than the easyconfig.
A GCCcore easyconfig that compiles code lists binutils in builddependencies; a host with a new enough as hides the omission.
Loading a toolchain module does not set $CC or $LIBBLAS.
Those variables exist only inside a build, from the toolchain class and toolchainopts.
Debugging by loading foss and re-running the failing command misses that injection.
Patches, and what a patch is relative to
A patch is a claim about a file at a path inside the unpacked archive.
The search is for a name, not for a hunk. A moved file is a missed patch.
The level is how many path components to strip, not a quality score.
Languages that bring their own package manager
A language ecosystem is a second package manager living inside an easyconfig.
exts_list is how that manager's packages become EasyBuild extensions.
An ecosystem with no easyblock is a residual, not a guess.
The modulefile is what a user loads. The easyconfig is how it was built.
modextravars and modluafooter write into that modulefile.
depends_on names another module, not a version range.
The exit code narrows the search before any log does: 127 is not found, 126 is not executable, 2 from make means a rule failed and the cause is higher up.
The step in the error message rules out every step before it.
find_base_dir descends while a directory has exactly one entry, so the start directory can differ from the archive's top.
A failed build keeps its build directory, and config.log or CMakeError.log in it says what the build system actually tried.
cmd.sh reproduces the failing command in the environment it failed in; eb -x cannot see any fault that depends on the contents of an archive.
Five parameters are enforced: name, version, homepage, description, toolchain. The MANDATORY category lists eight.
Search before writing; a copy of an existing easyconfig with a new version is almost always the cheaper correct answer.
--inject-checksums downloads, backs the file up, reorders parameters and does not overwrite an existing checksum without --force.
A local variable in an easyconfig has to be named local_*.
--check-style and --check-contrib are what upstream CI runs; neither runs the software, which is why a test report from a human is required.
Reach for configopts first, a generic easyblock second, and your own code only when more than one easyconfig will need it.
configure_step, build_step and install_step are the three the abstract class does not implement.
extra_options is how a parameter is born; setvar is how an environment variable is set so the dry run can see it.
--include-easyblocks shadows the shipped easyblock, and --list-easyblocks=detailed is how to confirm which file loaded.
Under --module-only fetch, extract, patch, configure, build, test and install do not run, so state carried between steps arrives as None.
Testing, and the four things it can mean
Four different things are called testing: the package's own suite, the sanity check, a test report, and a portable run on hardware that matters about. None of them subsumes another.
runtest is a whole command on the base class and an argument to make under ConfigureMake.
--ignore-test-failure records a failure, --skip-test-step avoids it, and skipsteps removes it for everyone.
The RPATH check already exempts libcuda.so.1 and the other driver libraries, so anything else it reports is real.
Upstream CI never builds an easyconfig, which is why the evidence is a human's test report and why an author cannot merge their own.
Standing where the build stood
Every install already carries the easyconfig, the log, a test report, a devel module and a reprod/ directory holding the easyblock source.
module load $EBDEVEL<NAME> reproduces the environment that build had, including $PATH and the modules tool's own bookkeeping.
eb --dump-env-script writes the toolchain's environment for a recipe before any build, and does not overwrite.
BuildEnv turns a toolchain's environment into a module other people can load, and copies in the RPATH wrappers.
What reprod/ does not carry: the framework version, the sources, the dependency installations, and the operating system.
Reading a stack off the wire: CernVM-FS
A repository is verifiable from one signature the root catalog's hash and every catalog names its children's.
Objects are named by the hash of their compressed bytes, so any client can check what it was given without trusting the transport.
A catalog is a SQLite database keyed by the MD5 of the absolute path, cut into nested catalogs that are fetched on first access.
Two keys: the repository key signs each revision, the master key signs the whitelist, and only the master public key has to reach clients out of band.
A cache miss is a sequence, and the client trusts its cache once the object is in it.
Three routes onto a machine: a native client with root, cvmfsexec or a container without it, and the same two on an ephemeral runner where the cache is the whole of the install.
The compatibility layer is why the host distribution stops mattering.
eessi_archdetect.sh -a cpupath returns an ordered list and the first target that exists on disk wins, so read $EESSI_SOFTWARE_SUBDIR rather than assuming.
Overrides are not validated: a target that does not exist gives an empty module avail.
A site adds software and drivers through a variant symlink, without writing to /cvmfs and without invalidating a signature.
Four things that actually happened
Every one of the four was green somewhere, so the question is never "did it pass" but what it passed and where.
A symlink such as /sw/arch resolves against the node underfoot, which makes an ldd on a login node evidence about the login node.
A failing check on a package that cannot be fixed is a report about something else; deleting the report does not fix the something else.
A parameter whose meaning depends on the shape of the source has to be re-read whenever the source changes.
Your first week on somebody’s stack
A site's pipeline, build lists and hooks are what stand between an easyconfig and a module; find all three before your first build.
A version bump is a change of source: re-read every parameter that describes the shape of the archive.
A dry run cannot see a fault that depends on the contents of an archive.
On a heterogeneous system, measure library resolution on the architecture that will run the binary, not on the one underfoot.
Every claim about a build carries the machine and the architecture it was measured on, or it is not a claim.
Extensions, and what an entry in exts_list inherits
exts_default_options reaches only entries that carry a version; a bare-string entry gets no options at all.
The merge is a shallow update, so a per-extension list replaces the default list instead of extending it.
An unknown per-extension option is dropped at debug level, with no warning: a typo in a bundle of two hundred extensions is silent.
Eight parameters plus start_dir are reset before an extension runs.
exts_filter is one mechanism doing two jobs, skipping what is already installed and checking what was just installed, and modulename: False switches off both.
An easystack is the list the bot will build
An easystack is a YAML list of easyconfig filenames. Per-item options: become eb flags with the dashes stripped, and last-wins means those flags beat the command line.
EESSI's path is easystacks/software.eessi.io/<ver>/eessi-<ver>-eb-<ebver>-<tc>.yml. The EasyBuild version in the name is the module the bot loads.
from-commit (or from-pr) is how an easyconfig that is not yet the default robot path is built. After a merge the pin can drop.
A rebuild lives under rebuilds/ and is the only case that passes --rebuild instead of --robot.
The bot is three processes and a comment
A bot instance is smee plus an event handler plus a job manager. A comment matching ^bot: is what the handler acts on. Opening a pull request is not.
for: is the installation prefix. on: is the allocated node. optarch stays native. CPU-to-CPU cross-compilation is not implemented.
software-layer/bot/build.sh pins a software-layer-scripts commit and execs that pinned bot/build.sh. The job's .diff is what selects the easystack files.
A green job produces a tarball. Signing is a detached .sig at deploy, and it is optional. bot:deploy uploads it. A Stratum-0 ingest is what makes a module. Three human permissions stand on that path.
A site stack next to EESSI, not inside it
EESSI_SITE_SOFTWARE_PREFIX is set before the EESSI module loads. EESSI-extend then writes an architecture-specific prefix and puts it on MODULEPATH.
Approach 1 is that prefix on a shared filesystem. Approach 2 is the same bot and the same scripts, aimed at a site CernVM-FS repository. The webinar recommends the owned prefix over stuffing everything into the host_injections symlink.
2025.06 aims NVIDIA driver symlinks with EESSI_NVIDIA_OVERRIDE_DEFAULT. 2023.06 used EESSI_HOST_INJECTIONS. They are not interchangeable.
A site MPI that is ABI-compatible with EESSI's OpenMPI goes in rpath_overrides/OpenMPI/system/lib. The modules above MPI do not get rebuilt.
Inspect writes a work queue, not a recipe
package inspect writes package.plan.json and a planned CycloneDX SBOM. It establishes no claim.
Residuals are the work queue. Notes are not. Severity is mechanical, judgment, or blocking.
package plan requires --stack-policy. The lock and the emitted .eb are what establish resolves.
Format, lint and check are three different questions
recipe format rewrites physical lines and nothing else.
recipe lint reports the same surface and does not write.
recipe check is the robot and metadata gate. A missing dependency stays a missing dependency.
A stack lock is not a package lock
stack solve is the multi-root annual-bump solver. Its policy is JSON. The package command's --stack-policy is TOML. They are not interchangeable.
build.list is the install order. stack.diff.md is the pull request. stack.lock.json is the Resolvo assignment.
A lock-only SBOM has no dependency edges. Edges are written only when --sbom-out is passed to stack solve.
A campaign finding is a typed failure, not a log
\*.campaign.json stores three independent claims. A completed run with no verify command is builds and not binary_verified.
A finding is a class, a disposition, a stage, and the exact routed command. The class names the first repair surface.
Claim is exclusive. Resolve records the action. Retry uses the same state path. The audit trail is not deleted.
A wrapper is the range EasyBuild will not write
An easyconfig still cannot write a range. A ModuleRC .modulerc is how a site answers an old name with a newer module.
check_version default True requires the requested version to be a prefix of the wrapped one.
In EasyBuild 5, -D does not imply -r. The plan is -Dr.
Four EasyBuild 5 surfaces the earlier pages skipped
An install lock is a mutex on a shared prefix. A Resolvo lock is an assignment of versions. They share a word and nothing else.
--bwrap installs in a namespace and then moves. /tmp noexec breaks the RPATH wrappers.
EBPYTHONPREFIXES makes a second Python prefix visible. Entry points are a 5.3 addition; an import test does not cover them.
A Dataset module is data. EASYBUILD_INSTALLPATH_DATA keeps it off the software prefix. Software depends on it like any other module.
A ReFrame test is a decorated Python class. Without @rfm.simple_test the class is never registered.
RunOnlyRegressionTest skips compile. The executable is assumed to exist.
valid_systems and valid_prog_environs are filters, not comments. ['*'] means any.
ReFrame does not look at the exit code. A sanity function is the verdict, and a missing pattern is a failure.
Six stages, in order: setup, compile, run, sanity, performance, cleanup. A run-only test no-ops compile.
Construction is not a stage. __init__ runs for tests that will be skipped.
Sanity and performance run on the ReFrame host, against files the job wrote. The stage directory has to be shared with the nodes.
run is submit; run_complete is wait. A queued job fails in the second.
The configuration file is the machine
The test describes what to run. The configuration describes where. A core count written in the test is a core count on one machine.
The builtin configuration always defines generic:default and builtin. A site file adds to it.
A partition is a scheduler, a launcher and a list of environments. valid_systems matches system:partition.
RFM_CONFIG_FILES, RFM_CHECK_SEARCH_PATH and $RFM_PREFIX are the three variables a site sets once.
A performance function is a number with a unit
A performance function returns a number and declares a unit. It does not decide correctness; the sanity function does.
A reference is (value, lower, upper, unit) per system:partition. A miss fails the performance stage only.
No reference still logs the number. That log is how a baseline is built.
Extract the application's own figure of merit. ReFrame's run timer includes the queue.
RegressionTest compiles, then runs. RunOnlyRegressionTest runs a binary that already exists. CompileOnlyRegressionTest compiles and stops.
An EasyBuild shop's application tests are run-only: the module is the thing under test, and valid_prog_environs = ['default'].
Sources live in src/ next to the test file and are copied into the stage directory.
The job script is generated. The test class does not write #SBATCH.
parameter multiplies the suite. variable does not; it is an overridable default.
The product is parameters x partitions x environments. Four small lists become a suite nobody will wait for.
Hooks after init see one case. Module names and task counts are set there.
--tag CI filters a run. It does not delete the other cases from the class.
A portable test leaves the machine in the config
A cluster name, a core count and a module string in the test are three facts about one machine.
The mixin reads the partition. "One full node" is a request; the configuration supplies the number.
find_modules builds the parameter list from modules available on the system, then the suite loads each case.
A missing feature or too little memory is a skip, not a failed job.
A mapping file is how a new module inherits tests. A missing line is a silent skip the next morning.
EESSI's suite is one tree with three filters: contribution (CI), nightly, end user.
CSCS lays out the config tree, the checks tree, the ci tree. The machine, the tests, and the invocation stay apart.
CSC's computing-environment course is not this course. ReFrame is taught as its own unit, which is why it is a part of this book.
Failing phase is the first field. The stage directory is the second.
rfm_job.sh is what ran. rfm_job.out is what it printed. On failure both stay in the stage; on success they move to output/.
--restore-session --failed reruns the failures and restores passed dependencies.
A skip is not a pass. A missing listing is a filter, and the debug log is the file that recorded it.
SeisSol, from a Spack recipe to a PR
The next file is a bump of the old easyconfig. Inspect Spack for residuals. Do not emit the EasyBuild file from Spack.
A parent bump that still needs companions exits 1 and prints companion=. The overlay is done when the parent exits 0.
PSpaMM is a package plan on gfbf. easi drops ImpalaJIT in package-config, not in a hand edit.
The generation is the one the site module environment already built. Newest-on-develop is a different consume.
The site wrapper is the install. Raw eb with hand-set prefixes is a different consume.
A checksum names one archive. A PythonPackage import names one module. Neither is the EasyBuild software name.