Testing, and the four things it can mean

What you will be able to do

1. Tell apart the four things called “testing” in an EasyBuild workflow, and say which one catches which failure.

2. Run a package’s own test suite from an easyconfig, and decide what to do when it fails.

3. Produce a test report, and say why upstream requires one from a human.

Before this chapter

• What “installed” means, for the sanity check.

• How a build executes, for where the test and testcases steps run.

“Did you test it” comprises four distinct questions, and answering the incorrect one leads to a broken module being shipped. They occur in the following order:

1. The package’s own test suite runs during the build. This is the test step.

2. EasyBuild’s sanity check runs after the install. Refer to What “installed” means.

3. An uploaded test report states that a build succeeded on a real machine.

4. A portable benchmark or correctness run is performed later on relevant hardware.

None of these steps subsumes another. A green test suite indicates that the code computes as intended. The sanity check indicates that the installation is usable. A test report indicates that a user’s machine successfully executed the build. A portable test indicates that the software still works on the node used by users.

The package’s own tests

Two parameters exist and they are not the same one. runtest drives the test step, which runs before install. On the base EasyBlock it is a whole command. Under ConfigureMake, which is what most easyconfigs use, it is appended to make, so runtest = 'test' means make test, and the composed command is pretestopts + test_cmd + runtest + testopts with test_cmd defaulting to make. That difference catches people: the same value means a command in one easyblock and an argument in another. tests is a different parameter and a different step. It is a list of scripts run in the testcases step, at the very end, after the module exists, and a script fails by returning non-zero. Three ways exist for a failing test step to proceed. --ignore-test-failure lets the build continue and records the failure. The framework wraps the test step precisely so this is possible: a failure is routed to report_test_failure rather than aborting. --skip-test-step does not run it at all. skipsteps = ['test'] in the easyconfig does the same thing permanently, for everyone, and that is the one to think twice about. Four things that actually happened’s third case is a skipsteps that was defensible and still wrong: the check was reporting a real defect somewhere else, and switching it off moved the discovery to a user. a skipped test needs a comment saying why. an ignored test needs somebody to have read the failure. Both are legitimate.

The sanity check is a test

Restating this is important. The sanity_check_paths assert that files and directories exist, and sanity_check_commands runs commands, which means the check executes the software. That is What “installed” means, and the only addition is the interaction with the options above: --skip-sanity-check exists, and under --skip the sanity check is deliberately not skipped. One misconception is that it leads people to disable the RPATH check on GPU packages. libcuda.so.1 lives with the driver and exists only on a node that has a GPU. EasyBuild knows that it exempts libcuda.so, libcuda.so.1, libnvidia-ml.so, and libnvidia-ml.so.1 by default. Anything else reported by the check is a real finding.

A test report

Upstream’s requirements for merging an easyconfig describe what CI can and cannot do. The pull request targets develop. CI is green. a successful test report has been submitted. A maintainer has approved it. The author does not merge their own. The third condition exists. Upstream CI parses every easyconfig, style‑checks it, verifies checksums are present, checks that an easyconfig exists for every dependency, and checks that no dependency graph has a version conflict. It does not build the contributed package. Therefore the evidence that a build works is provided by a human, and this is how it is produced.

eb --from-pr <PR#> --upload-test-report --force --robot

The --upload-test-report option posts the report as a GitHub gist, which requires a token with gist permission, and adds a summary comment to the pull request. The --dump-test-report option writes the report locally instead. Both options imply --keep-going, so a run that generates a report continues past the first failure. A report that covers nine of ten easyconfigs more useful than one that covers only one.

referenceWhat a test report contains

Every report carries:

• the easyconfigs processed, and the result for each;

• the time and date;

• the exact eb command line;

• the full EasyBuild configuration in effect;

• system information: hostname, operating system, architecture;

• the modules loaded at the time;

• the full environment of the session.

For each easyconfig that failed, a partial log is uploaded as its own gist and linked from the report.

That last item in the first list is why --test-report-env-filter exists: a regular expression whose matching variable names are excluded. The documented value to start from is

export EASYBUILD_TEST_REPORT_ENV_FILTER='^SSH|USER|HOSTNAME|UID|.*COOKIE.*'

A report is generated regardless of request, placed beside the installation log as <log basename>_test_report.md. Read a report before uploading the first one.

The contents (see What a test report contains) list the machine, the configuration, and the modules. That is why a test report is evidence: a reviewer can tell whether the success applies.

Testing on hardware the site does not have

The fourth kind. A build that works on the machine that built it tells nothing about the machine the users are on, which is Four things that actually happened’s login-node lesson in general form.

EESSI’s answer is a portable test suite, EESSI/test-suite, written with ReFrame. Tests are mapped to software in tests/eessi_test_mapping/software_to_tests.yml, so adding software declares which tests should now run rather than leaving somebody to remember. The build bot runs them through bot/test.sh on the architecture the build targeted.

The rule is small: a test has a location, and a result without its location is not a result. A green test on a login node and a green test on the GPU node a user submitted to are two different facts, and only one of them answers the ticket.

Part 2 of this book is that kind in full: chapters 39 to 47. A test class, the six-stage pipeline, the configuration file, a performance number, then the portable suite and a failed run. CSCS teaches it as a webinar separate from getting started; EESSI walks a job script, a site-specific test, then a mixin. The mapping file above is Software maps to tests.

EB-Testing-1 — Four reports, four responses

1. A package’s make test fails on two of 340 tests, in the floating-point comparison of an iterative solver.

2. The sanity check fails on lib/libthing.so, and the install directory contains lib64/libthing.so.

3. A colleague’s pull request has green CI and no test report. They ask you to merge it.

4. Your build passes everything on the build host. A user reports Illegal instruction on an older node.

For each: which of the four kinds of testing is involved, and what do you do?

Solution

1. The package’s own test suite. Read the two failures before deciding anything: an iterative solver disagreeing in the last digits across compilers is common and often acceptable, and a wrong answer is not. If they are tolerable, --ignore-test-failure for this build and a comment in the easyconfig saying which two and why. Not skipsteps, which would hide the other 338.

2. The sanity check, and it is correct. The library is in lib64, so either the check should say lib64 or the build should be told to use lib. Prefer the second where the build system supports it stack where half the packages use lib and half lib64 is a stack where every consumer easyconfig has to know which.

3. Neither of you can merge it: upstream requires a test report and forbids the author merging. What you can do is produce the report, with eb --from-pr <PR#> --upload-test-report --force --robot, which is the useful half of a review anyway. Then a maintainer merges.

4. The fourth kind, and no amount of the first three would have caught it. The binary was compiled for the build host’s microarchitecture, so this is What a toolchain injects’s optarch, and the fix is at the site level: build per architecture, or set a baseline every node can run. Testing on the build host was the mistake, not the build.

sourcesWhere this chapter comes from

Upstream documentation, fetched 2026-09-10:

• unit tests: the three suites, and the statement that they do not build the supported software

• contributing: the merge requirements, and the review process

• GitHub integration: --upload-test-report, its gist permission, and --test-report-env-filter

• the parameter reference: runtest against tests

• the option table: --ignore-test-failure, --skip-test-step, --keep-going

Code, from EasyBuild 5.3.1: EasyBlock._test_step and report_test_failure in easybuild/framework/easyblock.py for how a failure is routed; ConfigureMake.test_step for how runtest composes with make; and the RPATH check’s default exemptions for the driver libraries.

The portable-test material is EESSI’s: EESSI/test-suite and the mapping file it keys tests to software with.

What to remember

• Four different things are called testing: the package’s own suite, the sanity check, a test report, and a portable run on hardware that matters about. None of them subsumes another.

• runtest is a whole command on the base class and an argument to make under ConfigureMake.

• --ignore-test-failure records a failure, --skip-test-step avoids it, and skipsteps removes it for everyone.

• The RPATH check already exempts libcuda.so.1 and the other driver libraries, so anything else it reports is real.

• Upstream CI never builds an easyconfig, which is why the evidence is a human’s test report and why an author cannot merge their own.