Testing, and the four things it can mean

four distinct questions: suite, sanity, report, ReFrame

Four things are called testing. None of them substitutes for another.

four kinds of testing, not one pipeline

Testing names four questions. A green suite is not a sanity check, a report, or a ReFrame run.

smoke, sanity, unit, and ReFrame are four different objects

Testing names four objects. Only ReFrame is a class that runs on a partition.

What you will be able to do

  1. Tell apart the four things called “testing” in an EasyBuild workflow, and say which one catches which failure.

  2. Run a package’s own test suite from an easyconfig, and decide what to do when it fails.

  3. Produce a test report, and say why upstream requires one from a human.

Before this page

The test comprises four distinct questions. Answering the incorrect question puts a broken module in front of users. They occur in the following order:

  1. The package’s own test suite runs during the build.

That is the test step.

  1. EasyBuild’s sanity check runs after the install.

Refer to What “installed” means.

  1. An uploaded test report states that a build succeeded on a real machine.

  2. A portable benchmark or correctness run is performed later on relevant hardware.

None of these steps subsumes another. A green test suite indicates that the code computes as intended. The sanity check indicates that the installation is usable. A test report indicates that a user’s machine successfully executed the build. A portable test indicates that the software still works on the node used by users.

The package’s own tests

Two parameters exist and they are not the same. runtest drives the test step, which runs before install. On the base EasyBlock it is a whole command. Most easyconfigs use ConfigureMake, where it is appended to make, so runtest = 'test' means make test. The composed command is pretestopts + test_cmd + runtest + testopts, with test_cmd defaulting to make. That difference catches people: the same value means a command in one easyblock and an argument in another. tests is a different parameter and a different step. It is a list of scripts run in the testcases step, last of all, after the module exists. A script fails by returning non-zero. Three ways exist for a failing test step to proceed. --ignore-test-failure lets the build continue and records the failure. The framework wraps the test step precisely so this is possible: a failure is routed to report_test_failure rather than aborting. --skip-test-step does not run it at all. skipsteps = ['test'] in the easyconfig does the same thing permanently, for everyone, and that is the one to think twice about. Four things that actually happened’s third case is a skipsteps that was defensible and still wrong: the check was reporting a real defect somewhere else, and switching it off moved the discovery to a user. a skipped test needs a comment saying why. an ignored test needs somebody to have read the failure. Both are legitimate. Where the flag lives matters as much as which flag it is. --ignore-test-failure belongs in the build list, because it states one site’s tolerance for one package and says nothing about the software. That placement has a consequence worth naming. A production list and a test pipeline can carry different flags for the same easyconfig, and nothing reports the difference. The pipeline running without the flag tests something stricter than what the site installs, and its failure reads as a defect in the recipe. It is not one. A GROMACS build with PLUMED showed the shape. The CPU architectures passed every test. One GPU node failed the two-rank MPI tests that the production list had tolerated for that package for years. Four rebuilds went into making those tests pass, on the belief that the flag could not be carried where it was needed. It could. Check that a constraint exists before building around it. A rebuild is an expensive way to test a belief about a configuration file.

The sanity check is a test

Restating this is important. The sanity_check_paths assert that files and directories exist. The sanity_check_commands runs commands. The check executes the software. That is What “installed” means. The only addition is interaction with the options above. The --skip-sanity-check exists. Under --skip the sanity check is deliberately not skipped. One misconception is that it leads people to disable the RPATH check on GPU packages. libcuda.so.1 lives with the driver. libcuda.so.1 exists only on a node that has a GPU. EasyBuild knows that it exempts libcuda.so, libcuda.so.1, libnvidia-ml.so, and libnvidia-ml.so.1 by default. Anything else reported by the check is a real finding.

A test report

Upstream requirements for merging an easyconfig list CI capabilities and limits. The pull request targets develop. CI status is green. a successful test report has been submitted. A maintainer approved the pull request. The author did not merge their own change. The third condition is present. Upstream CI parses each easyconfig. Upstream CI style‑checks each easyconfig. Upstream CI verifies that checksums are present. Upstream CI checks for an easyconfig for every dependency. Upstream CI checks that no dependency graph contains a version conflict. The contributed package is not built. Human verification provides evidence that a build works. The process produces the evidence.

eb --from-pr <PR#> --upload-test-report --force --robot

--upload-test-report posts the report as a GitHub gist. --upload-test-report requires a token with gist permission. --upload-test-report adds a summary comment to the pull request.

--dump-test-report writes the report locally.

Both options imply --keep-going. A run that generates a report continues past the first failure. A report covering nine of ten easyconfigs is more useful than one covering only one.

The contents (see ) enumerate the machine, the configuration, and the modules. A test report is evidence. A reviewer can tell whether the success applies.

Testing on hardware the site does not have

The fourth kind. A build that works on the machine that built it tells nothing about the machine the users are on. That is Four things that actually happened’s login-node lesson in general form. EESSI’s answer is a portable test suite, EESSI/test-suite, written with ReFrame. Tests are mapped to software in tests/eessi_test_mapping/software_to_tests.yml. Adding software declares which tests should now run rather than leaving somebody to remember. The build bot runs them through bot/test.sh on the architecture the build targeted. The rule is small: a test has a location, and a result without its location is not a result. A green test on a login node and a green test on the GPU node a user submitted to are two different facts, and only one of them answers the ticket. ReframeReader is that kind in full: A ReFrame test is a class through Reading a failed ReFrame run. A test class, the six-stage pipeline, the configuration file, a performance number, then the portable suite and a failed run. CSCS teaches it as a webinar separate from getting started; EESSI walks a job script, a site-specific test, then a mixin. The mapping file above is Software maps to tests.

EB-Testing-1 — Four reports, four responses

  1. A package’s make test fails on two of 340 tests, in the floating-point comparison of an iterative solver.

  2. The sanity check fails on lib/libthing.so, and the install directory contains lib64/libthing.so.

  3. A colleague’s pull request has green CI and no test report. They ask you to merge it.

  4. Your build passes everything on the build host. A user reports Illegal instruction on an older node.

For each: which of the four kinds of testing is involved, and what do you do?

Solution

  1. The package’s own test suite. Read the two failures before deciding anything: an iterative solver disagreeing in the last digits across compilers is common and often acceptable, and a wrong answer is not. If they are tolerable, --ignore-test-failure for this build and a comment in the easyconfig saying which two and why. Not skipsteps, which would hide the other 338.

  2. The sanity check, and it is correct. The library is in lib64, so either the check should say lib64 or the build should be told to use lib. Prefer the second where the build system supports it stack where half the packages use lib and half lib64 is a stack where every consumer easyconfig has to know which.

  3. Neither of you can merge it: upstream requires a test report and forbids the author merging. What you can do is produce the report, with eb --from-pr <PR#> --upload-test-report --force --robot, which is the useful half of a review anyway. Then a maintainer merges.

  4. The fourth kind, and no amount of the first three would have caught it. The binary was compiled for the build host’s microarchitecture, so this is What a toolchain injects’s optarch, and the fix is at the site level: build per architecture, or set a baseline every node can run. Testing on the build host was the mistake, not the build.

What to remember

  • Four different things are called testing: the package’s own suite, the sanity check, a test report, and a portable run on hardware that matters about. None of them subsumes another.

  • runtest is a whole command on the base class and an argument to make under ConfigureMake.

  • --ignore-test-failure records a failure, --skip-test-step avoids it, and skipsteps removes it for everyone.

  • The RPATH check already exempts libcuda.so.1 and the other driver libraries, so anything else it reports is real.

  • Upstream CI never builds an easyconfig, which is why the evidence is a human’s test report and why an author cannot merge their own.