Software maps to tests¶

Questions

What runs when a new module lands in EESSI?
How does CSCS lay out a site suite?
Which of EESSI’s three uses is a pull request, and which is a nightly?

What you will be able to do

Say what a software-to-tests mapping is for, and what a missing entry means the next morning.
Read the CSCS and EESSI directory layouts and say which directory is the machine, which is the tests, and which is CI.
Pick the filter for a contribution check, and the filter for a daily regression, from the same suite.

Before this chapter

Testing, and the four things it can mean, for the fourth kind.
A portable test leaves the machine in the config, for a test that can run on more than one site.

A portable test that is not listed for its software will not run when the software is merged. EESSI provides a mapping file as its answer. CSCS provides a directory of checks and a daily invocation as its answer. Both implement the same concept: the suite names are collected in one location. Which tests a package requires.

EESSI: three uses, one suite¶

The writing-tests page lists three reasons the suite exists. A contribution. New software is being added to EESSI. The mapping says which tests now apply. The bot runs them on the architecture the build targeted, through bot/test.sh. A nightly. The same tests, on a set of clusters, looking for a performance regression. An end user. One test or the whole suite, to see whether this machine plus this EESSI mount actually runs. Those are three filters over one tree, not three suites. The CI tag from Parameters multiply the suite is the contribution filter: Crambin, five minutes, eight cores. The nightly drops the tag and keeps the HECBioSim ladder. The end user passes --name GROMACS or does not. The mapping lives next to the tests. Testing, and the four things it can mean already named it: software_to_tests.yml. Adding GROMACS to the stack without a line in that file is how a green build ships a module that the suite will not touch. The generalisable rule is the one Testing, and the four things it can mean ended on: a test has a location, and a result without its location is not a result. The mapping is that location, written down. Available tests in the public suite include GROMACS (six HECBioSim systems, implemented on top of ReFrame’s hpctestlib), TensorFlow (the multi-worker Keras tutorial), CP2K, OpenFOAM, and the OSU microbenchmarks. Each is a class under eessi/testsuite/tests/apps/. The mapping is what connects a module name to those classes.

CSCS: config, checks, CI¶

The CSCS suite, eth-cscs/cscs-reframe-tests, provides the layout referenced by the official ReFrame README as the most complete public tree. Three directories each perform a distinct job.

cscs-reframe-tests/
├── config/          the machine
├── checks/          the tests
├── ci/              how the site invokes them
├── uenv_checks/     the same idea for uenv stacks
└── portable_tests.md

Listing all information the suite knows about on a CSCS system.

reframe -C cscs-reframe-tests/config/cscs.py \
        -c cscs-reframe-tests/checks/ -R -l

-C is The configuration file is the machine. -c is the check path. -R recurses. -l lists. The daily run uses the command with -r instead of -l, from ci/. A site copying this layout does not need EESSI. It requires a configuration file describing its partitions, a directory of tests that do not name those partitions, and a nightly job that runs the listing. portable_tests.md in that tree defines the written rule for what a check may assume. EESSI’s mixin implements the same rule, enforced in code.

LUMI, CSC, and why this is a part¶

CSC Finland’s computing‑environment course consists of ten topics about using Puhti and Mahti: modules, batch jobs, Allas, containers. It is not a ReFrame course. LUMI’s user docs cover the same ground for that machine. The ReFrame course in this world is CSCS’s webinar, the official tutorial, and EESSI’s writing‑tests page. That is why these chapters are Part 2 and not a subsection of Testing, and the four things it can mean: the fourth kind of testing has its own layout, and the centres that run it teach it as its own course.

EB-ReFrame-46-1 — Which filter

Three requests. For each, say which invocation, and whether the mapping file has to contain the software yet.

A pull request adding GROMACS/2024.4-foss-2023b to EESSI. The reviewer asks for the five-minute case.
The nightly on a CPU partition, every GROMACS HECBioSim system the suite knows.
A user on a laptop with the EESSI CernVM-FS client, asking whether mpi4py imports and reduces.

Solution

reframe --name GROMACS --tag CI -r, after the mapping names GROMACS. Without the mapping entry the bot has nothing to run, and the PR is a build without the fourth kind of test. The mapping has to contain it in the same change.
reframe --name GROMACS -r on that partition, no CI tag. The mapping already contains it; the nightly is not the time to add the line.
reframe --name EESSI_MPI4PY -r against a configuration that describes the laptop (local scheduler, the EESSI modules system). The mapping is irrelevant to a human who named the test. find_modules supplies the module that the client actually mounted.

What to remember

A mapping file is how a new module inherits tests. A missing line is a silent skip the next morning.
EESSI’s suite is one tree with three filters: contribution (CI), nightly, end user.
CSCS lays out config/, checks/, ci/. The machine, the tests, and the invocation stay apart.
CSC’s computing-environment course is not this course. ReFrame is taught as its own unit, which is why it is a part of this book.