Software maps to tests

an installed module maps through YAML to a suite

Software maps to tests. The map is a file, not a convention in the class name.

module through YAML to a suite

Software maps to tests through a file.

a new module hits the map; PR and nightly are two uses

The map says what runs when a module is written. A missing entry is a silent next morning.

Questions

  • What runs when a new module is written in EESSI?

  • How does CSCS lay out a site suite?

  • Which of EESSI’s three uses is a pull request, and which is a nightly?

What you will be able to do

  1. Say what a software-to-tests mapping is for, and what a missing entry means the next morning.

  2. Read the CSCS and EESSI directory layouts and say which directory is the machine, which is the tests, and which is CI.

  3. Pick the filter for a contribution check, and the filter for a daily regression, from the same suite.

Before this page

Portable tests not listed for their software do not run when the software is merged. EESSI provides a mapping file as its answer. CSCS provides a directory of checks and a daily invocation as its answer. Both implement the same concept. The suite names are collected in one location. The collection determines which tests a package requires.

EESSI: three uses, one suite

The writing-tests page lists three reasons the suite exists. One reason is a contribution. New software is being added to EESSI. The mapping says which tests now apply. The bot runs them on the architecture the build targeted, through bot/test.sh. Another reason is a nightly. The same tests run on a set of clusters, looking for a performance regression. A third reason is an end user. The end user runs one test or the whole suite to see whether this machine plus this EESSI mount actually runs. Those are three filters over one tree, not three suites. The CI tag from Parameters multiplies the suite, acting as the contribution filter: Crambin, five minutes, eight cores. The nightly drops the tag and keeps the HECBioSim ladder. The end user passes --name GROMACS or does not. The mapping lives in EESSI/software-layer-scripts, not next to the test classes. Testing and the four things it can mean already named it: software_to_tests.yml. Adding GROMACS to the stack without a line in that file gives a green build a module the suite will not touch. The generalisable rule is that a test has a location, and a result without its location is not a result. The mapping is that location, written down. Available tests in the public suite include GROMACS (six HECBioSim systems, implemented on top of ReFrame’s hpctestlib), TensorFlow (the multi‑worker Keras tutorial), CP2K, OpenFOAM and the OSU microbenchmarks. Each class lives in EESSI/test-suite under eessi/testsuite/tests/apps/. The mapping connects a module name to those classes.

CSCS: config, checks, CI

The CSCS suite eth-cscs/cscs-reframe-tests provides the layout cited in the official ReFrame README, which is the most complete public tree. The three directories that run a job are config/, checks/, and ci/. The tree also carries containers/, deprecated_checks/, hpctestlib/, reframe_reporter/, uenv_checks/, and utility/.

cscs-reframe-tests/
├── config/ the machine
├── checks/ the tests
└── ci/ nightly --mode daily_production

List all information the suite knows about on a CSCS system.

reframe -C cscs-reframe-tests/config/cscs.py \
 -c cscs-reframe-tests/checks/ -R -l

-C denotes the configuration file that is the machine. -c denotes the check path. -R causes recursion. -l causes listing. The listing command is the inventory. The nightly job in ci/ runs --mode daily_production. A site copying this layout does not need EESSI. The site must have a configuration file describing its partitions, a directory of tests that do not name those partitions, and a nightly job that runs that pair. portable_tests.md in that tree defines the written rule for what a check may take for granted. EESSI’s mixin implements the same rule, enforced in code.

LUMI, CSC and why this is a part

The CSC Finland computing‑environment course contains ten topics on using Puhti and Mahti, including modules, batch jobs, Allas and containers. The course is not a ReFrame course. LUMI’s user documentation covers the same material for that machine. The ReFrame course consists of CSCS’s webinar, the official tutorial, and EESSI’s writing‑tests page. These pages are ReframeReader rather than a subsection of Testing. The fourth kind of testing has its own layout. The centres that run it teach it as an independent course.

EB-ReFrame-46-1 — Which filter

Three requests. For each, say which invocation, and whether the mapping file has to contain the software yet.

  1. A pull request adding GROMACS/2024.4-foss-2023b to EESSI. The reviewer asks for the five-minute case.

  2. The nightly on a CPU partition, every GROMACS HECBioSim system the suite knows.

  3. A user on a laptop with the EESSI CernVM-FS client, asking whether mpi4py imports and reduces.

Solution

  1. reframe --name GROMACS --tag CI -r, after the mapping names GROMACS. Without the mapping entry the bot has nothing to run, and the PR is a build without the fourth kind of test. The mapping has to contain it in the same change.

  2. reframe --name GROMACS -r on that partition, no CI tag. The mapping already contains it; the nightly is not the time to add the line.

  3. reframe --name EESSI_MPI4PY -r against a configuration that describes the laptop (local scheduler, the EESSI modules system). The mapping is irrelevant to a human who named the test. find_modules supplies the module that the client actually mounted.

What to remember

  • A mapping file is how a new module inherits tests. A missing line is a silent skip the next morning.

  • EESSI’s suite is one tree with three filters: contribution (CI), nightly, end user.

  • CSCS lays out the config tree, the checks tree, the ci tree. The machine, the tests, and the invocation stay apart.

  • CSC’s computing-environment course is not this course. ReFrame is taught as its own unit, which is why it is a part of this book.