Compile or run-only¶

Questions

When does a test compile, and when does it load a module?
What is valid_prog_environs = ['default'] claiming on a run-only test?
Where do the sources live relative to the test file?

What you will be able to do

Choose RegressionTest or RunOnlyRegressionTest for a given job, and say what the other choice would have done.
Point a compile test at its sources, and a run-only test at an installed module.
Say what valid_prog_environs means in each case.

Before this chapter

A ReFrame test is a class, for RunOnlyRegressionTest.
What “installed” means, for what a module being present actually claims.

Two kinds of work appear in an EasyBuild shop, represented by two base classes. A compile test builds a binary from source using the current programming environment and then runs it. That is rfm.RegressionTest. The compile stage is performed. valid_prog_environs lists the compilers with which the test is attempted, and a site defining gnu and intel receives two test cases from a single class. A run-only test executes an already existing binary. That is rfm.RunOnlyRegressionTest. The compile stage is omitted. valid_prog_environs = ['default'] is the typical value. That the environment is not the object under test. The object under test is a module specified in self.modules. EESSI’s suite consists almost entirely of the second kind. The software is pre‑built by EasyBuild, and the assessment concerns whether the module can run a known workload on this node. CSCS’s suite includes both system‑health tests that compile STREAM and OSU and application tests that load a module. There is a third class, CompileOnlyRegressionTest, whose run stage is omitted. It is the appropriate tool for “does this compiler build that file”, but not for any task described in this book involving an installed stack.

Sources next to the test¶

A compile test searches for sources in a directory adjacent to the test file, named src/ by default (sourcesdir). The CSCS webinar depicts the layout as:

tests/
├── src/
│   └── stream.c
└── stream1.py

ReFrame copies src/ into the stage directory and performs the build there. sourcepath, build_system and the Make, CMake, Autotools, and Spack backends constitute the How To; ExCALIBUR-tests exercises each of them. The default configuration performs a single-file compilation using the compiler from the current environment. A run-only test may also ship files adjacent to itself via readonly_files or by using the same src/. EESSI’s mpi4py test ships mpi4py_reduce.py in this manner; the script is included in the test while the interpreter and mpi4py are provided by the module.

Loading a module is the EasyBuild join¶

@rfm.simple_test
class GromacsRun(rfm.RunOnlyRegressionTest):
    valid_systems = ['*']
    valid_prog_environs = ['default']
    modules = ['GROMACS/2024.4-foss-2023b']
    executable = 'gmx'
    executable_opts = ['--version']

    @sanity_function
    def validate(self):
        return sn.assert_found(r'GROMACS version:', self.stdout)

The modules entry contains a list of module names. ReFrame generates a module load command in the job script, employing the modules system specified in the configuration. The sanity check corresponds to the same kind of assertion as EasyBuild’s sanity_check_commands: the binary launches and outputs its version. The check is neither GROMACS’s own test suite nor a HECBioSim benchmark. Those appear as later tests in Software maps to tests, mapped to the same software. The module name is a pin, as described in A pin is for one generation. GROMACS/2024.4-foss-2023b represents one installation. A test that hard‑codes this name will not encounter GROMACS/2024.4-foss-2024a. Parameters multiply the suite’s parameter and A portable test leaves the machine in the config’s find_modules illustrate how a suite avoids defining a separate class for each module.

What the generated job script actually contains¶

For a run‑only test, the script remains brief. The tutorial’s initial STREAM job consists of three lines.

#!/bin/bash
stream.x

Adding a module and a launcher expands the script by those two facts, which ReFrame obtains from the configuration and from self.modules, not from the test author writing Slurm directives. This is the productivity claim of Karakasis et al. [2020]: the job script is generated, and the test class does not mention #SBATCH. A compile test’s build script is generated in the same manner, from the build system and the current environment. The How To for Make and CMake is documented in the official docs; the only fact required here is that compilation occurs in the stage directory, and a failed compilation leaves the stage in place as a failed run does.

EB-ReFrame-43-1 — Which base class

Four jobs. For each, name the base class, and say what valid_prog_environs should be.

Build stream.c with every compiler the site defines, then run it.
module load GROMACS/2024.4-foss-2023b and run gmx mdrun on Crambin.
Confirm that mpicc from the foss environment can compile a hello-world, without running the binary.
Run python3 -c 'import mpi4py' from every mpi4py/ module the site has.

Solution

RegressionTest. valid_prog_environs lists those compilers, or ['*'] to take every environment the partition defines.
RunOnlyRegressionTest. ['default']: the environment is not under test, the module is.
CompileOnlyRegressionTest. ['foss'], or the name the configuration uses for that toolchain.
RunOnlyRegressionTest. ['default']. The modules come from a parameter (Parameters multiply the suite), not from the environment list.

What to remember

RegressionTest compiles, then runs. RunOnlyRegressionTest runs a binary that already exists. CompileOnlyRegressionTest compiles and stops.
An EasyBuild shop’s application tests are run-only: the module is the thing under test, and valid_prog_environs = ['default'].
Sources live in src/ next to the test file and are copied into the stage directory.
The job script is generated. The test class does not write #SBATCH.