A ReFrame test is a class

Questions

• What is the unit of work ReFrame runs?

• Who decides whether a test passed?

• What does valid_systems = ['*'] actually permit?

What you will be able to do

1. Read a run-only test and say what it will execute, and how it decides it passed.

2. Say why ReFrame does not look at the exit code.

3. Name the two constraints every test must declare.

Before this chapter

• Testing, and the four things it can mean, for the four kinds of testing. This part is the fourth kind, written out.

This file is a regression test. It contains twenty‑two lines. STREAM is McCalpin [1995]: Copy, Scale, Add, Triad, and a checksum line the binary prints when the arrays match. Both CSCS’s webinar and the official tutorial begin with it.

import reframe as rfm
import reframe.utility.sanity as sn
from reframe.core.builtins import sanity_function, performance_function

@rfm.simple_test
class stream_test(rfm.RunOnlyRegressionTest):
    valid_systems = ['*']
    valid_prog_environs = ['*']
    executable = 'stream.x'

    @sanity_function
    def validate(self):
        return sn.assert_found(r'Solution Validates', self.stdout)

    @performance_function('MB/s')
    def copy_bw(self):
        return sn.extractsingle(r'Copy:\s+(\S+)', self.stdout, 1, float)

    @performance_function('MB/s')
    def triad_bw(self):
        return sn.extractsingle(r'Triad:\s+(\S+)', self.stdout, 1, float)

The file reads as a configuration would: names on the left, values on the right. Line five breaks that reading. @rfm.simple_test is a decorator. The class is registered with the framework when the file is imported, and that registration is what reframe -c stream.py -l lists. Without the decorator the class is ordinary Python and ReFrame never sees it. RunOnlyRegressionTest is the other claim on that line. The compile stage is a no-op. The test runs an executable that already exists. The next two chapters say what that costs; a test that builds and a test that runs a module are two different base classes.

Two constraints, then a verdict

Every test must set valid_systems and valid_prog_environs. These are not documentation. The framework filters the test against the current system and the current programming environment, and a test that matches neither is not scheduled. ['*'] means any system, any environment: the tutorial’s starting point, and a claim that will have to be narrowed the moment the test leaves a laptop. A test must also define a validation function, decorated @sanity_function. ReFrame, by default, makes no assumption about whether a test succeeded. It does not look at the exit code. Karakasis et al. [2020] state the design goal as productivity: the writer of the test focuses on the logical requirements, and the framework handles the job script, the scheduler and the output files. The corollary is the one that surprises people coming from make test: a process that exits zero and prints nothing has failed. sn.assert_found looks in self.stdout, which is the name of the file where the job’s standard output was stored, not a live stream. The pattern is a regular expression. The STREAM binary prints Solution Validates when the checksum of the arrays matches; that string is the whole of this test’s correctness claim. The two @performance_function methods are optional. They extract figures of merit and give them a unit. A performance function is a number with a unit is those numbers. They do not decide whether the test passed; a missing Copy: line fails the performance phase, not the sanity phase.

What reframe -c actually loads

reframe -c stream/stream_runonly.py -l

The option -c specifies the check path, which may be a file or a directory. The option -l produces a listing. The option -r executes the check. The listing displays a single check, stream_test, using the builtin generic:default+builtin combination, since no configuration file is provided and the builtin definition always includes that pair. The hash following the name (/2e15a047 in the tutorial’s recording) is the test-case identity, comprising class, parameters, partition, and environment. This identifier enables rerunning a failed case without rerunning the entire suite.

EB-ReFrame-39-1 — What decides a pass

The STREAM binary is replaced by one that exits 0 and prints only Copy: 19000.0. The sanity function is unchanged. Does the test pass? Which phase fails, and why is the exit code not the answer?

Solution

It fails in the sanity phase. assert_found looks for Solution Validates in stdout and does not find it. The exit code is 0 and is never consulted. ReFrame’s default is that a test is unsuccessful until a sanity function says otherwise; an empty or partial log is a failed test, not a quiet success.

sourcesWhere this chapter comes from

The STREAM test is the opening example of the official ReFrame 4 tutorial (tutorial.html, fetched 2026-09-11) and of the CSCS webinar (Karakasis, 2022). The claim that ReFrame does not look at the exit code is from the pipeline documentation (pipeline.html): “ReFrame makes no assumption as of what a successful test is; it does not even look into its exit code.”

Karakasis et al. [2020] state the three design goals (productivity, portability, robustness) and the CSCS, NERSC and OSC deployments. McCalpin [1995] is STREAM.

What to remember

• A ReFrame test is a decorated Python class. Without @rfm.simple_test the class is never registered.

• RunOnlyRegressionTest skips compile. The executable is assumed to exist.

• valid_systems and valid_prog_environs are filters, not comments. ['*'] means any.

• ReFrame does not look at the exit code. A sanity function is the verdict, and a missing pattern is a failure.