A ReFrame test is a class

A ReFrame test is a class. The machine is not a field on that class. It lives in the config.

A ReFrame test is a class. The machine is not a field on it.

A ReFrame test is a class. Two constraints decide where it may run. Sanity, not the exit code, decides pass.

Questions

• What is the unit of work ReFrame runs?

• Who decides whether a test passed?

• What does valid_systems = ['*'] actually permit?

What you will be able to do

1. Read a run-only test and say what it will execute, and how it decides it passed.

2. Say why ReFrame does not look at the exit code.

3. Name the two constraints every test must declare.

Before this page

• Testing, and the four things it can mean, for the four kinds of testing. This part is the fourth kind, written out.

The file is a regression test. It contains twenty‑two lines. STREAM is McCalpin [1995]: Copy, Scale, Add, Triad. The binary prints a checksum line when the arrays match. Both CSCS’s webinar and the official tutorial begin with it.

import reframe as rfm
import reframe.utility.sanity as sn
from reframe.core.builtins import sanity_function, performance_function

@rfm.simple_test
class stream_test(rfm.RunOnlyRegressionTest):
    valid_systems = ['*']
    valid_prog_environs = ['*']
    executable = 'stream.x'

    @sanity_function
    def validate(self):
        return sn.assert_found(r'Solution Validates', self.stdout)

    @performance_function('MB/s')
    def copy_bw(self):
        return sn.extractsingle(r'Copy:\s+(\S+)', self.stdout, 1, float)

    @performance_function('MB/s')
    def triad_bw(self):
        return sn.extractsingle(r'Triad:\s+(\S+)', self.stdout, 1, float)

The file reads like a configuration: names on the left, values on the right. Line five breaks that reading. @rfm.simple_test is a decorator. The class registers with the framework when the file is imported. That registration appears in the output of reframe -c stream.py -l. Without the decorator the class remains ordinary Python. ReFrame does not see the class. RunOnlyRegressionTest is the other claim on that line. The compile stage performs no operation. The test runs an existing executable. The next two pages describe the associated costs. A test that builds and a test that runs a module use different base classes.

Two constraints, then a verdict

Each test must define valid_systems and valid_prog_environs. These entries are not documentation. The framework filters tests using the current system and the current programming environment. A test that matches neither is not scheduled. ['*'] denotes any system and any environment. The tutorial’s starting point is ['*']. A claim using this placeholder must be narrowed when the test leaves a laptop. Each test must also define a validation function decorated @sanity_function. ReFrame makes no default assumption about test success. ReFrame does not treat a zero process exit as a pass. The exit code is not examined. Karakasis et al. [2020] state the design goal as productivity. The test writer focuses on logical requirements. The framework handles the job script, the scheduler, and the output files. The corollary surprises users of make test. a process that exits zero and prints nothing has failed. sn.assert_found looks in self.stdout. self.stdout is the file of the job’s standard output, read after the job ends. The pattern is a regular expression. The STREAM binary prints Solution Validates when the array checksum matches. That string is the entire correctness claim of this test. The two @performance_function methods are optional. They extract figures of merit and give them a unit. A performance function is a numeric value with a unit. They do not determine test passed status. A missing Copy: line fails the performance phase. The sanity phase remains unaffected.

What reframe -c actually loads

reframe -c stream/stream_runonly.py -l

EB-ReFrame-39-1 — What decides a pass

The STREAM binary is replaced by one that exits 0 and prints only Copy: 19000.0. The sanity function is unchanged. Does the test pass? Which phase fails, and why is the exit code not the answer?

Solution

It fails in the sanity phase. assert_found looks for Solution Validates in stdout and does not find it. The exit code is 0 and is never consulted. ReFrame’s default is that a test is unsuccessful until a sanity function succeeds; an empty or partial log is a failed test, not a quiet success.

sourcesWhere this page comes from

The STREAM test is the opening example of the official ReFrame 4 tutorial (tutorial.html, fetched 2026-09-11) and of the CSCS webinar (Karakasis, 2022). The claim that ReFrame does not look at the exit code is from the pipeline documentation (pipeline.html): “ReFrame makes no assumption as of what a successful test is; it does not even look into its exit code.”

Karakasis et al. [2020] state the three design goals (productivity, portability, robustness) and the CSCS, NERSC and OSC deployments. McCalpin [1995] is STREAM.

What to remember

• A ReFrame test is a decorated Python class. Without @rfm.simple_test the class is never registered.

• RunOnlyRegressionTest skips compile. The executable is assumed to exist.

• valid_systems and valid_prog_environs are filters, not comments. ['*'] means any.

• ReFrame does not look at the exit code. A sanity function is the verdict, and a missing pattern is a failure.