Parameters multiply the suite

Questions

• What is the difference between a parameter and a variable?

• How many test cases do two lists of three produce?

• When does that product become a problem?

What you will be able to do

1. Read a test with parameter and variable and say how many cases ReFrame will generate.

2. Say which of those knobs a command line can override, and which it cannot.

3. Bound a product before it becomes a suite nobody will run.

Before this chapter

• A ReFrame test is a class, for the class.

• Compile or run-only, for a module as the thing under test.

One class does not correspond to a single run. ReFrame builds a cartesian product from every parameter on the class, then clones one test case for each (value‑tuple, partition, environment). This process causes a suite to grow and to explode. EESSI’s mpi4py tutorial states the trap in one sentence: four parameters with (3, 3, 2, 2) values produce 36 tests, and that is before partitions and environments multiply it again.

Two knobs

The parameter defines a dimension of the suite. Each parameter value corresponds to a distinct test case. The suite name is established when the suite loads. The command line cannot modify the list; it may only filter the cases to run using --name or --tag.

The variable is a field present on each case, has a default value, and can be overridden on the command line with -S name=value. It does not cause any multiplication.

import reframe as rfm
import reframe.utility.sanity as sn
from reframe.core.builtins import parameter, variable, run_after, sanity_function

@rfm.simple_test
class EESSI_MPI4PY(rfm.RunOnlyRegressionTest):
    valid_prog_environs = ['default']
    valid_systems = ['*']

    module_name = parameter([
        'mpi4py/3.1.4-gompi-2023a',
        'mpi4py/3.1.5-gompi-2023b',
    ])
    scale = parameter([2, 128, 256])

    n_iterations = variable(int, value=1000)
    n_warmup = variable(int, value=100)

    executable = 'python3'
    time_limit = '5m00s'

    @run_after('init')
    def set_modules(self):
        self.modules = [self.module_name]

    @run_after('init')
    def define_task_count(self):
        self.num_tasks = self.scale
        self.num_tasks_per_node = min(self.num_tasks, 128)

    @sanity_function
    def validate(self):
        total = round(self.scale * ((self.scale - 1) / 2))
        return sn.assert_found(
            r'Sum of all ranks: %s' % total, self.stdout
        )

Two modules multiplied by three scales yield six cases on one partition. n_iterations equals 1000 for all six cases, unless the invocation specifies -S n_iterations=200. The hooks execute after init, which occurs after the product exists. self.module_name is a single string, not a list. self.scale is a single integer. Therefore module load and task count belong in hooks rather than in the class body. num_tasks_per_node = min(self.scale, 128) remains a number pertaining to a single machine. A portable test leaves the machine in the config removes that 128.

Filtering is not shrinking the definition

reframe -n 'EESSI_MPI4PY.*3.1.5' --tag CI -l

-n matches the test‑case name. --tag CI indicates the five‑minute, few‑core case that EESSI marks for a pull request. The other five cases remain. They are omitted when the filter excludes them from the run, yet they stay in the class for the next nightly. A parameter that enumerates every GROMACS HECBioSim system, from Crambin to the 3M‑atom tetramer, constitutes a suite. Tagging only Crambin CI keeps that suite usable as both a check and a benchmark.

EB-ReFrame-44-1 — Count the cases

A run-only test has module_name = parameter over three GROMACS modules, bench = parameter over two HECBioSim systems, and nsteps = variable(int, value=10000). The configuration has two CPU partitions and one default environment. How many test cases does reframe -l print? Which of the three knobs does -S nsteps=1000 change, and does that change the count?

Solution

3 × 2 × 2 × 1 = 12. Three modules, two benches, two partitions, one environment. nsteps is a variable, so it does not appear in the product. -S nsteps=1000 changes the workload in all twelve; the listing still has twelve names.

sourcesWhere this chapter comes from

EESSI writing-tests tutorial, the mpi4py example: parameter against variable, the cartesian warning, and hooks after init. ReFrame builtins documentation for the same two decorators. EESSI available-tests page: GROMACS HECBioSim systems and the CI tag on Crambin.

What to remember

• parameter multiplies the suite. variable does not; it is an overridable default.

• The product is parameters × partitions × environments. Four small lists become a suite nobody will wait for.

• Hooks after init see one case. Module names and task counts are set there.

• --tag CI filters a run. It does not delete the other cases from the class.