Parameters multiply the suite

Parameters multiply the suite. Filtering a run does not shrink the definition.

Parameters multiply the suite. They do not shrink the class.

A parameter multiplies the suite. A variable does not. Two lists of three are nine cases.

Questions

• What is the difference between a parameter and a variable?

• How many test cases do two lists of three produce?

• When does that product become a problem?

What you will be able to do

1. Read a test with parameter and variable and say how many cases ReFrame will generate.

2. Say which of those knobs a command line can override, and which it cannot.

3. Bound a product before it becomes a suite nobody will run.

Before this page

• A ReFrame test is a class, for the class.

• Compile or run-only, for a module as the thing under test.

A single class may generate multiple runs. ReFrame creates a cartesian product of all parameter values defined in the class. ReFrame clones a test case for each combination of value‑tuple, partition and environment. The suite size increases dramatically. EESSI’s mpi4py tutorial warns that four parameters with (3, 3, 2, 2) values generate 36 tests. This count excludes the additional multiplication by partitions and environments.

Two knobs

parameter defines a suite dimension. Each parameter value maps to a distinct test case. Suite name sets when suite loads. Command line cannot modify the list. Filtering cases allowed only with --name or --tag. variable is a field on each case. variable has a default value. variable can be overridden on the command line with -S name=value. variable does not cause any multiplication.

import reframe as rfm
import reframe.utility.sanity as sn
from reframe.core.builtins import parameter, variable, run_after, sanity_function

@rfm.simple_test
class EESSI_MPI4PY(rfm.RunOnlyRegressionTest):
 valid_prog_environs = ['default']
 valid_systems = ['*']

 module_name = parameter([
 'mpi4py/3.1.4-gompi-2023a',
 'mpi4py/3.1.5-gompi-2023b',
 ])
 scale = parameter([2, 128, 256])

 n_iterations = variable(int, value=1000)
 n_warmup = variable(int, value=100)

 executable = 'python3'
 time_limit = '5m00s'

 @run_after('init')
 def set_modules(self):
 self.modules = [self.module_name]

 @run_after('init')
 def define_task_count(self):
 self.num_tasks = self.scale
 self.num_tasks_per_node = min(self.num_tasks, 128)

 @sanity_function
 def validate(self):
 total = round(self.scale * ((self.scale - 1) / 2))
 return sn.assert_found(
 r'Sum of all ranks: %s' % total, self.stdout
 )

Two modules multiplied by three scales yield six cases on one partition. n_iterations equals 1000 for all six cases. The invocation may specify -S n_iterations=200. Hooks execute after init. The product exists before init. self.module_name is a single string. self.scale is a single integer. Module load belongs in hooks. Task count belongs in hooks. num_tasks_per_node = min(self.scale, 128) remains a number on a single machine. A portable test leaves the machine in the config. The test removes that 128.

Filtering is not shrinking the definition

reframe -n 'EESSI_MPI4PY.*3.1.5' --tag CI -l

-n matches the test‑case name. --tag CI indicates the five‑minute, few‑core case that EESSI marks for a pull request. The other five cases remain. The filter omits them from the run. They stay in the class for the next nightly. A parameter enumerates every GROMACS HECBioSim system, from Crambin to the 3M‑atom tetramer, forming a suite. Tagging only Crambin CI keeps that suite usable as both a check and a benchmark.

EB-ReFrame-44-1 — Count the cases

A run-only test has module_name = parameter over three GROMACS modules, bench = parameter over two HECBioSim systems, and nsteps = variable(int, value=10000). The configuration has two CPU partitions and one default environment. How many test cases does reframe -l print? Which of the three knobs does -S nsteps=1000 change, and does that change the count?

Solution

3 x 2 x 2 x 1 = 12. Three modules, two benches, two partitions, one environment. nsteps is a variable, so it does not appear in the product. -S nsteps=1000 changes the workload in all twelve; the listing still has twelve names.

sourcesWhere this page comes from

EESSI writing-tests tutorial, the mpi4py example: parameter against variable, the cartesian warning, and hooks after init. ReFrame builtins documentation for the same two decorators. EESSI available-tests page: GROMACS HECBioSim systems and the CI tag on Crambin.

What to remember

• parameter multiplies the suite. variable does not; it is an overridable default.

• The product is parameters x partitions x environments. Four small lists become a suite nobody will wait for.

• Hooks after init see one case. Module names and task counts are set there.

• --tag CI filters a run. It does not delete the other cases from the class.