The pipeline has six stages

Questions

• In what order do the stages run?

• Which stages run on the login node, and which on the partition?

• What does a failure in run_complete mean that a failure in run does not?

What you will be able to do

1. Name the six pipeline stages, and say which ones a run-only test actually does work in.

2. Say where each stage executes: the ReFrame host, or the partition.

3. Attach a hook to a stage, and say when it runs relative to that stage.

Before this chapter

• A ReFrame test is a class, for the test class.

• How a build executes, for the comparison with EasyBuild’s eighteen steps.

EasyBuild executes eighteen named steps sequentially, using the same list for each package. ReFrame executes six stages sequentially, using the same list for each test. The stage names are taken from the pipeline documentation, ReFrame 4.10:

Stage	What it does
setup	partition, environment, stage path, job descriptor
compile	copy sources, build. No‑op on a run‑only test
run	write the job script, submit it
sanity	run the sanity function against the output
performance	extract figures of merit, compare to references
cleanup	copy artifacts out, delete the stage directory

A test that fails in sanity has already been compiled and run. A test that fails in setup has not been compiled or run. The failing phase is displayed in the FAILURE INFO block and appears as the first field to read, mirroring how EasyBuild’s error block names a step.

Loading is not the pipeline

Prior to any stage execution, ReFrame builds each discovered test by importing its file and invoking __init__. Each test is instantiated regardless of its suitability for the current system. Following loading, the framework filters tests using system, environment, and command-line criteria, and then clones each remaining test for every (partition, environment) combination. That clone is the test case. The original object is not scheduled for execution. work performed in __init__ or a post‑init hook executes for tests that will later be skipped, as noted on the pipeline page. Expensive setup should be placed in setup or a later stage, not during construction. init does not constitute a pipeline stage. Hooks may still attach to it (@run_after('init')). A portable test leaves the machine in the config illustrates this pattern.

Where each stage runs

Two execution contexts. The ReFrame context is the process that launched reframe, always the local host. The partition context is wherever the partition’s scheduler points, local or remote.

Stage	Context
setup	ReFrame
compile	ReFrame if build_locally or local, else the partition
run	ReFrame if local, else the partition
sanity	ReFrame
performance	ReFrame
cleanup	ReFrame

Sanity and performance therefore read files that the run stage wrote, on the host, after the job finished. A sanity function that tries to open a path that exists only on the compute node will not find it, unless that path is in the stage directory and the stage directory is shared. The EESSI configuration comments say this in so many words: the stagedir has to be a shared directory available on every node that will run a test. Compile and run are asynchronous from version 3.10. The stage splits into submit and wait, which is why a failure report can name compile_complete or run_complete rather than compile or run. run means the job script was submitted. run_complete means the job finished and the wait returned. A test stuck in the queue fails in run_complete, not in run.

Two execution policies

Serial: a test enters the pipeline only after the previous test has exited. The compile and run stages wait idle for the whole suite.

Asynchronous: several tests are in flight. When a test reaches compile or run, ReFrame does not block; it selects the next case. A concurrency limit (max_jobs on the partition, max_local_jobs on the host) caps the number of jobs submitted. This is the default configuration for cluster use and explains why a suite of thirty GROMACS cases does not submit thirty jobs simultaneously unless the partition permits it.

The time printed for the run stage reflects ReFrame’s time, not the scheduler’s. It includes queue time. It does not represent the binary’s runtime.

Hooks

A method decorated with @run_before('run') or @run_after('setup') executes at that edge. The official tutorial and the EESSI mixin place the machine-dependent numbers in hooks instead of the class body, since the class body is evaluated at load time before current_partition exists.

@run_after('init')
def set_modules(self):
    self.modules = [self.module_name]

@run_before('run')
def set_tasks_from_partition(self):
    ncpus = self.current_partition.processor.num_cpus
    self.num_tasks = ncpus
    self.num_tasks_per_node = ncpus

current_partition exists after setup, not at class-body time. The first hook can run after init. The second has to wait: a core count read before setup is a number the configuration has not supplied.

Cleanup of a successful test is deferred when another test depends on it. A failure in cleanup is reported as an error rather than as a test failure.

EB-ReFrame-40-1 — Name the failing stage

Four reports. For each, name the stage, and say whether the job ran.

1. pattern 'Solution Validates' not found in 'rfm_job.out'

2. compile_complete: the build job timed out in the queue

3. The listing shows the test, the run never starts, and the reason is valid_systems did not match

4. Copy: 8000 MB/s against a reference of 30000 with a 20% threshold

Solution

1. Sanity. The job ran; rfm_job.out exists. The verdict failed.

2. compile_complete, the wait half of compile. The compile job was submitted and did not finish. The test job was never submitted.

3. No stage. Filtering happens after load and before any test case is cloned. Nothing ran.

4. Performance. Sanity passed (or there would be no figure to compare). The job ran; the number is below the threshold.

sourcesWhere this chapter comes from

How ReFrame Executes Tests, ReFrame 4.10.3, fetched 2026-09-11: the six stages, the load-filter-clone sequence, the two execution contexts, serial against asynchronous, and the compile_complete / run_complete split (changed in 3.10.0).

Karakasis et al. [2020], figure of the pipeline, and the 5th EasyBuild User Meeting talk for the same six names in the EasyBuild-adjacent setting.

What to remember

• Six stages, in order: setup, compile, run, sanity, performance, cleanup. A run-only test no-ops compile.

• Construction is not a stage. __init__ runs for tests that will be skipped.

• Sanity and performance run on the ReFrame host, against files the job wrote. The stage directory has to be shared with the nodes.

• run is submit; run_complete is wait. A queued job fails in the second.