A performance function is a number with a unit¶

Questions

What is the difference between a sanity function and a performance function?
Where does a reference value live, and in what units?
What does a P: line report that a pass/fail does not?

What you will be able to do

Extract a figure of merit from a job’s output and give it a unit.
Attach a reference and a threshold per system, and say what a miss means.
Say where the number is logged, and why the log outlives the run.

Before this chapter

A ReFrame test is a class, for the STREAM test the numbers come from.
The pipeline has six stages, for the performance stage.

Sanity corresponds to a yes or a no. Performance consists of a number, a unit, and a comparison. The STREAM test in A ReFrame test is a class already includes both. This chapter provides the number.

@performance_function('MB/s')
def copy_bw(self):
    return sn.extractsingle(r'Copy:\s+(\S+)', self.stdout, 1, float)

@performance_function('MB/s')
def triad_bw(self):
    return sn.extractsingle(r'Triad:\s+(\S+)', self.stdout, 1, float)

The argument of the decorator represents the unit. The method name denotes the metric. The extractsingle function extracts group 1 from the first match and converts it to float. When the line is absent, the performance stage fails; this failure differs from a sanity failure, and the FAILURE INFO block reports it. A test lacking performance functions requires no work from the performance stage. A test containing performance functions but no references still records the numbers. Recording without comparison allows a site to build the baseline against which the next week’s run will be compared.

References¶

Each reference consists of a tuple for each metric and each system, formatted as (value, lower_thres, upper_thres, unit). Starting with version 3.0, the tuple must include the unit. The STREAM tutorial assigns these tuples to the test after the functions have been defined:.

reference = {
    'generic:default': {
        'copy_bw': (20000, -0.2, 0.2, 'MB/s'),
        'triad_bw': (15000, -0.2, 0.2, 'MB/s'),
    },
}

-0.2 and 0.2 represent relative thresholds, 20 % below and 20 % above the reference. An absolute floor is a number lacking a fractional component, and the documentation’s form should be copied rather than invented. A miss causes failure of the performance stage while not causing failure of sanity. The binary generated a valid STREAM run; the machine performed slower (or faster) than the site expected.

The key is system:partition. A reference for generic:default does not apply to example:cpu. A portable suite therefore either places references in the configuration (EESSI’s direction) or accepts that the first run on a new partition records rather than judges.

When the test sets no references, ReFrame 4.9 and later omit the reference tuple from the P: line. The number remains printed and logged.

The `P:` line and the perflog¶

A successful performance test outputs a separate line for each metric.

P: copy_bw: 19538.4 MB/s (r:0, l:None, u:None)
P: triad_bw: 14883.4 MB/s (r:0, l:None, u:None)

r is the reference, l and u are the thresholds. --performance-report prints the same facts as a table at the end of the session. RFM_PERF_INFO_LEVEL raises the log level of the P: lines if a suite’s stdout is too noisy. The durable record is the perflog, by default at perflogs/<system>/<partition>/<testname>.log, or under $RFM_PREFIX/perflogs/ when the prefix is set. It is a CSV whose columns are configurable. A site that graphs STREAM triad over six months reads this file, not the session JSON. The session JSON lives under $HOME/.reframe/reports/, with latest.json pointing at the last run. --restore-session in Reading a failed ReFrame run reads it. Past performance can be queried from the same database and compared to the current run; that is a How To in the official docs, not a mechanism this chapter needs.

What a number does not say¶

The run-stage timer includes queue time (The pipeline has six stages). A performance function that extracts a runtime from the application’s output does not. STREAM’s Copy: line is a bandwidth the binary measured. GROMACS’s ns/day is a throughput the binary measured. Both are independent of how long the job waited. A performance function that returns ReFrame’s own elapsed time is measuring the queue. EESSI’s rule of thumb for a new test is on the writing-tests page: at least one case tagged CI that finishes in under five minutes on about eight cores, and no case that needs more than about 1 GB per core. Those are constraints on the workload, so that the number being tracked is a number a site can afford to produce daily.

EB-ReFrame-42-1 — Sanity or performance

Three STREAM outcomes. For each, say which stage fails, or that both pass.

The output contains Solution Validates and Copy: 5000 against a reference of 20000 ± 20%.
The output contains Copy: 19000 and does not contain Solution Validates.
The output contains both lines, no reference is set, Copy: is

Solution

Sanity passes, performance fails. The binary validated the arrays; the bandwidth is outside the threshold.
Sanity fails. Performance is not the headline; the FAILURE INFO names sanity. A site that only watches perflogs will not see this run the artifacts in the stage directory and does not treat the Copy: line as a result.
Both pass. With no reference there is nothing to miss. The number is logged. That is the baseline run.

What to remember

A performance function returns a number and declares a unit. It does not decide correctness; the sanity function does.
A reference is (value, lower, upper, unit) per system:partition. A miss fails the performance stage only.
No reference still logs the number. That log is how a baseline is built.
Extract the application’s own figure of merit. ReFrame’s run timer includes the queue.

A performance function is a number with a unit¶

References¶

The P: line and the perflog¶

What a number does not say¶

The `P:` line and the perflog¶