A campaign finding is a typed failure, not a log¶

Questions

What three booleans does \*.campaign.json store, and which one can stay false on a completed run?
Who may resolve a finding, and what happens if a second owner tries?
When is a compile failure a resource finding rather than a compile finding?

What you will be able to do

Read a campaign state file and say which claim rungs it currently holds.
Classify an open finding by class, disposition, and stage, and name the first repair surface.
Claim, resolve, and retry a finding on the same state path without losing the audit trail.

Before this chapter

eb-stack ports a recipe and bumps one: campaign run is the command that calls eb.
Inspect writes a work queue, not a recipe: a bundle with a lock and a recipe.
Reading a failed build: the EasyBuild log the finding interned.

The campaign run operates synchronously. It remains in the foreground until all recipes and verification commands complete, or a typed failure is logged. The state file is the coordination surface. In the campaign reference, Hermes and OMP are role names: one denotes an owner that classifies and retries, and the other denotes optional concurrent workers whose sole shared lock is acquiring ownership. They are not required software components.

eb-stack campaign run \
  --bundle "$BUNDLE" \
  --config examples/targets/local-podman.toml \
  --target local-rocky9 \
  --state "$STATE"

eb-stack campaign status --state "$STATE"

The state path is reused for each retry. Package name and version must be identical. Each execution increments attempts. Build and binary claims are reset until the current attempt validates them. History and findings are retained.

Only one writer may operate at a time. An OS advisory lock together with a visible .lock file records host, PID, and Linux process-start identity. The kernel releases the guard when the controller exits, preventing stale metadata from permanently blocking a retry. A remote Slurm job may outlive that controller; verify its termination before retrying.

The three booleans are independent¶

CAMPAIGN.json is schema 1:

{
  "schema_version": 1,
  "package": "QMCPACK",
  "version": "4.3.0",
  "bundle": "/work/qmcpack",
  "target": "site-builder",
  "status": "failed",
  "attempts": 2,
  "claims": {
    "resolves": true,
    "builds": false,
    "binary_verified": false
  },
  "current_recipe": null,
  "findings": [],
  "history": []
}

status can be planned, running, failed, or completed. current_recipe is defined before EasyBuild starts. Another shell to monitor the run. resolves is true when each requested profile possesses both a lock and a recipe. builds is true when every emitted recipe succeeds via EasyBuild on this target. binary_verified is true only if at least one verification command is declared and all declared commands succeed. A completed campaign without verification commands has builds: true and binary_verified: false. The planned SBOM does not imply either condition. That represents eb-stack ports a recipe and bumps one’s ladder as a file. eessi-verified remains absent as a field. The EESSI bot controls that rung.

A finding is structured so a worker can pick it up¶

A terminal failure appends a finding with a stable attempt:N:finding:M id, a class, a disposition, a stage (preflight, stage, build, verify), the exact routed command and exit status, and the tail of stdout/stderr.

When EasyBuild reports a separate combined-output path, the campaign interns that nested log before classification, so the finding reflects the compiler line rather than the wrapper.

Class	Default disposition	First surface
`transport`	target-repair	SSH, rsync, staging path
`executor`	target-repair	Slurm syntax, partition, account
`runtime`	target-repair	image, mounts, ABI, environment
`interrupted`	retryable	abandoned `running` after the lock is free
`source`	requires-judgment	URL, mirror, cache
`checksum`	mechanical	digest and positional checksum list
`patch`	requires-judgment	patch source, order, applicability
`dependency-missing`	requires-judgment	manifest, policy, or robot
`configure` / `compile` / `link` / `test` / `install` / `sanity`	requires-judgment	the named step
`resource`	retryable	CPU, memory, GPU, filesystem, scheduler
`timeout`	retryable	routed or wall-time limit
`unknown`	requires-judgment	unclassified; read the command

A SAT-compatible dependency that fails to compile remains compile until evidence supports a stack exclusion.

A compiler killed with “virtual memory exhausted” is resource: keep the selected dependency and raise EASYBUILD_PARALLEL or the allocation.

The class is the first surface, not the diagnosis.

Claim, then resolve, then retry¶

eb-stack campaign finding claim \
  --state "$STATE" \
  --id attempt:2:finding:1 \
  --owner omp-worker-1

eb-stack campaign finding resolve \
  --state "$STATE" \
  --id attempt:2:finding:1 \
  --owner omp-worker-1 \
  --action "raised the Slurm memory request" \
  --evidence "target doctor and the isolated command exit successfully" \
  --change ~/.config/eb-stack/site.toml

Each claim is exclusive. If the same owner reclaims, the operation is idempotent. If a different owner attempts reclamation, an ownership error is returned. After a resolve operation, the identical state path is supplied to campaign run again. Open or in‑progress findings that match become superseded. Findings that have been resolved remain recorded in the audit trail. status displays the full record, including the exact command of any finding that has already been resolved. A compact view is provided during a long build:

eb-stack campaign status --state "$STATE" |
  jq '{status, attempts, claims, current_recipe,
       active_findings: [.findings[] |
         select(.status == "open" or .status == "in-progress") |
         {id, class, stage, recipe, summary, owner}]}'

Maintain the unfiltered state. The compact view serves the operator, not the handoff.

EB-Campaign-1 — Class this failure

Four terminal failures. Name class and the first repair, and say whether a second worker may resolve the finding without claiming it.

eb never started. rsync to the build host returned 255.
g++ died with “virtual memory exhausted” after 40 seconds.
A checksum in the emitted .eb does not match the tarball the campaign fetched.
Configure cannot find FFTW, and the lock named a version the robot overlay does not contain.

Solution

transport, target-repair. The host or the staging path. A second worker may not resolve it. Claim first.
resource, retryable. Raise memory or drop EASYBUILD_PARALLEL. Not compile: there is no source-level diagnostic. Still exclusive: claim, resolve, retry.
checksum, mechanical. Fix the positional digest. Same ownership rule.
dependency-missing, requires-judgment. The overlay or the stack policy, not a format pass. Same ownership rule.

No finding is resolved by a worker that does not own it. The state file is the lock.

What to remember

\*.campaign.json stores three independent claims. A completed run with no verify command is builds and not binary_verified.
A finding is a class, a disposition, a stage, and the exact routed command. The class names the first repair surface.
Claim is exclusive. Resolve records the action. Retry uses the same state path. The audit trail is not deleted.