The bot is three processes and a comment

The bot is three processes and a comment. A green job is not a module on a login node until ingest.

A bot: build comment starts a job. A module on a login node is a later process.

The bot is three processes and a comment. A human trigger is a comment, not a login on the build node.

Questions

• What does a bot: build comment start, and what does it not start?

• Why is software-layer’s bot/build.sh thirty-three lines?

• Where does a successful tarball go, and what still has to happen before a module appears on a login node?

What you will be able to do

1. Name the three processes a bot instance runs, and say which GitHub event each one reacts to.

2. Read a bot: build comment and say which node is allocated and which prefix is written.

3. Trace one artefact from the job directory to a module on MODULEPATH, including the human steps.

Before this page

• An easystack is the list the bot will build: the YAML the bot feeds to eb.

• EESSI, somebody else’s stack: a green local build is not the all-architecture claim.

• Reading a stack off the wire: CernVM-FS: ingestion is a Stratum-0 transaction, not a cp into /cvmfs.

All installations in software.eessi.io were constructed by the EESSI build-and-deploy bot. The 22 June 2026 Happy Hour reports roughly 30 000 installations, dating from autumn 2023. Contributors are not required to have an account on each CPU target. A pull request to EESSI/software-layer together with a comment is the complete interface. The procedure is described as semi-automatic. The term is load‑bearing. The 1 December 2025 design session enumerated ten goals on slides 3 and 4. Goal 4 reads “Avoid wasting scarce build resources”. The quoted human‑trigger sentence represents the Build/Deploy/Ingest Strategy decision (PDF page 12). The sentence is not a numbered goal. The quoted text states “as much as possible automatic but human involvement required to trigger certain actions (build, transfer, ingest)”. A comment initiates a build. A label initiates a deploy. A merge in a private staging repository initiates an ingest. None of those three events starts the next one. MultiXscale, the EuroHPC Centre of Excellence that funds most of the EESSI workforce, explains the existence of the Happy Hours and the webinars. CASTIEL2 is the tutorial portal; the build farm is elsewhere.

Three processes, one GitHub App

The bot is a GitHub App, written in Python and Bash, configured from an .ini file, and installed on software-layer, software-layer-scripts, and compatibility-layer. Caspar van Leeuwen’s 1 June 2026 webinar opens with a site. Three processes match those run by EESSI. Smee provides webhook relay. GitHub cannot reach a login node. smee.io (or site‑hosted smee) channel provides webhook URL for GitHub App. Smee client on login node forwards events to 127.0.0.1:3000.

apptainer run docker://ghcr.io/eessi/smee-client:latest --url http://smee.example/CHANNEL
# Forwarding ... to http://127.0.0.1:3000/

The event handler listens on that port. ./event_handler.sh prints “app is listening on port 3000”. Most GitHub events are ignored. A comment whose line matches ^bot:[ ]?(.*)$ is not ignored. The addition of a bot:deploy label is also not ignored. The job manager runs a cron-style loop, started with ./job_manager.sh. It releases held Slurm jobs, watches them, and writes SUCCESS or FAILURE back on the pull request from bot/check-result.sh. The 22 June 2026 revisit lists those three as a shortcoming of the current bot, together with .ini config, Slurm-only, GitHub-only, a fixed build-check-test-check pipeline, and no status page. They also form the architecture that runs today. The rewrite (“CPU”) does not build software.eessi.io. Interaction is a GitHub App that reads structured commands in comments and labels. No CI workflow file or external CLI is involved. The execution backend is an HPC cluster, one bot instance per site. GitHub-hosted runners are not the builders. for: and on: are separate (see below). The bot defines an interface. The target repository owns scripts/bot/build.sh and scripts/bot/test.sh. Builds run in a container so /cvmfs is writable, the view inside is controlled, and a machine that does not mount EESSI natively can still build.

The comment language

Instructions start with bot:. bot: help and bot: show_config do not initiate work. The command that initiates work is from bot 0.9.0:.

bot: build for:arch=x86_64/amd/zen4

A zen4 node is allocated and /cvmfs/software.eessi.io/versions/<ver>/software/linux/x86_64/amd/zen4 is written to. When on: is omitted, it is equivalent to for:. The outcome is a native build. A GPU build can run on a CPU‑only node:

bot: build on:arch=x86_64/amd/zen2 for:arch=x86_64/amd/zen2,accel=nvidia/cc80

Cross-compilation between CPU targets is not supported. The bot writes for: into cfg/job.cfg. The build scripts use that value as the installation path. The scripts do not set EasyBuild’s optarch from it. optarch stays native to the allocated node. A for:arch=x86_64/amd/zen4 job that ran on zen2 would write zen4‑labelled binaries compiled for zen2. The filter is there to stop that, not to implement a cross compiler. Filters are space‑separated with no space after the colon. Partial matching applies to on: (against the instance’s configured node types). Partial matching does not apply to for:.

Filter	Shorthand	Meaning
instance:...	inst:	which bot instance (e.g. aws)
repository:...	repo:	which EESSI version/repo
on:architecture=...,accelerator=...	on:arch=...,accel=...	where the job runs
for:architecture=...,accelerator=...	for:arch=...,accel=...	which prefix is written

A node type that the instance config declares as having a GPU is allocated only when accel= is also passed. Otherwise on:arch=x86_64/amd/zen4 would land on a GPU zen4 node when a CPU‑only zen4 node was the intent. Several instructions may share one comment, one per line. A first build usually targets one architecture, and the full matrix comes later. The building‑software page says so. Permission to comment bot: build is a configured allow‑list. Permission to deploy is also a configured allow‑list.

EB-Bot-1 — What this comment starts

A maintainer posts this on a software-layer pull request:

bot: build repo:eessi.io-2025.06-software on:arch=zen4 for:arch=x86_64/amd/zen4,accel=nvidia/cc90 inst:aws

The acting instance and the outcome for each other instance must be identified. The allocated node type and the written prefix must be determined. on:arch=zen4 is legal while for:arch=zen4 is not. A job starts only when the same comment lacks accel= on a site with a GPU‑only zen4 node type.

Solution

1. Only the instance whose configured name starts with aws. The others filter the instruction out. That is the whole point of inst:.

2. The prefix is zen4 plus the NVIDIA cc90 accelerator subdirectory. That is what for: names. The allocated node is whatever matches on:arch=zen4 only. accel= is not on on:, so a GPU-declared zen4 type is not allocated. A CPU-only zen4 node is used if the instance has one; otherwise nothing starts.

3. on: is matched against the instance’s configured node types, so a suffix is enough. for: has no such reference list; the documentation requires the full x86_64/amd/zen4 path.

4. No. A node type that the config marks as having an accelerator is allocated only when the corresponding accel= is passed. Without it the instance has nothing it is allowed to allocate.

The 33-line script is a pin

software-layer/bot/build.sh is not the build. It pins the scripts repository. It keeps a software-layer pull request from drifting unnoticed away from the scripts that EESSI 2025.06 runs.

commit_sha=$(cat ${TOPDIR}/software_layer_scripts_commit)
git clone --depth 1 --filter=blob:none --no-checkout \
 https://github.com/EESSI/software-layer-scripts
cd software-layer-scripts
git fetch --depth=1 origin ${commit_sha}
git checkout --detach ${commit_sha}
# symlink everything except bot/build.sh, then:
software-layer-scripts/bot/build.sh

The pin is the contents of bot/software_layer_scripts_commit in the target software-layer checkout. That is the design session’s “build logic ownership” decision in one file. The bot submits a job that runs the target bot/build.sh from the target repository. The target repository immediately hands off to a detached scripts commit. The real software-layer-scripts/bot/build.sh then reads cfg/job.cfg (written by the event handler from the comment’s for: / on: / repo). Sets EESSI_SOFTWARE_SUBDIR_OVERRIDE from .architecture.software_subdir. Falls back to $CPU_TARGET. If the target is not software.eessi.io, sets EESSI_SITE_INSTALL_FORCE=1 and EESSI_SITE_SOFTWARE_PREFIX, and still mounts software.eessi.io read-only as the compatibility layer. That is how Caspar’s software.caspar.nl demo builds on top of EESSI. Starts eessi_container.sh --access rw --mode exec with --save. The tmpdir can be resumed. Inside the container, runs install_software_layer.sh. It is a one-liner into EESSI-install-software.sh. After the build, resumes the same tmpdir. Runs create_tarball.sh to produce eessi-<ver>-software-<os>-<arch-with-hyphens>[-<accels>]-<timestamp>.tar.zst. EESSI-install-software.sh needs a [0-9]*.diff in the working directory. That is the pull request patch. Without it the script cannot decide which easystack files changed. Reproducing a failure locally is the same chain: container, prefix, configure_easybuild, the .diff, then either the full script or eb --easystack <file> --robot. Reading a failed build’s log-reading applies once that job has failed. The debugging-failed-builds page is the operator’s version of the same walk.

From artefact to module

Green jobs do not constitute modules. The job manager comment displays tarball contents. Users with deploy permission add the bot:deploy label. The event handler on software-layer executes eessi-upload-to-staging and places the tarball and a metadata file into an S3 bucket. A cron job on the CernVM-FS Stratum-0 creates a pull request in the private EESSI/staging_bundles repository. Merging that pull request performs the ingest. Tarballs that should not be included remain undeleted; their TaskState changes from APPROVED to REJECTED. The CI for the software-layer pull request starts with a red status. Re‑executing the workflow after ingest changes the status to green and merges the pull request. Sites that follow Caspar’s webinar omit staging_bundles. A script on the site Stratum-0 lists the bucket, downloads objects, optionally verifies a signature, calls filesystem-layer/scripts/ingest-tarball.sh, updates Lmod caches, and moves the tarball for single ingestion. The demo concludes with module load Biopython/1.86-gfbf-2025b resolving under /cvmfs/software.caspar.nl/versions/2025.06/... while python continues to be provided by software.eessi.io. Approach 2 targets EESSI’s dependencies, the site’s CVMFS repository, the same bot, and the same scripts.

EB-Bot-2 — Order the pipeline

These events all happened for one software-layer pull request. Put them in the order they must occur. Mark the ones that require a human with a configured permission.

1. event handler writes cfg/job.cfg and submits a held Slurm job
2. software-layer CI goes green
3. bot:deploy label added
4. bot: build for:arch=x86_64/amd/zen3 posted
5. staging_bundles pull request merged
6. job manager reports SUCCESS and the tarball listing
7. eessi-upload-to-staging writes the artefact to S3
8. EESSI-install-software.sh greps the pull request .diff
9. software-layer pull request opened, adding one easystack line

Solution

9, 4, 1, 8, 6, 3, 7, 5, 2.

9 is the contribution. 4 is the first human permission (build allow-list). 1 is the event handler; 8 is the job, which cannot run before the job exists. 6 is the job manager. 3 is the second human permission (deploy allow-list). 7 is the upload that 3 triggers. 5 is the third human step (ingest). 2 is CI after the bytes are on the Stratum-0; it cannot go green before 5.

Opening the pull request does not start 1. A green local EESSI-extend build does not replace 6. 6 does not replace 5.

sourcesWhere this page comes from

Dröge et al. [2023]: the three-layer design. The bot and the production repository post-date that paper.

Talks:

• Goals and Design Decisions Behind a Multi-Architecture Build Bot, EESSI Happy Hour, 2025-12-01 (EESSI-happyhour-2025-016-EESSI-Bot-Goals-Design.pdf): the ten goals on slides 3–4 (goal 4 is scarce build resources); GitHub App; HPC backend; for: / on:; repo-owned scripts bot/build.sh; containers; split components; Build/Deploy/Ingest Strategy (PDF page 12) for human triggers of build, transfer, ingest.

• Revisit the EESSI bot, EESSI Happy Hour, 2026-06-22 (EESSI-happyhour-2026-017-EESSI-bot.pdf): three processes (smee, event handler, job manager); ~30k installations; current shortcomings; the rewrite is not what is running.

• Caspar van Leeuwen, Bob Dröge and Pedro Santos Neves, Using EESSI as a base for a central software stack, MultiXscale webinar, 2026-06-01: smee URL, GitHub App registration, app.cfg, the three start scripts, bot: show_config, a for:arch build of Biopython into software.caspar.nl, ingest_tarballs.sh on the Stratum-0.

• Lara Peeters and Kenneth Hoste, Building software on top of EESSI + contributing, 2026-05-04: the workflow diagram this page’s pipeline is.

Code, read 2026-09-11: EESSI/software-layer/bot/build.sh and bot/software_layer_scripts_commit; EESSI/software-layer-scripts/bot/build.sh; EESSI/eessi-bot-software-layer/tools/commands.py (^bot:[ ]?(.*)$); tools/build_params.py; tasks/deploy.py.

Docs: bot.md (filters, artefacts, TaskState: REJECTED), debugging failed builds.

What to remember

• A bot instance is smee plus an event handler plus a job manager. A comment matching ^bot: is what the handler acts on. Opening a pull request is not.

• for: is the installation prefix. on: is the allocated node. optarch stays native. CPU-to-CPU cross-compilation is not implemented.

• software-layer/bot/build.sh pins a software-layer-scripts commit and execs that pinned bot/build.sh. The job’s .diff is what selects the easystack files.

• A green job produces a tarball. Signing is a detached .sig at deploy, and it is optional. bot:deploy uploads it. A Stratum-0 ingest is what makes a module. Three human permissions stand on that path.