What EasyBuild Is Doing

Three parts. A recipe. A stack. A test.

EasyBuild’s documentation is good at what to type. This book is why SYSTEM has no quotes, why a toolchain is a hierarchy rather than a name, and why a module in module avail can be invisible to the robot.

The samples run in the page. Escape opens the playground.

Start here

Read an easyconfig the way EasyBuild does. Twenty minutes. Nothing to install.

Then one of three parts. Enter anywhere.

• The recipe The file. The pin. The build.

• The stack A tree on a machine that never built it.

• The test Four kinds of testing. Then ReFrame.

Hour-sized levels, and a table by task, live on the path.

Where this runs

Not only on a supercomputer, and the book is careful about the difference.

EasyBuild is a Python package. It installs with pip into a virtual environment, needs a modules tool and a compiler, and builds software on a laptop exactly as it does on a compute node. Several of the recordings in this book were made that way, and their provenance says so: the failing build in Reading a failed build and the checks in Writing one from nothing ran on a workstation, with no cluster involved.

EESSI goes further. Its own installation instructions cover a laptop, and CernVM-FS ships a client profile for exactly that case, so a machine with no software stack of its own can mount one and load optimised modules in minutes. The same stack somewhere else does it on four different kinds of machine.

What changes on a large shared machine is the consequences, not the mechanism: many users, one tree, a scheduler, several microarchitectures, and a rebuild that costs somebody a night. Where a page is about those consequences it says so. Where it is about the mechanism, the mechanism is the same on a local machine, and the same commands run there while reading.

What this book requires

A browser.

EasyBuild’s own tutorial opens by asking for Linux, Python, an environment modules tool and a working EasyBuild install. That is a reasonable ask of someone who has already decided to use EasyBuild, and an unreasonable one of someone still deciding whether to.

Every sample here runs in the page. Edit one and the answer changes as the sample is typed. Press Escape anywhere and a playground opens: the same engine, not limited to the samples printed on the page.

Three of the four kinds work off tables exported from eb-stack at build time rather than retyped. So a page cannot drift from the framework’s own data.

A cross-check compares the browser’s answers against the binary’s on every build. A headless browser then checks that every sample on every page is live and answering.

The fourth reads an easyconfig into its evaluated model, and it is a reduced reader rather than the real parser. It follows single-line assignments and names what it skipped. Where the real parser is needed it says so instead of guessing.

That covers a lot, and not everything. Nothing in a browser can configure, compile or link, and nothing here pretends to.

Where a step needs a real toolchain the page shows a recording of a real run instead, with where and when it ran printed beside it. That turns out to be the right way to teach those steps. The operational skill is reading the log rather than watching the progress bar.

The book

One toctree on this page. Each part page carries its own. The sidebar nests.

• The recipe
  • Read a file
  • When it fails
  • How it runs
  • Write one
• The stack
  • EESSI, somebody else’s stack
  • Reading a stack off the wire: CernVM-FS
  • The same stack somewhere else
  • An easystack is the list the bot will build
  • The bot is three processes and a comment
  • A site stack next to EESSI, not inside it
• The test
  • Testing, and the four things it can mean
  • A ReFrame test is a class
  • The pipeline has six stages
  • The configuration file is the machine
  • A performance function is a number with a unit
  • Compile or run-only
  • Parameters multiply the suite
  • A portable test leaves the machine in the config
  • Software maps to tests
  • Reading a failed ReFrame run
• A path through this book
• Glossary
• What to remember
• Every exercise in this book
• References
• Afterword: when EasyBuild is the wrong tool

Who wrote this

Rohit Goswami, who maintains software stacks on a national machine, runs the same tools on a laptop, and wrote eb-stack, the updater and generator that emits the .eb files EasyBuild then builds. The samples on these pages borrow tables from that generator. They are not the generator, and they are not EasyBuild.