What EasyBuild Is Doing

An interactive book about EasyBuild: how a recipe is evaluated, how a toolchain is a hierarchy, and why the version next to a dependency is not a minimum.

EasyBuild’s documentation is good at what to type. It does not say why SYSTEM has no quotes, why a toolchain is a hierarchy rather than a name, or why a module visible in module avail can be invisible to the robot. Those three are usually learned by breaking a production stack.

This book is the explanation. The samples run in the page. After reading it, EasyBuild is one tool among others, not a default.

Start here

Read an easyconfig the way EasyBuild does. Twenty minutes, and nothing to install.

At the end a reader can predict what EasyBuild will download, name the easyblock it will look for, and say which dependencies a toolchain permits.

After that, either follow the path, which is six levels of about an hour each with the prerequisites named, or pick by the task at hand.

Task	Read
working out why a build failed	chapter 1, then chapters 5 to 9
moving a recipe to a newer stack	chapter 3
told a module exists, but it will not load	chapter 4 and chapter 6
reading a failure whose cause is not in the file	chapter 9
starting a job like this one next Monday	chapter 28
staring at a failed build right now	chapter 20
writing a recipe for software that has none	chapter 21
wondering how a stack gets onto a thousand machines	chapter 25
installing software with no admin rights	chapter 10
working on a machine with no stack of its own	chapter 11
adding a package to EESSI	chapter 30 then chapter 31
putting a site stack next to EESSI	chapter 32
porting a conda-forge or Spack recipe, or bumping one	chapter 12 then chapters 33 to 36
a site wrap or an EasyBuild 5 lock	chapter 37 and chapter 38
running any of this on your own machine	chapter 24 and chapter 26
deciding whether to use EasyBuild at all	the afterword
writing a ReFrame test of an installed module	chapter 39, then Part 2
running the EESSI or CSCS suite	chapter 45 and chapter 46
reading a failed ReFrame run	chapter 47

Reading straight through works too. Chapters 1 to 4 are the mechanisms everything else rests on. Chapters 5 to 9 are the parameters that cost the most time when misread, each built around a failure that actually happened. Chapters 10 to 12 are the ones where something is done. Chapters 13 to 19 are the machinery: the steps a build runs, what it is allowed to see, and what comes out. Chapters 20 to 24 are the working chapters, from reading a failure to writing a recipe, an easyblock and a development environment. Chapters 25 and 26 are how a stack reaches a machine that never built it. Chapters 30 to 32 are how a package enters EESSI and how a site stack is placed next to it. Chapter 27 is four failures in full. Chapters 39 to 47 are Part 2: ReFrame, the fourth kind of testing from chapter 23, written as its own course.

Where this runs

Not only on a supercomputer, and the book is careful about the difference.

EasyBuild is a Python package. It installs with pip into a virtual environment, needs a modules tool and a compiler, and builds software on a laptop exactly as it does on a compute node. Several of the recordings in this book were made that way, and their provenance says so: the failing build in chapter 20 and the checks in chapter 21 ran on a workstation, with no cluster involved.

EESSI goes further. Its own installation instructions cover a laptop, and CernVM-FS ships a client profile for exactly that case, so a machine with no software stack of its own can mount one and load optimised modules in minutes. Chapter 26 does it on four different kinds of machine.

What changes on a large shared machine is the consequences, not the mechanism: many users, one tree, a scheduler, several microarchitectures, and a rebuild that costs somebody a night. Where a chapter is about those consequences it says so. Where it is about the mechanism, the mechanism is the same on a local machine, and the same commands run there while reading.

What this book requires

A browser.

EasyBuild’s own tutorial opens by asking for Linux, Python, an environment modules tool and a working EasyBuild install. That is a reasonable ask of someone who has already decided to use EasyBuild, and an unreasonable one of someone still deciding whether to.

Every sample here runs in the page. Edit one and the answer changes as the sample is typed. Press Escape anywhere and a playground opens: the same engine, not limited to the samples printed in the chapter.

Three of the four kinds work off tables exported from eb-stack at build time rather than retyped. So a page cannot drift from the framework’s own data.

A cross-check compares the browser’s answers against the binary’s on every build. A headless browser then checks that every sample on every page is live and answering.

The fourth reads an easyconfig into its evaluated model, and it is a reduced reader rather than the real parser. It follows single-line assignments and names what it skipped. Where the real parser is needed it says so instead of guessing.

That covers a lot, and not everything. Nothing in a browser can configure, compile or link, and nothing here pretends to.

Where a step needs a real toolchain the page shows a recording of a real run instead, with where and when it ran printed beside it. That turns out to be the right way to teach those steps. The operational skill is reading the log rather than watching the progress bar.

The chapters

One toctree. The part headings below are for reading; Shibuya’s single-page build parses the sidebar as one XML document, so three captioned toctrees break it.

Part 1. EasyBuild

Chapters 0 to 38: the recipe, the build, the stack.

Part 2. ReFrame

The fourth kind of testing from chapter 23, as its own course. CSCS teaches it as a webinar separate from getting started; the official docs split Tutorial, How Tos and Advanced Topics; EESSI walks a job script, then a site-specific test, then a portable one. The chapters follow that sequence.

Back matter

• Tutorial: read an easyconfig the way EasyBuild does
• Playground
• Chapter 1. An easyconfig is Python
• Chapter 2. A toolchain is a hierarchy
• Chapter 3. A pin is for one generation
• Chapter 4. The robot
• Chapter 5. A versionsuffix is part of the name
• Chapter 6. What “installed” means
• Chapter 7. A checksum is a claim about bytes
• Chapter 8. One easyconfig, many installs
• Chapter 9. A hook changes a file you are not reading
• Chapter 10. Build your own, without asking anyone
• Chapter 11. EESSI, somebody else’s stack
• Chapter 12. eb-stack ports a recipe and bumps one
• Chapter 13. How a build executes
• Chapter 14. What a build is allowed to see
• Chapter 15. Build dependencies are not dependencies
• Chapter 16. What a toolchain injects
• Chapter 17. Patches, and what a patch is relative to
• Chapter 18. Languages that bring their own package manager
• Chapter 19. What a module exports
• Chapter 20. Reading a failed build
• Chapter 21. Writing one from nothing
• Chapter 22. Writing an easyblock
• Chapter 23. Testing, and the four things it can mean
• Chapter 24. Standing where the build stood
• Chapter 25. Reading a stack off the wire: CernVM-FS
• Chapter 26. The same stack somewhere else
• Chapter 27. Four things that actually happened
• Chapter 28. Your first week on somebody’s stack
• Chapter 29. Extensions, and what an entry in exts_list inherits
• Chapter 30. An easystack is the list the bot will build
• Chapter 31. The bot is three processes and a comment
• Chapter 32. A site stack next to EESSI, not inside it
• Chapter 33. Inspect writes a work queue, not a recipe
• Chapter 34. Format, lint, and check are three different questions
• Chapter 35. A stack lock is not a package lock
• Chapter 36. A campaign finding is a typed failure, not a log
• Chapter 37. A wrapper is the range EasyBuild will not write
• Chapter 38. Four EasyBuild 5 surfaces the earlier chapters skipped
• Chapter 39. A ReFrame test is a class
• Chapter 40. The pipeline has six stages
• Chapter 41. The configuration file is the machine
• Chapter 42. A performance function is a number with a unit
• Chapter 43. Compile or run-only
• Chapter 44. Parameters multiply the suite
• Chapter 45. A portable test leaves the machine in the config
• Chapter 46. Software maps to tests
• Chapter 47. Reading a failed ReFrame run
• A path through this book
• Glossary
• What to remember
• Every exercise in this book
• References
• Afterword: when EasyBuild is the wrong tool

Who wrote this

Rohit Goswami, who maintains software stacks on a national machine, runs the same tools on a laptop, and wrote eb-stack, the updater and generator that emits the .eb files EasyBuild then builds. The samples on these pages borrow tables from that generator. They are not the generator, and they are not EasyBuild.