Reading a failed build

What you will be able to do

1. Read EasyBuild’s error block and say which of its six fields to use first.

2. Find the log for a failed run, and search it in the order that finds the cause rather than the fallout.

3. Reproduce a failing command in the environment it failed in.

Everything before this chapter was about what EasyBuild does when it works. This is the one that gets used, and it is a reading skill rather than a repertoire of fixes. The material here is a real failure, produced on purpose, small enough to run anywhere. A tiny archive, wobble-1.0.tar.gz, containing one shell script at wobble-1.0/bin/wobble. A recipe that installs it with the Tarball easyblock and then runs it:

this sample fails on purpose, and working out how is the next block

easyblock = 'Tarball'

name = 'wobble'
version = '1.0'
toolchain = SYSTEM

sources = [SOURCE_TAR_GZ]

postinstallcmds = ['%(installdir)s/bin/wobble --check-licence']

The error block

Before you read the next block

That recipe installs one shell script and then runs it. Write down, in order: which of the eighteen steps fails, and what the exit code is.

What happens

postproc, and 127.

postproc during it, so configure, build and install all succeeded. 127 is the shell’s code for a command that is not there: the file being run does not exist at the path the easyconfig named.

If the prediction was a licence error from --check-licence, that is the reading the flag invites and the exit code rules out.

What EasyBuild says when a shell command fails

$ eb wobble-1.0.eb
ERROR: Shell command failed!
    full command    ->  /scratch/software/wobble/1.0/bin/wobble --check-licence
    exit code       ->  127
    called from     ->  'run_post_install_commands' function in
                        .../easybuild/framework/easyblock.py (line 3371)
    working directory ->
                        /scratch/build/wobble/1.0/system-system/wobble-1.0/bin
    output (stdout + stderr)  ->
                        /tmp/eb-ctnn91ni/run-shell-cmd-output/wobble-96mpj6ai/out.txt
    interactive shell script  ->
                        /tmp/eb-ctnn91ni/run-shell-cmd-output/wobble-96mpj6ai/cmd.sh

ERROR: Installation of wobble-1.0.eb failed: shell command 'wobble ...'
failed with exit code 127 in postproc step for wobble-1.0.eb

Recorded: EasyBuild 5.3.1 from PyPI, local workstation, 2026-09-10; long scratch paths elided to /scratch

Six fields are presented. They are not equally useful. The order to read them is not the order they are printed.

The exit code, first. 127 is the shell’s code for “command not found”. It is not a licence problem nor a permissions problem; the command being run is absent. A single number has ruled out most of the search space. 126 indicates “found but not executable”. 2 usually signals a usage error. A make failure is typically 2 with the real cause further up.

Then the step. in postproc step indicates that the install already happened. The problem is not in configure, build or install.

Then the working directory The command ran in .../wobble-1.0/bin, which is one level deeper than the archive’s top. Nothing in the easyconfig requested that.

Why the start directory moved

EasyBuild does not treat the extraction directory as the build directory. It looks for the real top of the source tree. The rule is worth knowing exactly.org][An easyconfig is Python]]’s failure.

referenceHow the start directory is guessed

find_base_dir in easybuild/tools/filetools.py, verified against EasyBuild 5.3.1:

lst = get_local_dirs_purged()
new_dir = get_cwd()
while len(lst) == 1:
    new_dir = os.path.join(get_cwd(), lst[0])
    if not os.path.isdir(new_dir):
        break
    change_dir(new_dir)
    lst = get_local_dirs_purged()

It descends while there is exactly one entry, skipping dotfiles and a directory named easybuild.

The following table shows the mapping.

The archive contains	The start dir becomes
pkg-1.0/ with several entries	pkg-1.0
pkg-1.0/src/ and nothing else	pkg-1.0/src
pkg-1.0/bin/wobble and nothing else	pkg-1.0/bin
two top-level directories	the extraction directory itself

An explicit start_dir is resolved relative to that result rather than instead of it, which is why a start_dir that was right for a git checkout can be wrong for a release tarball of the same software. #+end_detail That rule (see How the start directory is guessed) represents the entire failure. wobble-1.0 contained exactly one entry, bin, which also contained a single entry. So EasyBuild descended twice, Tarball copied the contents of that directory into the install prefix, and the install tree is:

%(installdir)s/bin/wobble does not exist. %(installdir)s/wobble does. Exit 127 was telling the truth. Two ways to fix it, and they are not equivalent. start_dir = '' stops the descent, so bin/wobble is where the recipe expects; or the postinstallcmds line is corrected to the path that exists. The first changes the layout the module exports, which is what a user meets, so it is usually the right one.

Not the tree the easyconfig assumed

$ ls -la ebtest/software/wobble/1.0/
total 4
drwxr-xr-x 2 rgoswami rgoswami 60 Sep 10 07:06 .
drwxr-xr-x 3 rgoswami rgoswami 60 Sep 10 07:06 ..
-rwxr-xr-x 1 rgoswami rgoswami 26 Sep 10 07:06 wobble

Recorded: EasyBuild 5.3.1 from PyPI, local workstation, 2026-09-10

%(installdir)s/bin/wobble does not exist. %(installdir)s/wobble does. Exit 127 was telling the truth.

Two ways to fix it, and they are not equivalent. start_dir = '' stops the descent, so bin/wobble is where the recipe expects; or the postinstallcmds line is corrected to the path that exists. The first changes the layout the module exports, which is what a user meets, so it is usually the right one.

EB-Failure-2 — Read this log fragment

A log you have never seen, opened at the end and scrolled up:

== ... (took 4 min 18 sec)
== sanity checking...
  >> file 'lib/libfoo.so' or 'lib64/libfoo.so' found: FAILED
== FAILED: Installation ended unsuccessfully

1. Identify the step that failed and the steps that then succeeded.

2. The build executed make for four minutes and terminated with exit code 0.

3. What conclusions can be drawn from that, and what remains uncertain?

4. Specify the two subsequent locations to inspect, in sequence, and indicate what each will resolve.

>> running shell command:
 make -j 16
 [working dir: /scratch/build/foo/1.2/gcc-13.2.0/foo-1.2]
>> command completed: exit 0, ran in 4 min 12 sec

1. Which step failed, and which steps therefore succeeded?

2. The build ran make for four minutes and exited 0. What does that let you rule out, and what does it not?

3. Name the two places you would look next, in order, and say what each one would settle.

Solution

1. sanitycheck failed, so configure, build and install all succeeded: something was compiled and something was installed. The twelve steps before the check are not suspects.

2. It rules out a compile failure and a link failure of the thing make built. It does not tell you that make install put anything where the easyconfig expected, and it does not tell you that what got built is named libfoo.so: a library built as libfoo.so.1.2 with no development symlink passes make and fails this check.

3. First the install directory, which is still there whether the file exists under another name or in another directory. Then the install step’s own commands in the log, which settle whether the build system was told a prefix it disagreed with. Nothing in the configure output is worth reading until those two are answered.

The error block lists the command, the exit code, the cwd, and a path to out.txt. The output is not inlined. Open that file. Two logs exist, and identifying which appears on the page prevents confusion. A temporary log per eb invocation is announced as Temporary log file in case of crash /tmp/eb-*/easybuild-*.log and is removed when the run succeeds. A log per installation is named using easybuild-%(name)s-%(version)s-%(date)s.%(time)s.log, and on success it is copied into <installdir>/easybuild/ where it remains with the software. The line at the end of a failed run, Results of the build can be found in the log file(s)..., refers to the second log. eb --last-log prints the path to the most recent log. It globs the log path, retains only files owned by the current uid, and returns the newest file by modification time. On a shared build host it therefore finds your last log rather than the machine’s, and it implies --terse, which enables vim $(eb --last-log) to work. The reading order is often misunderstood. Start at the end and work upward. The last error usually represents fallout. Search for the step markers, not for the word error. Starting configure step and Running method configure_step part of step configure split the log into the eighteen sections described in How a build executes, and knowing the current environment section narrows the search. Search for a command, not a complaint. running shell command (or INFO running cmd in older logs) lists every command with its working directory and exit code. Those are the facts. Searching for error yields a compiler’s opinion of a header file.

The log, and how to search it

The error block names the command, the exit code, the cwd, and a path to out.txt. The output is not inlined. Open that file.

There are two of them, and knowing which one is on the page saves confusion.

A temporary log per eb invocation, announced as Temporary log file in case of crash /tmp/eb-*/easybuild-*.log, removed when the run succeeds. And a log per installation, named from easybuild-%(name)s-%(version)s-%(date)s.%(time)s.log, which on success is copied into <installdir>/easybuild/ and stays with the software. The line at the end of a failed run, Results of the build can be found in the log file(s)..., points at the second.

eb --last-log prints the path to the most recent one. It globs the log path, keeps only files owned by your uid, and returns the newest by modification time. So on a shared build host it finds your last log rather than the machine’s, and it implies --terse, which is why vim $(eb --last-log) works.

Then the reading order, which is the part people get wrong.

Start at the end and work upward. The last error is usually fallout.

Search for the step markers, not for the word error. Starting configure step and Running method configure_step part of step configure divide the log into the eighteen sections from How a build executes, and knowing which one the environment is in narrows everything.

Search for a command, not a complaint. running shell command (or INFO running cmd in older logs) finds every command with its working directory and exit code beside it, which is a list of facts. Searching for error finds a compiler’s opinion of a header file.

referencePatterns worth searching a build log for

From EasyBuild’s own troubleshooting guidance, with what each is good for:

Pattern	Finds
ERROR	EasyBuild’s own failures, capitalised
Error 1	a make rule that failed
error:	a compiler diagnostic, with the colon to exclude prose
failure	test-suite reports
not found	a missing library or header
No such file or directory	a path assumption, like this chapter’s
Segmentation fault	a compiler or a test binary crashing
bazel	Bazel’s own log, which is elsewhere

And the one to avoid: a bare error, which matches every occurrence of the word in every configure script’s prose.

The rule that matters more than the list: look for the first error, not the last. Later errors are usually consequences.

The two paths in the error block constitute the portion that merits altering your habits. out.txt is the command’s own output, complete:

Reproducing the failure

The cmd.sh script runs in the shell of the environment where the command was executed.

out.txt, which is the whole story here

$ cat /tmp/eb-ctnn91ni/run-shell-cmd-output/wobble-96mpj6ai/out.txt
/usr/bin/bash: line 1: /scratch/software/wobble/1.0/bin/wobble:
No such file or directory

Recorded: EasyBuild 5.3.1 from PyPI, local workstation, 2026-09-10

Executing it launches a shell whose environment originates from the sibling env.sh, with the failing command placed as the previous history entry. The loop consists of pressing up, editing, running, and repeating, using the environment that the build actually used rather than an approximation. This corresponds to What a build is allowed to see: the login shell differs from the build’s environment, and a command that succeeds in one and fails in the other provides no output until the environment is identified. To examine a failure before it occurs rather than after, eb --dump-env-script <easyconfig> creates <name>.env in the current directory, containing the module load lines for the toolchain and its dependencies, as well as the build environment defined by the toolchain object. Sourcing that file gives the shell the same environment as the build. This topic comprises the entire content of Standing where the build stood.

The generated cmd.sh, in full

$ cat /tmp/eb-ctnn91ni/run-shell-cmd-output/wobble-96mpj6ai/cmd.sh
#!/usr/bin/env bash
# Run this script to set up a shell environment that EasyBuild used to
# run the shell command
EB_SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
echo "# Shell for the command: '.../bin/wobble --check-licence'"
echo "# Use command history, exit to stop"
if [ "$#" -eq 0 ]; then
  bash --rcfile $EB_SCRIPT_DIR/env.sh -i
else
  BASH_ENV=$EB_SCRIPT_DIR/env.sh bash "$@"
fi

Recorded: EasyBuild 5.3.1 from PyPI, local workstation, 2026-09-10

Successful installation cleans up its build directory. Failed installation does not clean up its build directory, and this is deliberate. The mechanism is the step list. cleanup is the fourteenth of the eighteen steps, so a build that fails in configure never reaches it, whatever --cleanup-builddir says. For an autotools build, config.log remains in the build directory and contains the actual compiler invocations configure tried and their actual output. For CMake, CMakeFiles/CMakeOutput.log and CMakeFiles/CMakeError.log remain. Both logs contain information that the EasyBuild log does not. Nobody cleans the tree afterwards. A build path filling with failed attempts is a normal way for a shared build host to run out of disk. Three options change where a failure happens: --stop=<step> ends the run cleanly after a named step, so the run can stop at configure and allow inspection of its output. -x, the extended dry run, walks the whole procedure without performing any actions and reports what it would run. It ends with (no ignored errors during dry run) when nothing went wrong. Its limit is exactly the one from An easyconfig is Python: it never extracts anything, so a fault that depends on the contents of an archive is invisible to it. This chapter’s failure would pass a dry run. --trace is on by default in EasyBuild 5, and it causes the run above to print the working directory and the source checksums as it proceeds. Turning --trace off is possible but rarely advisable.

What is left behind, and what is not

A successful installation cleans up its build directory. A failed one does not, and that is deliberate: the wreckage is the evidence.

The mechanism is the step list. cleanup is the fourteenth of the eighteen steps, so a build that fails in configure never reaches it, whatever --cleanup-builddir says. What remains, for an autotools build, is config.log in the build directory, which contains the actual compiler invocations configure tried and their actual output. For CMake it is CMakeFiles/CMakeOutput.log and CMakeFiles/CMakeError.log. Both say things the EasyBuild log does not.

Nobody cleans the tree afterwards. A build path filling with failed attempts is a normal way for a shared build host to run out of disk.

Three options change where a failure happens:

--stop=<step> ends the run cleanly after a named step, so the run can stop at configure and go look at what it produced.

-x, the extended dry run, walks the whole procedure without doing any of it, and reports what it would run. It ends with (no ignored errors during dry run) when nothing went wrong. Its limit is exactly the one from An easyconfig is Python: it never extracts anything, so a fault that depends on the contents of an archive is invisible to it. This chapter’s failure would pass a dry run.

--trace is on by default in EasyBuild 5, and it is the reason the run above printed the working directory and the source checksums as it went. Turning it off is possible and rarely wise.

EB-Failure-1 — Four failures, four first moves

For each, say what you read first and what you would rule out with it.

1. ERROR: Shell command failed! with exit code 2 and in build step, and the output ends with make: *** [all] Error 2.

2. A build that ran for forty minutes and failed in sanitycheck, with a missing libfoo.so.

3. Couldn't find file thing-2.1.tar.gz anywhere, and downloading it didn't work either... Paths attempted (in order):...

4. A build that succeeds on your machine and fails on a colleague’s, same easyconfig, same EasyBuild version.

Solution

1. The Error 2 is make reporting that a rule failed, not the failure itself. Read out.txt and search upward for the first error:, which is the compiler’s. The exit code has told you the build system failed rather than EasyBuild, so nothing in the easyconfig’s structure is suspect yet.

2. The step, and then the install tree. Reaching sanitycheck means configure, build and install all succeeded, so the software built something and it is not what the recipe said it would be. Compare sanity_check_paths against what is actually in the install directory, which is still there. This is What “installed” means’s check doing its job, and the forty minutes are not wasted: the tree is evidence.

3. The list of attempted paths, in order, which is the whole diagnosis. It shows every local directory searched, then source_urls, then EasyBuild’s own mirror at sources.easybuild.io. If the URL you expected is absent from that list, source_urls is wrong; if it is present and failed, the upstream moved.

4. Neither log alone. This is a difference in environment or configuration, so start with eb --show-config on both and diff it, then compare the loaded modules. What a build is allowed to see’s point applies: the environments are constructed, and the two of you constructed different ones.

sourcesWhere this chapter comes from

The failing build, the error block, the log paths, out.txt and cmd.sh were produced on a workstation on 2026-09-10 with EasyBuild 5.3.1 installed from PyPI, using a one-file archive made for the purpose. No cluster was involved.

Upstream documentation, fetched 2026-09-10:

• interactive debugging of failing shell commands: the six fields, out.txt, cmd.sh and the eb-shell> prompt

• log files: the temporary log, the per-installation log, and the filename template

• tracing progress: what --trace adds, and that it is on by default in EasyBuild 5

• the extended dry run: what -x reports, and its closing line

• the tutorial’s troubleshooting page: the search patterns, the advice to read upward, and the note that a failed build keeps its build directory

• the generated option table: --last-log, --stop, --cleanup-builddir

Code, from EasyBuild 5.3.1 as installed: find_base_dir in easybuild/tools/filetools.py for the single-child descent, and find_last_log in easybuild/tools/config.py for what --last-log actually returns.

What to remember

• The exit code narrows the search before any log does: 127 is not found, 126 is not executable, 2 from make means a rule failed and the cause is higher up.

• The step in the error message rules out every step before it.

• find_base_dir descends while a directory has exactly one entry, so the start directory can differ from the archive’s top.

• A failed build keeps its build directory, and config.log or CMakeError.log in it says what the build system actually tried.

• cmd.sh reproduces the failing command in the environment it failed in; eb -x cannot see any fault that depends on the contents of an archive.