Reading a failed build

Read the error block first. Then the log, then the directory the command ran in. Reproduce that command.

A failed build is a reading problem. The log names a directory. That directory names the command.

Read the stage, then start_dir, then the pin. The error text is not the cause.

What you will be able to do

1. Read EasyBuild’s error block and say which of its six fields to use first.

2. Find the log for a failed run, and search it in the order that finds the cause rather than the fallout.

3. Reproduce a failing command in the environment it failed in.

Everything before this page was about what EasyBuild does when it works. That one is the one that gets used. It is a reading skill rather than a repertoire of fixes. The material here is a real failure. It was produced on purpose. It is small enough to run anywhere. A tiny archive, wobble-1.0.tar.gz, contains one shell script at wobble-1.0/bin/wobble. A recipe installs it with the Tarball easyblock. The recipe then runs it.

this sample fails on purpose, and working out how is the next block

easyblock = 'Tarball'

name = 'wobble'
version = '1.0'
toolchain = SYSTEM

sources = [SOURCE_TAR_GZ]

postinstallcmds = ['%(installdir)s/bin/wobble --check-licence']

The error block

Before you read the next block

That recipe installs one shell script and then runs it. Write down, in order: which of the eighteen steps fails, and what the exit code is.

What happens

postproc, and 127.

postproc during it, so configure, build and install all succeeded. 127 is the shell’s code for a command that is not there: the file being run does not exist at the path the easyconfig named.

If the prediction was a licence error from --check-licence, that is the reading the flag invites and the exit code rules out.

What EasyBuild says when a shell command fails

$ eb wobble-1.0.eb
ERROR: Shell command failed!
 full command -> /scratch/software/wobble/1.0/bin/wobble --check-licence
 exit code -> 127
 called from -> 'run_post_install_commands' function in
 .../easybuild/framework/easyblock.py (line 3371)
 working directory ->
 /scratch/build/wobble/1.0/system-system/wobble-1.0/bin
 output (stdout + stderr) ->
 /tmp/eb-ctnn91ni/run-shell-cmd-output/wobble-96mpj6ai/out.txt
 interactive shell script ->
 /tmp/eb-ctnn91ni/run-shell-cmd-output/wobble-96mpj6ai/cmd.sh

ERROR: Installation of wobble-1.0.eb failed: shell command 'wobble ...'
failed with exit code 127 in postproc step for wobble-1.0.eb

Recorded: EasyBuild 5.3.1 from PyPI, local workstation, 2026-09-10; long scratch paths elided to /scratch

Six fields are presented. They are not equally useful. The order to read them is not the order they are printed. The exit code, first. 127 is the shell’s code for “command not found”, not a licence or permissions problem. The command being run is absent. A single number has ruled out most of the search space. 126 indicates “found but not executable”. 2 usually signals a usage error. A make failure is typically 2. The real cause lies further up. Then the step. in postproc step indicates that the install already happened. The problem is not in configure, build or install. Then the working directory. The command ran in .../wobble-1.0/bin, which is one level deeper than the archive’s top. Nothing in the easyconfig requested that.

Why the start directory moved

EasyBuild does not treat the extraction directory as the build directory. It looks for the real top of the source tree. The rule is worth knowing An easyconfig is Python’s failure.

referenceHow the start directory is guessed

find_base_dir in easybuild/tools/filetools.py, verified against EasyBuild 5.3.1:

lst = get_local_dirs_purged()
new_dir = get_cwd()
while len(lst) == 1:
 new_dir = os.path.join(get_cwd(), lst[0])
 if not os.path.isdir(new_dir):
 break
 change_dir(new_dir)
 lst = get_local_dirs_purged()

It descends while there is exactly one entry. It skips dotfiles and a directory named easybuild. The following table shows the mapping.

The archive contains	The start dir becomes
pkg-1.0/ with several entries	pkg-1.0
pkg-1.0/src/ and nothing else	pkg-1.0/src
pkg-1.0/bin/wobble and nothing else	pkg-1.0/bin
two top-level directories	the extraction directory itself

An explicit start_dir resolves relative to that result. A start_dir suitable for a git checkout may be wrong for a release tarball of the same software.

That rule (see How the start directory is guessed) represents the entire failure. wobble-1.0 contained exactly one entry, bin. bin also contained a single entry. EasyBuild descended twice. Tarball copied the contents of that directory into the install prefix. The install tree is: %(installdir)s/bin/wobble does not exist. %(installdir)s/wobble exists. Exit 127 reported the condition. There are two ways to fix it. The ways are not equivalent. start_dir = '' stops the descent. bin/wobble is the location the recipe expects. Alternatively, the postinstallcmds line is corrected to the existing path. The first changes the layout the module exports. The layout is what a user encounters. The layout is usually the correct one.

Not the tree the easyconfig assumed

$ ls -la ebtest/software/wobble/1.0/
total 4
drwxr-xr-x 2 user     user     60 Sep 10 07:06 .
drwxr-xr-x 3 user     user     60 Sep 10 07:06 ..
-rwxr-xr-x 1 user     user     26 Sep 10 07:06 wobble

Recorded: EasyBuild 5.3.1 from PyPI, local workstation, 2026-09-10

%(installdir)s/bin/wobble is missing. %(installdir)s/wobble is present. Exit 127 indicates failure. Two fixes exist. The fixes differ. start_dir = '' stops the descent. bin/wobble matches the recipe expectation. postinstallcmds can be corrected to the existing path. The first fix changes the exported module layout. The exported module layout is what the user sees. The exported layout is usually correct.

EB-Failure-2 — Read this log fragment

A log you have never seen, opened at the end and scrolled up:

== ... (took 4 min 18 sec)
== sanity checking...
 >> file 'lib/libfoo.so' or 'lib64/libfoo.so' found: FAILED
== FAILED: Installation ended unsuccessfully

Identify the step that failed. Identify the steps that then succeeded. The build executed make for four minutes and terminated with exit code 0. The termination indicates that the overall build completed without error. Uncertainty remains regarding the cause of the earlier failure. The first location to inspect is the log file of the failed step. The first location will reveal the reason for the failure. The second location to inspect is the summary of the succeeding steps. The second location will confirm the successful completion of those steps.

>> running shell command:
 make -j 16
 [working dir: /scratch/build/foo/1.2/gcc-13.2.0/foo-1.2]
>> command completed: exit 0, ran in 4 min 12 sec

1. Determine the failed step.

2. Determine the successful steps.

3. The build executed make for four minutes and returned exit code 0.

4. Immediate failures in the make process are excluded.

5. Later runtime errors remain possible.

6. List the next two locations to examine, in sequence.

7. Indicate the outcome each location would resolve.

Solution

1. sanitycheck failed, so configure, build and install all succeeded: something was compiled and something was installed. The twelve steps before the check are not suspects.

2. It rules out a compile failure and a link failure of the thing make built. It does not tell you that make install put anything where the easyconfig expected, and it does not tell you that what got built is named libfoo.so: a library built as libfoo.so.1.2 with no development symlink passes make and fails this check.

3. First the install directory, which is still there whether the file exists under another name or in another directory. Then the install step’s own commands in the log, which settle whether the build system was told a prefix it disagreed with. Nothing in the configure output is worth reading until those two are answered.

The error block lists the command, the exit code, the cwd, and a path to out.txt. Open that file. Two logs exist. Identifying which appears on the page prevents confusion. A temporary log per eb invocation is announced as Temporary log file in case of crash /tmp/eb-*/easybuild-*.log. The temporary log is removed when the run succeeds. A log per installation is named using easybuild-%(name)s-%(version)s-%(date)s.%(time)s.log. On success the log is copied into <installdir>/easybuild/. The log remains with the software. The line at the end of a failed run is Results of the build can be found in the log file(s).... The line refers to the second log. eb --last-log prints the path to the most recent log. It globs the log path. It retains only files owned by the current uid. It returns the newest file by modification time. On a shared build machine it finds your last log rather than the machine’s. It implies --terse. It enables vim $(eb --last-log) to work. The reading order is often misunderstood. Start at the end and work upward. The last error usually represents fallout. Search for the step markers, not for the word error. Starting configure step and Running method configure_step part of step configure split the log into the eighteen sections described in How a build executes. Knowing the current environment section narrows the search. Search for a command rather than a complaint. running shell command (or INFO running cmd in older logs) lists every command with its working directory and exit code. Those are the facts. Searching for error yields a compiler’s opinion of a header file.

The log, and how to search it

The error block lists the command, the exit code, the cwd, and a path to out.txt. The output is not inlined. Open that file. There are two of them. Knowing which one is on the page saves confusion. A temporary log per eb invocation is announced as Temporary log file in case of crash /tmp/eb-*/easybuild-*.log. The temporary log is removed when the run succeeds. A log per installation is named from easybuild-%(name)s-%(version)s-%(date)s.%(time)s.log. On success the log is copied into <installdir>/easybuild/. The log stays with the software. The line at the end of a failed run is Results of the build can be found in the log file(s).... The line points at the second. eb --last-log prints the path to the most recent one. It globs the log path, keeps only files owned by your uid, and returns the newest by modification time. On a shared build machine it finds your last log rather than the machine’s. It implies --terse. vim $(eb --last-log) works. Then the reading order is the part people get wrong. Start at the end and work upward. The last error is usually fallout. Search for the step markers, not for the word error. Starting configure step and Running method configure_step part of step configure divide the log into the eighteen sections from How a build executes. Knowing which one the environment is in narrows everything. Search for a command rather than a complaint. running shell command (or INFO running cmd in older logs) finds every command with its working directory and exit code beside it. It is a list of facts. Searching for error finds a compiler’s opinion of a header file.

referencePatterns worth searching a build log for

From EasyBuild’s own troubleshooting guidance, with what each is good for:

Pattern	Finds
ERROR	EasyBuild’s own failures, capitalised
Error 1	a make rule that failed
error:	a compiler diagnostic, with the colon to exclude prose
failure	test-suite reports
not found	a missing library or header
No such file or directory	a path assumption, like this page’s
Segmentation fault	a compiler or a test binary crashing
bazel	Bazel’s own log, which is elsewhere

And the one to avoid: a bare error, which matches every occurrence of the word in every configure script’s prose.

The rule that matters more than the list: look for the first error, not the last. Later errors are usually consequences.

The two paths in the error block form the portion requiring habit changes. out.txt is the command’s own complete output.

Reproducing the failure

cmd.sh runs in the environment’s shell where the command was run.

out.txt, which is the whole story here

$ cat /tmp/eb-ctnn91ni/run-shell-cmd-output/wobble-96mpj6ai/out.txt
/usr/bin/bash: line 1: /scratch/software/wobble/1.0/bin/wobble:
No such file or directory

Recorded: EasyBuild 5.3.1 from PyPI, local workstation, 2026-09-10

Executing it launches a shell whose environment originates from the sibling env.sh, with the failing command placed as the previous history entry. The loop consists of pressing up, editing, running and repeating, using the environment that the build actually used rather than an approximation. This corresponds to The isolation policy. The login shell differs from the build’s environment. A command that succeeds in one and fails in the other provides no output until the environment is identified. To examine a failure before it occurs rather than after, eb --dump-env-script <easyconfig> creates <name>.env in the current directory. The file contains the module load lines for the toolchain and its dependencies. The file also contains the build environment defined by the toolchain object. Sourcing that file gives the shell the same environment as the build. That dump is the subject of Standing where the build stood.

The generated cmd.sh, in full

$ cat /tmp/eb-ctnn91ni/run-shell-cmd-output/wobble-96mpj6ai/cmd.sh
#!/usr/bin/env bash
# Run this script to set up a shell environment that EasyBuild used to
# run the shell command
EB_SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
echo "# Shell for the command: '.../bin/wobble --check-licence'"
echo "# Use command history, exit to stop"
if [ "$#" -eq 0 ]; then
 bash --rcfile $EB_SCRIPT_DIR/env.sh -i
else
 BASH_ENV=$EB_SCRIPT_DIR/env.sh bash "$@"
fi

Recorded: EasyBuild 5.3.1 from PyPI, local workstation, 2026-09-10

Successful installation cleans up its build directory. Failed installation does not clean up its build directory. The behavior is deliberate. The mechanism is the step list. cleanup is the fourteenth of the eighteen steps. A build that fails in configure never reaches cleanup, regardless of --cleanup-builddir. For an autotools build, config.log remains in the build directory and contains the actual compiler invocations configure tried and their actual output. For CMake, CMakeFiles/CMakeOutput.log and CMakeFiles/CMakeError.log remain. Both logs contain information that the EasyBuild log does not. No process cleans the tree afterwards. A build path filling with failed attempts is a normal way for a shared build machine to run out of disk. Three options change where a failure happens. --stop=<step> ends the run cleanly after a named step. The run can stop at configure and allow inspection of its output. -x, the extended dry run, walks the whole procedure without performing any actions and reports what it would run. The dry run ends with (no ignored errors during dry run) when nothing went wrong. The limit is exactly the one from An easyconfig is Python. The dry run never extracts anything. A fault that depends on the contents of an archive is invisible to the dry run. This page’s failure would pass a dry run. --trace is on by default in EasyBuild 5. --trace causes the run above to print the working directory and the source checksums as it proceeds. Turning --trace off is possible. Disabling --trace is rarely advisable.

What is left behind, and what is not

A successful installation cleans up its build directory. A failed one does not. That is deliberate. The wreckage is the evidence. The mechanism is the step list. cleanup is the fourteenth of the eighteen steps. A build that fails in configure never reaches it, whatever --cleanup-builddir says. What remains, for an autotools build, is config.log in the build directory. config.log contains the actual compiler invocations configure tried and their actual output. For CMake it is CMakeFiles/CMakeOutput.log and CMakeFiles/CMakeError.log. Both say things the EasyBuild log does not. Nobody cleans the tree afterwards. A build path filling with failed attempts is a normal way for a shared build machine to run out of disk. Three options change where a failure happens. --stop=<step> ends the run cleanly after a named step. The run can stop at configure and go look at what it produced. -x, the extended dry run, walks the whole procedure without doing any of it. It reports what it would run. It ends with (no ignored errors during dry run) when nothing went wrong. Its limit is exactly the one from An easyconfig is Python. It never extracts anything. A fault that depends on the contents of an archive is invisible to it. This page’s failure would pass a dry run. --trace is on by default in EasyBuild 5. It is the reason the run above printed the working

EB-Failure-1 — Four failures, four first moves

For each, say what you read first and what you would rule out with it.

1. ERROR: Shell command failed! with exit code 2 and in build step, and the output ends with make: *** [all] Error 2.

2. A build that ran for forty minutes and failed in sanitycheck, with a missing libfoo.so.

3. Couldn't find file thing-2.1.tar.gz anywhere, and downloading it didn't work either... Paths attempted (in order):...

4. A build that succeeds on your machine and fails on a colleague’s, same easyconfig, same EasyBuild version.

Solution

1. The Error 2 is make reporting that a rule failed, not the failure itself. Read out.txt and search upward for the first error:, which is the compiler’s. The exit code has told you the build system failed rather than EasyBuild, so nothing in the easyconfig’s structure is suspect yet.

2. The step, and then the install tree. Reaching sanitycheck means configure, build and install all succeeded, so the software built something and it is not what the recipe said it would be. Compare sanity_check_paths against what is actually in the install directory, which is still there. This is What “installed” means’s check doing its job, and the forty minutes are not wasted: the tree is evidence.

3. The list of attempted paths, in order, which is the whole diagnosis. It shows every local directory searched, then source_urls, then EasyBuild’s own mirror at sources.easybuild.io. If the URL you expected is absent from that list, source_urls is wrong; if it is present and failed, the upstream moved.

4. Neither log alone. The two logs differ in environment or configuration, so start with eb --show-config on both and diff it, then compare the loaded modules. The isolation policy’s point applies: the environments are constructed, and the two of you constructed different ones.

sourcesWhere this page comes from

The failing build, the error block, the log paths, out.txt and cmd.sh were produced on a workstation on 2026-09-10 with EasyBuild 5.3.1 installed from PyPI, using a one-file archive made for the purpose. No cluster was involved.

Upstream documentation, fetched 2026-09-10:

• interactive debugging of failing shell commands: the six fields, out.txt, cmd.sh and the eb-shell> prompt

• log files: the temporary log, the per-installation log, and the filename template

• tracing progress: what --trace adds, and that it is on by default in EasyBuild 5

• the extended dry run: what -x reports, and its closing line

• the tutorial’s troubleshooting page: the search patterns, the advice to read upward, and the note that a failed build keeps its build directory

• the generated option table: --last-log, --stop, --cleanup-builddir

Code, from EasyBuild 5.3.1 as installed: find_base_dir in easybuild/tools/filetools.py for the single-child descent, and find_last_log in easybuild/tools/config.py for what --last-log actually returns.

What to remember

• The exit code narrows the search before any log does: 127 is not found, 126 is not executable, 2 from make means a rule failed and the cause is higher up.

• The step in the error message rules out every step before it.

• find_base_dir descends while a directory has exactly one entry, so the start directory can differ from the archive’s top.

• A failed build keeps its build directory, and config.log or CMakeError.log in it says what the build system actually tried.

• cmd.sh reproduces the failing command in the environment it failed in; eb -x cannot see any fault that depends on the contents of an archive.