Reading a stack off the wire: CernVM-FS

What you will be able to do

  1. Verify the content-addressing and signing chain of a real repository with curl, python3 and sqlite3, from a machine with nothing mounted.

  2. Open a live file catalog and say what each column and flag means.

  3. Say what a cache miss does, in order, and which step each failure mode belongs to.

EESSI, somebody else’s stack referenced a software stack located at /cvmfs without defining /cvmfs. CernVM-FS provides a read‑only filesystem distributed via plain HTTP. Complete access through a shell. Blomer et al. [2011] created it for LHC software, handling several gigabytes per release, hundreds of thousands of files, and an object count around 10^7 throughout its history. Blomer et al. [2015] is the later survey of that design at global scale. Meusel et al. [2015] is the server backend: union publish, content-addressed objects, the transaction that Reading a stack off the wire: CernVM-FS’s Stratum 0 is. The primary challenge involves metadata rather than bulk data volume. Transport uses HTTP, so any web cache works; hashes ensure authenticity across untrusted caches while preserving cacheability. All commands in this chapter execute on a standard machine lacking a CernVM-FS client, mount, elevated privileges, or a specific account, with the public EESSI repository responding to plain GET requests. this chapter performs verification instead of description, and each numeric value returned originates from the network on 2026-09-10. Apply this setting once, and subsequent steps follow automatically.

base=http://s1eessieu1-cvmfs.openhtc.io/cvmfs/software.eessi.io

The manifest is the entry point

The whole of .cvmfspublished for software.eessi.io

$ curl -sS $base/.cvmfspublished
C7ca4fb6b5db9c1432243b47e306e46327cdeda19
B26624
Rd41d8cd98f00b204e9800998ecf8427e
D240
S23927
Gyes
Ano
Nsoftware.eessi.io
Xd61b84cdd59ae8a88814558553af1c33f7da6c75
H0db1e2ec784d23942d4678696dd94af71b5e5218
T1788943292
M536bcbab2288d369f4fe5af85e785d9520442a11
Yfe0b764e723f0fdba0522e72b59f86f66f5234d7
--
5647f8f42af43e4fb10ec4c3586fa5fbd823982d
<256 bytes of RSA signature>

Recorded: curl against the EESSI Stratum 1 CDN endpoint, local workstation, 2026-09-10; the signature block after the – separator is binary and elided

The format consists of thirteen fields, each represented by a single letter, followed by a separator, a 40‑character certificate fingerprint, and a 256‑byte signature, totaling 603 bytes for a repository containing three generations of a software stack. Only four of these fields are used. C is the content hash of the root file catalog. B is that catalog’s size, 26624 bytes. S is the revision, 23927, which is the number of times this repository has been published. D is the time‑to‑live of the root catalog in seconds: 240, four minutes, which is the interval after which a client will look for a newer revision. R is d41d8cd98f00b204e9800998ecf8427e, which is the MD5 of the empty string. T is a Unix timestamp, here 2026-09-09 08:41:32 UTC. X is the hash of the certificate that signed this, and H, M and Y are hashes of the tag history, the repository metadata, and the reflog checksum. The same request against a different Stratum 1 comes back byte for byte identical:

Two replicas, one answer

$ curl -sS http://aws-eu-central-s1.eessi.science/cvmfs/software.eessi.io/.cvmfspublished \
>   | head -2
C7ca4fb6b5db9c1432243b47e306e46327cdeda19
B26624

Recorded: curl against two EESSI Stratum 1 endpoints, 2026-09-10

Replication means that a Stratum 1 mirrors immutable objects and includes a copy of the signed manifest, rather than being an independent service with its own state.

Content addressing, checked rather than believed

Two numbers, before you look

The manifest above says the root catalog hashes to 7ca4fb... and B26624. The next step is to fetch that object over HTTP and hash it.

Predict two things: what sha1sum of the downloaded bytes will be, and how many bytes the download will be.

What happens

sha1sum returns 7ca4fb6b5db9c1432243b47e306e46327cdeda19: exactly the name it was requested under. That is what content addressing means, and it is checkable without trusting the server that sent it.

The download is 5511 bytes, not 26624. The hash covers the compressed form, and B in the manifest is the uncompressed size. Decompressing the 5511 bytes gives exactly 26624, and what comes out is a SQLite database.

The manifest indicates that the root catalog is 7ca4fb.... Objects are stored under data/, split into 256 directories based on the first two hex digits of their hash:.

Fetch by hash, then check the hash

$ h=7ca4fb6b5db9c1432243b47e306e46327cdeda19
$ curl -sS -o cat.obj $base/data/${h:0:2}/${h:2}C
$ ls -l cat.obj
-rw-r--r-- 1 rgoswami rgoswami 5511 Sep 10 07:28 cat.obj

$ sha1sum cat.obj
7ca4fb6b5db9c1432243b47e306e46327cdeda19  cat.obj

$ python3 -c "import zlib; d=open('cat.obj','rb').read();
> u=zlib.decompress(d); open('cat.db','wb').write(u);
> print(len(u), 'bytes uncompressed')"
26624 bytes uncompressed

$ file cat.db
cat.db: SQLite 3.x database, page size 1024, ... database pages 26, ...

Recorded: curl and python3 against the EESSI Stratum 1, local workstation, 2026-09-10

Four facts fall out of those five commands, and each of them is a claim the documentation makes and this checks. The name is the hash. sha1sum of the bytes that came back equals the name they were requested under. Nothing in the transport had to be trusted for that to be checkable. The hash covers the compressed form. The documentation says “The cryptographic content hash refers to the zlib-compressed version of the file”. The numbers agree: 5511 bytes stored, hashing to the object name; 26624 bytes after decompression. ``B`` in the manifest is the uncompressed size. 26624, exactly what zlib.decompress produced. A catalog is a SQLite database. Not a format to reverse-engineer; a file that can be opened.

Inside the catalog

md5path_1 and md5path_2 form the two halves of an MD5, with paths omitted and only their hashes retained. To conserve space, absolute paths are not stored. Instead, MD5 hash values of the absolute path names are stored. a lookup consists of a hash and an index probe at any depth, but this prevents a catalog from being listed by prefix. previous_revision is a hash, creating a chain in the repository where each published revision references the preceding one. The flags column encodes the type of each entry, and the live histogram is sufficiently small to be read in full:

What a file catalog actually holds

$ sqlite3 cat.db '.tables'
bind_mountpoints  catalog  chunks  nested_catalogs  properties  statistics

$ sqlite3 cat.db 'pragma table_info(catalog)' | cut -d'|' -f2 | tr '\n' ' '
md5path_1 md5path_2 parent_1 parent_2 hardlinks hash size mode mtime
flags name symlink uid gid xattr mtimens

$ sqlite3 cat.db 'select key, value from properties order by key'
TTL|240
last_modified|1788943289
previous_revision|fb61f4ecc41ee66b7db06be75cfe0e538fedb220
revision|23927
schema|2.5
schema_revision|8

Recorded: sqlite3 on the decompressed root catalog of software.eessi.io, revision 23927, 2026-09-10

Sixteen directories (flag 1), three regular files (flag 4), thirty-three symlinks (12, which is 4 for a file and 8 for a symlink), and three entries with flag 3. The count is three, not two. The documentation states that a nested-catalog transition directory “appears as empty directory in the parent catalog with flags set to 2”, and that in the live catalog those entries carry flag 3. The statistics table resolves the numbers arithmetically: in the /versions/2025.06 catalog, self_nested equals 501 and exactly 501 entries have flags 3, while self_dir equals 755, which is 501 plus the 253 entries with flags 1 plus the single entry with flags 33, the nested catalog’s own root directory. Transition points are counted as directories. The three entries in the root catalog correspond exactly to the three names listed in the nested_catalogs table:

Every entry in the root catalog, by kind

$ sqlite3 cat.db 'select flags, count(*) from catalog group by flags order by 2 desc'
12|33
1|16
4|3
3|3

Recorded: sqlite3 on the same root catalog, 2026-09-10

The root catalog of the entire EESSI software stack is 26 kilobytes and contains minimal information. It records the top‑level layout and, for each EESSI version, stores a hash and a promise. Requesting any path under /versions/2025.06 causes the client to fetch that catalog first, verify it in the same manner, and then proceed. There is no limit to the nesting depth, and a client that never enters a subtree does not download its metadata. This explains why signing the manifest alone is sufficient: the root catalog’s hash appears in the signed manifest, each nested catalog’s hash appears in its parent, and each file’s hash appears in the catalog that lists it. The structure forms a Merkle tree with a single signature at the top.

Where the repository is cut

$ sqlite3 cat.db 'select path, sha1, size from nested_catalogs order by path'
/versions/2023.06|e466e9788ded...|91136
/versions/2025.06|36562d2b1128...|247808
/versions/2026.06|dc9037b9aa27...|174080

$ sqlite3 cat.db "select name from catalog where flags=3"
2023.06
2025.06
2026.06

Recorded: sqlite3 on the same root catalog, 2026-09-10

So the root catalog of the entire EESSI software stack is 26 kilobytes and knows almost nothing. It knows the top-level layout, and for each EESSI version it has a hash and a promise. Ask for anything under /versions/2025.06 and the client fetches that catalog first, verifies it the same way, and continues. There is no limit on the nesting, and a client that never enters a subtree never downloads its metadata.

Which is also why signing the manifest is enough: the root catalog’s hash is in the signed manifest, each nested catalog’s hash is in its parent, and each file’s hash is in the catalog that lists it. A Merkle tree, with one signature at the top.

The point at which a repository is cut is a maintenance decision, and EESSI’s is a file in the repository. Its hash resides in the catalog. It is fetched like any other file, and regular files have no suffix:

The cut points are declared, and the declaration is readable

Read the fourth software-layer line: .../software/*/* cuts a catalog per installed package version. Opening one module’s files pulls in metadata for that package and nothing else. The line above it, .../reprod, is Standing where the build stood’s reproducibility directory. EESSI ships the easyconfig, the environment script and the easyblock source for everything in the stack, catalogued separately so they cost nothing until requested.

EESSI’s own .cvmfsdirtab, fetched by content hash

$ sqlite3 cat.db "select name, hex(hash) from catalog where flags=4"
README.eessi|A3494CC2E08BF1EA8AB6948019459E95A2AAC036
.cvmfsdirtab|1AA5539113BC1FFC8EF599B00FE16ABAECAC6597
.modulerc.lua|8E0885D42CDB423496AC22E6DCBE47EDA93C6BD3

$ h=1aa5539113bc1ffc8ef599b00fe16abaecac6597
$ curl -sS -o dirtab.z $base/data/${h:0:2}/${h:2}
$ python3 -c "import zlib;print(zlib.decompress(open('dirtab.z','rb').read()).decode())"
# Nested catalog for each EESSI version
/versions/*

# Compatibility layer paths:
/versions/*/compat/*/*
/versions/*/compat/*/*/var
/versions/*/compat/*/*/var/db/repos

# Software layer paths
/versions/*/software/*/*/*/*/software
/versions/*/software/*/*/*/*/software/*/*
/versions/*/software/*/*/*/*/modules
/versions/*/software/*/*/*/*/reprod
# Accelerator targets: .../accel/<vendor>/<type>/software
/versions/*/software/*/*/*/*/accel/*/*/software
...

Recorded: curl and zlib against the EESSI Stratum 1, 2026-09-10; the file is 1283 bytes, abridged here

Read the fourth software-layer line: .../software/*/* cuts a catalog per installed package version. So opening one module’s files pulls in metadata for that package and nothing else. And the line above it, .../reprod, is Standing where the build stood’s reproducibility directory: EESSI ships the easyconfig, the environment script and the easyblock source for everything in the stack, catalogued separately so they cost nothing until asked for.

EB-CVMFS-2 — Verify a repository yourself

Do this one rather than reading it. You need curl, python3 and sqlite3, and no privileges.

  1. Fetch .cvmfspublished from a public Stratum 1 and read off C, B, S and D.

  2. Fetch the root catalog object by its hash and check sha1sum against the name you requested.

  3. Decompress it and compare the byte count with B.

  4. Open it with sqlite3 and count the entries by flags.

  5. Then answer: what would you have to compromise to serve a reader a modified file without them noticing?

Solution

Steps 1 to 4 are the transcripts in this chapter, against whichever repository you pick; the numbers will differ and the relationships will not. If sha1sum disagrees with the object name, you have found either a corrupted mirror or a much more interesting problem.

Step 5 is the point of the exercise. To change one file you need a catalog that names its new hash; to change a catalog you need its parent to name that new hash, recursively, up to the root; and the root catalog’s hash is in the signed manifest. So you need the repository’s private key, and then also to defeat the whitelist, which is signed by a different key whose public part reached the client out of band.

What you do not need to compromise: any Stratum 1, any proxy, or the network. Those carry immutable objects that the client checks for itself, which is why the transport can be plain HTTP.

The one place the chain stops is the local cache, which the client trusts once an object is in it.

EB-CVMFS-3 — Where would that number come from

The root catalog of a repository holding 51.7 million files is 26 kilobytes and lists twenty-five entries.

  1. Where does the 51.7 million come from, if the catalog lists twenty-five things? Name the table and the column.

  2. A client mounts the repository and runs ls /cvmfs/<repo>. How many catalogs does it fetch, and how many objects?

  3. The same client then opens one file inside one package. What does it fetch, in order?

  4. Somebody proposes flattening the repository into a single catalog for simplicity. Using the numbers, say what that would cost the client in part 2.

Solution

  1. The statistics table, column subtree_regular. Every catalog carries counters for itself (self_*) and for everything below it (subtree_*), so a 26 kB file can account for a subtree it does not list.

  2. One catalog, the root, and the objects for whatever ls needs, which is metadata rather than content: the entries are in the catalog it already fetched. The three version directories are transition points, and their catalogs are not fetched until something enters them.

  3. The version catalog, then the CPU target’s software catalog, then the nested catalog for that installation directory, then the file’s own object, verified against the hash the catalog gave. Four fetches where a flat design would have needed one enormous one.

  4. The client would download metadata for 51.7 million files and 8.4 million directories to answer ls on a directory with three entries in it. That is the argument for nesting, in one sentence, and it is why 40,070 catalogs is a feature rather than an accident.

The manifest’s bottom signature uses the repository key located on the publisher. The allowed signing certificates are listed in .cvmfswhitelist, which is signed with a different key:

Two keys

The master public key cannot arrive through the repository it authenticates. “The public key used to ultimately verify a repository’s signature needs to be distributed to clients through a channel different from CernVM-FS content distribution.” For EESSI that channel is a package, cvmfs-config-eessi, which installs /etc/cvmfs/keys/eessi.io/eessi.io.pub.

Whitelists expire. “Signatures are only good for 30 days by default, so cvmfs_server resign must be run again before they expire”. The validity period is seven days rather than thirty when the master key is on a hardware token. A compromised repository key is handled with a blacklist that “has precedence over the white-list”. One limit is stated plainly upstream:

The whitelist is signed with a different RSA key, the **repository master key**. Only the public part of this master key needs to be distributed to
clients.

Blacklisted fingerprints prevent clients from loading future repository
publications by a corresponding compromised repository key, but they do
not prevent mounting a repository revision that had previously been
mounted on a client, because the catalog for that revision is already in
the cache.

New content is introduced at a single location, the Stratum 0, on a machine referred to as the release manager or publisher. The publisher mounts the repository read‑only and overlays it with a union filesystem, causing changes to be written to a scratch area instead of the repository. cvmfs_server transaction opens that, publish converts the modifications into the storage format described above, and abort discards them. Upstream describes the relevant property with three words: “Publishing is an atomic operation.” everything a client observes is a published revision: the S field, accompanied by a previous_revision hash, and a client does not apply an older revision. Once a particular revision of a file system resides in a client’s local cache, the client will no longer apply an older revision.

One writer

A Stratum 1 is a standard web server that uses the CernVM-FS server toolkit to create and maintain a mirror of a CernVM-FS repository served by a Stratum 0 server. It is registered with cvmfs_server add-replica and kept current by cvmfs_server snapshot -a from cron. Squid appears twice in a deployment with different jobs: a reverse proxy in front of a Stratum 1 and a forward proxy in front of clients. The recommended forward-proxy configuration includes one line that decides how a cluster behaves at the start of a job array.

Getting it to a thousand machines

Combines multiple simultaneous requests for the same object into a single request to a Stratum 1 server. Eight hundred nodes starting the same job request the same library in the same second, resulting in one upstream request.

collapsed_forwarding on

The client is a FUSE filesystem that is mounted by autofs upon first access and is unmounted after a period of inactivity. It initially runs as root, then drops root privileges and changes the persona to the cvmfs user early in the file system initialization.

The local client

The client is a FUSE filesystem mounted by autofs on first access, and unmounted again when nothing has used it for a while. It starts as root and “drops root privileges and changes the persona to the cvmfs user early in the file system initialization”.

There are three independent reasons, each sufficient on its own. Objects are named by their content. Changing a file’s bytes changes its name, so there is nowhere to write a modification of a file that is still that file. The signature covers the whole tree. The signed manifest fixes the root catalog hash, which fixes every nested catalog and every file hash below it. A local write breaks the chain that makes the repository verifiable at all. There is no client write path. Publishing is defined as a server-side atomic operation through a union filesystem on the release manager machine. The site-specific escape hatch cannot be a write, and it is not. It is a symlink whose stored target is a variable reference, expanded by the client from its own configuration. Three of them are in the live catalog above:

Why it is read-only

The signed entity is that string. A site that redirects EESSI_HOST_INJECTIONS elsewhere alters the symlink target without invalidating the signature. That constitutes the entire trick, and it is described in The same stack somewhere else.

Variant symlinks, as stored

$ sqlite3 cat.db "select name, symlink from catalog where flags=12
>   and symlink like '\$(%'"
host_injections|$(EESSI_HOST_INJECTIONS:-/opt/eessi)
nvidia|$(EESSI_NVIDIA_OVERRIDE_DEFAULT:-/dev/null)
override|$(EESSI_LIB_OVERRIDE_DEFAULT:-/dev/null)

Recorded: sqlite3 on the root catalog of software.eessi.io, revision 23927, 2026-09-10

What is signed is that string. A site that points EESSI_HOST_INJECTIONS somewhere else changes what the symlink resolves to without invalidating a signature contains exactly the bytes it was signed with. That is the whole trick, and it is The same stack somewhere else.

EB-CVMFS-1 — Six symptoms, six layers

For each, name the layer and the check.

  1. A new module appeared on one node twenty minutes ago and is still invisible on another.

  2. A job’s first minute is slow every time, on every node, and fast afterwards.

  3. ls /cvmfs/software.eessi.io works; a file under it gives an input/output error.

  4. A site with 800 nodes saturates its uplink whenever a job array starts.

  5. Your laptop’s /var fills up.

  6. A colleague proposes editing a file under /cvmfs to test a fix.

Solution

  1. The root catalog’s time to live, D in the manifest, 240 seconds for this repository, plus the kernel attribute cache. The second node has not looked for a new revision yet. Check with cvmfs_config stat -v <repo> and compare the revision each node has against curl $base/.cvmfspublished.

  2. Cache misses, then hits. If it recurs on every node on every run, the working set exceeds CVMFS_QUOTA_LIMIT and eviction is taking it to half the quota between runs, or the cache directory is being cleared.

  3. Metadata resolved, content did not: the catalog is cached and an object fetch or its verification failed. Check the proxy chain, then fetch the object by hand the way this chapter does and compare sha1sum against the name.

  4. Missing forward proxies, or proxies without collapsed_forwarding on. The requests are identical and simultaneous, which is exactly the case that line exists for.

  5. CVMFS_CACHE_BASE is /var/lib/cvmfs by default and the quota is 4000 MB. Move the cache or lower the quota, and set CVMFS_CLIENT_PROFILE=single on a single machine.

  6. Not possible, and the reason is the format rather than the permissions: content-addressed objects, a signed Merkle tree, and no client write path. What they need is a variant symlink or a site tree under host_injections.

What to remember

  • A repository is verifiable from one signature the root catalog’s hash and every catalog names its children’s.

  • Objects are named by the hash of their compressed bytes, so any client can check what it was given without trusting the transport.

  • A catalog is a SQLite database keyed by the MD5 of the absolute path, cut into nested catalogs that are fetched on first access.

  • Two keys: the repository key signs each revision, the master key signs the whitelist, and only the master public key has to reach clients out of band.

  • A cache miss is a sequence, and the client trusts its cache once the object is in it.