Reading a stack off the wire: CernVM-FS

CernVM-FS is a catalog plus a cache, not a copy of the tree. A miss is a network fetch.

The stack on the wire is a stratum and a catalog, not a tarball of the prefix.

CernVM-FS is a catalog plus chunks. The client mounts the catalog; objects arrive on read.

What you will be able to do

1. Verify the content-addressing and signing chain of a real repository with curl, python3 and sqlite3, from a machine with nothing mounted.

2. Open a live file catalog and say what each column and flag means.

3. Say what a cache miss does, in order, and which step each failure mode belongs to.

EESSI, somebody else’s stack referenced a software stack located at /cvmfs without defining /cvmfs. CernVM-FS provides a read‑only filesystem distributed via plain HTTP. Complete access is provided through a shell. Blomer et al. [2011] created it for LHC software. It handled several gigabytes per release. It handled hundreds of thousands of files. It handled an object count around 10^7 throughout its history. Blomer et al. [2015] is the later survey of that design at global scale. Meusel et al. [2015] describes the server backend. The backend supports union publish. The backend uses content-addressed objects. The backend implements the transaction that reads a stack off the wire. The transaction starts on Stratum 0. The primary challenge involves metadata rather than bulk data volume. Transport uses HTTP. Any web cache works with this transport. Hashes ensure authenticity across untrusted caches. Hashes preserve cacheability. All commands in this page execute on a standard machine lacking a CernVM-FS client. All commands in this page execute on a standard machine lacking a mount. All commands in this page execute on a standard machine lacking elevated privileges. All commands in this page execute on a standard machine lacking a specific account. The public EESSI repository responds to plain GET requests. This page performs verification instead of description. Each numeric value returned originates from the network on 2026-09-10.

base=http://s1eessieu1-cvmfs.openhtc.io/cvmfs/software.eessi.io

The manifest is the entry point

The whole of .cvmfspublished for software.eessi.io

$ curl -sS $base/.cvmfspublished
C7ca4fb6b5db9c1432243b47e306e46327cdeda19
B26624
Rd41d8cd98f00b204e9800998ecf8427e
D240
S23927
Gyes
Ano
Nsoftware.eessi.io
Xd61b84cdd59ae8a88814558553af1c33f7da6c75
H0db1e2ec784d23942d4678696dd94af71b5e5218
T1788943292
M536bcbab2288d369f4fe5af85e785d9520442a11
Yfe0b764e723f0fdba0522e72b59f86f66f5234d7
--
5647f8f42af43e4fb10ec4c3586fa5fbd823982d
<256 bytes of RSA signature>

Recorded: curl against the EESSI Stratum 1 CDN endpoint, local workstation, 2026-09-10; the signature block after the – separator is binary and elided

The format is thirteen single-letter fields, a separator, a 40‑character certificate fingerprint, and a 256‑byte signature. For a repository holding three generations of a software stack, the whole file is 603 bytes. The next ten letters are the ones this page uses. C is the content hash of the root file catalog. B is the catalog size, 26624 bytes. S is the revision number, 23927. The revision number is the count of times this repository has been published. D is the time‑to‑live of the root catalog in seconds, 240 (four minutes). The time‑to‑live is the interval after which a client will look for a newer revision. R is d41d8cd98f00b204e9800998ecf8427e. The value is the MD5 of the empty string. T is a Unix timestamp, 2026-09-09 08:41:32 UTC. X is the hash of the certificate that signed this. H, M and Y are hashes of the tag history, the repository metadata, and the reflog checksum. The same request against a different Stratum 1 returns the same bytes.

Two replicas, one answer

$ curl -sS http://aws-eu-central-s1.eessi.science/cvmfs/software.eessi.io/.cvmfspublished \
> | head -2
C7ca4fb6b5db9c1432243b47e306e46327cdeda19
B26624

Recorded: curl against two EESSI Stratum 1 endpoints, 2026-09-10

Replication means that a Stratum 1 mirrors immutable objects and includes a copy of the signed manifest. Replication does not make a Stratum 1 an independent service with its own state.

Content addressing, and how to check it

Two numbers, before you look

The manifest above says the root catalog hashes to 7ca4fb... and B26624. The next step is to fetch that object over HTTP and hash it.

Predict two things: what sha1sum of the downloaded bytes will be, and how many bytes the download will be.

What happens

sha1sum returns 7ca4fb6b5db9c1432243b47e306e46327cdeda19: exactly the name it was requested under. That is what content addressing means, and it is checkable without trusting the server that sent it.

The download is 5511 bytes, not 26624. The hash covers the compressed form, and B in the manifest is the uncompressed size. Decompressing the 5511 bytes gives exactly 26624, and what comes out is a SQLite database.

The manifest shows the root catalog 7ca4fb.... Objects are stored in data/, organized into 256 directories based on the first two hex digits of their hash.

Fetch by hash, then check the hash

$ h=7ca4fb6b5db9c1432243b47e306e46327cdeda19
$ curl -sS -o cat.obj $base/data/${h:0:2}/${h:2}C
$ ls -l cat.obj
-rw-r--r-- 1 user     user     5511 Sep 10 07:28 cat.obj

$ sha1sum cat.obj
7ca4fb6b5db9c1432243b47e306e46327cdeda19 cat.obj

$ python3 -c "import zlib; d=open('cat.obj','rb').read();
> u=zlib.decompress(d); open('cat.db','wb').write(u);
> print(len(u), 'bytes uncompressed')"
26624 bytes uncompressed

$ file cat.db
cat.db: SQLite 3.x database, page size 1024, ... database pages 26, ...

Recorded: curl and python3 against the EESSI Stratum 1, local workstation, 2026-09-10

Four facts fall out of those five commands. Each of them is a claim the documentation makes. This checks. The name is the hash. sha1sum of the bytes that came back equals the name they were requested under. Nothing in the transport had to be trusted for that to be checkable. The hash covers the compressed form. The documentation says “The cryptographic content hash refers to the zlib-compressed version of the file”. The numbers agree: 5511 bytes stored, hashing to the object name; 26624 bytes after decompression. ``B`` in the manifest is the uncompressed size. 26624, exactly what zlib.decompress produced. A catalog is a SQLite database. Not a format to reverse-engineer; a file that can be opened.

discrepancyThe trailing C, and where it is not written down

The object was fetched from data/7c/a4fb...cdeda19C, with a C appended. Without it:

$ curl -sS -o obj.bin -w 'HTTP %{http_code}\n' $base/data/7c/a4fb...cdeda19
HTTP 404

$ curl -sS -o obj.bin -w 'HTTP %{http_code}\n' $base/data/7c/a4fb...cdeda19C
HTTP 200

The server infrastructure appendix of CernVM-FS documents the 00..ff split and the content-addressed naming. It does not document this suffix. It was established here by trying both against the live repository. The outcomes were a 404 and a 200. Treat the suffix as a fact about the deployed format rather than a documented interface. Expect regular file objects to have no suffix. The .cvmfsdirtab fetch later in this page relies on that.

Inside the catalog

md5path_1 and md5path_2 form the two halves of an MD5, with paths omitted and only their hashes retained. Absolute paths are not stored to conserve space. MD5 hash values of the absolute path names are stored instead. A lookup consists of a hash and an index probe at any depth, but this prevents a catalog from being listed by prefix. The catalog field previous_revision is a hash. Each published revision names the one before it. The flags column encodes the type of each entry. The live histogram is small enough to read in full.

What a file catalog actually holds

$ sqlite3 cat.db '.tables'
bind_mountpoints catalog chunks nested_catalogs properties statistics

$ sqlite3 cat.db 'pragma table_info(catalog)' | cut -d'|' -f2 | tr '\n' ' '
md5path_1 md5path_2 parent_1 parent_2 hardlinks hash size mode mtime
flags name symlink uid gid xattr mtimens

$ sqlite3 cat.db 'select key, value from properties order by key'
TTL|240
last_modified|1788943289
previous_revision|fb61f4ecc41ee66b7db06be75cfe0e538fedb220
revision|23927
schema|2.5
schema_revision|8

Recorded: sqlite3 on the decompressed root catalog of software.eessi.io, revision 23927, 2026-09-10

Sixteen directories have flag 1. Three regular files have flag 4. Thirty-three symlinks have flag 12. Flag 4 is a regular file. Flag 8 is a symbolic link. The 33 entries with flag 12 are the catalog’s symlink count. Three entries have flag 3. The count is three, not two. The CernVM-FS docs name transition directories as flag 2 and symlinks as flag 8. The live catalog stores those as bitmasks: flag 3 is 1|2, flag 12 is 4|8. The statistics table resolves the numbers arithmetically. In the /versions/2025.06 catalog, self_nested equals 501. Exactly 501 entries have flag 3. In the same catalog, self_dir equals 755. The value 755 equals 501 plus 253 entries with flag 1 plus the single entry with flag 33, the nested catalog’s own root directory. Transition points are counted as directories. The three entries in the root catalog correspond exactly to the three names listed in the nested_catalogs table.

Every entry in the root catalog, by kind

$ sqlite3 cat.db 'select flags, count(*) from catalog group by flags order by 2 desc'
12|33
1|16
4|3
3|3

Recorded: sqlite3 on the same root catalog, 2026-09-10

The root catalog of the entire EESSI software stack is 26 kilobytes and contains minimal information. It records the top‑level layout and, for each EESSI version, stores a hash and a promise. Requesting any path under /versions/2025.06 causes the client to fetch that catalog first, verify it in the same manner, and then proceed. There is no limit to the nesting depth. A client that never enters a subtree does not download its metadata. The root catalog’s hash appears in the signed manifest. Each nested catalog’s hash appears in its parent. Each file’s hash appears in the catalog that lists it. The structure is a Merkle tree with a single signature at the top.

Where the repository is cut

$ sqlite3 cat.db 'select path, sha1, size from nested_catalogs order by path'
/versions/2023.06|e466e9788ded...|91136
/versions/2025.06|36562d2b1128...|247808
/versions/2026.06|dc9037b9aa27...|174080

$ sqlite3 cat.db "select name from catalog where flags=3"
2023.06
2025.06
2026.06

Recorded: sqlite3 on the same root catalog, 2026-09-10

referenceWhat a 26-kilobyte catalog accounts for

Every catalog carries a statistics table covering itself and its whole subtree. Read out of the root catalog of software.eessi.io at revision 23927, on 2026-09-10:

Counter	Value
self_dir	19
self_regular	3
self_symlink	33
self_nested	3
subtree_dir	8,381,258
subtree_regular	51,703,169
subtree_symlink	3,507,752
subtree_nested	40,070
subtree_chunks	47,045
subtree_file_size	6,808,996,874,851

Fifty-one million files and 6.8 terabytes, accounted for by a 26-kilobyte database that itself lists twenty-five entries. Forty thousand nested catalogs is what makes that work: a client mounting the repository fetches this one file, and the metadata for anything it does not open costs it nothing.

One level down, /versions/2025.06 is a 247-kilobyte catalog with 891 entries, of which 501 are transition points into further catalogs, and its own subtree accounts for 3.68 terabytes. Its properties table carries root_prefix /versions/2025.06 and revision 9291, and no TTL row: the time to live lives in the root catalog only.

Deeper again, for one CPU target, the split is visible in the shape of the two catalogs: x86_64/amd/zen4/modules has 3241 entries and no nested catalogs; x86_64/amd/zen4/software has 1609 entries and 1078 nested catalogs, one per installation directory, accounting for 114.7 gigabytes. So listing the modules is cheap and opening one package’s files is cheap, which is the pair of properties a software stack needs.

The cut point of a repository is a maintenance decision. EESSI’s file is in the repository. The hash resides in the catalog. The file is fetched like any other file. The suffix C is the catalog object. A regular file has no suffix. Other catalog types carry a letter of their own.

The cut points are declared, and the declaration is readable

The fourth software-layer line .../software/*/* cuts a catalog per installed package version. Opening one module’s files pulls in metadata for that package only. The line above it, .../reprod, is the reproducibility directory of Standing where the build stood. EESSI publishes the easyconfig, the environment script, and the easyblock source for everything in the stack. The items are catalogued separately. The items cost nothing until requested.

EESSI’s own .cvmfsdirtab, fetched by content hash

$ sqlite3 cat.db "select name, hex(hash) from catalog where flags=4"
README.eessi|A3494CC2E08BF1EA8AB6948019459E95A2AAC036
.cvmfsdirtab|1AA5539113BC1FFC8EF599B00FE16ABAECAC6597
.modulerc.lua|8E0885D42CDB423496AC22E6DCBE47EDA93C6BD3

$ h=1aa5539113bc1ffc8ef599b00fe16abaecac6597
$ curl -sS -o dirtab.z $base/data/${h:0:2}/${h:2}
$ python3 -c "import zlib;print(zlib.decompress(open('dirtab.z','rb').read()).decode())"
# Nested catalog for each EESSI version
/versions/*

# Compatibility layer paths:
/versions/*/compat/*/*
/versions/*/compat/*/*/var
/versions/*/compat/*/*/var/db/repos

# Software layer paths
/versions/*/software/*/*/*/*/software
/versions/*/software/*/*/*/*/software/*/*
/versions/*/software/*/*/*/*/modules
/versions/*/software/*/*/*/*/reprod
# Accelerator targets: .../accel/<vendor>/<type>/software
/versions/*/software/*/*/*/*/accel/*/*/software
...

Recorded: curl and zlib against the EESSI Stratum 1, 2026-09-10; the file is 1283 bytes, abridged here

The fourth software-layer line .../software/*/* cuts a catalog per installed package version. Opening one module’s files pulls in metadata for that package only. The line above it .../reprod is Standing where the build stood’s reproducibility directory. EESSI publishes the easyconfig, the environment script and the easyblock source for everything in the stack. These items are catalogued separately. They cost nothing until asked for.

EB-CVMFS-2 — Verify a repository yourself

Do this one rather than reading it. You need curl, python3 and sqlite3, and no privileges.

1. Fetch .cvmfspublished from a public Stratum 1 and read off C, B, S and D.

2. Fetch the root catalog object by its hash and check sha1sum against the name you requested.

3. Decompress it and compare the byte count with B.

4. Open it with sqlite3 and count the entries by flags.

5. Then answer: what would you have to compromise to serve a reader a modified file without them noticing?

Solution

Steps 1 to 4 are the transcripts in this page, against whichever repository you pick; the numbers will differ and the relationships will not. If sha1sum disagrees with the object name, you have found either a corrupted mirror or a much more interesting problem.

Step 5 is why the exercise exists. To change one file you need a catalog that names its new hash; to change a catalog you need its parent to name that new hash, recursively, up to the root; and the root catalog’s hash is in the signed manifest. So you need the repository’s private key, and then also to defeat the whitelist, which is signed by a different key whose public part reached the client out of band.

What you do not need to compromise: any Stratum 1, any proxy, or the network. Those carry immutable objects that the client checks for itself, which is why the transport can be plain HTTP.

The one place the chain stops is the local cache, which the client trusts once an object is in it.

EB-CVMFS-3 — Where would that number come from

The root catalog of a repository holding 51.7 million files is 26 kilobytes and lists twenty-five entries.

1. Where does the 51.7 million come from, if the catalog lists twenty-five things? Name the table and the column.

2. A client mounts the repository and runs ls /cvmfs/<repo>. How many catalogs does it fetch, and how many objects?

3. The same client then opens one file inside one package. What does it fetch, in order?

4. Somebody proposes flattening the repository into a single catalog for simplicity. Using the numbers, say what that would cost the client in part 2.

Solution

1. The statistics table, column subtree_regular. Every catalog carries counters for itself (self_*) and for everything below it (subtree_*), so a 26 kB file can account for a subtree it does not list.

2. One catalog, the root, and the objects for whatever ls needs, which is metadata rather than content: the entries are in the catalog it already fetched. The three version directories are transition points, and their catalogs are not fetched until something enters them.

3. The version catalog, then the CPU target’s software catalog, then the nested catalog for that installation directory, then the file’s own object, verified against the hash the catalog gave. Four fetches where a flat design would have needed one enormous one.

4. The client would download metadata for 51.7 million files and 8.4 million directories to answer ls on a directory with three entries in it. That is the argument for nesting, in one sentence, and it is why 40,070 catalogs is a feature rather than an accident.

The manifest’s bottom signature uses the repository key located on the publisher. The allowed signing certificates are listed in .cvmfswhitelist. The file is signed with a different key.

Two keys

Delivery of the master public key via its repository is not possible. Distribution of the public key for repository signature verification must use a channel other than CernVM-FS content distribution. For EESSI, the channel is the package cvmfs-config-eessi installing /etc/cvmfs/keys/eessi.io/eessi.io.pub. Whitelists have an expiration. Signatures are valid for 30 days by default. cvmfs_server resign must be run again before expiration. When the master key resides on a hardware token, the validity period is seven days instead of thirty. A compromised repository key is handled with a blacklist that precedes the white‑list. One limit is stated plainly upstream.

The whitelist is signed with a different RSA key, the **repository master key**. Only the public part of this master key needs to be distributed to
clients.

Blacklisted fingerprints prevent clients from loading future repository
publications by a corresponding compromised repository key, but they do
not prevent mounting a repository revision that had previously been
mounted on a client, because the catalog for that revision is already in
the cache.

New content is introduced at a single location, the Stratum 0. The machine is called the release manager or publisher. The release manager mounts the repository read‑only. The release manager overlays the repository with a union filesystem. Changes are written to a scratch area instead of the repository. cvmfs_server transaction opens that. publish converts the modifications into the storage format described above. abort discards them. Upstream describes the relevant property with three words: “Publishing is an atomic operation.” Everything a client observes is a published revision, indicated by the S field and a previous_revision hash. A client does not apply an older revision. When a particular revision of a file system resides in a client’s local cache, the client will no longer apply an older revision.

One writer

Stratum 1 is a standard web server. It uses the CernVM-FS server toolkit to keep a mirror of a repository served by Stratum 0. It is registered with cvmfs_server add-replica and kept current by cvmfs_server snapshot -a from cron. Squid appears twice in a deployment, with different jobs. One is a reverse proxy in front of a Stratum 1; the other is a forward proxy in front of clients. The recommended forward-proxy configuration includes one line that decides how a cluster behaves at the start of a job array.

Getting it to a thousand machines

Multiple simultaneous requests for the same object combine into a single request to a Stratum 1 server. Eight hundred nodes start the same job and request the same library in the same second. This produces one upstream request.

collapsed_forwarding on

The client is a FUSE filesystem mounted by autofs on first access and unmounted after a period of inactivity. It initially runs as root, then drops root privileges and changes the persona to the cvmfs user early in the file system initialization.

The local client

The client is a FUSE filesystem mounted by autofs on first access. The client is unmounted when no access occurs for a period. The client starts with root privileges. The client drops root privileges. The client changes to the cvmfs user early in the file system initialization.

referenceThe four caches, and their shipped defaults

From /etc/cvmfs/default.conf as shipped (cvmfs/cvmfs, devel branch) and the client documentation:

Layer	Setting	Default
kernel attribute cache	(FUSE)	1 minute
in-memory metadata cache	CVMFS_MEMCACHE_SIZE	not stated in the documentation
local disk cache	CVMFS_CACHE_BASE	/var/lib/cvmfs
disk cache soft limit	CVMFS_QUOTA_LIMIT	4000 megabytes
shared between repositories	CVMFS_SHARED_CACHE	yes
HTTP timeout, with a proxy	CVMFS_TIMEOUT	5 seconds
HTTP timeout, direct	CVMFS_TIMEOUT_DIRECT	10 seconds

Replacement is least-recently-used, and the threshold is not the quota: “When exceeding the cache quota, files are deleted until the overall cache size is less than or equal to the cache threshold. The cache threshold is hard-wired in this client to half of the cache quota.”

For a single machine the shipped configuration documents CVMFS_CLIENT_PROFILE=single for “laptops or fewer than 5 machines”, and EESSI’s own instructions raise the quota to CVMFS_QUOTA_LIMIT=10000.

Three independent reasons each suffice on their own. Objects are named by their content. Changing a file’s bytes changes its name. There is nowhere to write a modification of a file that is still that file. The signature covers the whole tree. The signed manifest fixes the root catalog hash. The root catalog hash fixes every nested catalog and every file hash below it. A local write breaks the chain that makes the repository verifiable at all. There is no client write path. Publishing is defined as a server-side atomic operation through a union filesystem on the release manager machine. The site-specific escape hatch cannot be a write. It is a symlink whose stored target is a variable reference, expanded by the client from its own configuration. Three of them appear in the live catalog above:

Why it is read-only

The signed entity equals that string. A site that redirects EESSI_HOST_INJECTIONS elsewhere changes the symlink target. The signature stays valid. The entire trick consists of this. The description appears in The same stack somewhere else.

Variant symlinks, as stored

$ sqlite3 cat.db "select name, symlink from catalog where flags=12
> and symlink like '\$(%'"
host_injections|$(EESSI_HOST_INJECTIONS:-/opt/eessi)
nvidia|$(EESSI_NVIDIA_OVERRIDE_DEFAULT:-/dev/null)
override|$(EESSI_LIB_OVERRIDE_DEFAULT:-/dev/null)

Recorded: sqlite3 on the root catalog of software.eessi.io, revision 23927, 2026-09-10

The signed data is that string. A site that points EESSI_HOST_INJECTIONS somewhere else changes what the symlink resolves to without invalidating a signature. It contains exactly the bytes it was signed with. That is the whole trick. It is The same stack somewhere else.

sourcesWhere this page comes from

Live evidence: every transcript in this page was produced on 2026-09-10 against http://s1eessieu1-cvmfs.openhtc.io/cvmfs/software.eessi.io and http://aws-eu-central-s1.eessi.science/cvmfs/software.eessi.io, at repository revision 23927, with curl, python3 and sqlite3 and no CernVM-FS client installed.

CernVM-FS documentation, fetched 2026-09-10. Note the URL form: the stable documentation uses directory URLs, so .../cpt-repo/ resolves and .../cpt-repo.html returns 404.

• overview: what CernVM-FS is, the feature list, the figure captions on verification

• repository creation and publishing: transactions, atomicity, .cvmfsdirtab, variant symlinks, resign and the 30-day signature

• implementation details: catalog columns and flags, MD5 paths, nesting, the Merkle-tree argument, the manifest field table, the compressed-form hash, cache management

• Stratum 1: replication and its reuse of the client’s integrity checks

• Squid: collapsed_forwarding and the recommended configuration

• client configuration and the parameter list

• security: the two keys, the whitelist, the blacklist limit, cache trust

• server infrastructure: the storage layout

• mount/default.conf on the devel branch: the shipped defaults, which the documentation does not restate

EESSI: the filesystem layer and native installation for the endpoints, keys and quota.

EB-CVMFS-1 — Six symptoms, six layers

For each, name the layer and the check.

1. A new module appeared on one node twenty minutes ago and is still invisible on another.

2. A job’s first minute is slow every time, on every node, and fast afterwards.

3. ls /cvmfs/software.eessi.io works; a file under it gives an input/output error.

4. A site with 800 nodes saturates its uplink whenever a job array starts.

5. Your laptop’s /var fills up.

6. A colleague proposes editing a file under /cvmfs to test a fix.

Solution

1. The root catalog’s time to live, D in the manifest, 240 seconds for this repository, plus the kernel attribute cache. The second node has not looked for a new revision yet. Check with cvmfs_config stat -v <repo> and compare the revision each node has against curl $base/.cvmfspublished.

2. Cache misses, then hits. If it recurs on every node on every run, the working set exceeds CVMFS_QUOTA_LIMIT and eviction is taking it to half the quota between runs, or the cache directory is being cleared.

3. Metadata resolved, content did not: the catalog is cached and an object fetch or its verification failed. Check the proxy chain, then fetch the object by hand the way this page does and compare sha1sum against the name.

4. Missing forward proxies, or proxies without collapsed_forwarding on. The requests are identical and simultaneous, which is exactly the case that line exists for.

5. CVMFS_CACHE_BASE is /var/lib/cvmfs by default and the quota is 4000 MB. Move the cache or lower the quota, and set CVMFS_CLIENT_PROFILE=single on a single machine.

6. Not possible, and the reason is the format rather than the permissions: content-addressed objects, a signed Merkle tree, and no client write path. What they need is a variant symlink or a site tree under host_injections.

What to remember

• A repository is verifiable from one signature the root catalog’s hash and every catalog names its children’s.

• Objects are named by the hash of their compressed bytes, so any client can check what it was given without trusting the transport.

• A catalog is a SQLite database keyed by the MD5 of the absolute path, cut into nested catalogs that are fetched on first access.

• Two keys: the repository key signs each revision, the master key signs the whitelist, and only the master public key has to reach clients out of band.

• A cache miss is a sequence, and the client trusts its cache once the object is in it.