Image store internals
Image bytes live in a content-addressed store in packages/server/src/image-store/. This page explains how the code works. To choose and configure a store, read Image storage.
The interface
Section titled “The interface”getImageStore() returns a BlobStore with put, get, stat, delete and list. hashBytes() gives the sha256 of the stored bytes.
Which store you get comes from IMAGE_STORE, which store-config.ts parses. initializeImageStore() builds the store at start-up in both the server and the collector. A missing or invalid setting stops the process with one line that names the setting. It never prints a secret’s value.
Adapters
Section titled “Adapters”local.tskeeps files at<root>/sha256/<first two hex>/<64 hex>, with no extension. A write goes to<root>/tmp/, is flushed, is renamed into place, and then its shard directory is flushed. Building the store sweeps temporary files older than an hour.s3.tsandsigv4.tsare one implementation of the S3 interface and a small Signature Version 4 signer. Provider presets are data inS3_PRESETS. The project chose this over the AWS software development kit, which adds about twenty megabytes for the same six requests.gcs.tsuses the native JSON API with service-account sign-in. It does not use S3 interoperability keys.azure.tssupports Shared Key or a shared access signature token.ipfs.tsandcid-index.tspin through a Kubo remote procedure call node or Pinata, and read through gateways whose bytes are hashed. Theimage_store_cidtable maps thesha256to the content identifier, and it answersstatandlist.
The remote adapters share object-blob-store.ts:
- Keys are
[prefix]sha256/<2 hex>/<64 hex>. - A
putof a blob that is already stored renews its age with a server-side copy (or a metadata change) and sends no bytes. listfollows paging.
Composites
Section titled “Composites”composite.ts builds composites only from other stores, so they nest:
tieredis a bounded local cache, by recency, in front of another store. It reads through and writes through.mirroredwrites to all its stores and reads from the first that answers. Itslistis the union.fallbackreads a second store that it never writes to, lists or deletes from.images gctherefore cannot touch it.
Rules every adapter keeps
Section titled “Rules every adapter keeps”__testing__/blob-store-contract.ts runs these rules against every adapter:
- A malformed digest throws
TypeErrorbefore storage is touched. getandstatanswernull, anddeleteanswersfalse, for a missing blob.- Only a real “not found” answer counts as missing. A 403 is an error.
statandlistgive amodifiedAtMsthat a repeatedputmoves forward.
An S3 HEAD request gives whole seconds, and a listing gives milliseconds. The adapter truncates the listing so both agree.
Unit tests inject fetch, the clock and the file system, so the coverage run needs no Docker.
The emulator suites (*.emulator.test.ts) run the same contract against real servers in Docker: MinIO (cgr.dev/chainguard/minio), Azurite, fake-gcs-server and Kubo. Run them with:
yarn workspace server run test:emulatorsThat sets IMAGE_STORE_EMULATORS=1. The suites are skipped otherwise.
There is one known emulator limit. fake-gcs-server does not move updated on a metadata PATCH. Its renewal test is skipped there and covered against the unit fake.
Image bytes: the store and the database
Section titled “Image bytes: the store and the database”The store holds image bytes. The server and the collector must share the same store, which for local means the same directory (IMAGE_STORE_DIR).
image_hashis unchanged. It is the primary key and the public/image/direct/<hash>address. It is derived from the downloaded bytes, the source address and the extension. It is not a hash of the bytes served.image.content_sha256is the hash of the sanitised bytes, and the file’s name.image.byte_sizeis their length. A link-only row has length 0, no digest and no file.- Write order.
insertImagewrites the file before the row, because the row may sit in a long transaction. A rollback leaves an orphan file, whichimages gcremoves. A row never points to a missing file. Aputrenews the age of a file it reuses. - Read path. Queries select
CASE WHEN content_sha256 IS NULL THEN content END(db/image-columns.ts). Routes callwithImageBytesorloadImageBytes(image-store/image-bytes.ts). They take the bytes from the row when it has no digest, and from the file otherwise. A missing file is logged, and the request falls through to the ordinary redirect or 404 branches. Never selectimage.contentdirectly. Keepres.contentType(ext).send(buffer), so the ETag and 304 answers stay identical. IMAGE_BYTES_IN_DATABASE(defaulttrue) decides whether the database copy is still written. A trigger from migration 0022 clears a digest that stops matching its bytes. That covers the previous release writingcontentduring a rolling deploy. See Moving between stores.- The operator command is
src/bin/images.ts. See Moving between stores forexport,verify,import,gcandcopy. They all go through theBlobStoreinterface. - The
contentcolumn is dropped in a later migration, after production has run with the bytes switched off.