When HEAD said 138 bytes and GET returned zero

There was one line in the failing build that eventually made the incident comprehensible:

ERROR: short read: expected 138 bytes but got 0: unexpected EOF

At first, it did not seem like much to build on. host-app builds were intermittently failing at build-image. Some failed at a COPY step with that error. Others appeared to run for roughly sixteen minutes and then reported no useful failure in the Host log. The same application could get through after we set build_cache_enabled=false.

We used that workaround three separate times. It was useful, and it was not a fix. It removed the remote BuildKit cache from the path, which let an affected build complete. More importantly, it made the cache a credible suspect.

The answer turned out not to be corrupted BuildKit state in the usual sense. It was a registry that could still describe a file after garbage collection had removed the file itself.

The decisive observation looked like this:

HEAD /v2/.../blobs/sha256:...  -> 200, Content-Length: 138
GET  /v2/.../blobs/sha256:...  -> 200, 0-byte body

The registry knew the size of a blob that no longer existed.

The workaround that narrowed the problem

Forest runs Dockerfile builds as Kubernetes Jobs in a build Grove. The worker starts BuildKit, imports a registry-backed cache when caching is enabled, builds the image, and exports the cache back to the registry. A cache miss is ordinary: BuildKit does the work and records new layers. A poisoned cache is different. It can make the attempt to reuse old work fail before the build reaches the useful part.

build_cache_enabled=false stopped affected builds from importing or exporting that remote cache. The builds then passed. That did not tell us whether the bad state was an individual local worker, BuildKit’s own content store, the cache manifest, or the registry. It did establish that the application source and Dockerfile were not sufficient to reproduce the failure on their own.

We also tried this on the build host:

docker buildx prune -a -f

It was a red herring. Forest’s builds do not use the host’s ordinary Docker daemon or its buildx cache. They run in Kubernetes Jobs inside the Grove. Pruning the host was a plausible response to a cache failure, but it was pruning a cache the failing build never consulted.

There was a separate BuildKit failure later that day involving # syntax=docker/dockerfile:1.7 and the Kata-VM sandbox. That failure was real, but it was deterministic and came from an external Dockerfile frontend that the sandbox could not hand off to. Removing the unnecessary syntax pin fixed that path. It was not the source of the short reads, and combining the two would have made both investigations less clear.

A blob that existed and did not exist

The registry was registry:2 with its default in-memory blob-descriptor cache. A descriptor is metadata about a blob: among other things, its digest and size. It makes existence checks cheap. In the normal case, a server that can answer a blob HEAD request with a digest and a content length can also serve the bytes.

The normal case stopped applying after registry garbage collection.

Our retention process deletes old manifests and then, in its scheduled weekly pass, runs the registry’s garbage collector. That pass had deleted 172 blobs. The important part is how it runs: the long-lived registry server continues serving requests, while a separate process runs this command inside the container:

docker exec canopy-registry bin/registry garbage-collect /etc/docker/registry/config.yml

The garbage collector scans registry storage and removes blob files that are no longer referenced at that instant. It does not coordinate with, or invalidate, the in-memory descriptor cache held by the server process. That server had been alive for about five weeks.

So garbage collection changed one layer of the registry’s state but not another:

We call that a ghost blob: a blob the registry reports as existing, with a plausible descriptor, but whose backing bytes have gone away.

That is more precise than saying the registry returned stale data. The two endpoints disagreed in a very particular way. HEAD returned 200 and the correct historical Content-Length: 138. A full GET returned 200 but no bytes. The metadata had survived garbage collection. The bytes had not.

How an efficient existence check poisoned a cache

BuildKit’s registry-cache exporter needs to decide whether each blob already exists at the remote registry. Uploading an identical layer again is needless work, so it asks the registry first. If the registry says a digest exists, BuildKit can skip the upload. That is the efficient and correct thing to do when the registry’s answer represents its storage.

In this incident, it did not.

A later build regenerated byte-identical material: base-image content and deterministic COPY layers are common examples. BuildKit asked whether the digest was already in the cache registry. The ghost descriptor made the existence check pass. BuildKit skipped the upload because, according to the registry, there was nothing to upload.

The registry then accepted a cache manifest that referred to that digest. Its manifest validation made the same underlying existence assumption. The manifest became the cache reference future builds imported, even though one of its blobs had no readable file behind it.

flowchart TD
  gc["Registry GC runs in a separate process"] --> deleted["Blob file deleted from storage"]
  deleted --> stale["Running registry descriptor cache still says present"]
  stale --> head["BuildKit HEAD existence check succeeds"]
  head --> skipped["BuildKit skips uploading the identical blob"]
  skipped --> manifest["Registry accepts cache manifest referencing ghost blob"]
  manifest --> imported["Later build imports that cache reference"]
  imported --> eof["Blob read returns zero bytes: unexpected EOF"]

The poisoned manifest is why this was not only a one-time missing file. Garbage collection made a descriptor stale. A subsequent export turned that stale answer into a durable cache reference. Later imports trusted the reference until one needed to read the missing content and received the short read.

Neither component needs a villain in this story. BuildKit assumed that a registry returning “this blob exists” meant the bytes existed. The registry assumed its in-memory descriptor cache still represented its storage. Those are both reasonable assumptions in isolation. Garbage collection changed storage underneath the registry process without making the second assumption true.

Why it looked less orderly than it was

The cache did have a second problem, but it was not the root cause of ghost blobs.

Forest originally identified a cache reference by (team, git_url). That prevents caches from crossing team boundaries, and it separates different repositories. It is incomplete for a monorepo with multiple Dockerfiles. host/api and host/app share a Git URL but build from different paths.

They therefore imported and exported the same remote-cache reference. One application’s export displaced the other application’s useful cache state. Usually that meant a cold miss or a less useful cache. During this incident, it also made failures appear to jump between two different applications instead of pointing cleanly at one poisoned reference.

The corrected identity is:

(team, git_url, build_path)

This matters for cache usefulness and for isolating failures. It did not make the registry claim a deleted blob existed. The stale descriptor cache did that. Keeping those causes separate made the remediation less tempting to overstate.

There was another misleading effect: a failed BuildKit process could write its final useful line and terminate before Host’s log-chunk path delivered that final chunk. That is why some of the roughly sixteen-minute attempts arrived without an informative error even though BuildKit had produced one. Missing evidence made the failures feel more random than their shared cache dependency really was.

The self-heal that was not running, and would not have healed this

We had already added heal_cache_ref() after a different cache problem: an interrupted export could leave a missing or truncated blob behind. Before importing a cache manifest, the worker fetched its descriptors and compared each blob’s HEAD Content-Length with the size declared in the manifest. A mismatch removed the bad cache reference, allowing the next build to proceed as a cache miss.

That was a sensible check for the failure it had seen. It did not cover a registry whose metadata and bytes disagreed.

In the ghost-blob case, the manifest declared 138 bytes and HEAD reported 138 bytes. The old integrity check saw a match. It was asking whether two pieces of metadata agreed, not whether the blob could be read.

There was a second, independent problem. The build-dockerfile worker image pinned on the Grove had been built on 8 September, before heal_cache_ref() existed. The source contained a self-heal that the failing Jobs were not running.

This is a smaller version of the deployment lesson from the earlier BuildKit incident. A repository change is not evidence of the image a particular Grove dispatches. In this case there were two gaps at once:

1. The original check trusted the same stale HEAD answer that BuildKit trusted.
2. The Grove’s pinned worker image predated even that insufficient check.

The test for the revised worker deliberately recreates the shape we found: HEAD reports the manifest’s real size, while a GET returns an empty body. A test that merely substitutes a 404 would miss the essential failure mode.

Repairing the invariants

The changes address different boundaries. They are complementary, not five descriptions of one fix.

Make registry existence reflect registry storage

The root-cause change was to disable the registry’s blob-descriptor cache:

REGISTRY_STORAGE_CACHE_BLOBDESCRIPTOR=none

The registry provisioning and reconciliation paths now set that environment variable. With the in-process descriptor cache disabled, an existence answer reflects the backing storage rather than metadata retained in a long-running process after GC. A later BuildKit exporter that asks whether a blob exists gets an answer that can safely decide whether to upload it.

The expected post-GC path is consequently boring. If GC has removed a blob, the registry answers that it is absent. BuildKit uploads the byte-identical content again before publishing a manifest that refers to it. The cache costs an upload on that export; it does not acquire a reference to content that cannot be imported later.

This addresses the coherence failure directly. Restarting the registry also cleared the stale cache immediately, but restarting is an operational repair, not a lasting guarantee that a future garbage-collection pass will not recreate the same state.

Separate deleting a manifest from reclaiming blob space

canopy-gc-registry-artifacts.sh no longer runs garbage collection inline after normal artifact-retention deletions. It removes or tombstones the manifest. Physical blob-space reclamation remains in the scheduled off-hours retention and GC process.

That change does not replace disabling descriptor caching. It removes an unnecessary daytime window in which a cleanup action and concurrent cache pushes could interact. A manifest deletion is the immediate retention decision. Reclaiming unreachable bytes is a separate operation with a different operational cost and risk profile.

Verify a byte, not a size

heal_cache_ref() now makes a ranged request:

Range: bytes=0-0

It considers the blob healthy only if one byte actually arrives. A correct Content-Length is no longer proof that an object exists.

flowchart LR
  oldHead["Old: HEAD Content-Length equals manifest size"] --> oldTrust["Assume healthy"]
  newGet["New: GET Range bytes=0-0"] --> byte{"Did one byte arrive?"}
  byte -->|yes| keep["Keep cache reference"]
  byte -->|no| remove["Delete poisoned cache reference"]

The ranged request is intentionally small. The worker does not need to redownload every cache blob before every build. It needs to distinguish a descriptor from readable content. Reading the first byte establishes the property that the old check lacked.

Give each Dockerfile its own cache identity

The worker now hashes the Git URL and build path together under the team namespace. host/api and host/app therefore have separate registry-cache tags even though they are in the same repository. This does not cure a ghost blob, but it prevents different Dockerfile builds from repeatedly displacing each other’s cache and makes a bad reference easier to localise.

Keep the final error after the pod exits

When BuildKit fails, the worker now extracts the final #N ERROR line and writes it into the Kubernetes termination-message result path as well as the ordinary log stream. The termination message survives pod completion reliably enough for Host to report the actual failure reason when the final log chunk does not arrive.

That change did not prevent this incident. It makes the next cache or sandbox failure less likely to begin as an unexplained wait.

Making the repaired worker the worker that runs

The live remediation had two phases.

First, we repaired the poisoned state. We restarted the registry, which cleared its in-memory descriptors; the previously ghost blob then returned 404 rather than 200 with a historical content length. We deleted the poisoned cache manifest. We recreated the registry with descriptor caching disabled and verified the effective container configuration.

Then we made the worker fixes live. We rebuilt forest/images/build-dockerfile from the current source, pushed the resulting image, and explicitly repinned the staging Grove’s build-dockerfile image to its new digest. Finally, we checked the effective image through the Grove image-plan path.

That last verification is not administrative detail. The real-byte check and termination-message propagation only protect builds once the Grove dispatches the rebuilt image. A source tree can contain the right shell function while a pinned worker continues to run an earlier one. The image-plan check closes the loop from source, to image, to the value the next build will actually use.

What changed

The short read was accurate. BuildKit expected 138 bytes because the cache manifest and the registry descriptor both said 138 bytes. The read returned zero because garbage collection had removed the file behind that descriptor.

The useful correction was to stop treating an existence answer as an abstract fact. In this system, “the blob exists” crosses a process boundary: BuildKit asks a registry server, the server may consult a descriptor cache, and the bytes live in registry storage that a separate garbage collector can change. The answer is only useful if those layers agree.

They agree now because the registry no longer holds a blob-descriptor cache across garbage collection. The worker also tests readable content before trusting a remote cache reference, separate Dockerfiles stop sharing a tag, and the failure path preserves the error that explains what actually happened.

The metadata surviving after the bytes were gone was the whole incident. Once we could see that contradiction directly, the rest of the failure chain became ordinary systems behaviour.