Blog
Engineering notes.
Things we built, things that broke, and the work needed to make them less likely to break the same way again.
When HEAD said 138 bytes and GET returned zero
How registry garbage collection left BuildKit cache blobs visible to metadata checks after their files were gone, and how that poisoned later cache exports in Forest.
Read the articleHow the Canopy Pipeline Graph made our pipeline incremental
How Canopy's Pipeline Graph makes incremental execution possible: execution identity, scoped source inputs, order-only dependencies, verified workspaces, shadow mode, and determinism checks.
Read the articleThe package name was not the network boundary
A Cortex API build failure that looked like flaky package downloads, and turned out to be a CUDA dependency graph, a 2 GiB egress ceiling, an HTTP proxy boundary, and a model distributed across several hosts.
Read the articleThe pipeline is code. The execution is a graph.
Inside canopy.pipeline.ts and @canopy/ci: a synchronous authoring API, the Canopy Pipeline Graph, content identity, planning, and the boundary where execution begins.
Read the articleWhy a WORKDIR took five minutes
A technical account of the BuildKit performance incident in Forest: native snapshotting, a misleading benchmark, configuration precedence, and a separate one-hour PVC retention bug.
Read the articleForest, and the problem with pretending one machine is not enough
A technical account of Forest, the runtime behind Canopy Host: its workload model, Talos and Kubernetes layer, capacity mistakes, migration failures, and what production changed.
Read the article