Engineering

Notes from building the systems.

We write about architecture, incidents, tradeoffs, and the work behind the products. Each article comes from a real engineering investigation.

Compute · Giovanni

We taught Canopy to use your GPU

A technical account of Canopy Host's heterogeneous compute model: native Apple and NVIDIA execution, capability-based placement, Cortex training, artifact verification, real hardware results, and the gaps still open.

Canopy Source · Giovanni

When 3,100 files became 300

An engineering investigation into silent Git provider diff limits and how Canopy Source records the difference between a complete comparison and the files a provider happened to return.

Forest · Giovanni

A Grove should know what it can run

The execution boundary behind Canopy Host is changing: requirements travel with work, Groves report capabilities, and placement records the decision. Here is what runs, what is gated, and what still lacks hardware or a dispatch path.

Reliability · Giovanni

When HEAD said 138 bytes and GET returned zero

How registry garbage collection left BuildKit cache blobs visible to metadata checks after their files were gone, and how that poisoned later cache exports in Forest.

Pipelines · Giovanni

How the Canopy Pipeline Graph made our pipeline incremental

How Canopy's Pipeline Graph makes incremental execution possible: execution identity, scoped source inputs, order-only dependencies, verified workspaces, shadow mode, and determinism checks.

Security · Giovanni

The package name was not the network boundary

A Cortex API build failure that looked like flaky package downloads, and turned out to be a CUDA dependency graph, a 2 GiB egress ceiling, an HTTP proxy boundary, and a model distributed across several hosts.

Pipelines · Giovanni

The pipeline is code. The execution is a graph.

Inside canopy.pipeline.ts and @canopy/ci: a synchronous authoring API, the Canopy Pipeline Graph, content identity, planning, and the boundary where execution begins.

Builds · Giovanni

Why a WORKDIR took five minutes

A technical account of the BuildKit performance incident in Forest: native snapshotting, a misleading benchmark, configuration precedence, and a separate one-hour PVC retention bug.

Forest · Giovanni

Forest, and the problem with pretending one machine is not enough

A technical account of Forest, the runtime behind Canopy Host: its workload model, Talos and Kubernetes layer, capacity mistakes, migration failures, and what production changed.

---