Canopy Host’s own API pipeline had become long enough that rerunning every verification step on every change was painful.
That is not an unusual CI problem. A monorepo has several reasonable checks. Some install dependencies. Some compile. Some run a large test suite. Some inspect generated specifications. Eventually a one-line documentation change asks the same expensive questions again, even when the answer cannot have changed.
The tempting fix is to inspect the Git diff and skip tests that probably do not matter.
We deliberately did not build that.
“Probably” is the problem. A changed lockfile, executable bit, generated input, secret, image, dependency output, or remote service can make a step different even when a diff heuristic says its directory looks untouched. CI is where an optimization that is usually right is particularly expensive to get wrong.
The question became more demanding:
Can Canopy prove that this execution would be equivalent to one that already succeeded?
That distinction drives the system now running on our own Host API pipeline. We did not teach Canopy to skip CI steps. We taught it to prove when the work had already been done.
The interesting part is not a cache backend. It is that we had already spent time making canopy.pipeline.ts compile into an inspectable Canopy Pipeline Graph, or CPG. The previous engineering note, The pipeline is code. The execution is a graph., was about what the platform can know before execution starts.
Incremental execution is where that architectural choice began to pay us back.
The graph was already the model of the work
canopy.pipeline.ts is TypeScript, but it does not start containers when it calls step(). It declares nodes and edges. @canopy/ci compiles those declarations into a bounded graph that Host can validate and plan before it dispatches anything.
flowchart TD author["canopy.pipeline.ts"] --> compile["@canopy/ci<br/>compile"] compile --> cpg["Canopy Pipeline Graph"] cpg --> validate["Validate"] cpg --> scheduling["Scheduling"] cpg --> identity["Execution identity"] cpg --> policy["Policy / trust"] cpg --> reuse["Incremental execution"] cpg --> explain["Plan / provenance"] identity --> execute["Execute or reuse"] reuse --> execute scheduling --> execute
The CPG is not simply the representation Canopy executes. It is the representation Canopy reasons over.
Here, reasoning has a deliberately unromantic meaning. It is deterministic platform analysis over explicit declarations. The platform can see dependencies, source boundaries, requested egress, runtime requirements, data flow, scheduling constraints, and later the recorded outputs of an execution. It does not infer those things from a shell transcript after the fact.
That distinction matters because a node definition digest is not an execution identity. Consider an unchanged declaration:
step('test', {
image: 'registry.example.test/test@sha256:...',
run: 'bun test',
});The command and image can be identical while the repository has changed completely. A Git diff is not sufficient in the other direction either. It says what changed between commits, not what the command reads, which runtime it will receive, or whether an upstream workspace is the same thing it was before.
The graph gave us a place to make that distinction explicit.
What makes two executions the same?
For reusable exec and file nodes, Canopy computes an ExecutionKey. It is a canonical, schema-versioned hash of the actual inputs Canopy can establish before the node runs. The key is scoped to the team and repository, so a result is not casually shared across trust boundaries.
flowchart TD key["ExecutionKey"] key --> definition["node definition"] key --> sources["source fileset digest"] key --> dependencies["data-dependency execution keys"] key --> runtime["runtime identity"] key --> secrets["secret identity"] key --> scope["team and repository scope"] key --> schema["key schema version"]
The definition includes the operation declaration: command, digest-pinned image, declared environment configuration, egress declaration, and other meaningful operation fields. The runtime identity separately records the selected substrate and the pieces that can change what actually runs, including the worker or host-script identity, workspace-merge image where relevant, isolation tier, and sandbox and engine key versions.
Dependencies contribute their execution keys, but only for real data edges. A source checkout is a special case. Folding its whole-tree identity into every descendant would make scoped sources useless, because any change anywhere would invalidate every child. Descendants instead identify their own declared source files against the same checkout tree.
Secrets need their own treatment. Canopy records no secret value in a graph or execution key. For environment-variable secrets it derives an opaque version signal from the encrypted stored value. A Vault-backed secret without a version that Canopy can inspect makes the key unresolvable. An unresolved key is not a best guess. It means execute normally.
The implementation also has a real key schema version. It is currently version 2 because an earlier version did not include file mode in source identity. Two keys made under different schemas must not compare equal just because their hash payload happened to look similar.
This is intentionally stronger than hash(command + git diff). It is still not an attempt to prove arbitrary software is deterministic. It is a conservative identity for a concrete execution under a concrete runtime.
When any required part cannot be established, Canopy fails closed. An unavailable or truncated tree manifest, an unsupported source-tree entry, an unknown runtime, or an unknown secret version produces no reusable key. The step runs.
Reuse is an optimization. Correct execution remains the fallback.
sources is a statement about the work
Source identity is where the CPG becomes especially useful in a monorepo.
The Host API pipeline now has real, hand-traced declarations like this:
const specCoverage = step('spec-coverage', {
image: OPENAPI_DRIFT_TEST_IMAGE,
run: 'cd host/api && bun scripts/openapi-coverage.ts',
inputs: [src],
orderOnlyInputs: [specMergeUnit],
sources: ['host/api/**'],
reuse: 'auto',
});The important part is not the syntax. sources says that the result of this step is a function of this subset of the repository. Canopy obtains the checkout tree manifest and hashes the matched paths, Git blob identities, and file modes. A change from non-executable to executable is observable even if the file bytes did not change.
No sources declaration means the whole repository tree. That is deliberately conservative. A step without a proven narrower boundary can still reuse when the whole tree is identical. It just will not survive an unrelated monorepo change by pretending it knows more than it does.
The author must also state reuse: 'auto' or reuse: 'never' when declaring sources. That requirement is useful friction. A narrow source set is a correctness assertion, not a performance hint. It deserves an explicit decision about whether the author is willing to reuse work based on that assertion.
There are a few hard edges here. Gitlinks and submodules are not silently omitted from the fileset digest. Canopy cannot represent their checkout contents through its ordinary blob manifest, so it refuses to form an execution identity. The same applies to a missing or uncertain manifest. More execution is cheaper than reusing a result with an under-declared input.
The same pipeline also contains deliberately broad work. sdk-drift reads across host/api, clients/ts, cortex/api, and root scripts while it builds generated SDK types. We have not assigned it a convenient-looking but incomplete source boundary. It remains unscoped, which means whole-tree identity. The system is only as honest as the declarations we can defend.
Network access changes the default
Files are not the only source of variability.
An ExecOp that declares egress can contact a package registry, API, clock source, or anything else permitted by its network boundary. Matching source, dependency, runtime, and secret identity does not prove that an external response will be the same.
The rule is therefore simple:
networked ExecOp + no explicit reuse decision = reuse: neverAn author can explicitly choose reuse: 'auto' for a networked step, but has to own that claim. The Host API test step does this because it runs bun install --frozen-lockfile against its lockfile and declares the registry egress it needs. The lockfile and the relevant source tree are part of its identity. That is different from treating an arbitrary network call as deterministic by default.
@canopy/ci also exposes reuseAdvice(), a static check over the authored graph. It can point out an unscoped reusable step, an empty source set, a source declaration paired with reuse: 'never', or egress that quietly defaults to non-reuse. It does not inspect history or guess at a cache hit. It explains what the graph itself says.
Again, the graph is carrying facts. There is no heuristic deciding that a package registry feels safe today.
One graph needed two kinds of edge
The CPG initially had inputs, which means both “this must finish before I run” and “I consume its result.” That works until those two ideas diverge.
Our own pipeline had a concurrency problem on the build Grove. The pipeline needed these checks to proceed in sequence:
flowchart TD build["build"] --> merge["spec-merge-unit"] --> coverage["spec-coverage"] --> drift["sdk-drift"]
The first version used ordinary input edges to force that order. But spec-merge-unit does not read build output. spec-coverage does not consume the merge test output. They each read the checkout, or perform their own installation. The arrows had been doing two jobs, and one of them was false.
Once execution-key reuse existed, the false data edges had a cost:
unrelated build input changes
↓
build execution key changes
↓
downstream dependency key changes
↓
checks re-execute despite not consuming build outputWe added Ninja-style orderOnlyInputs to express the real relationship:
const specMergeUnit = step('spec-merge-unit', {
image: OPENAPI_DRIFT_TEST_IMAGE,
run: 'node --test scripts/merge-openapi.test.mjs',
inputs: [src],
orderOnlyInputs: [build],
sources: ['scripts/merge-openapi.mjs', 'scripts/merge-openapi.test.mjs'],
reuse: 'auto',
});inputs means a data dependency and a scheduling dependency. orderOnlyInputs means a scheduling dependency only.
That is not a naming cleanup. orderOnlyInputs participates in topological ordering, cycle detection, critical-path calculation, and dispatch readiness. It deliberately does not participate in node digesting, execution-key dependency folding, dependency reuse invalidation, or workspace chaining. There is no data to seed into a child, and a changed predecessor should not invalidate work that never read it.
The distinction made the graph more truthful. It also found a scheduler bug.
Terminal was not the same as released
A reused node is seeded as terminal immediately. At first, the scheduler treated an order-only parent being terminal as sufficient to release its child. That was one hop too shallow.
flowchart TD build["build is still executing"] -->|order only| merge["spec-merge-unit is reused and terminal"] -->|order only| coverage["spec-coverage"] build -. "old scheduler could release coverage" .-> coverage
spec-merge-unit had a terminal result, but its own order barrier, build, was not yet satisfied. Releasing spec-coverage preserved data correctness because neither check consumed build output. It violated the concurrency constraint that motivated the edges in the first place.
The fix introduced a separate concept of an order barrier being satisfied. A terminal node satisfies its barrier only when its full order-only ancestry is satisfied too. The scheduler now checks that recursively.
flowchart TD build["build terminal"] -->|order only| merge["spec-merge-unit terminal"] -->|order only| coverage["spec-coverage can dispatch"]
This is the kind of detail a job list does not naturally force you to model. Once the graph distinguishes data flow, execution identity, and scheduling order, the executor has to preserve all three meanings, including when a shortcut result appears before a normal scheduling transition.
A reused result is not necessarily reusable files
Artifact-free reuse is comparatively straightforward. If a previous successful node produced no workspace for a downstream node, Canopy can seed a successful reused result with its recorded exit code and output digest.
Artifact-producing nodes are harder. A row in pipeline_node_executions can say a prior execution succeeded and point at a workspace digest. That does not establish that the workspace is still present where this run will execute.
Before real reuse, Canopy pins the run to an execution target. It then verifies artifact-bearing candidates on that selected target:
flowchart TD key["execution identity matches"] --> success["previous success found"] success --> digest["workspace digest recorded"] digest --> physical["physical artifact verified"] physical --> retain["artifact retained against GC"] retain --> reuse["reuse result"] physical -->|"missing or uncertain"| execute["execute normally"]
The workspace storage path is checked against the real object:
| Target | Workspace storage | Verification |
|---|---|---|
| Grove | S3-compatible object storage | HeadObject for the recorded object |
The database is an index, not proof that storage exists. An object can be missing. Target resolution or a storage request can fail. Every one of those cases downgrades the candidate to a normal execution, and a downstream data consumer is re-evaluated accordingly.
That is why the implementation does not merely mark a matching row as reused during planning. Planning can find a provisional match. The engine makes it real only after the target has been pinned and artifact availability has been checked where the artifact must actually be used.
Retention is part of correctness
The physical check exposed another detail we initially got wrong.
Canopy retains a workspace when it verifies it for reuse, so garbage collection does not immediately remove something a new run has committed to consume. The Grove object-store path tracks that through a last_used_at value. The since-removed local-spool path had the same database bookkeeping, but its garbage collector made its decision from the physical archive file’s mtime.
Those were two definitions of “recently used.”
Commit bf8ac02d fixed the mismatch. Verifying a local workspace now also updates the physical file mtime the local collector actually reads. Both records agree that the workspace was just retained.
There was a second race on both backends:
sequenceDiagram participant GC as garbage collector participant Run as new pipeline run GC->>GC: snapshot old candidate Run->>Run: verify and retain workspace GC->>GC: re-check before deletion GC-->>Run: skip if freshness changed
For local storage, GC re-stats the file immediately before unlinking it. For Grove storage, it re-reads last_used_at immediately before deleting the object. If a reuse retained the artifact after GC took its snapshot, the artifact survives that sweep and is considered again later.
This is not the glamorous part of incremental execution, but it is the part that makes the word “safe” meaningful. Result identity alone is insufficient. Safe reuse also includes the lifecycle of the thing being reused.
Shadow before skip
We did not move directly from always executing to automatic reuse.
Environments have an incremental-execution mode of off, shadow, or on. In shadow mode, Canopy computes the same keys and determines what it would have reused, but still executes the real node. It then compares the previous and fresh semantically meaningful results: exit code, output digest, and workspace digest.
If those differ, the reuse model was wrong about something. Canopy records a shadow-divergence signal rather than silently treating the prediction as success.
The platform feature host.pipeline_incremental_execution adds an operational control above the environment mode. When enabled, a shadow environment is promoted to real reuse. When disabled, an environment that would use real reuse is downgraded to shadow, not to off. Operators can stop actual reuse globally while retaining visibility into what would have happened and whether it still agrees with real execution. --no-reuse is the per-run escape hatch for execution-key reuse when a particular build needs to run its nodes again.
This feature is internal and marked planned in Canopy’s feature registry. We are using it on Canopy’s own infrastructure. That is deliberately not a claim about customer rollout or a broad maturity label. The architecture is live; the interesting part of this note is how it decides reuse is safe, not a cherry-picked benchmark.
When the identity lies
Shadow mode tests the model when a prior result would have been reused. There is another useful case: two independent real executions can resolve to exactly the same ExecutionKey and still produce different outcomes.
That is evidence of a determinism failure in the identity model, even though neither run reused the other.
Canopy now emits pipeline.execution_identity_divergence when it sees this case. It compares exit code, output digest, and workspace digest before the execution-store upsert can overwrite the earlier record. The event does not fail an otherwise successful node. It makes the contradiction visible instead of letting whichever write arrived last erase the evidence.
That is the most interesting consequence of this work for us. The same model that decides when work can be avoided can also produce evidence that its own assumptions are incomplete.
The graph was worth the extra work
Building the CPG looked like extra work when a list of CI jobs would have been easier. Incremental execution is where that decision started paying us back.
Canopy now has enough structure to distinguish source contents from source names, a scheduling edge from a data edge, a prior success from a physically available workspace, and a cache prediction from evidence that the prediction was wrong. The platform can validate the graph, constrain its requested capabilities, schedule it, calculate its critical path, explain reuse, retain provenance, and detect a contradiction in execution identity using the same underlying model.
It does not mean every pipeline result is reusable. It should not. The conservative path is still to execute when Canopy cannot prove the equivalence.
But Canopy no longer only knows how to execute a pipeline. It has enough structure to decide, conservatively and explainably, when an execution does not need to happen at all.