A build can finish successfully and still produce an image the destination machine cannot run.
That is a rather late place to discover an architecture mismatch. The builder did its job. The registry stored real bytes. The deployment has a Grove. Yet the image might target linux/amd64 while the machine reports a different platform. A scheduler can eventually say no, but by then we have already lost the more useful question: why did Host consider this destination suitable in the first place?
The same question gets harder with a native process or an accelerator. An image is not a native bundle. A Mac with Metal is not an NVIDIA worker with CUDA. A Grove that runs ordinary containers should not acquire a GPU Operator because another Grove may someday have a GPU. gpu: true is not enough information to distinguish any of these cases.
Forest began with a narrower job. Host held the desired state, Forest turned it into Kubernetes workloads, and Forest reported what happened. That remains a useful arrangement. The first Forest note argued that a one-node Grove should be allowed to stay a one-node Grove. The new work extends that argument: a Grove should also be allowed to be one kind of execution environment. The cost of supporting more kinds should fall on the Groves that actually need them.
What changed in October is the boundary between work and the place that runs it. Host now has an execution requirement. A Grove reports roles, substrate, platforms, capacity units, and, where present, accelerator inventory. Placement can retain both the requirement and its decision. Some compatibility refusals are written but switched off. The native path is substantial code, but its Mac path has not been demonstrated on real hardware. The committed GPU environment has never been applied. A compute job API exists, but its first Grove step still hits a build-bound dispatch contract.
Those differences in status are part of the design story, not footnotes to it.
The old question was too small
The original Forest path asked, in effect, “which Grove should receive this build or deployment?” Groves were already separate units of capacity, including build and runtime Groves, but the dominant execution substrate was Kubernetes. The controller watched Host’s intent and reconciled Pods, Jobs, Deployments, networking, and observed status. Kubernetes was doing real work: container scheduling, resource accounting, lifecycle, and reconciliation. There is no plan here to make it disappear.
The question became insufficient when the work itself stopped having one honest shape. A pipeline node might request a container image for linux/amd64. A native node might require a host process on darwin/arm64. An accelerator request might require NVIDIA and CUDA, or Apple and Metal. Isolation and trust matter as well. An untrusted step cannot simply be sent to a native host process because both places happen to have enough CPU.
A cluster name, or even a broad “build Grove” role, cannot carry all of that meaning. If we make the runtime infer it from a failed Pod or process launch, the platform knows the important fact only after it has made the decision.
The recent placement work gives the decision an input object. The relevant type is deliberately more specific than a machine label:
interface ExecutionRequirement {
family: ExecutionFamily;
resources: { cpuMilli: number; memoryBytes: number };
accelerators?: AcceleratorRequirement;
compatibility: {
platform: PlatformConstraint;
artifactKind: ArtifactKind;
changeKind: ChangeKind;
isolation: IsolationMode;
};
trust: 'trusted' | 'untrusted';
target?: { groveId: string };
}family answers which role the Grove must hold. platform is a set of acceptable OS and architecture pairs, with sources recording where that constraint came from. artifactKind distinguishes an OCI image, native bundle, host command, or no artifact. changeKind says which Forest wire operation the target must understand. isolation distinguishes a container, VM, and host process. Resource and accelerator requests are separate because a CPU and memory fit does not imply an accelerator fit. An explicit target does not waive compatibility.
The current compatibility checker tests Grove role, platform intersection, substrate, isolation, and trust. For example, NativeJob requires the native substrate; BuildJob requires Kubernetes. Untrusted work is restricted to a Kubernetes Grove. Accelerator eligibility goes through separate admission and policy code rather than that checker. These are actual branches of code, but their rollout state matters: host.deploy.forest_compatibility is an internal experimental feature whose registry default is disabled. With it off, placement records a requirement and logs an incompatibility that would have been refused. It does not globally refuse that work. Plan-time never-placeable checks and artifact compatibility refusals are under the same switch.
That separation lets us observe the new model before asking it to reject old decisions. It also means we cannot point to the checker and claim that every deployment is currently protected by it.
A placement is a decision worth keeping
placeExecution() is the Grove chooser for these paths. It may use an explicit configured Grove or the existing build-Grove selection. A new placement can write an execution_placements row with the subject, Grove, requirement, and decision. The decision includes the selected execution platform and, while enforcement is off, any incompatibility code. The persisted-placement feature is enabled by default. A retry reads its stored Grove instead of silently making a new choice. Requirement derivation happens when a placement is made, not when the stored one is read back.
That is a useful distinction. A deployment history that says only “ran on a Grove” is missing the reason. A retry that silently switches Groves can change architecture, substrate, cache locality, or artifact availability while appearing to be the same attempt. Persisting the choice makes the execution path inspectable and stable across a retry. It does not yet mean Forest has a complete capability-ranking scheduler. The chooser is still simple, and compatibility enforcement is a separate, disabled gate.
The image work follows the same principle. Grove-built artifacts now record a platform from the image configuration when it can be read. External image import resolves an OCI index for the destination Grove’s reported platforms, in its reported order, with a fallback when no usable Grove platform is available. Under the compatibility flag, a build_image node can fail if its pushed image platform disagrees with the node’s target, and runtime compilation can refuse an incompatible artifact. A platform is therefore something an artifact can retain, rather than a guess that vanishes after the build.
There is a related boundary in artifact storage. An artifact row now says whether its bytes belong to the OCI registry, npm registry, the content-addressed artifact store, or the evidence path. A consumer that needs an image reference has to require an OCI artifact. It cannot treat every digest as if it named a container. The object-store path verifies bytes by digest and provides scoped blob operations, but a missing bucket is a 503 for uploads while garbage collection quietly skips it. Staging found an earlier provisioning helper’s limit: its dry run was clean, then the MinIO client image could not be pulled. The revised helper creates the bucket through the S3 API using curl SigV4, without an image pull. The bucket was still unprovisioned on staging at the latest handoff. This is useful operational detail because the more kinds of work we add, the more tempting it is to call every output an “artifact” and then forget how it can actually be fetched and executed. Storage kind and platform are separate facts, and both have to survive the build.
It is tempting to describe this as CPU architecture selection. That misses most of the failures. A right-architecture image can still be sent to the wrong Grove role. A native command cannot become a Kubernetes Job through a matching arm64 string. A host process has a different trust boundary from an isolated container. A GPU may exist on a node but expose the wrong API or be unavailable for allocation. The requirement is a way to ask all of those questions before dispatch.
flowchart TD work["Pipeline node or desired workload"] --> requirement["Execution requirement"] grove["Grove roles, substrate, platforms, capacity"] --> placement["Host placement"] requirement --> placement placement --> decision["Persisted Grove and execution-platform decision"] decision --> controller["Kubernetes Grove: forest-controller"] decision -.->|native path, gated and unproven on a Mac| agent["Native Grove: forest-agent"] controller --> observed["Observed state reported to Host"] agent -.-> observed
The diagram is an architectural path through code, not a claim that every kind of work shown by the model can already traverse it in production.
The second substrate has to be honest
A native machine is not an oddly configured Kubernetes worker. Some work needs the machine’s own process model, hardware APIs, keychain, signing, or local device access. Apple Silicon and Metal make this particularly concrete. Packaging a process as a container does not conjure Metal into the ordinary Linux container substrate.
Forest now has a forest-agent binary alongside forest-controller. The controller remains the Kubernetes adapter. The agent has enrollment and persisted identity, a single-instance state lock, heartbeat and poll loops, capacity reporting, and an agent log ring. Its native execution code includes verified and bounded bundle download and extraction, code-signing checks, generation storage, a process supervisor with probes and restart backoff, secret handling, an nginx proxy manager, cancellation, timeout, and a restart journal for jobs. Native workloads and native jobs each require their own agent flag; both default off. Host also checks advertised change kinds before sending native workloads, and native workload emission requires FOREST_NATIVE_WORKLOAD_DISPATCH=true.
This is not a second unrelated scheduler. Host still owns intent and placement. The Forest side reports what the machine can do and what happened after it accepted a change. The substrate-specific part is the executor: a Kubernetes controller applies Kubernetes objects; a native agent manages signed bundles and host processes. The common question remains whether this Grove can execute this requirement.
The native path also has hard boundaries today. Host can compile a native workload spec and normalize agent status, but no Host writer yet creates the native workload generation that normal deployment would need. Native ingress is behind another default-off internal feature. Its design keeps public TLS at host nginx and sends plain HTTP over the WireGuard mesh to the native agent’s proxy; it does not make a Mac into a public edge by itself. Native job handling exists in the agent, but Host’s ordinary Grove exec driver remains build-bound. We have not run a real native Mac workload through this path. Tests of process supervision and wire behavior are valuable; they are not a hardware acceptance result.
There is a similar distinction inside Apple accelerator reporting. The agent can inspect system_profiler on darwin/arm64, report an Apple GPU with Metal, and use a reported working-set limit when available. The inventory describes unified memory as not memory-enforced and cooperative sharing. That is materially different from claiming a dedicated, enforceable GPU memory partition. The native slot and admission code can serialize a claimed accelerator slot. It does not prove production MLX or Metal execution on a deployed Mac.
An accelerator is a resource, not a Boolean
Forest’s accelerator inventory item contains a kind, vendor, model, APIs, memory size when known, memory kind, whether memory is enforced, and share mode. Placement units relate inventory IDs to allocatable and free counts, alongside CPU and memory. NVIDIA inventory is assembled from GPU Feature Discovery labels on Kubernetes nodes. The Apple agent reports its own inventory and a machine placement unit. These are descriptions of available capacity, not measured utilization of a running model.
A pipeline accelerator request likewise has a count, optional vendor, and optional minimum memory. Host normalizes NVIDIA to CUDA and Apple to Metal; an Apple request must have count one. Admission can check the reported units. Separately, an accelerator-policy feature can enforce a team’s entitlement, budget in accelerator-seconds, and concurrent use. That feature is internal and default-off. A requested GPU, a reported device, permission to consume it, and successful execution on it are four different facts.
The Forest renderer contains NVIDIA-specific behavior for accelerated jobs: GPU resource requests, nvidia RuntimeClass, and affinity from GPU Feature Discovery labels, including a memory threshold when requested. There are golden tests for the GPU smoke pipeline and the renderer’s affinity cases. The smoke example is deliberately CUDA-free: a CPU preparation step, an accelerated step that inspects the device, and a CPU evaluation step. This keeps the acceptance test focused on placement, device visibility, and work returning through the pipeline. The example’s checklist is still unchecked on real hardware.
The normal Grove module keeps this machinery optional. accelerators_enabled defaults off. Module tests assert that a default Grove plans no GPU Operator and that its controller ConfigMap and deploy triggers gain no accelerator keys. An accelerated worker requires the switch and a NVIDIA Talos schematic ID. When enabled, the module has explicit GPU Operator values with driver and toolkit disabled, a RuntimeClass, node patches, and an external bare-metal worker path. That opt-in behavior matters as much as the accelerated path itself. A routine Grove upgrade must not install privileged GPU infrastructure or restart a controller merely because the repository learned to describe a GPU.
The Grove that does not exist
The gpu-bare-metal-1 environment is in version control. It describes one control-plane VM and one bare-metal NVIDIA worker. It has not been planned or applied. There is no GPU machine to join it.
The committed environment is deliberately hard to turn on accidentally. Its apply-policy row names gpu-host-unassigned, a host that does not exist, and sets plan-only with zero permitted creates and updates. The OpenTofu configuration checks /etc/hostname against the same expected host, so a manual plan on a different box fails. There is no state-authority entry for it. A static test asserts those guards together. Activation requires assigning a real host in both places, preparing the Talos schematic and mesh address, reviewing a plan, applying manually, and then doing the GPU smoke acceptance run.
This is useful infrastructure code precisely because it is inert. We can review topology, module inputs, node policy, RuntimeClass, tests, and an acceptance procedure before buying or attaching hardware. We cannot turn a directory of OpenTofu into evidence that a GPU Grove is serving work. The infrastructure README still lists the driver and Talos extension combination, operator values, device injection, and machine-to-control-plane network path as unverified on real hardware.
A remote job should not own a runner process
A different October change addresses execution lifetime. Previously a pipeline runner could dispatch an exec step to a Grove and remain occupied until the Grove job finished. The remote job was durable; the waiting process was not the right place to represent its lifetime. Holding a runner slot for the duration also couples queue capacity to the longest remote steps.
The new detached path allows a supported exec step to dispatch a Grove job, write that job ID into the node’s awaiting_job_id, and return. When the Grove reports a terminal job, Host queues pipeline.node_resume in the same transaction as that terminal status. The resume reads the persisted job outcome and runs the normal node success or failure tail. A recurring sweep covers terminal jobs reached by another path or a lost resume, and times out an overdue live job after the step timeout and scheduling grace. Cancellation clears the wait, so a late report cannot revive a cancelled node.
sequenceDiagram participant Runner as Pipeline runner participant Host as Host persisted state participant Grove as Grove job Runner->>Host: Create Grove job row for eligible exec step Grove->>Host: Poll for assigned job Runner->>Host: Store node awaiting_job_id Note over Runner: Runner slot released Grove->>Host: Report terminal job state Host->>Host: Queue pipeline.node_resume with terminal update Host->>Runner: Resume node from stored job result Note over Host: Recurring sweep recovers missed resumes and timeouts
That is a change of ownership. The work’s lifetime belongs to persisted platform state, not to whichever process first dispatched it. The process may leave. Host can still explain what the node is awaiting and resume it later.
This needed the pipeline run-identity work that arrived just before it. A graph digest is an identity for work, not an identity for a particular attempt to execute that work. Previously two runs of the same graph could share node rows, and the second could silently inherit the first run’s success. Host now creates per-run node rows, carries the run ID through job payloads and idempotency keys, and cancels the run’s own Grove jobs. The old accidental reuse became an explicit GRAPH_DEDUP seed-time decision for eligible container nodes with suitable inputs and trust level. Native host-command nodes do not inherit that result. Without a run-specific node to await and resume, a terminal remote job could be attributed to the wrong execution of an identical graph.
Detached waiting is also controlled. host.pipeline_detached_wait defaults off, and the environment must be in PIPELINE_DETACHED_WAIT_ENVIRONMENT_IDS. It applies to eligible Grove exec steps. Build-group members, accelerator steps, native steps, and other node kinds still wait in place. The code is a durable boundary for a subset of execution, not a statement that all pipelines now detach.
A new surface should reuse the same truth
Compute jobs were added as a pipeline-run kind, not as a second batch scheduler. POST /compute/jobs builds a Canopy Pipeline Graph with exec nodes, runs the existing planner, creates a pipeline_runs row of kind compute, and uses the normal node machinery. Steps are marked reuse: 'never'. An explicit Grove pin is validated and stored as a placement before the run starts. Cancellation targets the run and its Grove jobs. This gives compute work the same place to apply graph validation, placement, capacity admission, budgets, accelerator policy, and cancellation as build work. It also gives later artifact lineage a place to attach without inventing a parallel job history.
The API, CLI, and MCP surface exist behind host.deploy.forest_compute_jobs, an internal feature that defaults off. There is a decisive unfinished edge: the Grove exec driver still rejects a run without buildId, and forest_build_jobs.build_id is still required. A compute run has no build. Its first real Grove exec step therefore cannot dispatch through that driver today. Tests exercise much of the service with a fake driver, but they do not convert this into an end-to-end compute execution path. Native steps and input artifacts are also rejected by the compute request parser rather than quietly accepted.
The same care applies to observability. Staff routes and CLI/MCP commands can inspect accelerator inventory, native Grove status, per-Grove placements, and execution placements when their internal observability feature is enabled. It defaults off. These surfaces answer questions that heterogeneity makes unavoidable: which Grove received the work, what substrate and platform it reported, what requirement placement recorded, and how fresh the capacity report is. They expose inventory and placement evidence. They do not yet prove accelerator utilization or a successful real GPU job.
What is running, and what is still a boundary
Forest already runs ordinary Canopy workloads on Kubernetes Groves. Host’s persisted placement, Grove roles and platforms, capacity reports, per-run pipeline node state, and explicit graph reuse are real foundations in the code. The corporate site itself is Canopy dogfooding. It is not evidence of customer deployment or of every new execution path.
The compatibility refusal, accelerator entitlement and budget, detached wait, compute-job API, native ingress, and staff execution-observability surfaces are implemented behind default-off or additional rollout controls. Native workload dispatch has its own environment switch and advertised-capability check. The native agent’s workload and job modes also default off. Where a flag is off, its code should be read as a controlled path, not as universal behavior.
The GPU Grove is code-only and inert. The GPU smoke run awaits hardware. The real Mac workload path is unproven on a Mac, and Host still lacks a normal native-generation writer. Compute jobs do not yet dispatch buildless work to a real Grove. Metal and MLX production execution are not established by the inventory and tests. Forest has not replaced Kubernetes, and a multi-node Grove alone does not establish high availability.
These are not all the same kind of unfinished. Some are feature gates on implemented paths. Some need a missing writer or dispatch contract. Some need physical hardware and an acceptance run. Treating them alike would make it hard to decide what to do next.
The larger direction is simple. Work should carry what it needs. A Grove should report what it can actually run. Host should keep the placement decision and Forest should report the observed result. When those facts disagree, the platform should be able to say so before a scheduler failure and explain it afterward.
None of that requires an ordinary Grove to stop being ordinary. A one-node Kubernetes Grove without accelerators remains a valid place to run the work it can run. The architecture gets more capable when each new capability has an explicit requirement, an explicit provider, and a boundary that does not quietly enlist every other Grove.