6.3.9 Remote Executor Environments and Isolation

A remote executor is more than a machine that happens to have the requested CPU and operating system. After 6.3.2 Remote Executor Matching has shown that an action can be placed on a compatible worker, you still need evidence that it ran in the promised environment and that the worker did not carry files, processes, credentials, or residue across an action boundary. Even a remote attempt with a terminal result does not prove those things.

Treat the executor as a sequence of boundaries

Do not report all remote-worker time as “execution.” A worker commonly has to make inputs available, prepare a filesystem view, start its isolation mechanism, run the command, capture outputs, and clean up. The portable protocol does not dictate this worker lifecycle, so name the implementation and version that does for your fleet.¹ 6.3.4 Remote Action Tracing supplies the action and attempt joins; this page supplies the worker-stage joins.

Stage	Evidence to retain	A failure here is not necessarily
CAS traversal and input fetch	action/input digests, storage endpoint or tier, cache state, timestamps	a scheduler or toolchain failure
Execution-root preparation	worker identity, input-view method, image/runtime identity, preparation duration	command runtime
Isolation startup	selected process, container, or VM mode and its configuration revision	proof that the command could read every input
Command execution	attempt identity, exit status, stdout/stderr, resource and network observations	output publication or cleanup success
Output capture and upload	output digests, upload/reachability evidence, timestamps	a clean execution root
Cleanup	deletion/reconciliation witness, leftover-process check, reusable-local-state disposition	harmless bookkeeping

The first three stages are particularly easy to hide in an aggregate “remote execution” span. One maintained local REAPI implementation, for example, keeps its CAS and action filesystem inside a long-lived Linux guest, selects packaged read-only runtime layers through platform properties, and creates a per-action sandbox in that guest.² That is a useful concrete lifecycle, not a portable REAPI promise. A different backend may materialize inputs and start its boundary differently.

Specify what the worker promises

Matching answers “which worker class may receive this action?” It does not make the class reproducible. Record a versioned environment contract for each class:

• the executor image or runtime release, operating system and architecture;
• installed runtime libraries and tools that are intentionally outside the declared input tree;
• certificates, locale, timezone, and other ambient configuration that can change behavior; and
• the isolation mode, network policy, and credentials or endpoints that the spawned command can reach.

The contract needs an identifier that is meaningful in evidence, not merely an image label that can move. The practical reason is visible in multi-platform remote execution: a service may route actions using properties for a requested isolation type and container image, while the Bazel platform still represents the execution system at a higher level.³ Unless the backend treats a property as a versioned image identity, the property alone does not prove the exact image release or its compatibility with a previously produced result.

Bazel's action API can explicitly set an action environment or request its default shell environment; those are inputs to the action contract.⁴ They are not a complete inventory of the worker's ambient state. An action that works only because an undeclared shared library, certificate, locale, or tool is present has a worker-environment dependency. Keep that dependency visible in the class contract. Repairing a rule's undeclared inputs or tools is the action author's work; 6.3.1 Making Actions Work Remotely explains how to route that repair instead of hiding it in fleet configuration.

When changing an image or runtime, test representative action classes against the old and new environment, including cache acceptance and a forced execution path. Then drain or roll out the worker population with the compatibility and recovery plan from 6.3.17 Remote Execution Upgrades. “Latest” does not tell a future incident investigator which runtime actually executed an action.

Four boundaries with different jobs

The word sandbox can make several independent protections sound like one. Keep these questions separate when you choose and test a worker design.

Boundary	Question it answers	It does not establish
Bazel action sandboxing	Does this action get a restricted filesystem view of its declared inputs and outputs?	A security boundary against a hostile process, or a fresh process/container/VM.
Process, container, or VM isolation	How strongly can the action process affect the worker host or another running action?	That the supplied inputs are complete, that network access is restricted, or that cleanup happened.
Credential and network policy	Which services can the command contact, and which authority can it use?	That the filesystem or process boundary prevents cross-action state.
Cleanup and reuse policy	What local state, processes, mounts, snapshots, and caches may survive an attempt?	That surviving state is safe for another action or tenant.

The local Docker-sandbox documentation illustrates the first distinction: its model gives each spawned action a fresh container and limits crossing the container boundary to declared inputs and outputs.⁵ It is a useful diagnostic model for an action contract, not evidence that a particular remote fleet has equivalent host protections, network policy, or cleanup. Likewise, local per-action input trees such as the symlink forests in 5.8.1 Symlink Forests explain how declared paths can be staged; they do not choose a remote worker's trust boundary.

Container and VM labels are also insufficient security conclusions. Stripe describes moving from a gVisor-plus-containerd design to Firecracker microVMs because its workloads needed a different security/performance trade-off and because filesystem-heavy workloads paid substantial overhead in the earlier design.⁶ That is a production case study, not a rule that microVMs are always necessary or that containers are always inadequate. Pick a boundary from the action trust level, required runtime features, startup and I/O cost, host exposure, and the evidence your implementation can emit.

Keep the case-study components attached to the boundary they actually change. Firecracker replaced the process-isolation boundary in Stripe's design; an LVM copy-on-write snapshot supplied an input root to that boundary. A VM snapshot, which may preserve guest memory and emulated device state, is another lifecycle mechanism again. Calling all three “the sandbox” hides separate startup, security, compatibility, and cleanup obligations. The input-root side—LVM, read-only images, writable layers, and CAS directory traversal—is developed in 6.3.10 Remote Input Presentation. Here, the durable lesson is that the selected container or microVM must have a versioned lifecycle and an observable teardown boundary of its own.⁶

Do not give a build command a credential merely because the worker needs one to fetch inputs or publish outputs. A worker that evaluates untrusted code must prevent that code from directly changing shared results or influencing other co-tenants; Stripe calls out cache-write authority as one such risk.⁶ For each worker class, test the command's reachable endpoints and effective credentials explicitly: allowed dependency service, disallowed metadata or control endpoint, cache-write path, and any logging/upload path. Keep secret material out of ordinary action inputs and outputs, where it could be recorded, uploaded, or reused as build data. Identity, storage visibility, queue limits, and cross-tenant guarantees form the larger system in 6.3.14 Multi-Tenant Remote Execution; one secure-looking container is only one layer of that system.

Make isolation evidence adversarial enough to matter

Run a small, versioned qualification suite per worker class and after a material image, runtime, or isolation-policy change. It should include the following probes:

1. Stage attribution: add a controlled delay to one preparation stage at a time and confirm the trace assigns it to that stage rather than command execution.
2. Filesystem and process leakage: have one action leave a distinctive filename or background-process marker, then verify a later action cannot observe it. Test failure cleanup as well as success cleanup.
3. Escape and authority: try to access a host-only path, prohibited network endpoint, and an operation reserved for the worker control plane. Preserve the denial or the observed authority—not just a final action failure.
4. Residue and reuse: repeat the probe after cancellation, worker restart, and an image transition. Record which local caches or snapshots are deliberately reusable and how they are invalidated or partitioned.

These are worker-operation tests, not a substitute for repairing a broken action contract or for a general container/VM hardening guide. 6.3.10 Remote Input Presentation chooses how remote inputs reach the worker; it does not prove that the process, VM, credentials, or residue are contained. This article's pass condition is narrower: the named executor environment is reproducible, its selected isolation and authority boundaries work as claimed, and no unaccounted-for prior action state crosses into the next attempt.