6.4.4 Worker State Isolation

A persistent worker deliberately survives after an action finishes. That reuse saves startup work, but it also creates a boundary that an ordinary subprocess does not have: request B can inherit memory, caches, temporary files, current directories, or other state left by request A. A green build is therefore not enough to qualify a worker. You need evidence that each request produces the same result regardless of which requests the process served before it.

6.4.3 Operating Persistent Worker Pools establishes whether reuse is worthwhile and whether the pool is stable. This article starts after that decision. Its question is whether the pool preserves the action contract across successive requests.

Separate the two leak paths

Worker isolation has two independent layers:

Boundary	What can cross it	What worker sandboxing changes
Request filesystem	undeclared source files, stale generated files, shared temporary files, writable caches	gives each singleplex request a sandbox directory containing its declared inputs
Long-lived process	static variables, compiler sessions, in-memory caches, environment mutations, working-directory state	nothing inside the process; the same worker can retain memory across sandboxed requests

Bazel's worker documentation makes this distinction explicit. In ordinary worker mode the action is not sandboxed by default. --worker_sandboxing gives each singleplex request a separate sandbox directory, but the tool can still leak information through internal state such as a cache.¹ Input digests included in work requests let a cooperative worker validate cached file data; they do not erase other retained state or prevent a cache from being keyed incorrectly.¹

This yields a useful diagnosis rule. If the failure disappears with worker sandboxing, suspect filesystem visibility or persistence first. If it survives sandboxing but disappears when the action runs without a worker, suspect in-process state. Neither observation proves the exact defect, but each removes a large class of candidates.

Do not confuse worker sandboxing with the general per-spawn sandbox model from 2.3.2 Sandboxing. An ordinary sandbox starts a new process for the spawn; a worker sandbox narrows the filesystem view of one request while the worker process remains the reused execution subject.

Build a sequence-sensitive reproduction

A state leak may remain invisible when every test begins with a fresh Bazel server or sends requests in one friendly order. Preserve the suspect pool and test sequences, not isolated labels.

Choose two small requests from the same observed worker pool:

• Seeder A introduces a distinctive value: a conflicting basename, package name, compiler option, generated symbol, or source content.
• Probe B must not depend on that value and has a known output from a non-worker or fresh-worker baseline.

Keep each action invariant between cohorts: every occurrence of A uses the same configuration, toolchain, arguments, and declared inputs, and the same is true for B. A and B may have different declared inputs from each other; that difference is often what exposes the leak. They must nevertheless have compatible startup dimensions that place them under the same WorkerKey.

Run at least these sequences:

B on a fresh worker
A, then B on the same worker
B, then A, then B on the same worker
repeated A/B sequences in randomized order

The dedicated request-isolation snippet turns the smallest deterministic pair into executable evidence. Its verification harness uses a dependency edge to order A before B, limits each mnemonic to one worker, compares the emitted PIDs, and checks stable result bytes separately. Under Bazel 9.1.0 it captured:

fresh B: clean
A -> B, worker filesystem: contaminated (same_pid=yes)
A -> B, sandboxed worker filesystem: clean (same_pid=yes)
A -> B, sandboxed worker memory: contaminated (same_pid=yes)
A -> B, standalone memory: clean
stable result digests: clean=2e22da2ab13713309ac75219e525b8e06ed02f3f1963b8feef203fa25827f93d contaminated=bea58e9872ac65173da889cd0da3efcbba133b34c561d87cb0f2a7bbcc031696

Constrain the test pool to one instance for the mnemonic and prove placement rather than inferring it from the flag. The execution-placement evidence configuration uses --worker_max_instances=EvidenceWorker=1; its worker cohort then builds two requests in one invocation. The verifier requires two worker execution records, exactly one verbose worker-creation identity, and equal process IDs emitted by both requests. That project is pinned to Bazel 8.0.1, so reuse the evidence pattern rather than its captured values and rerun it for the Bazel and ruleset versions you are qualifying.

Compare output bytes and required diagnostics, not only exit codes. Bazel's generic evidence gives you worker log paths, IDs, mnemonics, execution records, and lifecycle messages from --worker_verbose.¹ It does not promise to log the complete WorkerKey, request arguments, input digests, or request order. Capture those only when the concrete worker or a protocol test hook emits them, and include a per-request correlation marker. A singleplex request_id is zero, so it cannot distinguish successive requests by itself. Without process identity and request correlation, an apparent A -> B run has not proved that A and B used the same process.

The conflicting value should be deliberate. For a filesystem hypothesis, give an input or temporary filename to A that B must not see. For a process-state hypothesis, vary a setting that a faulty global cache might omit from its key. The test succeeds diagnostically when A -> B differs from fresh B and the order can be reproduced. Randomization is useful after the smallest failing pair exists; it is not a substitute for reducing the sequence.

Use a containment ladder

Contain the affected action class narrowly while the implementation defect is investigated. Move from the smallest intervention that addresses the observed boundary to a more expensive one only when the evidence requires it.

1. Enable supported singleplex worker sandboxing. For the repository's pinned Bazel 9.1.0, bazel help build --long documents --worker_sandboxing for singleplex workers and states that the dynamic strategy always sandboxes them. This isolates declared request inputs, at the cost of sandbox setup. It does not promise an in-process reset.
2. Require an explicit request reset. If retained memory is the cause, use a ruleset/tool version whose worker clears request-scoped state or correctly keys safe caches. The reset guarantee belongs to that worker implementation, not to the generic Bazel worker protocol. Rule authors should repair and test it through 4.4.4 Persistent Workers for Rule Authors.
3. Recycle at a documented boundary. Restarting a worker can contain a leak when a fixed implementation is not yet available. Define which action cohort is recycled, when, and how the restart is observed. Bazel's --worker_quit_after_build is documented mainly as a debugging and profiling control; it ends workers after a build, not after each request.¹
4. Fall back to a non-worker strategy. When correctness cannot be restored with a supported sandbox or reset, route the affected mnemonic to an isolated subprocess strategy. Bazel's generic strategy override is mnemonic-scoped. A finer cohort requires the ruleset to expose a distinct mnemonic or another documented configuration boundary. This gives up process reuse but preserves a valid action boundary while the ruleset or tool is repaired.

Avoid treating retries as containment. A retry may land on a clean process and turn an order-dependent defect into intermittent success. It neither identifies the retained state nor demonstrates that later requests are safe. Likewise, clearing every worker before saving logs and the failing order destroys the most valuable evidence.

Multiplex workers add concurrent request IDs and per-request sandbox support requirements. Qualify those separately in 6.4.5 Multiplex Worker Qualification; success with singleplex --worker_sandboxing does not establish multiplex isolation.

Prove the repair under both boundaries

Repeat the original minimal sequence with the chosen containment mode. Then run the randomized cohort long enough to exercise worker reuse and require:

• output equivalence with the fresh-worker and non-worker baselines;
• no undeclared file from one request visible to another;
• no result change when request order changes;
• worker and request evidence showing the intended mode actually ran;
• an explicit placement assertion that fails if Bazel selects a fallback instead of the intended sandboxed worker; and
• a tested non-worker fallback or rollback configuration.

For the pinned Bazel version, the command help verifies the generic singleplex sandbox control, but it does not guarantee that a particular ruleset resets its compiler session or caches correctly. Nor does Bazel promise one universal "unsupported sandbox" diagnostic: an ineligible worker strategy can allow a later configured strategy to run. Pin the Bazel and ruleset versions, test the concrete worker, and make the canary inspect execution records plus verbose worker creation evidence. Fail the canary when the selected runner or sandboxed worker identity is absent; do not treat a green fallback build as proof of the intended containment mode.

The snippet's negative placement canary does exactly that: worker,standalone produces a successful build for an action that is not worker-eligible, then the canary exits nonzero because the execution record contains no worker runner and verbose output contains no matching worker-creation evidence.

Promote only the smallest passing scope. A fleet-wide sandbox flag may be a reasonable default, but it is not proof that every worker is hermetic. Keep the sequence test as a regression check because a toolchain or ruleset upgrade can change cache keys, cleanup behavior, temporary paths, or sandbox compatibility.