6.3.1 Making Actions Work Remotely

Remote execution starts with an admission question, not an endpoint: can a worker run this class of actions from the contract Bazel sends, without relying on the client machine? A local green build cannot answer that question because the local machine may quietly provide files, tools, services, environment, or state that the action never declared.

This article begins after 6.1.3 Choosing Remote Cache or Execution has established that remote execution is worthwhile for a measured workload. The task now is to turn that workload into an evidence-backed inventory: which action classes are ready, which have a repairable contract gap, and which must remain intentionally local.

Readiness is a property of an action contract

An action is remote-ready when its declared inputs, tools, command, environment, outputs, and execution requirements are sufficient on a compatible worker. Bazel must know the inputs before execution so it can identify the action, transport its input closure, and reuse or execute it correctly.¹ A remote worker does not inherit the developer's checkout layout, shell startup files, PATH, background processes, or files installed during an earlier local action.

This is the infrastructure consequence of 2.3.1 Hermeticity, not a second definition of hermeticity. 2.3.2 Sandboxing already explains how a local sandbox makes undeclared files disappear from an action's prepared working directory. For remote admission, that sandbox result is one piece of evidence: it can expose a missing input, but it does not prove that the selected tool can run on the worker or that every required runtime surface is present. Bazel's official rule guidance therefore treats isolated actions, explicit dependencies, toolchain-provided tools, and host/execution-platform differences as separate compatibility concerns.²

Do not solve every gap by adding more software to the executor image. A stable, versioned worker runtime can be a legitimate part of the execution contract, but it must be named as such. Otherwise the image becomes an untracked copy of the developer host, and a passing action says little about portability. The design and isolation of that runtime continue in 6.3.9 Remote Executor Environments and Isolation.

Inventory action classes before changing strategy

Audit actions in groups that share a meaningful execution contract. A mnemonic is a useful starting key, but not always the whole key: the same mnemonic can select different tools, environments, or platforms under different configurations. Record representative target labels and configurations so the class can be reproduced.

Use several evidence surfaces because each answers a different question:

• 5.2.3 bazel aquery — Action Graph shows the post-analysis plan: command lines, declared inputs, tools, outputs, mnemonic, and configuration. It does not prove where an action ran or whether it succeeded.
• A profile shows which classes matter to the measured workload and helps avoid spending the audit only on visible but insignificant actions. It must not be used to exclude rare release, test, or code-generation classes from readiness coverage.
• An ordinary execution log records what Bazel actually executed and supports comparisons between environments. Keep it distinct from the experimental execution graph discussed in 5.4.6 Execution Log Graph. Comparing fresh execution logs can locate the first action whose inputs or outputs diverge and the actions that consume the divergence.³
• A preserved sandbox failure tests a concrete hidden-input hypothesis. The retained directory and exact failed command can be reconciled with the action declaration rather than bypassing isolation.⁴ The full local workflow is in 5.8.2 Diagnosing Sandbox Issues.
• Platform and toolchain resolution identify which executable was selected and what environment it is meant to run in. If the gap is in a rule's action declaration, route it to 4.4.3 Action Execution Contract; if selection chose a host-only tool, continue with 4.6.3 Toolchain Resolution.

Deep tracing inside a tool is an escalation, not a first step. Use 6.6.2 Action-Internal Telemetry only after the action graph, profile, execution log, and sandbox have isolated an opaque action whose internal behavior still cannot be explained.

The result should be a blocker register, not a pile of logs:

This format forces a useful distinction: an observed failure is evidence, while “non-hermetic” is only a broad diagnosis. The register must name the missing contract and who can change it.

Turn host assumptions into named blockers

Most blockers fit a small set of questions:

1. Files: Did the action read a file absent from its declared inputs?
2. Tools: Did it discover an executable through PATH, JAVA_HOME, an absolute path, or state retained by another action?
3. Environment: Did an inherited variable, locale, user identity, or secret change behavior without becoming an explicit, appropriate contract input?
4. Setup and state: Did a repository-time probe, earlier CI step, mutable directory, background process, or network service prepare something Bazel cannot reproduce from the action declaration?
5. Outputs: Did the process create or consume files outside the declared output contract, or depend on an undeclared side effect?

The local sandbox is strongest at the first question. It deliberately stages known inputs, and its debugging mode preserves the failed environment for inspection, but sandboxing does not hide every host file or eliminate every source of nondeterminism.⁴ Environment inheritance, unsandboxed execution, host tools, repository rules, and nondeterministic outputs therefore need separate evidence.⁵

Hidden setup deserves its own row in the register. Legacy WORKSPACE repository rules can execute host commands, inspect the OS, locate programs with which, or create absolute symlinks; those observations happen on the Bazel host and may describe a machine unlike the remote worker.⁶ The diagnostic is WORKSPACE-specific, but the decision is general: host discovery is not a worker input merely because it happened before execution.

Integration tests reveal the same mistake in another form. If CI starts a database before invoking Bazel, a local test may pass because the service happens to exist. A remote test action receives no such lifecycle automatically, and the external state also undermines isolation and cache correctness.⁷ Record the missing service contract. Then either make the test lifecycle self-contained in an execution environment that supports it, or approve a visible local-only exception; do not inject credentials or an endpoint into a broadly cacheable action and call the problem fixed.

Absolute tool paths are equally concrete. A toolchain generated locally may point into /nix/store, yet the remote executor has no corresponding path. The failure is not “Nix versus RBE”; it is a tool input or runtime closure that never crossed the execution boundary.⁸ T6.2 RBE Incompatibility explains the Nix-specific incompatibility, just as iOS and Windows fleet cases continue in T2.3 iOS RBE & Build Optimization and T5.5 Windows RBE Infrastructure. The blocker-register method stays the same across those specializations.

Make one of three explicit decisions

Classify a class as remote-compatible only when the declaration and selected tools are complete, the required environment is available by an explicit contract, and a representative remote-only attempt has placement evidence.

Classify it as repairable when the missing dependency or portability gap has a concrete owner. The repair may add an input, move tool selection into a toolchain, make a test own its setup, remove an ambient read, or declare the worker capability the action truly requires. The readiness register records the acceptance criterion; the responsible ruleset or language integration implements the change.

Classify it as intentionally local when the remote fleet will deliberately not satisfy the requirement. This is an exception, not an unknown bucket. Keep the evidence, owner, reason, affected class, and condition for review. That makes local execution a controlled design choice instead of silent fallback.

Prove the repair without fallback

Retest the repaired class in a mode that cannot silently execute it locally. Join the success to placement evidence from the execution log, Bazel's process summary, or the backend. A green invocation with local fallback enabled proves only that some strategy produced an output; policy for failures, retries, and fallback is handled in 6.3.13 Remote Action Failure Handling.

Keep the local reproduction path as well. A preserved sandbox case is faster to iterate on and makes a future regression diagnosable without requiring a fleet incident. 6.3.19 Local REAPI Development adds a later, versioned diagnostic option for exercising the protocol and a distinct worker environment on one machine.

One accepted class is not a production-ready deployment. Workload coverage, isolation, overload, failure paths, recovery, and unsupported classes are joined into the broader decision in 6.3.18 Remote Execution Production Readiness. For the class that has passed this gate, the next handoff is narrower: give its declared tools, platform needs, and runtime requirements to 6.3.2 Remote Executor Matching so the backend can route it only to compatible executors.