6.3.19 Local REAPI Development
extraYou do not need a worker fleet to exercise Bazel's Remote Execution API. A local REAPI executor lets the Bazel client upload an action to an endpoint, execute it in a separate worker environment, and retrieve the result while every component still lives on one development machine.
This makes the protocol traced in 6.3.4 Remote Action Tracing concrete without requiring a shared service.
That workflow is useful for three different reasons:
- expose actions that accidentally depend on the developer host.
- exercise a Linux execution boundary from a supported macOS, Windows, or Linux host.
- share one persistent CAS and ActionCache between several local Bazel servers.
It is not a performance promise. It adds protocol, VM, and data-transfer work, and it cannot model the scheduler, network, contention, or failure behavior of a production cluster. Nor does it reproduce a production executor image: toolchains, runtime layers, environment variables, and execution properties must be compared with CI or RBE separately.
actiond's topology
actiond is a concrete implementation. Bazel
connects to a host-side gRPC listener. The host starts a small Linux VM and
forwards REAPI traffic to a guest process that implements Execution, CAS,
ByteStream, ActionCache, and Capabilities services.1
Bazel client
|
| REAPI over local gRPC
v
host actiond process
|
| vsock / Hyper-V socket bridge
v
Linux guest: executor + CAS + ActionCache
The VM is long-lived. Actions do not boot one VM apiece. Each action receives a
separate chroot with private mount and network namespaces. Compared with the
local execution strategies in 2.3.2 Sandboxing, this places the action
inside a separate Linux guest. An actiondfs filesystem exposes the Merkle
input tree lazily from guest CAS blobs, so sandbox setup does not perform one
host-side filesystem operation for every declared input.1
This is a stronger boundary than merely changing Bazel's local spawn strategy, but “runs in a VM” is not synonymous with “fully hermetic.” The action contract still depends on the selected toolchain, declared environment, execution properties, and any runtime layers supplied by the executor.
The published release matrix is narrower than “any host”: macOS ARM64, Windows
ARM64/x86_64, and Linux ARM64/x86_64. Windows needs Hyper-V. Linux needs
io_uring plus read/write access to KVM and vhost-vsock. The README contains
the current checks and startup commands.1
Configure Bazel as an REAPI client
After installing the host-specific release and starting its documented VM
server on 127.0.0.1:8980, first define and register an execution platform
whose constraints describe the guest OS and CPU and whose toolchains produce
Linux executables. The runnable
local-reapi-executor snippet uses
//:local_linux_exec for a Linux x86-64 guest and selects it with a named
.bazelrc configuration:
build:local-reapi --remote_executor=grpc://127.0.0.1:8980
build:local-reapi --remote_cache=grpc://127.0.0.1:8980
build:local-reapi --spawn_strategy=remote
build:local-reapi --genrule_strategy=remote
build:local-reapi --remote_local_fallback=false
build:local-reapi --remote_upload_local_results=false
build:local-reapi --noremote_cache_compression
build:local-reapi --extra_execution_platforms=//:local_linux_exec
Then:
bazel build --config=local-reapi //:local_reapi_probe
The linked BUILD file
defines both labels, so this command is concrete rather than a placeholder.
For another workspace, the execution-platform label must describe the actiond
guest and its registered Linux-executable toolchains. Cross-host builds may
also need an explicit --host_platform when exec-configured tools still follow
a host-platform contract, and an explicit --platforms when the requested
output targets a different OS or CPU. The upstream LLVM smoke test selects all
three surfaces.
Copy its intent, not its repository-specific labels.
This named configuration is one example of the repository-local workflows discussed in 3.5 Developer Experience & Local Tooling: developers opt into a different execution boundary without changing the target graph.
These flags describe the inspected actiond contract, not a universal local-RBE profile. In particular, compression was not supported at that revision, and disabling local fallback is a deliberate diagnostic choice: it prevents an incompatible action from silently succeeding on the host.1
Verify the boundary with upstream evidence
Do not infer remote execution merely from a green build. Use the upstream checks as observable evidence:
tools/e2e.sh vmcovers macOS and Linux: it starts the host-specific VM and builds thetest/workspace with remote-only spawn strategies, cache acceptance disabled, and local fallback disabled. Itstest/stress.bzlworkload also checks that external networking is blocked while loopback and the packaged/etc/hostswork. Success therefore means those actions executed through the guest endpoint and satisfied the checked action-sandbox behavior. It is not a general security proof.- On Windows, use
e2e/run_llvm_windows_vm_smoke.ps1. It starts the Hyper-V-backed executor, builds with remote-only strategies and local fallback disabled, and requires a remote process summary. A completed actiond measurement is the Windows evidence route. The bash route above does not support Windows. e2e/llvm_tblgen_smoke.shselects a Linux-musl host platform because generated exec tools run in the VM. A successfulllvm-tblgenbuild with--remote_local_fallback=false, together with Bazel's process summary reporting remote processes, proves that the Linux exec-configured actions ran remotely. An exec-format or toolchain-resolution failure instead exposes a host-only tool assumption.- For cache reuse, keep the same actiond
--rootand endpoint but use a fresh Bazel--output_basefor the second build, as the platform runners undere2e/do. With cache acceptance enabled, remote cache hits in Bazel's process summary are the observable criterion that the new Bazel server reused the guest ActionCache/CAS. A second success without cache hits proves execution, not sharing.
These are upstream development routes, not lightweight prerequisites for every workspace. For your own target, the smallest useful check is a build with local fallback disabled plus an execution log or process summary that shows remote actions.
A repo-local proof with an explicit fixture boundary
Packaging actiond inside this snippet would make the asset host-specific and
would still require VM support. Instead, the snippet keeps the live command
available for an existing endpoint and makes the portable regression check
deterministic. Its reduced
execution record
and cache-hit record
were captured from separate Bazel 9.1.0 output bases using the same local
Buildfarm endpoint. The first build rejected cache hits; the second used a
fresh client. The
capture manifest
records normalized reproduction commands, distinct output bases, endpoint,
versions, and fixture hashes. It also states that ephemeral paths and the
Docker-published port were normalized; it is an inspectable provenance
assertion, not independent attestation of the capture host. The verifier checks
that contract, requires the same action digest, and distinguishes
runner: remote, cacheHit: false from
runner: remote cache hit, cacheHit: true.
Running the snippet's deterministic command builds //:local_reapi_probe
locally, then checks those captured fields and prints:
PASS //:local_reapi_probe: first client selected remote execution (cacheHit=false)
PASS //:local_reapi_probe: fresh client selected a remote cache hit (cacheHit=true)
BOUNDARY checked-in records prove Bazel's client observations, not executor isolation
That output proves the fixture remains internally consistent and shows which Bazel-side observations to require. It does not prove that actiond is running now, that Buildfarm has actiond's VM boundary, or that either backend provided a particular isolation property. Run the named config against actiond and retain its upstream executor/process evidence when the VM boundary is the claim under test.
Your toolchains must run in the guest
On a macOS laptop, actiond does not make a Darwin compiler executable inside Linux. Bazel resolves toolchains before sending the action. The selected tools and all their inputs must be compatible with the Linux guest's CPU and declared execution constraints.
Keep the roles from 3.3.2 Platform Vocabulary distinct:
- the host runs the Bazel client and actiond VM process.
- the execution platform is the Linux guest that runs compiler/test actions.
- the target platform describes the produced output.
A Linux executor can still cross-build a Windows or embedded target when the registered toolchain supports that pair. Conversely, a native macOS toolchain cannot execute remotely merely because the endpoint is local.
When resolution fails, inspect:
- the active execution platforms.
- the selected toolchain's
exec_compatible_with. - the executable format/CPU of every tool.
- hidden shell or system-library dependencies exposed by the VM.
Runtime mounts are explicit compatibility debt
The executor can mount packaged Bash or selected glibc versions when an action requests the corresponding execution property. This is useful for legacy tools, but the runtime is executor infrastructure rather than an ordinary input declared by the target.1
Use the property to make a boundary visible and unblock a migration. Do not use it to claim that the action is independent of the executor image. A truly portable tool should normally arrive through a declared toolchain or input closure.
What local REAPI proves
| A successful run can prove... | It does not prove... |
|---|---|
| The action's tools execute in the selected Linux environment | A production scheduler will route it correctly |
| The declared input root is sufficient for this worker runtime | No hidden worker runtime is involved |
| REAPI/CAS/AC client paths work against this endpoint | Conformance with every REAPI server or feature |
| Several local Bazel servers can reuse one CAS/AC | Networked or multi-region cache performance |
| VM isolation contains action execution better than host execution | Complete protection against every hostile workload |
Use it as a development and diagnostic layer between native local sandboxing and a shared RBE service. Promote findings into explicit toolchain inputs, platform constraints, or worker-image contracts instead of making the local executor a prerequisite for understanding the build.
A local REAPI executor is valuable because it changes the execution boundary without requiring a cluster. It catches host assumptions and exercises remote client paths, but the workspace still owns Linux-compatible toolchains and a production rollout still needs scheduler, capacity, isolation, and failure-mode validation.
Footnotes
-
actiond repository map and upstream
ARCHITECTURE.md— host/guest topology, REAPI subset, actiondfs input model, runtime properties, tests, and limitations. ↩1 ↩2 ↩3 ↩4 ↩5