6.3.5 Remote Execution Architecture

One remote endpoint can make a remote execution system look like one service. Operationally, it is a chain of responsibilities: an endpoint accepts a client request, storage supplies bytes, scheduling chooses where work may go, executor control delivers that work, an executor runs it, and evidence systems make the boundaries attributable. You need that role map before an outage can be assigned to a component or team.

6.3.4 Remote Action Tracing followed one representative compile action through portable protocol observations and backend-owned events. This article places that same action in a deployable topology. It starts with a candidate already selected through 6.3.3 Choosing a Remote Execution Backend; the implementation cases below show different ways to realize the roles, not a new product ranking.

REAPI defines a surface, not a cluster design

The Remote Execution API standardizes the client-facing contracts needed for interoperability. Its portable surface includes Execution, ActionCache, ContentAddressableStorage, and Capabilities; the ByteStream API carries blob data. Execute and WaitExecution expose long-running Operations to the client.¹ Those names tell you which protocol a client can speak. They do not require one process per service or prescribe a frontend, queue, scheduler, worker protocol, storage engine, coordination store, or telemetry system.

Keep these two columns separate when reading a deployment:

Even the word endpoint names an address, not the component behind it. A deployment may terminate client RPCs in a frontend that routes to several services, expose the scheduler directly, or combine frontend and scheduler in one process. Reaching that endpoint proves neither that storage is healthy nor that an executor has received work.

Build the topology from logical roles

Use logical roles first, then map each role to the actual processes, workloads, and teams in your deployment. A small system may co-locate several roles. A larger system may split one role across tiers or replicas. Co-location changes the blast radius and the evidence available at a boundary; it does not erase the responsibility.

Client and ingress/frontend

The Bazel client or CI runner chooses the endpoint and instance, presents its identity, sends the portable RPCs, and observes RPC and Operation results. An ingress or frontend may authenticate, authorize, route, and load-balance those requests. These duties are common deployment choices, not additional REAPI services.

The owner must be able to answer which client configuration and identity reached which frontend revision and where each request was routed. Without that join, “the remote executor is down” may describe only an inaccessible or misconfigured edge.

AC/CAS and blob transfer

The storage role serves the declared input closure and preserves content-addressed outputs and result metadata. Clients and executors can both use this data plane: the client uploads missing inputs or retrieves results, while an executor fetches inputs and uploads captured outputs. Storage therefore sits on several important paths, but it does not decide which action enters a queue or which executor runs it.

This is the most important separation in the topology: AC/CAS latency or a missing blob is a data-path problem until evidence shows otherwise. The exact ordering and reachability rules that make a published result safe belong to 6.3.7 Remote Execution Storage.

Scheduler, queues, and coordination

The scheduler role accepts eligible execution work, associates it with an implementation queue or compatible destination set, and chooses an executor-control path. Queue and matching records are backend evidence. They are not implied merely because the client has an Operation.

Schedulers often depend on coordination state such as a backplane, task store, worker registry, or service-specific operation state. Name that dependency as a role even when it is embedded in the scheduler. Do not yet label its state durable, reconstructible, or disposable: 6.3.6 Remote Execution State and Recovery makes that recovery classification. Admission, priority, fairness, backpressure, and dispatch policy belong to 6.3.8 Scheduling Actions on Remote Executors.

Executor control and executors

Executor control connects scheduling decisions to the worker fleet. Depending on the backend, it may push an assignment, grant or observe a lease, or let an executor pull work. It also accounts for worker registration, availability, and lifecycle signals that the scheduler relies on. None of those internal worker protocols is a portable REAPI requirement.

The executor role performs the work: obtain inputs, prepare an execution environment, run the command, capture outputs, and send results back through the backend's data and control paths. Keep control separate from work. A lost assignment or stale worker registration is not the same failure as a command crash on a correctly selected executor. Environment preparation, credentials, cleanup, and cross-action isolation continue in 6.3.9 Remote Executor Environments and Isolation.

Observability and evidence sinks

Every role should emit evidence to logs, metrics, traces, or health systems. That evidence plane observes execution; it does not schedule actions, store result blobs, or prove progress merely because a dashboard is green. Missing telemetry creates an attribution gap. It does not prove that the corresponding work did not happen. 6.6.1 Tracing a Build Across Services develops the cross-service join and retention design.

Name four paths through the roles

A component diagram becomes operational only when its arrows have meanings. Annotate at least these four paths in the deployment you operate:

1. Control path: client execution request to ingress, scheduler and queue, executor control, executor, and terminal observation back to the client.
2. Data path: action and input bytes into CAS; input fetch by the executor; captured output blobs back to CAS; the ActionResult returned through the terminal ExecuteResponse; and, only when caching is allowed and publication succeeds, an Action Cache entry for later consumers.
3. Evidence path: request, Operation, assignment or lease, worker, storage, and component-health observations sent to evidence sinks with explicit join keys and source clocks.
4. Failure path: the dependency chain by which one unavailable or contradictory boundary becomes a client-visible symptom. A frontend failure can hide healthy storage and executors; storage failure can block both client and worker data transfer; coordination failure can stop assignment while CAS remains healthy; evidence failure can leave successful or failed work unattributable.

The failure path is not a recovery arrow. It is the bounded claim you can make now: which dependency first lacks or contradicts expected evidence. 6.3.6 Remote Execution State and Recovery classifies what can survive or be reconstructed; 6.3.13 Remote Action Failure Handling handles retry and fallback decisions after a failed or ambiguous attempt.

Real implementations compose the roles differently

Representative implementations are useful precisely because they disagree on component shape:

• Buildfarm exposes server, worker, shared backplane, storage, and queue configuration. Its minimal route deliberately starts with cache behavior and then adds execution, while focused configuration separates the server, worker, and queues.² This is a Buildfarm topology, including its backplane and worker protocol.
• Buildbarn has a runnable deployment map with frontend, storage, scheduler, runner, worker, and monitoring responsibilities. Its Compose and Kubernetes configurations make those boundaries inspectable.³ Sharded storage, Jsonnet configuration, and Buildbarn's worker protocol are not REAPI requirements.
• NativeLink provides deployment examples with separately configured CAS, scheduler, and executors, plus a distinct metrics route.⁴ That composition demonstrates another valid split; its store and scheduler configuration remain NativeLink contracts.
• BuildBuddy documents client, scheduler, executor, CAS, and build-observability roles. Its implementation map places scheduling and remote execution in separately licensed enterprise paths.⁵ A BuildBuddy architecture account also names an API receiving actions, a scheduler matching them, and remote workers doing the work.⁶ Those claims describe BuildBuddy, not every REAPI server.

A deployment can also add a domain-specific dependency that none of those role maps requires. In one Buildbarn case, Kubernetes exposed frontend, scheduler, and storage services while a separate Nix server exported a read-only /nix/store to executors.⁷ That Nix/NFS path is an additional executor data and availability boundary. It is not part of REAPI and should not be hidden inside a generic “worker” box.

A managed service changes the operating-owner column, not the protocol facts. For example, Aspect describes its managed remote execution offering as Buildbarn-based while keeping action hermeticity, execution-platform compatibility, and multi-platform worker requirements visible at the client and workload boundary.⁸ The provider may operate named internal roles, but the customer still owns client configuration, workload compatibility, identity integration, data acceptance, and the evidence used for its own release decision. A managed endpoint does not justify inventing a hidden topology or assuming high availability for components the service contract does not name.

Assign responsibility for the traced action

Return to the representative compile action from 6.3.4. Its trace already separated the action digest, Operation, backend assignment or lease, and worker identity. The table below does not repeat its timeline. Instead, it asks where each responsibility lives and what a failure at that boundary would mean now.

For an incident, find the earliest boundary whose expected observation is missing or contradicts another one, then assign only that much responsibility. If the scheduler recorded an assignment but executor control has no delivery or lease observation, start at that interface. If executor evidence records a blob read failure while scheduler evidence is complete, start with the executor-to-storage data path. Do not jump from either symptom to a recovery prescription.

The topology now supplies the inputs for the next decisions. Classify and test state loss in 6.3.6 Remote Execution State and Recovery; prove input and result publication correctness in 6.3.7 Remote Execution Storage; design admission and queue policy in 6.3.8 Scheduling Actions on Remote Executors; and qualify the executor environment in 6.3.9 Remote Executor Environments and Isolation. Those articles answer different questions because this architecture keeps their responsible roles and paths distinct.