6.3.3 Choosing a Remote Execution Backend

Choosing a remote execution backend is not a search for the longest feature list. It is a compatibility decision: which versioned implementation can satisfy this workload's portable client and protocol contract, which extra backend contracts would you accept, and how could you leave without redesigning the build?

Select a backend from requirements, not a feature list

A capability record narrows the field. It does not certify a client, workload, operating commitment, or exit.

Begin with the audited workload and versioned Bazel client contract from earlier readiness and matching work. Intersect the exact client, required REAPI services, and each candidate's Capabilities record to form a candidate shortlist. Capabilities are necessary but not sufficient: a scored must-have and exit matrix must include exercised positive and negative probes. A score of 2 is direct version-pinned evidence and a passed check; a must-have score of 0, 1, or unknown rejects the candidate. The output is a shortlist with open evidence and a feasible exit, not a product ranking or deployment design.

WORKLOAD + VERSION CONTRACT

Freeze representative action classes before naming candidates

Pinned Bazel client and configuration, platforms, toolchains, runtime needs, and visible unsupported cases

EXACT CLIENT

Record each Bazel version and remote configuration

Client feature use is part of the contract

REQUIRED PROTOCOL

Name services, API range, digests, compression, and output behaviour

Portable REAPI requirements

CANDIDATE CAPABILITIES

Capture the endpoint and instance response for the pinned backend

Necessary evidence, not sufficient proof

does not prove worker matching does not prove operations or exit

Keep only candidates with a non-empty client / protocol / backend intersection. This is a shortlist for investigation, not a ranking.

Must-have requirement

Evidence score

Owner and exit condition

Pinned client / REAPI intersection

2 exercised

Client + backend owner. Reject a missing range or service.

Identity, data, lifecycle, and required extension

? open

Service owner. Reject until removal or migration is feasible.

Exit path for configuration, credentials, data, and control APIs

1 advertised

Architecture owner. Evidence must name time, cost, rollback, and retention.

Score: 2 = version-pinned evidence plus a passed check. Any 0, 1, or ? on a must-have rejects the candidate, whatever its total.

POSITIVE PROBE

Run the exact client and one representative action through the required intersection

Proves only this pinned client / backend / workload combination

NEGATIVE PROBE

Request one documented unsupported compatibility condition

Require visible rejection before execution, or record an evidence gap

SHORTLIST OUTPUT

Candidates that clear every must-have, with open evidence and a feasible exit

Later work qualifies the action trace, architecture, operations, and production readiness

pinned evidence named owners exit record

Capabilities answer what a server advertises. Selection evidence answers whether this versioned client and workload can use it, which commitments the candidate adds, and how the organization can leave.

Start after 6.3.1 Making Actions Work Remotely has classified representative action classes and 6.3.2 Remote Executor Matching has reduced their platform, toolchain, and execution-property needs to a bounded contract. Those are the requirements. Product documentation and capability responses are evidence against them, not a substitute for them.

Freeze the workload contract before naming products

Write one versioned requirement sheet for the workload you intend to move. Pin the Bazel binary and relevant client configuration, then name representative action classes rather than saying “the monorepo.” For each class, record its execution platform, configured tools, required runtime surfaces, output shape, and expected behavior when no compatible executor exists. Bazel's rule guidance explains why isolated actions, explicit dependencies, toolchain-provided tools, and execution-platform-compatible binaries are prerequisites for this step.¹

Classify every requirement by where its meaning comes from:

Requirement class	Examples	Why the distinction matters
Portable protocol	Required REAPI services and API range; digest functions; compression; action-cache, CAS, and output or symlink semantics	A different implementation can satisfy the same client-facing contract.
Backend extension	Property vocabulary, pool selector, worker protocol, image convention, authentication integration, or product API	Migration requires an equivalent mapping, an adapter, or removal of the extension.
Organization policy	Data location, writer trust, support response, cost boundary, retention, or change approval	Neither Bazel nor REAPI can decide whether the candidate satisfies it.

This classification prevents two opposite errors. A portable field should not become vendor folklore, while a scheduler, pool, worker image, or authentication model should not be presented as if REAPI standardized it. The Remote APIs repository defines the client-facing services and capability records, but explicitly leaves server internals such as worker protocols and matching design to implementations.²

A useful requirement sheet includes at least:

• every supported Bazel client version and the configuration that changes its remote behavior;
• the required CAS, Action Cache, ByteStream, Capabilities, and Execution RPCs and their acceptable API-version intersection;
• required cache and execution digest functions; compressed-ByteStream and inline-batch compressor support where each path is used; size limits; the absolute-symlink-target policy; and output forms required from the negotiated API version;
• representative platforms, toolchains, action classes, and the bounded backend properties needed to place them;
• authentication, authorization, namespace, data-handling, and evidence needs;
• the expected visible result for an unsupported service, feature combination, or workload class.

Do not add architecture answers yet. Storage layout, scheduler topology, executor control, isolation, geography, and observability belong on the sheet as questions and later acceptance gates. 6.3.5 Remote Execution Architecture provides the system model needed to evaluate those answers after a candidate has passed this portable selection gate.

Compatibility is an exercised intersection

The REAPI Capabilities service is necessary selection evidence. For a chosen endpoint and instance, GetCapabilities advertises server API versions plus distinct cache and execution fields.² Check cache and execution digests separately. Treat compressed ByteStream and inline BatchUpdateBlobs compressor lists as different transfer contracts; neither advertises gRPC-level compression. The symlink field describes policy for absolute symlink targets, while other output forms follow the negotiated API surface and still need representative probes. The response can expose an immediate mismatch such as no common API range, no required digest, or execution disabled, unavailable, or unauthorized for this endpoint and account. Verify the required CAS, Action Cache, ByteStream, Capabilities, and Execution RPCs separately rather than treating one response as proof that every service path works.

It cannot prove that the pinned Bazel client implements and selects every advertised feature. It also cannot prove that the backend understands your property vocabulary, has a matching worker, supplies the promised toolchain environment, or meets your isolation and operating requirements. A capability response describes one server-side protocol surface; it does not certify a working client/backend/workload combination.

Keep those claims separate in a compatibility matrix:

Workload class	Bazel client and config	Required REAPI surface	Backend release, config, and instance	Advertised intersection	Exercised result	Classification
representative compile	exact version and remote config	services, API range, digest, compression, output needs	exact release/configuration and endpoint instance	captured Capabilities record	fresh remote attempt joined to backend evidence	pass, protocol mismatch, action/fleet failure, or unknown
representative test	exact version and remote config	same fields plus its distinct environment needs	exact release/configuration and endpoint instance	captured Capabilities record	fresh remote attempt joined to backend evidence	pass, protocol mismatch, action/fleet failure, or unknown

Use one row per materially different client/configuration and action class. A server's high API version is not a proxy for feature parity, and a release note is not a negotiation record. Preserve the exact Bazel binary, backend revision, endpoint instance, worker or deployment revision, capability response, and observed result together.

Use positive and negative probes with narrow claims

A positive probe begins with the captured capability record and a requirement row whose advertised intersection is non-empty. Run the exact Bazel client against the candidate's pinned, maintained test or deployment route and retain evidence that one representative action was accepted and executed remotely. Buildfarm's quick start separates cache reuse from remote execution, while NativeLink provides distinct cache and remote-execution integration-test routes; these are examples of the evidence shape, not interchangeable commands for every backend.^3,4

The result supports one sentence: this pinned client/backend/fleet combination executed this action class through the required intersection. It does not show that every action is ready, that capacity is sufficient, or that the deployment survives failures.

A negative probe starts from a condition that the candidate's pinned protocol or implementation evidence says is unsupported. Select a required service, API range, digest or compression combination, or other compatibility condition that can be changed without also changing the action's source behavior. Submit it through the same client and evidence route. The useful outcome is an explicit client or server compatibility rejection before the action is accepted for execution. Record that as a protocol or configuration mismatch.

If the request is accepted and a worker later fails to run the command, the probe crossed the protocol boundary: you observed an action, matching, or environment failure instead. If the candidate has no maintained way to create and observe the intended unsupported condition, mark the matrix cell ?. Endpoint unreachability, an invented toy server, or plausible-looking console output would not close that evidence gap.

After a candidate clears this selection gate, 6.3.4 Remote Action Tracing follows one normal action through its observable execution lifecycle. That trace is deliberately later: endpoint reachability and a capability response do not establish queueing, placement, execution, and result publication.

Use maintained candidates as evidence routes, not rankings

Candidate discovery should point you to maintained primary evidence. It should not turn the article into an endorsement or imply that similar protocol ranges mean similar products. Current repository evidence provides several useful, different routes:

• Buildfarm is an Apache-2.0 implementation with a cache-first, execution-second quick start and separate server, worker, queue, and backplane configuration.³
• Buildbarn's maintained deployment repository makes its frontend, storage, scheduler, runner, worker, and monitoring responsibilities inspectable. Those components and its Jsonnet, Compose, or Kubernetes choices are Buildbarn design, not REAPI requirements.⁵
• NativeLink provides configuration, deployment, metrics, and integration-test routes. Its current FSL-1.1-Apache-2.0 license has a future Apache grant, so the exact release and license date are part of the adoption and exit record; it must not be labeled simply Apache-2.0 today.⁴
• BuildBuddy exposes architecture, configuration, deployment, cache, and remote execution source paths, but its MIT-licensed core and separately licensed enterprise/ remote-execution and scheduling code must remain distinct in a licensing and exit assessment.⁶

Use the official remote-execution overview's maintained service-discovery route to find additional self-service and commercial candidates, then demand the same versioned evidence from each.⁷ Do not fill a comparison table from a historical launch post, an unpinned conference claim, or a product landing page. Remote systems have workload-specific limits, and organizations retain value by preserving the ability to change systems as their requirements evolve.⁸

Backend-specific features can still be legitimate requirements. For example, one BuildBuddy case describes pools, affinity routing, action merging, hedging, runner recycling, and tenant queues.⁹ Record a feature you actually need as an extension, along with its compatibility test, owner, and removal or migration cost. Do not copy it into the portable REAPI column.

Score requirements without averaging away a blocker

Use a small evidence score:

• 2 — direct, version-pinned evidence exists and the named check passed;
• 1 — documented or advertised, but not exercised for this client/workload;
• 0 — absent, incompatible, or contradicted;
• ? — not known yet.

A must-have row passes only with 2. Any 0, 1, or ? on a must-have rejects the candidate from the shortlist, regardless of its total. Totals can order the remaining investigation; they cannot compensate for an unproven protocol intersection or missing workload platform.

Requirement	Must-have?	Evidence required for 2	Candidate A	Candidate B	Owner	Exit condition
Pinned Bazel/REAPI overlap	yes	captured capabilities plus exact-client result			client/backend	reject on missing range or service
Digests, transfer compression, symlink/output behavior	as required	separate cache/execution digests; exercised ByteStream/batch path; absolute-target policy and versioned representative outputs			client/backend	reject or deliberately change the workload contract
Representative platform and action classes	yes	positive remote result; unsupported class remains visible			workload/fleet	reject if required class cannot be represented
Failure boundary	yes	protocol mismatch is distinguishable from action/fleet failure			backend	reject while failures remain ambiguous
Authentication and integration	yes	pinned identity path and least-required operations exercised			service/security	reject on unresolved access or data boundary
Operations and lifecycle	yes	named owners, upgrade/support evidence, and cost model			service owner	reject on unowned duty or unsupported lifecycle
Backend extensions	no unless required	each extension has a test and removal/migration story			architecture owner	remove, replace, or accept explicit coupling
Exit path	yes	documented configuration, data, identity, and operations transition			service owner	reject a long-lived commitment without a feasible exit

The exit row is not “uses an open protocol.” Record how to detach client configuration and credentials; whether stored data can be migrated or must be rebuilt cold; which property vocabulary, images, dashboards, and control APIs must change; which replacement candidate exists; and who owns the time, cost, rollback, and data-retention consequences.

Managed and self-hosted move responsibility; they do not erase it

Self-hosting exposes more implementation and operating choices, but source availability alone does not make the workload or stored state portable. Buildbarn's deployment materials, for example, make storage, monitoring, certificate, worker, configuration, and upgrade work concrete.¹⁰ Managed service transfers only the duties named in its service and support contract.

Responsibility	Self-hosted route	Managed route
Workload and client compatibility	Your team proves and maintains it.	Your team still proves it with the supported service combination.
Service operation	Your team owns deployment, capacity, monitoring, upgrades, incidents, and recovery.	The provider may own specified service operations; your team owns integration, escalation, and acceptance evidence.
Identity and data policy	Your team implements and audits controls.	Your team verifies contract, configuration, retention, residency, export, and revocation behavior.
Lifecycle and cost	Infrastructure, staffing, and dependency upgrades are explicit internal costs.	Usage, support, contract, and switching exposure are explicit supplier costs.
Exit	Source and data access help only if a replacement deployment and migration are feasible.	Endpoint replacement helps only if identity, data, extensions, and operating dependencies can also move or be retired.

The selection record should end with a shortlist and open evidence, not a claim that an RBE deployment is ready. Architecture, isolation, storage, scheduling, capacity, observability, recovery, and rollout gates remain to be tested. Only 6.3.18 Remote Execution Production Readiness combines those results into a production decision.