6.3.18 Remote Execution Production Readiness

A staging build that finishes remotely proves one path through the system. It does not prove that the deployment can carry the production workload, remain correct under contention, expose a useful failure, or recover from change. Production readiness is therefore a decision backed by joined evidence, not a feature checklist and not a larger demonstration build.

Begin only after a candidate backend has passed the compatibility and capability checks in 6.3.3 Choosing a Remote Execution Backend. Then qualify the deployment against representative workload classes and explicit failure conditions. The result should say what may enter production, what must remain local, which degraded modes are allowed, and what change makes the decision invalid.

Turn readiness into a set of falsifiable claims

Write each qualification gate as a claim that an operator could disprove. “CAS enabled” is a feature statement. “Every required consumer can retrieve a declared output after the producing client exits, until the published retention boundary” is a claim with a subject, an observation, and a limit.

For every gate, retain four fields:

Keep three authorities separate in those fields. Bazel and REAPI define portable client/service behavior; the selected backend defines its scheduler, worker, storage, and recovery behavior; the organization decides which limits and risks it accepts. REAPI itself is a wire contract, not a production topology or an operational recipe.¹ A Buildbarn deployment, for example, can provide concrete frontend, storage, scheduler, runner, and worker evidence, but those components are not universal REAPI roles.²

That separation prevents two common errors. A backend dashboard cannot turn an implementation-specific queue transition into a portable protocol guarantee. Conversely, a successful Execute response cannot establish that the backend's coordinator survives a restart or that its worker cleanup boundary prevents state leakage.

Qualify the workload, not the showcase target

Partition the intended production workload into classes that can fail differently. Include representative languages, action families, execution platforms and toolchains, input sizes, output shapes, durations, concurrency profiles, and trust levels. Add release-only generators, tests with unusual runtime needs, and large-output consumers even if they are rare. Remote systems enforce implementation-specific CPU, memory, I/O, and other limits, so one ordinary compile action cannot discover the useful boundary of the service.³

For each class, preserve a remote-only success path and a negative or boundary probe. Disable or otherwise exclude local fallback while collecting placement evidence; otherwise a green build may hide a remote failure. If a class is unsupported, record it as an explicit production restriction with an owner and a review condition. Do not silently omit it from the test corpus.

A compact coverage table makes gaps visible:

6.3.1 Making Actions Work Remotely defines the admission decision for each action class; this qualification uses that register as input. A production decision is not a reason to repair action declarations, toolchains, or host assumptions inside the readiness exercise.

Require a gate across every deployment boundary

The following gates form the minimum integration argument. Each points to the article that defines the mechanism and its deeper evidence:

1. Compatibility and placement. Pin the Bazel client, backend, API and feature matrix; prove representative actions land only on compatible executors. Continue with 6.3.3 Choosing a Remote Execution Backend and 6.3.2 Remote Executor Matching.
2. Trace and topology. Join client execution to backend queue, attempt, worker, storage, and terminal observations without pretending they share one universal identifier. Use 6.3.4 Remote Action Tracing and 6.3.5 Remote Execution Architecture.
3. State and storage. Test loss and restart of coordination state, CAS and Action Cache reachability, retention, corruption handling, and the recovery promises the implementation actually makes. Use 6.3.6 Remote Execution State and Recovery and 6.3.7 Remote Execution Storage.
4. Scheduling and capacity. Exercise matching, admission, quotas, backpressure, fairness, autoscaling lag, and headroom under a representative arrival and duration distribution. Use 6.3.8 Scheduling Actions on Remote Executors and 6.3.12 Remote Execution Capacity.
5. Worker and data boundaries. Prove the versioned runtime, isolation, cleanup, input availability, output publication, and required consumer paths. Use 6.3.9 Remote Executor Environments and Isolation, 6.3.10 Remote Input Presentation, and 6.3.11 Remote Output Availability.
6. Failure and tenant behavior. Test cancellation, retry, duplicate attempts, allowed fallback, identity, authorization, namespace, quota, and cross-tenant isolation. Use 6.3.13 Remote Action Failure Handling and 6.3.14 Multi-Tenant Remote Execution.
7. Planned change. Canary a client, backend, scheduler, executor, or image change; prove drain, abort, rollback, evidence preservation, and cold-cache protection. Use 6.3.17 Remote Execution Upgrades.

Multi-region routing and action deduplication are additional gates only when the deployment promises those behaviors. In that case qualify them through 6.3.15 Multi-Region Remote Execution and 6.3.16 Action Deduplication and Racing rather than treating them as implicit benefits of a distributed service.

This list is deliberately a set of joins, not a retelling of the preceding articles. A gate passes only when evidence from the relevant mechanism is attached to the same versioned deployment and workload cohort. Nine individually credible documents about nine different configurations do not qualify one production system.

Exercise degraded operation, not only total failure

The difficult production states are often partial: one worker pool is unmatchable, storage is slow but reachable, an API server restarts, cancellation arrives after a duplicate attempt starts, or the client loses its stream while the backend continues execution. State the permitted result for each condition: fail closed, retry within a bound, fall back locally for named classes, serve a cached result, or reject new work while preserving existing operations.

Retain evidence that distinguishes those outcomes. A production incident in one remote-execution service remained invisible as build failure because GOMA fell back locally; autoscaling and logging regressions instead appeared as intermittent slowness for days.⁴ That is why a successful user-facing build and a healthy remote service are different claims.

The same incident record shows why qualification must include flow control, timeouts, guardrails, and failure attribution rather than only component availability. Missing proxy flow control exhausted API-server memory; a cyclic RPC dependency could deadlock workers; and a buggy build submitted 172,000 protocol-valid compile actions that saturated a pool, motivating build-scoped cancellation and per-build quota guardrails.⁴ These are implementation cases, not universal failure modes, but they demonstrate the class of evidence a production exercise must seek: behavior under pressure, bounded impact, and a diagnosable reason.

Publish a decision that expires

The final artifact is a versioned qualification record, not “RBE approved.” It names the tested client and backend versions, deployment configuration, worker runtime identities, workload corpus, tenant model, evidence locations, known limits, allowed degraded modes, recovery expectations, and decision owner.

Also state invalidation conditions. Requalification is required when a change can alter an established claim: client/API compatibility, scheduler policy, storage layout or retention, executor image or isolation, identity and authorization, topology, material workload distribution, or the rollback and recovery mechanism. The scope may be targeted to affected gates, but the impact analysis must preserve their joins. For example, an executor-image change is not only an isolation probe when it also changes tool compatibility and cache acceptance.

Keep the operational record usable after launch. Dashboards and alerts should link from symptoms to the same cohort, pool, attempt, storage, and tenant evidence used during qualification. Publish the escalation route alongside the limits; production post-mortems show that unclear incident escalation can turn a technical regression into prolonged user impact even when the underlying fix is quick.⁴