6.3.17 Remote Execution Upgrades

A remote-execution upgrade changes a live producer of shared results. Replacing the binaries or worker images is only one part of the transition: queued and running operations still have owners, old and new workers may coexist, and candidate workers may already have published results by the time you decide to roll back. Treat the change as a controlled migration of traffic, work, and state—not as a fleet restart.

6.1.5 Safe Infrastructure Rollouts provides the general compatibility-matrix, canary, abort, fallback, and rollback method. Here we apply it to the parts that make remote execution different: executor images, placement, draining, Operations, accepted attempts, and shared AC/CAS state.

Name the upgrade unit before moving traffic

“Upgrade RBE” is too broad to test or reverse. Record the exact old and candidate identity of each component that can change independently:

REAPI defines the client-facing execution, Operation, storage, and capability contracts, but it does not define a scheduler database, worker protocol, draining algorithm, lease recovery, or deployment order.¹ Concrete backends expose different component splits: for example, Buildfarm separates servers, workers, queues, and a backplane, while Buildbarn deployments separate frontends, storage, schedulers, runners, and workers.^2,3 Your upgrade runbook must therefore name the pinned backend and configuration that supply every non-portable behavior.

Build the smallest compatibility matrix that covers the real coexistence window. If an old frontend can reach a new scheduler, or a new scheduler can dispatch to an old worker, that mixed cell needs a passing probe or a deliberate rejection. Testing only “all old” and “all new” misses the path the rolling upgrade actually creates.

Keep image skew from becoming hidden action skew

Suppose linux-amd64-v42 and linux-amd64-v43 workers run together during an image rollout. If v43 changes an output-relevant ambient library but that library is absent from the action's declared identity, the same apparent action can produce different bytes depending on placement. A rolling Linux upgrade can create exactly this problem with glibc: workers in old and new partitions may generate mutually incompatible cached outputs while Bazel cannot see the ambient version difference.⁴

The first gate is therefore not “does the new image boot?” It is:

1. Which runtime differences can affect outputs or execution semantics?
2. Which of those differences are represented by declared tools, inputs, environment, or execution properties?
3. Can old and new producers safely share the same consumer and result cohort?

6.3.9 Remote Executor Environments and Isolation explains how to inventory and test the promised executor environment. For an output-relevant dependency, prefer a declared, versioned toolchain or sysroot so the selected runtime participates in the action contract. The glibc case shows why this is safer than relying forever on fleet homogeneity: a sysroot lets the version change with repository state and keeps old revisions reproducible.⁴

When you cannot repair the identity before the rollout, keep old and new worker pools distinct and route only a bounded action cohort to the candidate. If the two producer environments are not proven cache-compatible, isolate their write paths or result namespace according to the backend's supported mechanism. That temporary isolation costs reuse, but it preserves a boundary around candidate results. Do not invent backend-side cache-key rewriting; use 6.2.1 How Remote Cache Keys Work to reason about the action identity clients actually send.

Define drain as a state transition, not a shutdown signal

A useful drain contract answers what happens to three populations:

• Unassigned work: new matching actions stop entering the retiring pool, or are rejected with a visible reason. Moving a queue entry to another pool is safe only if the target pool satisfies the same execution contract.
• Assigned or running work: the backend either lets the current attempt finish, cancels it, requeues it under a fenced ownership rule, or marks its outcome unknown. The runbook must name which behavior is guaranteed and what evidence proves completion.
• Completed or publishing work: output upload and result publication may outlive the command. A worker is not drained merely because its child process exited; preserve the action, Operation, attempt, result, blob, worker, and image joins until publication reaches a terminal known state. 6.3.7 Remote Execution Storage develops the underlying output-capture, blob-reachability, and ActionResult-publication chain.

This is where the state inventory from 6.3.6 Remote Execution State and Recovery becomes operational. REAPI permits Operations to disappear and does not promise at-most-once execution, so an absent Operation or a replacement attempt cannot be silently translated into “nothing ran.”¹ Cancellation, duplicate attempts, and late completion are handled in depth by 6.3.13 Remote Action Failure Handling; the upgrade procedure must preserve enough evidence for that policy to decide which result, if any, is accepted.

For each retiring pool, define a drain-complete predicate. A practical predicate requires all of the following:

• placement of new work into the pool is disabled and observed as disabled;
• every accepted Operation is terminal, deliberately transferred under the backend's tested contract, explicitly abandoned, or recorded as unknown;
• every result allowed to escape the candidate cohort has reachable required blobs and an attributable producer; and
• no retiring worker or controller can reconnect and publish late work into the restored path.

Killing the worker satisfies none of those conditions by itself. Even a backend with a strong reconnect-and-cleanup protocol supplies an implementation case, not a portable guarantee; Buildbarn's periodic worker protocol, for example, has its own reconnect and cleanup behavior.⁵

Promote through observable gates

Use one representative journey as the running canary: for example, a presubmit cohort containing C++, JVM tests, code generation, and a large-output action on linux-amd64-v43. Keep an old-pool control cohort with the same revisions, targets, configurations, and cache conditions.

Promote in this order:

1. Capture the baseline. Record client, component, configuration, image, platform/toolchain, AC/CAS instance, and representative action identities.
2. Start an isolated candidate pool. Admit only the named cohort. Require remote-only placement evidence, correct outputs, complete result publication, and expected consumer access.
3. Exercise coexistence cells. Probe every old/new client, frontend, scheduler, worker, and stored-result combination that real routing permits. Reject unsupported cells visibly instead of allowing a catch-all pool or local fallback to hide them.
4. Inject the planned failure. Stop admission to the candidate, drain it, restart or roll back one component, and reconcile queued, running, completed, and published work. Measure the cold storage, image, and executor load caused by replacement.
5. Expand one boundary. Add one action class, platform, pool, client cohort, or journey, then repeat the affected cells and gates.

Correctness and complete evidence are promotion gates. Latency and throughput come afterward. A production RBE incident in which an autoscaler rollout made scale-down too aggressive and scale-up too slow produced intermittent long queues without build failures because clients could fall back locally; the rollout was reverted only after the service symptoms were joined to the changed capacity behavior.⁶ That case is a reminder that a green build and aggregate fleet health can conceal a broken remote path.

Recover the service you can prove, not the state you hope for

When a stop signal fires, first abort exposure: prevent new clients and actions from entering the candidate path. Then preserve evidence before restarting components, modifying queues, or deleting state. Record the last known routing configuration, component and image versions, Operations, assignments or leases, worker liveness, result/blob reachability, and candidate write cohort.

Choose the recovery action per state population:

• Roll back a stateless binary or configuration only after proving it can read the current protocol and stored schema.
• Drain or fence candidate workers so a late completion cannot repopulate the restored path.
• Restore durable state only from a tested, compatible recovery point.
• Reconstruct queue or coordination state only from named authoritative evidence; retain an explicit unknown outcome where ownership cannot be reconciled.
• Isolate results produced by an incompatible or unidentified environment, then recompute the bounded affected cohort with a trusted producer.
• Gate returning traffic against the cold-cache, image-distribution, CAS, network, and executor demand caused by recovery.

Rollback is not time travel. Reinstalling an old scheduler does not retract candidate ActionResults, undo a schema migration, restore rotated credentials, or prove that an old lease is current. When state has crossed the rollback boundary, forward repair or explicit reconstruction may be safer than starting old code against new state.

Finish with a fresh-client proof. Run a known passing old cell and a representative candidate-disabled action with local fallback off. Confirm the intended executor placement, output correctness and availability, accepted result producer, and absence of late candidate writes. Then test the next planned upgrade as a game day, including abort, drain, rollback or forward repair, and cold-start protection. 6.3.18 Remote Execution Production Readiness consumes that evidence as a production qualification gate; a coupled real incident across cache, execution, BES, network, or regions continues in 6.6.6 Build-Service Incident Recovery.