6.1.5 Safe Infrastructure Rollouts

A remote build can stay green while a fleet change makes only one client, toolchain, worker image, or stored-result cohort unsafe. Roll out shared Bazel infrastructure by tested compatibility combinations, not by component version alone: expose a bounded canary, reject unsupported combinations before useful work depends on them, and know separately how to abort, fall back, and roll back.

6.1.1 Bazel as a Shared Service established the protected journeys, owners, and stop signals. This article turns those service outcomes into a rollout method. It does not replace the repository-upgrade work in M4.1 Ruleset & Toolchain Compatibility Matrix or the remote-execution draining and recovery procedure in 6.3.17 Remote Execution Upgrades.

Compatibility Is a Matrix, Not a Version Line

A shared build path is a combination of independently changing parts. Record the exact old and candidate identities for each dimension, the combinations you intend to support during the transition, and the evidence that makes each cell pass or fail. Compatibility is meaningful only with respect to actual consumers and combinations; a version number alone cannot establish it.¹

Dimension	Compatibility question	Evidence for one matrix cell
Bazel client	Can this client use the required cache, execution, and result path?	Capability observation plus a representative request and result
Rules and configuration	Does this revision create the intended actions and remote settings?	Effective configuration and representative action evidence
Digest, compressor, and protocol capabilities	Do client and service share every required wire behavior?	Advertised capability range and a positive and negative probe
Toolchain	Do selected tools run in the promised execution environment and declare output-relevant inputs?	Resolution evidence plus a remotely executed representative action
Platform	Do Bazel's selected constraints and action properties route to a compatible pool?	Requested properties joined to placement evidence
Executor image	Does the worker supply the intended ambient runtime without changing outputs invisibly?	Immutable image identity and an output-equivalence or rejection test
Backend components	Can frontend, storage, scheduler, and executor roles interoperate in the candidate topology?	Cross-component cache and execution integration tests
Stored results	May old and new producers safely share Action Cache and CAS state?	Matched action/result/blob identities and controlled old/new producer-consumer tests

REAPI's Capabilities service reports supported API ranges, digest functions, and compression behavior, while the Action, Command, Platform, and result contracts describe the portable exchange.² This makes capability evidence necessary, but not sufficient. An endpoint can advertise a feature that an older Bazel client does not use correctly for your workload; a client and server can also agree on the protocol while the selected toolchain cannot run in the worker image.

Semantic version order does not collapse those questions. “Both releases are newer” or “the major versions match” says nothing about whether the required capability intersection exists, whether two independently released backend components interoperate, or whether an action produced under one ambient runtime is safe to reuse under another. Buildfarm's focused server, worker, queue, and integration configurations make those separate roles visible.³ SemVer is a lossy compatibility estimate: real safety depends on which consumers and combinations are being tested.¹

Do not use a target or platform label as a substitute for an action-identity argument. A toolchain or platform change can alter declared tools, command, inputs, environment, or execution properties. An executor-image change can instead alter an ambient library or runtime that the action failed to declare. The former may naturally produce a different action identity; the latter can leave apparently matching identities with different real behavior. The matrix must test the consequence, not merely compare names.

Mark Every Cell Pass, Reject, or Unknown

Build the smallest matrix that covers the actual transition window. If old and new clients, backends, and images may coexist, test those mixed combinations; testing only old-everything and new-everything misses the risky cells.

For each cell, record one of three outcomes:

• Pass: the required path completes with correct outputs and complete evidence, and the action's placement and result producer match the intended combination.
• Reject: the combination is deliberately unsupported and fails before it can publish or consume an untrusted result.
• Unknown: evidence is missing or cannot distinguish fallback, cache reuse, and candidate execution. Unknown is a stop signal, not permission to promote.

A useful negative canary asks for a combination that the candidate is meant to reject—for example, a required protocol capability outside the observed client/server intersection, or an execution-platform request for which no compatible candidate pool exists. A visible rejection proves more than a green happy path: it shows that incompatibility is not silently hidden by local fallback or ambiguous worker matching. The remote-apis protocol map supplies the capability and platform-property boundary, while implementation repositories supply the scheduler and worker evidence.^2,3

The positive canary should exercise a representative action class from a protected journey. Retain the invocation, action, result, producer, client, backend, toolchain, platform, and image identities. Then require the expected cache or execution path rather than accepting “build succeeded.” Buildbarn's runnable deployments separate frontend, storage, scheduler, runner, worker, image/configuration, and monitoring surfaces, which makes a mixed-version cell observable instead of treating the deployment as one binary.⁴

Expand Exposure Only While the Matrix Holds

Canary by a boundary you can identify and reverse: named Bazel client cohort, CI journey, action class, execution platform, worker pool, or cache instance. Do not choose a random percentage if it mixes every language, toolchain, and platform while omitting the rare combination most likely to fail.

Bounded canaries shorten the feedback loop, but coexistence can introduce version skew across code, data, and configuration; static configuration should be tested with the candidate it accompanies.⁵

Use this promotion loop, applying progressively broader feedback only while the candidate and its configuration remain a tested unit.⁵

1. Observe compatibility before exposure. Capture advertised capabilities, effective client configuration, candidate component and image identities, and the intended cache namespace or instance.
2. Run rejecting cells. Confirm unsupported combinations stop visibly and cannot reach a writer path through fallback.
3. Run passing cells. Exercise representative cache misses, hits, and remote actions. Join the client observation to backend placement and stored-result evidence.
4. Protect the journey. Apply the correctness and completeness gates and stop signals defined for the service, then compare latency or throughput only within the matched cohort.
5. Expand one compatibility boundary. Add a client cohort, action class, platform, or worker pool. Re-run cells whose producer-consumer combinations changed.

Invocation evidence remains useful at the canary boundary. Use separate cache, queueing, critical-path, parallelism, and placement signals to detect a displaced cost, but do not let one invocation stand in for the fleet compatibility matrix.

Abort, Fallback, and Rollback Are Different Controls

Define all three before exposure:

• Abort stops new exposure. It freezes promotion but does not by itself cancel queued or running work, erase candidate results, or move existing clients.
• Fallback gives affected clients a bounded alternate path, such as the old endpoint, a cache-only path, an isolated instance, or local execution when that path satisfies the journey's correctness and trust requirements.
• Rollback restores a previous client, configuration, backend component, or image. It changes only the surfaces controlled by that reversal.

These controls must name state. A scheduler or coordinator may still own queued or running operations when exposure stops. A candidate producer may already have published ActionResults and output blobs. Reverting deployment configuration does not retract either class of state. Buildfarm and Buildbarn show queues, storage, schedulers, and workers as distinct implementation roles; their lifetime and recovery behavior must therefore be established from the chosen backend rather than inferred from REAPI alone.^3,4

For queue and operation state, decide whether old work drains, is rejected, moves to an isolated pool, or is explicitly abandoned with preserved evidence. Do not call a worker kill a drain. Use 6.3.17 Remote Execution Upgrades for the RBE-specific draining mechanics and recovery tests.

For cache state, classify results produced during the canary:

• Results whose complete output-relevant environment is represented by the action identity may remain valid across the rollback.
• Results from a combination that is proven incompatible or whose ambient image dependency is missing from action identity must not remain readable by an affected consumer cohort.
• Results with incomplete producer or compatibility evidence remain unknown; isolate them rather than declaring the whole cache safe or flushing it all.

The recovery action might be namespace or instance isolation, read/write policy change, selective quarantine, or controlled recomputation, depending on the backend contract. A global cache clear is not a rollout plan: it destroys evidence, creates a cold-load event, and does not repair the omitted identity or incompatible producer.

After rollback, rerun a known passing old cell with a fresh client observation. Confirm the intended placement or cache path, validate required outputs, and verify that late candidate work cannot silently repopulate the restored path. Only then has the previous service behavior been recovered rather than merely the previous deployment files.