6.6.6 Build-Service Incident Recovery

A shared build-service incident creates two risks at once. The first is the failure already affecting builds. The second is an over-broad recovery that destroys the evidence, invalidates healthy state, or sends every client back through the same cold path. Recovery therefore has two distinct jobs: contain the smallest defensible failure cohort, then restore service under explicit correctness and capacity gates.

Begin from the retained classification record in 6.6.5 Classifying Failures Before Retrying. It may identify deterministic work, transient infrastructure, incompatibility, or corrupt shared state, or it may still say unknown and name the next falsifying check. While the class remains unknown, restrict intervention to reversible containment that preserves evidence; do not turn uncertainty into a destructive repair or an unbounded retry. A restart may be appropriate for a deadlocked service, but it is not a generic diagnosis. One remote-execution incident was mitigated by restarting two unresponsive workers; the subsequent post-mortem found a cyclic RPC call graph that could exhaust HTTP/2 streams and deadlock.¹ The restart restored availability. The retained incident evidence made the durable repair possible.

Freeze the evidence before changing state

Create an incident record that identifies both the affected cohort and the state you are about to change. At minimum, retain:

• invocation and CI run or attempt identities;
• cache instance and action digest, the returned ActionResult or a backend-specific result-record identity, and every relevant referenced blob digest;
• client, rules or configuration, platform, toolchain, worker image, and backend versions;
• cache instance, writer identity, executor pool, region, and placement;
• queue, transfer, execution, and result-delivery observations;
• the required-result ledger and what evidence is missing;
• the first failure, including a local BEP file or execution record where available.

These fields turn “remote builds are broken” into a set you can test. They also preserve the joins established in 6.6.1 Tracing a Build Across Services. Capture them before flushing a cache, restarting a service, cancelling operations, changing routing, or scaling a replacement fleet. If you capture only after the repair, the surviving records describe the recovered system rather than the incident.

Record each intervention as a timestamped hypothesis: what state it changes, which cohort it protects, which evidence it may erase, what success looks like, and how to reverse it. A successful intervention is not proof that the changed component was the root cause. It is evidence that the intervention altered the failure path.

Contain by identity, not by fear

Containment should reduce new harm while disturbing as little healthy state as possible. The available boundary depends on the failure. For each case, pair the intervention with the evidence that keeps it bounded:

• Unsafe cache result
  • Containment: disable or quarantine the identified writer, namespace, action results, or affected consumer cohort.
  • Preserve: producer, action/result/blob identities, write time, instance, and divergent recomputation.
• Incomplete AC/CAS state
  • Containment: stop publication from the suspect path and route a fresh-client canary around it.
  • Preserve: the ActionResult and every referenced object, plus backend lookup and upload observations.
• Incompatible executors
  • Containment: drain or exclude the affected image, platform, pool, or version cohort.
  • Preserve: operation, placement, image, toolchain, and worker-attempt identities.
• Scheduler or queue overload
  • Containment: reject, defer, cancel, or cap the identified workload cohort.
  • Preserve: admission, queue age, request owner, action class, and cancellation outcome.
• BES or consumer failure
  • Containment: preserve local event output and isolate publication or consumption without declaring execution failed.
  • Preserve: invocation, announced and received event sets, acknowledgements, and referenced artifacts.
• Region failure
  • Containment: stop new placement in the affected region and fail over only the supported cohort.
  • Preserve: routing decision, replicated-state availability, and in-flight operation disposition.

These records name desired containment boundaries, not portable REAPI administrative operations. REAPI addresses an ActionResult by instance_name plus the digest of its encoded Action; the result then references output, stdout, and stderr blobs. It does not define a standard result digest, producer field, writer-disable control, or delete/quarantine RPC. Selective writer or result quarantine therefore requires a backend control plane that retained producer attribution and exposes that boundary. When it does not, use a coarser portable path: stop cache reads or writes for the evidenced instance or client cohort, preserve the suspect entry, and route a fresh canary through a clean instance or cache-bypass path.²

A global cache flush is rarely the first safe move. It mixes known-bad and known-good objects, removes producer attribution, and converts every former hit into lookup, execution, upload, and materialization demand. Likewise, restarting an entire executor fleet can erase attempt evidence while forcing image and input cold starts.

Containment can also be a deliberate service fallback. During Airbnb's Bazel migration, Bazel and Gradle builds coexisted, and the organization could disable Bazel during remote-cache or remote-execution incidents so developers fell back to Gradle.³ That is a useful case, not a universal recipe: a fallback is safe only if it still produces the required correct result, its supported cohort is known, and its capacity has been tested. A local-execution fallback that shares the same corrupt cache writer or cannot absorb the action volume has not isolated the failure.

Recover a dependency graph, not isolated components

Build-service components impose load on one another. Replacing cache state causes action-cache misses, CAS transfers, and recomputation. Replacing workers causes image and input downloads. Moving traffic to another region consumes its storage, network, scheduler, and executor headroom. Falling back locally moves work onto developer or CI machines and may still use shared storage.

Write the recovery as a staged demand plan:

1. Protect correctness. Keep suspect writers, results, images, or routes out of the canary path. Do not publish results from an unqualified replacement.
2. Prove a narrow path. Use fresh client state and a representative action cohort. Verify action placement, result bytes, required event delivery, and artifact availability—not merely a zero exit code.
3. Rate-limit re-entry. Increase traffic by a declared cohort while watching lookup, queue, transfer, execution, upload, materialization, and result completeness indicators from 6.6.3 Measuring Build-Service Latency and Reliability.
4. Coordinate cold demand. Budget storage reads, network transfer, image pulls, executor work, result writes, and autoscaling lag together. Pause or reverse promotion when any protected gate leaves bounds.
5. Retire temporary state deliberately. Drain old operations, remove quarantine only with evidence, restore ordinary routing, and record which fallback results may be reused.

Do not restore every component independently to its own “healthy” dashboard. An empty queue can coexist with a failed admission path; a cache can be available while referenced blobs are absent; a replacement pool can be ready while a synchronized miss storm overwhelms storage. The recovery criterion is an end-to-end cohort producing correct, complete results within the declared service bounds.

Remote-execution post-mortems show why the combined view matters. Aggressive scale-down plus slow scale-up produced alternating slow and normal builds, while an unrelated logging change consumed CPU and the logging pipeline dropped the excess evidence.¹ In another incident, one build submitted a very large body of long-running compile actions; the backend was behaving according to its API while the accepted workload exhausted useful capacity.¹ Recovery controls therefore need workload admission and evidence completeness as well as nominal component health.

Treat state identity as part of the repair

Some incidents survive a source revert because the shared service retained a different identity. A warm Bazel server can preserve state that a fresh process would recompute. A cache result can outlive its producer. A merge queue can keep testing work based on the state that existed before the fix.

In one in-repository module-registry incident, warm CI Bazel servers retained registry files that were expected to be immutable. The defective change was not detected until a scale-up introduced fresh processes. Recovery required more than reverting the commit: the module version changed, the fix was fast-tracked, running merge-queue jobs were stopped, and the queue was rebased.⁴ The lesson is not to repeat those exact steps for every incident. It is to inventory every stateful consumer and version boundary affected by the repair.

For each repaired component, ask:

• What makes old and replacement state distinguishable?
• Which clients, servers, queues, or consumers can still hold the old identity?
• Can the old state be drained safely, or must its bounded cohort be discarded?
• Which completed results remain valid, and which must be recomputed?
• What proves that new work cannot be joined to the old unsafe cohort?

This is narrower than a planned remote-execution upgrade. Planned draining, compatibility matrices, and rollback for executor evolution are developed in 6.3.17 Remote Execution Upgrades. Incident recovery starts with an unplanned, partly observed condition and prioritizes evidence preservation and containment.

Define recovery before declaring it

“The graph is green” is not a recovery criterion. Declare recovery only when a named representative cohort has met all of these conditions:

• suspect state and producers remain contained or have been proven safe;
• correct outputs were produced or reused through a fresh-client path;
• required BEP/BES results and referenced artifacts are complete;
• queueing, storage, network, executor, and fallback load remain inside the staged capacity gates;
• old in-flight work and retained state have an explicit disposition;
• ordinary routing can resume without relying on the incident override;
• the retained record supports a follow-up hypothesis and a reproducible game day.

Then test the recovery design deliberately. Seed one bounded cache, executor, queue, result-delivery, or regional failure. Measure time to containment, the scope of changed state, evidence retained, cold-load peak, fallback saturation, and time to a complete representative result. A useful game day can fail its recovery target and still succeed as an experiment if it exposes a false assumption before production does.

The build-service recovery rehearsal makes that decision contract inspectable without pretending to simulate an RBE backend. Its versioned cases cover bounded recovery, safe handling of an unknown class, a destructive unknown-state intervention, and transferred load that exceeds a protected stage's budget. The verifier requires the five stages in order, preserves the first failure, rejects destructive action while classification is unknown, and blocks promotion when correctness, result completeness, load, old-state disposition, or ordinary routing remains unsafe. These records are a teaching schema. Replace their fields and thresholds with evidence and limits from your deployment.

The post-mortem follows mitigation rather than replacing it. It should assess impact, identify technical and process causes, and assign concrete corrective work.¹ Incident command, escalation policy, organizational risk thresholds, and service ownership belong to H.10.2 Service Reliability and H.7.8 Main Recovery. Here, the Bazel-specific deliverable is a recovery path whose state identities, capacity consequences, and evidence are explicit.