6.3.13 Remote Action Failure Handling

An interrupted remote build can leave more than one thing in motion: a gRPC call, a REAPI Operation, an executor process, and perhaps a backend lease. Treating them as one object produces the two dangerous responses: retrying at every layer, or accepting whichever result arrives first. The operational goal is narrower: give one named cross-layer policy owner accountability for the whole recovery path, preserve the first failure, and either accept one evidenced result or stop with uncertainty.

Start with the normal trace in 6.3.4 Remote Action Tracing. It supplies the action digest, Operation.name, worker and backend correlations that a failure decision must retain. This article decides what to do after that trace breaks; 6.6.5 Classifying Failures Before Retrying separately decides whether a fleet-wide incident is transient, incompatible, corrupt, flaky, or deterministic.

Four lifecycles can disagree

The terms retry and cancel are too vague until you name their subject.

Subject	Portable fact	What must be proven locally
RPC and Bazel client recovery	Bazel 9.1.0 has more than one built-in retry site around remote work; an RPC failure is not one invocation-wide action-attempt counter.	Which Bazel retry site, method, status, delay source, and per-site counter applied.
REAPI execution	Execute yields a streamed long-running Operation; after an operation name is known, WaitExecution can resume observation. WaitExecution may return NOT_FOUND, in which case REAPI tells the client to call Execute again.	The action digest, operation name, every stream loss, and each terminal ExecuteResponse.
Backend attempt	A scheduler may assign an executor, issue a lease, reclaim it after a disconnect, or run duplicate work.	The backend's revision, attempt/lease identity, expiry, requeue rule, executor death handling, and orphan-cleanup witness.
Result acceptance	A terminal response can report an ActionResult; it is not automatically the result your protected workflow may trust.	The chosen response, result and output digests, cache/publication witness, placement/worker trust evidence, and conflict rule.

REAPI intentionally does not make Execute at-most-once. A server may run the same action more than once, even in parallel, and redundant executions may keep running after an Operation completes.¹ A lost client stream therefore does not prove that the process stopped, and a backend lease is not a portable REAPI field. In one BuildBarn design, a worker periodically asks whether it should continue; following a client interruption, its scheduler waits about a minute for reconnect before cleanup.² That is a useful implementation contract, not a timeout to copy into another service.

Give the whole recovery path one accountable owner

For a Bazel 9.1.0 deployment, name one cross-layer policy owner for the action's total recovery behavior. That owner does not pretend to be the only component that can repeat work. It sets the outer deadline and cancellation budget, decides when CI may start another invocation, records all retry sites, and rejects a result that crosses a placement or trust boundary. The actual mechanisms remain separate and must all be accounted for.

Bazel 9.1.0 has two independent action re-execution and Operation-recovery sites that use the same --remote_retries default of five, but do not share a counter:^3,4

• GrpcRemoteExecutor uses a RemoteRetrier around its Execute/WaitExecution state machine. It retries WaitExecution on a known Operation after a retriable stream failure, and retries Execute when the Operation is lost or a retryable execution result needs a new Operation.
• RemoteSpawnRunner separately wraps input upload, remote execution, server-log/output handling, and forced re-execution in an ExecuteRetrier.

The same action path has a separate cache-RPC retry site: GrpcCacheClient uses its own RemoteRetrier for Action Cache and CAS work, including action result lookup/upload, FindMissingBlobs, and blob download; its ByteStreamUploader uses that retrier for input upload.⁴ Configured remote-output or remote-downloader services can add further users of the generic retrier. Inventory every enabled site rather than treating these three as a fixed complete list.

Each retry execute() call creates a fresh backoff, so --remote_retries=5 is a per-call, per-site maximum, not a cap on an action or invocation. Nor does --remote_retry_max_delay=5s cap every delay: it limits only the exponential backoff used by RemoteRetrier, while ExecuteRetrier uses a server-provided RetryInfo.retry_delay directly.⁴ When an Execute or WaitExecution stream ends successfully with a nonterminal Operation, the remote executor also reconnects in a loop outside RemoteRetrier. Treat that as a distinct observation-recovery site, not as proof that the five-attempt setting bounds client traffic.⁴

Make the ownership rule operational:

1. Give the policy owner an explicit outer deadline or cancellation budget that applies across client recovery, backend attempts, and CI. Test that it stops new work and records what may remain orphaned; no Bazel retry flag supplies this global counter by itself.
2. Inventory every enabled Bazel retry site—at minimum cache RPC, executor/Operation recovery, and spawn-level execution—their per-site attempt/delay behavior, and every Operation name they observe or replace. Record a nonterminal stream reconnection as its own event.
3. Let a backend reclaim a dead worker or expired lease only under its pinned reconciliation contract. This is an independent attempt controller, not a reason to hide the old lease, new attempt, or recovery reason from the policy owner.
4. Permit a CI rerun only after the policy's outer stop condition and evidence rule say it may begin. It must retain the first invocation and action evidence rather than replacing it with a later pass.

This prevents amplification. One action can retry cache lookup or input/output transfer, consume retries in the remote executor, then in the spawn runner, wait through a server-provided retry delay, reconnect repeatedly to a nonterminal Operation, and be requeued by a backend before CI considers another invocation. Optional remote services can add still more traffic. No single local setting gives the sum. The point is not to find a magic global number; it is to have one accountable policy enforce the outer stop condition while its ledger exposes the work each mechanism creates. The 2024 post-mortem of a build that sent 172,000 timing-out compile actions is a useful reason to make cancellation, per-build guards, and admission visible rather than relying on retries to resolve overload.⁵

Cancellation requests; it does not erase work

First decide what the client needs: stop waiting, request cancellation, or prove that execution stopped. They are different outcomes.

The REAPI Execution service does not require a server to implement the other Operations methods. Where CancelOperation is available, the long-running Operations contract says it begins asynchronous, best-effort cancellation. A successful RPC reply does not guarantee the operation stopped; the client must observe a later terminal state, which may show completion despite cancellation.¹

Use this sequence when an invocation is interrupted or a deadline expires:

1. Preserve the first failure before cancelling: invocation and action IDs, action digest, Operation name, gRPC status/deadline, cache decision, last Operation stage, backend attempt/lease and worker identity when available.
2. If the backend supports it, issue the documented cancellation request and mark its acknowledgement as requested, not stopped.
3. Continue the backend's documented reconciliation path long enough to obtain a terminal response, a confirmed cleanup, or an explicit unknown state. A client disconnect alone satisfies none of those.
4. Fence any late work by the backend's attempt identity. If it cannot prove which attempt still owns outputs or capacity, do not reuse its result.

BuildBuddy's public routing case cancels related actions after it observes an interrupted Bazel build event, and its hedged executions cancel losing copies. Those are BuildBuddy-specific controls; REAPI neither requires build-event driven cancellation nor says that the first hedged completion wins.⁶

Choose one terminal interpretation

For an ordinary cacheable, deterministic action, a successful terminal ExecuteResponse can be accepted only after the result is reachable and the action ran in an allowed trust domain. Confirm the output closure and result publication using 6.3.7 Remote Execution Storage, then join it to the compatible placement and worker-environment evidence from 6.3.2 Remote Executor Matching and 6.3.9 Remote Executor Environments and Isolation.

Do not implement a universal “first completion wins” rule. If a cancelled, reclaimed, or duplicate attempt completes late, compare its action digest, attempt identity, status, result/output digests, cacheability, and trust attributes with the selected terminal response. A matching duplicate is evidence to retain; it does not need a second publication. A conflicting, incomplete, untrusted, or unjoinable completion is an explicit uncertainty: quarantine its result, preserve its evidence, and fail closed until the backend-specific reconciliation contract explains it. Re-running locally does not prove which remote output was valid.

Degrade by trust and placement, not by convenience

The safest degraded mode depends on what the action and its consumer require. Write the permitted mode in the action-class contract before an incident.

Mode	Use it only when	Required proof	Do not infer
Cache-only	A previously trusted result is acceptable, but fresh remote work is not.	A named remote-cache read path, result/blob reachability, and proof that the cohort did not use a remote execution strategy.	That a cache hit proves a worker is healthy today.
Remote disabled	Local execution is an approved exception for this action class.	Execution-log evidence of local placement, plus the stated exception owner.	That the local green result repaired a remote failure.
Read-only cache	A local result may be consumed but must not become shared evidence.	Client upload disabled and server-side writer authorization showing the producer cannot publish.	That --noremote_upload_local_results alone prevents every remote writer.
Reduced capacity	The service can safely admit a bounded cohort while it sheds load.	Backend admission, compatible-pool, queue, and failure-headroom evidence.	That all queued work will finish merely because the endpoint responds.
Local fallback	The class is locally reproducible, the output is acceptable under local trust, and the service policy permits it.	Fallback event, actual local placement, and an explicit decision about whether a local result may be uploaded.	That fallback tested remote readiness or preserved the original failure.
Fail closed	Remote placement, a protected executor, or a trusted remote result is mandatory.	Local fallback disabled, a visible failure, and retained first-failure/placement evidence.	That availability authorizes a silent trust-boundary bypass.

The Bazel controls make several rows concrete. --remote_local_fallback is off by default and invokes the standalone local strategy when remote execution fails; in 9.1.0 it does not apply after an execution timeout. Locally executed results may be uploaded by default when the remote cache supports it and the client is authorized, so a fallback policy must decide whether to disable --remote_upload_local_results and separately enforce server-side write authority.^3,4 A diagnostic or protected-builder cohort normally chooses fail-closed, because a fallback success is not remote placement evidence.

Keep a retry and fallback ledger

For each incident or game-day case, keep a small row per action rather than overwriting it with the final result:

Field	Why it matters
First failure	Preserves the original status, phase, deadline, and time before retries change the symptom.
Attempt chain	Connects every enabled retry site (at minimum cache, executor, and spawn) and its per-site counter, RPC call/stream, Operation names, and backend lease or executor attempts without pretending they are one ID or one shared counter.
Placement and trust	Distinguishes remote, local fallback, cache hit, compatible worker/image, and allowed producer.
Decision	Names retry, wait, cancel-requested, cache-only, local fallback, fail-closed, or unknown, plus the cross-layer policy owner and outer stop condition.
Terminal witness	Records the selected ExecuteResponse or cache result, digest closure, publication check, and treatment of late/duplicate completions.
Capacity effect	Counts orphan work, requeues, and fallback load against the named pool or service budget.

Exercise this ledger with a controlled stream loss after an Operation is known, a worker death during execution, an unavailable backend, and a late duplicate completion. The pass condition is not simply that a build turns green. It is that the action has one accepted result with complete evidence—or an explicit uncertainty that prevents a protected workflow from silently using an untrusted output.