6.6.4 Locating Build-Service Failures

A message such as “the remote build failed” names a journey, not a failing component. The same visible symptom can begin in the source tree, Bazel's analysis, credentials, cache storage, scheduling, execution, result delivery, or the system that consumes the result. Reliable triage finds the earliest transition whose observed outcome contradicts its contract, then asks for the cheapest evidence that could prove that attribution wrong.

Follow transitions, not error words

Start with the correlated lifecycle from 6.6.1 Tracing a Build Across Services. Mark each transition as confirmed, contradicted, or unknown:

source/configuration
  -> Bazel loading and analysis
  -> remote authentication and capability negotiation
  -> action-cache lookup and CAS input availability
  -> scheduler admission and placement
  -> worker input fetch and action execution
  -> output upload and result publication
  -> Bazel materialization and result interpretation
  -> BEP/BES delivery
  -> CI or dashboard consumption

This is not one mandatory pipeline. A local execution path can use a remote cache without a scheduler or remote worker. A successful action can finish even when BES delivery or a downstream dashboard fails. The Build Event Protocol (BEP) and Build Event Service (BES) are also separate: Bazel produces structured events, while BES transports those events to another service.¹

The portable Remote Execution API defines capabilities, CAS and Action Cache operations, Execute and streamed Operation messages, and the ActionResult that refers to output blobs. It does not define a backend's admission queue, placement algorithm, lease or attempt identity, worker identity, or worker telemetry.² Treat those scheduler and worker joins as contracts of the implementation you operate, not as fields every REAPI service must expose.

The diagnostic rule is simple: do not jump past a transition you have not established. If Bazel never produced an action, executor telemetry cannot explain the failure. If a worker completed and published a result but the dashboard lacks it, “remote execution failed” is too broad. If a cache hit returns the wrong bytes, the symptom appears at the consumer, but the unsafe state may have been published by an earlier producer.

Build a layer hypothesis

Use one row per candidate boundary. Record the identity you followed, what evidence supports the boundary, what remains missing, and one next check that could falsify your current conclusion.

The table routes an investigation; it does not assign root cause automatically. For example, a scheduler queue can grow because executors are unhealthy, because no platform-compatible worker exists, or because one client submitted excessive work. A production incident in which one build submitted a very large set of long-running compile actions illustrates this distinction: the queue was the visible bottleneck, while the triggering workload originated at the client.³

Preserve uncertainty explicitly

A gap in telemetry is not evidence that the unobserved layer failed. Write the narrowest defensible result:

The action was accepted by the remote service, but no scheduler-admission or worker-receipt evidence was retained. The failure is bounded to that missing segment; it is not yet attributable to either scheduler or executor.

This wording is operationally useful. It tells the incident owner what is known, prevents an unsupported hand-off, and identifies the next instrumentation or query. It also avoids timestamp-only joins. Two records occurring near each other are weaker evidence than records joined by invocation, action, operation, result, or blob identity.

BEP completeness needs the same care. Its event graph announces child events, but a Bazel crash or failed transport can leave announced events unposted.⁴ Therefore a missing event can mean “the work did not happen,” “Bazel did not finish producing evidence,” or “delivery was incomplete.” Compare local BEP, BES acknowledgement, and consumer receipt before choosing among those layers.

Choose a discriminating check

A useful next check separates at least two plausible layers. “Retry the build” usually does not: it changes timing, placement, local state, and perhaps cache state at once. Prefer a bounded comparison that changes one boundary or inspects both sides of it.

• If local execution with the same declared action succeeds while remote-only execution fails before tool launch, inspect remote compatibility, input transfer, and worker setup—not the source-level test assertion.
• If a fresh client reads a remote result but cannot materialize a referenced blob, inspect the ActionResult-to-CAS relationship rather than scheduler latency.
• If a local BEP file contains the terminal result and the BES consumer does not, execution is no longer the leading hypothesis. Compare publication, acknowledgement, and consumer persistence.
• If cache access is suspected, first distinguish endpoint communication from action-identity mismatch. Bazel can report remote-cache read/write warnings for local execution, while execution-log comparison can expose different inputs, arguments, environment, or outputs across invocations.^5,6

Do not repair yet. Clearing shared state, restarting a fleet, or enabling broad fallback may erase the producer, version, placement, or incomplete-transfer evidence needed to identify the boundary. First bound the failing segment and retain the first-failure record. 6.6.5 Classifying Failures Before Retrying then uses that location to decide whether the failure is deterministic, flaky, incompatible, transient, or corrupt—and whether retry is justified.

Once a build-cop or on-call rotation receives an incident, this bounded layer, the retained identities, and the first falsifying check form a much stronger handoff than an error-message category. H.7.8 Main Recovery develops the human ownership and coordination policy around that handoff.