6.6.5 Classifying Failures Before Retrying

A failed build is an observation, not yet a diagnosis. Repeating it immediately may produce a green result, but that does not tell you whether the original failure was a flaky test, a temporarily unavailable service, an incompatible worker, or shared state returning the wrong bytes. The safe order is: preserve the first failure, locate its layer, vary one relevant condition, and only then choose retry or repair.

6.6.4 Locating Build-Service Failures narrows the failing lifecycle transition. This article takes the next step: classify the failure mechanism well enough to decide whether another attempt is evidence, recovery, or merely more load. 6.3.13 Remote Action Failure Handling covers the attempt and cancellation semantics inside remote execution; here the unit of reasoning is the complete build service, including Bazel, tests, CI, cache, workers, and their nested retry loops.

Give each class a falsifiable meaning

Use classes that imply a different next action, not labels that merely restate the symptom.

Class	Stable explanation	Evidence that strengthens it	Appropriate response
Deterministic build or test failure	The same declared work fails under the same relevant state	Comparable attempts reproduce the same failure with the same inputs, configuration, toolchain, and environment	Do not retry for recovery; fix or revert the responsible change
Flaky test or action	Nominally equivalent executions sometimes disagree because the work itself is nondeterministic or depends on uncontrolled state	Repeated attempts under controlled placement and inputs alternate outcomes, with evidence pointing inside the test or action	Preserve attempt history; quarantine or repair the flaky work according to policy
Transient infrastructure failure	A service condition temporarily prevents otherwise valid work from completing	Structured service status, queue or capacity evidence, transport failure, or a bounded cohort incident explains the failed interval; later comparable work succeeds after that condition clears	Retry with a bounded budget and backoff at one chosen layer
Incompatible worker or service cohort	One version, image, platform, or capability cohort cannot correctly serve the request	The failure follows the cohort while a matched compatible cohort succeeds	Drain, roll back, or quarantine the incompatible cohort; retry away from it only as containment
Corrupt shared state	A shared entry or referenced object is invalid, inconsistent, or untrustworthy	The failure follows a specific digest, writer, cache instance, or stored-result cohort; bypass changes the observation and content checks provide positive evidence	Quarantine the smallest evidenced state cohort and repair its publication or storage path

These classes are hypotheses until the evidence distinguishes them. A second pass after a first failure is not sufficient to declare flakiness: the attempts may have used different workers, images, cache paths, credentials, repository state, or service conditions. Conversely, two identical-looking failures do not prove determinism if both landed on the same broken worker cohort.

Google's CI account describes test-history-based flake classification rather than asking each change author to infer flakiness from one retry.¹ A large-scale Bazel test system described at BazelCon likewise records granular test history and applies configurable analyzers, while deliberately using different execution behavior for blocking CI and flake-detection jobs.² The durable lesson is that classification needs a population and history; retrying one job is only one sample.

Read each status at its own layer

There is no single Bazel "failure status." Preserve the hierarchy instead of flattening it into one string:

Evidence surface	What it establishes	What it does not establish
Bazel command exit code and BEP BuildFinished	The outcome of the complete Bazel command; BuildFinished.failure_detail, when present, adds a structured Bazel cause	Which remote attempt failed or whether retry is safe
BEP action/target FailureDetail and TestResult.status	A structured Bazel failure at that subject, or the status of one identified test attempt, shard, and run	That every later pass is the same class, or that a test failure is infrastructure
gRPC status from Execute or its stream	The remote API call failed while creating or observing the long-running Operation	The executed tool returned a nonzero exit code
Terminal ExecuteResponse.status	The remote execution system reports an error after the Operation was created	The tool's own process exit status
ActionResult.exit_code	The remotely executed command completed with that process exit code	Whether the command was deterministic, flaky, or run on a compatible cohort

<a class="cross-ref" href="/book/5~6~1" title="5.6.1 Exit Code Taxonomy"><span class="cross-ref-id">5.6.1</span> Exit Code Taxonomy</a> is the lookup table for Bazel command outcomes. An exit code narrows the command-level symptom; it is not the service-wide diagnosis.³ Likewise, a google.longrunning.Operation is the envelope: for REAPI Execute, creation errors are gRPC statuses and execution-system errors belong in terminal ExecuteResponse.status; the server must not put an Execute failure in Operation.error. Backend worker-attempt statuses may add useful detail, but they are implementation contracts rather than portable Operation fields.⁴

BEP connects the invocation and test layers. BuildFinished carries the command outcome, failed action or target events may carry FailureDetail, and each TestResult identifies one attempt, shard, and run.⁵ Classification records these surfaces separately, then asks whether comparable observations support one of the hypotheses below.

Preserve a classification record

Start with the original invocation and attempt. Retain its structured Bazel command exit code and FailureDetail when available, per-attempt test status, gRPC status, terminal ExecuteResponse.status, ActionResult.exit_code, action and result identity, CI attempt identity, worker and image cohort, cache instance and bypass state, and queue or capacity signals. Keep the raw first-failure evidence even if a later attempt passes.

Then write the hypothesis before changing the experiment:

1. Located layer: the earliest inconsistent transition supported by the trace.
2. Candidate class: deterministic, flaky, transient infrastructure, incompatible cohort, or corrupt shared state.
3. Discriminator: the one condition whose controlled change should separate that class from its nearest alternative.
4. Allowed action: retry, reproduce, bypass, quarantine, roll back, repair, or stop.
5. Stopping rule: the evidence or budget that ends further attempts.

For example, suppose one remote test fails and the next CI attempt passes. The first trace shows worker image v42; the second shows v41. The correct record is not “flaky.” It is “failure associated with v42; image compatibility is the leading hypothesis.” A controlled rerun on both cohorts can falsify that hypothesis. If failures follow v42, drain or roll it back. If outcomes vary within both cohorts while all other recorded conditions remain comparable, test flakiness becomes more plausible.

The same discipline prevents an unsafe cache conclusion. A build that succeeds with cache reads disabled has shown that the cache path affects the outcome. It has not proved which entry is corrupt, whether the original producer was wrong, or whether bypass also changed execution placement. Corruption requires positive identity and content evidence. Preserve the implicated action result, referenced blob identities, writer or producer version, and affected cohort; quarantine narrowly instead of flushing globally.

Use repetition as an experiment

A useful repetition keeps the suspected cause stable or changes it deliberately. Record at least source revision, target scope, Bazel and rules configuration, platform and toolchain, relevant environment, action identity, worker/image cohort, cache policy, and service interval. When one of these changes, call the attempt non-comparable for the conclusion it cannot support.

Choose the experiment from the competing hypotheses:

• To distinguish deterministic failure from flakiness, repeat equivalent work while retaining per-attempt test or action evidence.
• To distinguish flakiness from incompatible placement, compare matched worker or image cohorts rather than pooling all attempts.
• To distinguish transient infrastructure from deterministic work, correlate structured service and capacity evidence with other comparable requests in the same interval.
• To investigate shared-state corruption, isolate the smallest cache instance, result, blob, or producer cohort justified by identities, then compare a controlled bypass without destroying the original evidence.

Granularity matters. Marking an entire test target flaky because one subtest is unstable can suppress hundreds of stable cases. The BazelCon case study tracks test suites and subtests where the runner exposes them, and points out that timeouts and dynamically generated test names can erase that granularity.² When evidence exists only at target level, state that limitation rather than claiming a more precise classification.

Put retry at one accountable layer

Retries can exist in the test runner, Bazel action or RPC handling, remote backend, and CI job. Their budgets multiply. If CI permits three attempts and each attempt can independently cause several lower-layer repeats, one visible retry policy can produce many executions and requests during the exact period when a dependency is degraded.

Create a retry ledger for the journey:

Layer	Failure classes it may retry	Maximum additional attempts or elapsed time	Backoff and jitter	Evidence retained	Stop signal
Bazel test scheduling	Explicitly designated test instability via --flaky_test_attempts; controlled sampling via the distinct --runs_per_test control	Defined per test cohort and purpose	Usually immediate only when classification work requires samples	Every Bazel test attempt	Any policy or time budget exhausted
Test framework or runner, if customized	Deployment-specific in-process case retry	Counted only when every internal attempt is observable	Framework-specific	Every internal case attempt	Framework and journey budget exhausted
Bazel or RPC client	Named transient transport/service conditions	One journey-level allocation	Bounded exponential backoff with jitter	Original structured status and subsequent attempts	Deadline, non-retryable status, or load gate
Remote backend	Backend-declared transient attempt loss	Counted in the same journey budget	Backend-specific and versioned	Operation and backend attempt chain	Accepted terminal result or budget exhausted
CI job	Only when the whole invocation must be reconstructed	Normally the last resort, not an extra independent budget	Delayed and admission-controlled	Original invocation plus new invocation identity	Journey budget or degraded-service gate

The numbers belong to the deployment's capacity and reliability design; there is no universal safe count. What matters is one accountable owner, a global view of amplification, and an admission rule that can reduce or stop retries when failure rate, queue depth, or dependency health deteriorates. Backoff spreads repeated requests in time; jitter prevents clients from synchronizing; neither makes an unjustified failure class retryable.

The distinction in the first two rows matters: Bazel schedules additional test attempts for --flaky_test_attempts, while --runs_per_test deliberately collects repeated samples. A language runner can also repeat cases internally, but that is a separate, deployment-specific retry site. Inventory it only when its attempts are visible; otherwise the journey ledger undercounts load.²

Do not let retry success erase the incident. Report “recovered after one classified transient retry” or “passed on a later, non-comparable CI attempt,” not simply “passed.” CI feedback must remain accessible and actionable, and test history is essential to deciding whether failures are related to the change or already unstable.¹

Turn the worksheet into a failure-load test

Validate the policy before an outage. Inject a bounded failure into one service or worker cohort and observe the complete journey:

• the first failure and its identities remain retrievable;
• the classifier reports a justified class or explicit unknown;
• only the designated layer retries;
• backoff, jitter, and the journey budget bound added requests and executions;
• admission closes or degrades safely before retries overload the dependency;
• a successful later attempt remains linked to, rather than replacing, the original failure;
• quarantine or rollback uses the evidenced cohort, not the entire fleet or cache.

The failure-classification fixture makes this policy executable. Its versioned journey cases cover deterministic action failure, transient transport failure, an incompatible worker image, and a missing scheduler-to-worker trace segment. The verifier treats these JSON records as a teaching schema—not as BEP or REAPI wire messages—and its negative controls reject retry for deterministic work, loss of first-failure evidence, guessing across an incomplete trace, two retry owners, and a nested plan above the journey budget.

Run a second case where the failure is deterministic. The system should stop, surface the actionable build or test evidence, and create no recovery retry load. Then run an ambiguous case with a missing trace segment. A trustworthy classifier must return unknown and request the next falsifying check rather than guessing “transient.”

The output is a decision, not merely a label: retry a bounded transient failure, repair or revert deterministic work, retain and route flaky test evidence, drain or roll back an incompatible cohort, quarantine and repair evidenced corrupt state, or remain unknown until a discriminator is available. 6.6.6 Build-Service Incident Recovery uses that decision to contain and recover shared services without destroying the evidence.

Repository policy still needs a human accountability model. After the technical classifier produces a bounded decision, H.7.8 Main Recovery defines the rotation that coordinates reverts, escalation, quarantine, and ownership; it does not replace the evidence hierarchy or choose a technical class by policy.