6.2.4 Wrong Cache Result Diagnostics

A wrong result from a shared cache is an incident about shared state, not just a failed build. By the time you arrive here, 5.5.3 Remote Cache Diagnostics has already isolated a suspect action and compared it with controlled, cache-bypassing builds. The next job is to determine which shared objects and producer cohort are implicated, contain only that scope, and restore safe reuse without erasing the evidence.

Build an evidence chain before changing state

Start from the action identity described in 6.2.1 How Remote Cache Keys Work. For each affected invocation, retain a record that connects:

consumer invocation
  -> instance or namespace + effective read policy
  -> action digest
  -> cache lookup outcome
  -> action result
  -> referenced output and metadata digests
  -> observed output bytes

producer invocation
  -> producer identity + Bazel/rules/toolchain/executor versions
  -> effective write policy
  -> action digest
  -> uploads and backend observations

The client-side execution log records action inputs, arguments, environment, outputs, execution strategy, and cache behavior. Backend request observations add the shared-service side of the exchange. Retain both: a successful client completion does not by itself prove that every referenced object was uploaded, and a backend aggregate does not identify which invocation produced a suspect result.^1,2

Capture the exact evidence that exists in your deployment: invocation ID, action digest, returned AC value or backend audit identity, referenced blob digests, instance or namespace, endpoint and credential path, producer and consumer identity, component versions, timestamps, cache read/write policy, and relevant request outcomes. The names and availability of backend fields are implementation-specific. NativeLink, for example, publishes cache metrics and recording-rule examples; bazel-remote exposes status, metrics, and disk-cache inspection surfaces. Those are useful evidence sources, not portable REAPI incident commands.^2,3

Do not begin by deleting entries or republishing a result. Either action can destroy the producer trail or replace the state you are trying to explain.

Classify the shared state

Use the evidence chain to make one of three decisions.

Classification	Decisive observation	Operational meaning
Healthy	Comparable clients used different action identities, a cache read was disabled, or the requested entry was absent under the observed policy	There is no demonstrated bad shared object. Fix identity, configuration, endpoint, or credential drift
Incomplete	An action result is found but one of its referenced objects cannot be retrieved, or the publication evidence stops before the required objects are reachable	The recorded result cannot be materialized completely. Investigate upload, storage, proxy, or lifecycle paths
Unsafe	A cache hit for the suspect action identity yields bytes that consistently disagree with matched, isolated rebuilds	Consumers must not reuse the implicated shared result while attribution continues

Treat an ordinary miss as healthy until evidence shows otherwise. Different command lines, environments, tool inputs, or host-derived values create different action identities and therefore different entries. Official Bazel guidance uses paired execution logs to find those differences across machines; implicit compilers and system libraries are common ways for client environments to diverge.^1,4

An incomplete result is not automatically corrupt storage. It can reflect a failed or partial publication, an object removed by retention policy, a request sent to a different endpoint or instance, or a backend fault. Retention and garbage-collection design is treated separately in 6.2.6 Remote Cache Lifecycle. Here, the important distinction is whether the producer ever established a reachable result or whether previously available state later disappeared.

Unsafe means only that reuse is unsafe for the proven scope. A matching digest establishes byte identity for the object addressed by that digest; it does not show that the action declared every input that influenced those bytes. Hidden toolchains, libraries, or environment dependencies can let a producer publish semantically wrong output while the cache stores and returns exactly what it was given.⁴

Attribute the incident without skipping a link

Work backward from affected consumers. Group hits by action digest and result or blob identity, then compare producer identity, Bazel and rules versions, toolchain or executor cohort, instance, and write window. Request-level views that can be ordered by time are useful for locating the earliest suspicious action or transfer rather than its downstream cascade.⁵

Each additional correlation narrows a different hypothesis:

• One producer cohort writes the bad result while other matched producers do not: investigate that cohort's tools, environment, executor image, rules, and credentials.
• Several independent producers disagree for the same declared identity: investigate nondeterminism, undeclared inputs, or unsafe normalization before accusing a writer or backend.
• One result consistently references an unavailable blob: trace publication, proxy/storage routing, and retention observations for that result and blob.
• Correctly uploaded bytes later change or become unreachable under the same object identity: escalate with backend integrity evidence.

These observations narrow attribution; they do not finish it. Stable disagreement with a controlled rebuild proves a bad reuse observation, not whether the cause was a compromised writer, an accidental producer defect, an underdeclared input, unsafe normalization, or a backend defect.

Bound the blast radius

The blast radius is the intersection of what was written and what could read it. Bound it along concrete dimensions rather than saying “the cache is poisoned”:

• implicated action results and referenced blobs;
• producer identities, versions, executor pools, and write interval;
• instance, namespace, endpoint, and storage route;
• consumer cohorts whose read policy can reach that state;
• downstream artifacts or releases that incorporated the observed bytes.

Absence of evidence is not evidence of absence. If producer attribution or backend telemetry is incomplete, widen the quarantine boundary explicitly and record why. Do not silently widen it to every cache object. A global flush turns an attribution problem into a cold-cache event, removes forensic state, and forces unaffected actions to execute again.

Quarantine, recover, and prove the repair

Contain the smallest boundary your evidence can defend. Depending on capabilities your actual deployment has already established, that may mean suspending write authority for an implicated producer, preventing affected consumers from reading an implicated instance, routing a bounded cohort to known-good isolated state, or invalidating a proven-bad result. None of these is a universal Bazel or REAPI feature: document the exact identity, state, and traffic boundary enforced by your cache service, gateway, or deployment policy.

Keep diagnostic rebuilds from reading or writing the suspect state. Preserve the original objects and logs until the causal defect is understood. Then remove or repair the defect before repopulation. If selective object invalidation is supported and its semantics have been verified for your backend, apply it only to the proven-bad result and any state whose reachability or correctness is also proven bad. Otherwise keep the affected cohort isolated and let correctly fixed producers populate a separate safe boundary; do not invent a deletion primitive or fall back to a global flush.

Recovery is complete only after a controlled canary shows all of the following:

1. the fixed producer executes with the intended identity, versions, and write authority;
2. backend evidence shows the expected result and required objects were accepted;
3. a fresh consumer in the intended cohort reuses that result;
4. its materialized bytes match independent controlled rebuilds;
5. the previously affected cohort can be reopened without observing the suspect state.

Use the bounded canary and abort discipline from 6.1.5 Safe Infrastructure Rollouts when reopening producers or consumers. Keep monitoring the implicated action and cohorts after recovery; a repopulated entry that merely hides the defect is not a repair.