5.5.3 Remote Cache Diagnostics

A remote-cache percentage is useful only after you define what was eligible to hit the cache. A clean CI build after a broad source change, a repeated build of the same revision, and a developer's incremental build exercise different action sets. Mixing them into one number can hide a regression or manufacture one.

The same discipline applies to correctness incidents. A cache miss means Bazel did not reuse an entry. It is not evidence that an entry was poisoned. Start by classifying the observation, then collect the narrowest data that can distinguish changed action identity, disabled caching, and an incorrect reused result.

Turn hit rate into a comparable signal

Track cache behavior by a stable cohort: repository and branch class, command or workflow, platform, Bazel version, relevant configuration, and whether the run is cold, repeated, or incremental. Record action counts as well as ratios. Ten misses among twenty eligible actions and ten misses among twenty thousand need different responses.

Do not adopt a universal “healthy” percentage. The mapped sources describe cache-hit trends and drill-downs, but they do not establish an 80% threshold. Set a baseline from comparable successful invocations, then alert on a sustained change from that baseline. Segment by mnemonic or target when an aggregate stays flat: one expensive action class can lose all of its hits while a large volume of cheap hits keeps the overall ratio reassuring.¹

Bazel's final process summary is a quick invocation-level check. It labels results reused from the remote cache as remote cache hit. Local cache hits are not included in that summary. Treat it as triage, not as a complete historical metric.² A BES-backed dashboard can retain invocation trends, while request- level telemetry can answer which action transferred an artifact and when the first cascade of executions began.³ 5.5.1 Build Event Protocol (BEP) introduces the event stream, and 5.5.2 Metrics & Monitoring develops the monitoring pipeline.

Diagnose misses from the first divergence

Use a controlled pair of invocations. First repeat the same target, revision, flags, and platform. Only after reuse works on one machine should you compare CI with a developer host or two worker pools. This separates a basic cache or configuration problem from cross-environment action-key drift.²

For unexpected rebuilding, --explain=<path> with --verbose_explanations provides a quick, high-level account of why Bazel considered work out of date. It is a lead, not proof of why a remote lookup missed. The decisive comparison is between execution logs from the two builds:

bazel build //app:release \
  --execution_log_compact_file=/tmp/exec-a.log

bazel build //app:release \
  --execution_log_compact_file=/tmp/exec-b.log

An execution-log record includes the action's command line, environment, inputs, outputs, execution strategy, and cache behavior. Bazel's compact format is recommended because it is much smaller and has little runtime overhead. The portable operation is to capture comparable Bazel logs, align the relevant action records, and compare their declared identity inputs and cache observations. Bazel's source-tree execution-log parser can produce an aligned textual comparison. As a version-checked convenience case, bb explain can automate a structural comparison and trace an early changed action through its transitive effects.^2,4 Neither tool invocation proves that the first reported divergence is the ultimate cause; it supplies a lead to verify.

Interpret the first meaningful divergence before looking at the rebuild cascade:

One documented cross-machine investigation found HOME inherited into a test environment. The different values changed the action between CI and the developer host. Replacing the inherited value with the constant the test actually required restored sharing.¹ Other common causes include timestamps embedded in archives and absolute host paths in generated metadata.⁵ These are action or output determinism defects, not cache-server corruption.

If actions appear identical and cacheable, inspect the effective configuration before blaming storage. A BEP text file contains a structured_command_line message labeled canonical. It shows options after rc-file expansion. In particular, --noremote_accept_cached disables accepting cached results.² 5.7.2 Execution Log Analysis covers deeper log comparison and tooling.

Require stronger evidence for cache poisoning

Use cache poisoning narrowly: a lookup for a particular action identity reuses an output that is wrong for that action. Low hit rate cannot establish this—a poisoned entry must be hit to have an effect. Nor does a changed output with a changed action key establish it. That is a different cache entry.

When wrong output appears only with cache reads enabled, preserve the failing invocation, the reused action, and the suspect action-result and blob identities. Then run independent builds with remote reads and writes disabled under matched inputs and compare their outputs in isolated local state. Two uncached builds that disagree point first to non-determinism. Stable independent outputs that agree with each other but disagree with the reused result are much stronger evidence that the cached action result is bad. Compare the earliest divergent artifact with a binary-aware tool such as Diffoscope, and trace it to its producer rather than deleting the whole cache and losing the evidence.⁴

Use an exact matched-build test to make that claim reproducible:

1. Pin the same revision, target, Bazel version, platform, toolchains, environment, and non-cache flags. Record the digest of the wrong artifact from a read-enabled build and retain its BEP and compact execution log.
2. Create two new, empty output bases. Do not clean or delete the original output base or any shared cache. In each new output base, repeat the build with remote cache reads and writes disabled and with --disk_cache=, then capture the same evidence. Use the supported flags for the pinned Bazel version and confirm from the canonical command line and backend observations that the diagnostic builds neither read from nor wrote to the suspect cache.
3. Confirm in the canonical command line and execution logs that the same producer action was compared. Hash its output from all three builds. If the two bypass outputs match each other while the read-enabled build records a cache hit for the same action identity and yields different bytes, the test is evidence of wrong reuse. If the bypass outputs differ, investigate non-determinism instead.

The controls have separate jobs. Disabling remote reads prevents reuse, while disabling remote writes prevents the diagnostic rebuilds from overwriting or adding evidence in the suspect shared cache. --disk_cache= disables a configured disk cache for these runs.

A fresh --output_base prevents the local action cache and existing output tree from satisfying the action. The repository cache stores downloaded external artifacts, not action results, so disabling or deleting it does not strengthen this test. Keep remote execution and every other execution setting matched so the test changes cache reuse rather than the execution environment. Stable disagreement establishes only that the cached result is not equivalent to the controlled rebuild for the observed action identity. It does not alone distinguish a compromised writer, an underdeclared input, unsafe key normalization, a producer-version defect, or a backend defect. Continue with 6.2.4 Wrong Cache Result Diagnostics when the investigation must attribute a bad shared result to infrastructure or a writer trust domain.

A non-hermetic producer can populate a shared cache with semantically bad bytes even when the cache service stored exactly what it received. That makes provenance important: record which invocation and trust domain wrote the suspect result. 6.2.4 Wrong Cache Result Diagnostics develops containment policy, such as restricting which builds may write to a shared cache, and infrastructure controls for untrusted writers. Here, the key decision is whether the evidence points to action-key drift, a non-cacheable or read-disabled configuration, a non-deterministic producer, or a wrong result actually reused for the same action identity.