6.2.6 Remote Cache Lifecycle

A remote cache is often recomputable, but its recovery class depends on the service promise, authoritative sources, and a measured rebuild bound. When an entry disappears, Bazel may be able to execute the action again; when thousands of clients discover a cold cache together, that same recovery mechanism can overwhelm the execution fleet, network, and storage service. Lifecycle design is therefore a capacity and availability decision built on a correctness invariant: a reported cache hit must lead to every required byte, or fail in a way that safely permits re-execution or an explicit error.

6.2.1 How Remote Cache Keys Work separated the Action Cache (AC) from the content-addressable store (CAS). Lifecycle policy has to preserve that relationship over time rather than treating “the cache” as one undifferentiated bucket.

Retain relationships, not just objects

An AC entry maps an action digest to an ActionResult. That result names output files and refers to their content by CAS digest. Several results may refer to the same CAS blob, and one result may require many blobs. These two object classes consequently have different access patterns:

• AC entries are read when clients ask whether an action already has a result.
• CAS objects are read when clients, workers, or later consumers need the bytes named by a result.
• A metadata hit may not immediately fetch every output, especially in a build-without-the-bytes workflow. Required bytes can be requested much later.

Deleting an old AC entry merely turns a future lookup into a miss. Deleting a still-referenced CAS blob can leave an apparently valid ActionResult whose output cannot be obtained. BuildBuddy documents a versioned case in which the client trusted remote blob metadata longer than the backend's LRU retention and later received Missing digest failures.¹ BuildBarn's history supplies a different implementation lesson: storage consistency and insufficient control over eviction were serious enough to drive the project away from its original S3 and Redis arrangement toward a specialized disk-backed store.²

Do not turn those cases into a universal garbage-collection algorithm. REAPI object identity does not choose your backend's TTL, LRU approximation, size limit, replication layout, or deletion order. Instead, define and test the invariant your particular backend promises. Common safe shapes include:

1. retain referenced CAS data for at least the usable life of the AC results that name it;
2. remove or invalidate AC results before their required bytes can no longer be served; or
3. make a missing referenced blob an explicit, tested condition that the supported client/backend combination converts to bounded re-execution or a clear failure.

The third shape is not permission to tolerate arbitrary dangling state. It is a recovery contract, and it must be tested under the Bazel and backend versions you operate. Bazel 7 introduced client-side handling for remote eviction and stale metadata, but the exact flags and behavior changed across releases.³ Carry those changes through the compatibility and partition plan in M4.8 Cache, Output Tree & Infra Impacts rather than assuming every client reacts identically.

Choose AC and CAS policies from a reuse horizon

Begin with the reuse you are trying to preserve. For each important workload cohort, measure the time between publication and later AC lookup, then the time between that lookup and actual CAS download. A postsubmit compile result reused within hours has a different horizon from an image layer consumed by a release job days later.

Record at least:

• AC lookup and hit age by action class;
• CAS object age at download, bytes transferred, and missing-blob outcomes;
• eviction rate and the execution/upload work it causes;
• storage occupancy by object size and age; and
• which clients keep remote references without local bytes, and for how long.

Then choose backend policy explicitly. A size-bounded LRU-like backend, a fixed TTL store, and a sharded or mirrored store can all be reasonable, but each makes different promises. State the backend and version when documenting the choice. Do not copy BuildBarn's disk-store recommendation or BuildBuddy's LRU behavior as if either were a Bazel requirement.^2,4 For example, bazel-remote exposes a backend-specific maximum-size setting and applies LRU eviction to stay within it; that is a concrete deployment choice, not REAPI lifecycle semantics.⁵

Pay particular attention to consumers outside the immediate build. The rules_img documentation, for example, distinguishes eager publication from strategies that expect image-layer bytes to remain available in a remote CAS. The CAS-backed paths inherit cache eviction and availability risk; eagerly materializing the blobs is their portable fallback.⁶ The general decision is the same: if a downstream consumer must outlive the cache's supported retention window, materialize or copy its required artifact into a store with the right durability contract. Do not silently turn a performance cache into an artifact archive.

Replication improves some failure modes but does not, by itself, lengthen the logical retention window. Replicas can apply the same eviction policy, mirror a deletion, or all lack the same object. Likewise, backing up every cache byte is not automatically sensible. Compare the cost and recovery time of backup and consistent restore against controlled recomputation. If you do restore, AC and CAS state must form a coherent point or the restore can manufacture dangling results.

Test data loss as a relationship failure

A lifecycle test should prove more than “old files were deleted.” In an isolated cache population:

1. Populate one known action result and retain its action digest, returned AC value, and referenced output digests.
2. Confirm that a clean client obtains the result and its required bytes.
3. Use the selected backend's supported administration path to make one referenced CAS object unavailable while retaining the AC result.
4. Repeat the request with the exact client/backend versions you operate.
5. Require one documented safe outcome: the state is treated as a miss and the action is re-executed, or the build fails clearly without presenting partial output as success.
6. Confirm that controlled re-execution republishes a complete result and that a fresh client can obtain it.

Preserve backend and client evidence throughout the test. An ordinary action-key miss, an expired AC entry, a dangling ActionResult, and a failed upload are different events even if all eventually execute more work. Use 6.2.4 Wrong Cache Result Diagnostics when the observed state is incomplete or its cause is uncertain. For remote execution, atomic-enough result publication and input/output availability are developed in 6.3.7 Remote Execution Storage; lifecycle starts after a valid result has been published.

A cold cache is a fleet event

A new cache instance, a deliberate purge, broad expiry, or storage loss can make an otherwise correct cache cold. The first miss is unsurprising. The risk is synchronized recovery: many clients miss the same popular actions, execute them, upload duplicate bytes, and compete with normal traffic. Cache storage, CAS network, schedulers, executors, and local fallback hosts can saturate in sequence.

Prewarming everything is usually the wrong default. It spends capacity on objects that may never be requested and can evict useful data while the real working set is still arriving. Prewarm only named, measured cohorts whose reuse horizon and criticality justify it. Prefer a bounded trusted producer set, and let observed demand establish the rest of the working set.

Define degraded behavior before the event:

• which journeys and action classes receive priority;
• how many clients or trusted writers may repopulate concurrently;
• what queue, execution, storage, and network headroom is protected;
• whether a class may execute locally, must use remote execution, or must fail closed because its trust or platform requirements forbid fallback;
• which cache observations trigger admission reduction or rollback; and
• when normal concurrency may resume.

Read-only mode is not automatically a recovery mode for an empty cache: it can prevent repopulation entirely. Conversely, opening writes to every client may amplify load and weaken the producer policy from 6.2.3 Securing a Shared Remote Cache. Keep a bounded trusted writer cohort while other clients reuse what is present or follow their approved fallback.

Run a cold-start game day

Use a separate instance or namespace so the exercise cannot erase production evidence. Establish a warm baseline first, including cache age distribution, action-class hit rate, execution queueing, CAS read/write traffic, end-to-end latency, and correctness gates. Then expose a small representative client cohort to an empty population.

Increase the cohort only while all declared gates hold. Measure:

• concurrent misses and duplicate executions for the same popular actions;
• scheduler queue time and executor saturation;
• CAS ingress, egress, request errors, and publication completion;
• local fallback CPU and disk pressure, if fallback is allowed;
• time until the important working set reaches a stable reuse level; and
• completed, correct results for every protected journey.

Include a negative stage: remove cache availability during repopulation and verify the chosen fallback or fail-closed behavior. Stop if correctness evidence is incomplete, if a protected queue or service exceeds its bound, or if a fallback runs actions outside their approved trust environment. Recovery time is the time to restore the declared journeys and evidence, not merely the time until the cache process answers health checks.

A cache-side plan is sufficient only while cache loss and repopulation remain isolated from the rest of the build service. When they interact with scheduler state, worker capacity, BES, CI admission, or another shared service, use 6.6.6 Build-Service Incident Recovery to plan recovery for the coupled incident.