6.2.3 Securing a Shared Remote Cache

A shared remote cache is a trust boundary, not just a performance service. A cache hit lets one producer's bytes stand in for another client's execution, so the security question is not merely “can this client connect?” It is “which identity may publish a result that other clients will trust, where may it publish, and how will we contain results if that identity is compromised?”

6.2.1 How Remote Cache Keys Work established that an action key is a cross-client equivalence claim. Security adds a second claim: the producer authorized to bind that key to an ActionResult was entitled to do so and faithfully published the action's actual outputs.

Start with the cache-specific threat model

Protecting the connection with TLS and establishing identities through your IAM system are prerequisites. They do not decide the cache policy. Write down which failure you are preventing before choosing credentials or namespaces:

  • Accidental poisoning: a trusted producer publishes an incorrect result because an action depended on an undeclared host tool, environment value, or other ambient input. Different machines can then reuse incompatible output under the same declared action identity.1,2
  • Malicious poisoning: an attacker able to write action-cache entries binds a legitimate action digest to attacker-chosen output. A malicious CAS blob by itself is normally inert; it becomes a build result when trusted metadata refers to it.2
  • Disclosure: a reader can retrieve cached build outputs. The official guidance explicitly treats the cache as storage for your binaries, so read access is a confidentiality decision, not a harmless optimization.1
  • Availability abuse: a writer can consume storage or bandwidth and can displace useful CAS content. Even when it cannot create a trusted result, an unrestricted upload identity can still degrade the service.2

This model makes “CI is trusted” too coarse. A presubmit that evaluates an unreviewed change may run modified build definitions, generators, and CI scripts. Giving that job the postsubmit writer credential lets the proposed change use a trusted identity to publish its own output. Restricting writes to CI is therefore a useful first step, not proof that every CI invocation is a trusted producer.2 Stateless workers and separation between trusted and untrusted worker networks reduce persistence and cross-job influence, but they do not replace cache authorization.3

Authentication, authorization, and namespace are different controls

Authentication answers which principal made a request. Authorization answers which operations that principal may perform on which cache data. Namespace selection decides which logical cache population the request addresses. Keep all three visible in the design.

An authenticated developer must not automatically receive write permission. A writer for the release branch must not automatically write the presubmit or another team's cache population. Likewise, a REAPI instance name is not, by itself, an access-control rule. It selects an instance-scoped protocol namespace; it becomes a security boundary only when the chosen backend independently authenticates the caller and enforces an ACL for that instance. Apply the same reasoning to a backend-specific HTTP path or storage prefix: a different name is containment only if access policy prevents crossing it.

The Bazel client can be configured as a cache reader without uploading locally executed results by setting --remote_upload_local_results=false.1 That is a client behavior, not an authorization barrier: a stolen credential or a different client can ignore it. Enforce write denial at the cache service or its storage/API boundary, then use the Bazel flag as defense in depth.

A starting least-privilege matrix might look like this:

PrincipalReadWriteNamespaceWhy
Developer workstationApproved developer resultsNoneDeveloper-compatible populationReuse without trusting an unmanaged machine as a producer
Untrusted presubmitOnly data the change is allowed to observeNone to a trusted populationSeparate or read-only presubmit populationProposed code cannot publish a result for later trusted consumers
Reviewed postsubmit builderIts required inputs/resultsAC and CAS for approved action classesOne bounded platform/configuration populationPublishes results only after the source and build path meet the trust policy
Trusted remote-execution publisherData needed for assigned actionsResults produced by the trusted execution pathIts authorized execution populationSeparates the authority to request work from the authority to publish trusted results
Incident responderEvidence needed for investigationQuarantine/invalidation operations, not ordinary build publicationExplicit affected populationsEmergency authority is auditable and is not reusable as a build credential

The exact rows depend on your organization, but the separations are stable: reader is not writer, presubmit is not postsubmit, and an administrative recovery identity is not a routine producer. Give credentials short lifetimes and bind them to the smallest practical operation and namespace. Keep cache writer, builder, signer, and publisher roles separate; 6.7.5 Build Identities and Secrets develops that credential and evidence boundary.

Remote execution can make a controlled worker service the publisher rather than an unmanaged client. Stripe describes a design in which only trusted execution sources are authorized to upload action results.4 That still creates a harder problem: untrusted actions execute inside the trusted publishing system, so the worker must prevent an action from escaping its execution boundary or directly exercising the cache writer's authority. End-to-end worker, queue, storage, quota, and tenant isolation are developed in 6.3.14 Multi-Tenant Remote Execution. Remote execution is not a one-line substitute for cache policy.

Make every trusted write attributable

A cache object tells a consumer which result and blobs match a digest. It does not, on its own, answer which human, CI invocation, source revision, worker image, or credential caused that binding. Your service's audit plane must retain that context when accepting a trusted write.

For an AC publication, record at least:

  • authenticated principal and authorization decision;
  • instance, backend namespace, and operation;
  • action digest, resulting ActionResult, and referenced output digests;
  • invocation or job identity, source revision, producer cohort, and relevant client/worker version; and
  • timestamp plus credential or policy version.

Log rejected writes too. They prove that the intended boundary was exercised, and a sudden series of denials can expose a misrouted client or attempted abuse. Keep the audit data outside the authority of the producer being audited. A postsubmit-only policy improves attribution because a suspicious publication can be connected to reviewed source and retained CI logs, but review and logging reduce risk rather than eliminate compromise.2

Design containment before granting write access

Suppose the credential used by one postsubmit producer is stolen at 14:00, and at 14:20 a developer receives a wrong result from the shared cache. A safe response is bounded and evidence-preserving:

  1. Revoke that principal's write authority immediately. If the scope is uncertain, switch affected consumers to a known-safe read-only population or fail closed rather than silently accepting new results.
  2. Preserve the writer's accepted and rejected request logs, action/result/blob digests, invocation identities, policy versions, and timestamps. Revocation prevents future publication; it does not repair entries already accepted.
  3. Quarantine the affected producer cohort and namespace. Bound the suspect set by identity, time window, source revision, platform/configuration population, and result digests before deleting anything.
  4. Rebuild affected actions through an independently trusted producer into a clean or quarantined population. Compare the controlled results and restore readers only after the intended producer path and access decisions are observable.
  5. Invalidate proven-bad action-result bindings and handle referenced CAS objects according to backend reachability and retention rules. Do not turn a global cache flush into the first diagnostic step: it destroys attribution, widens the outage, and does not remove the compromised credential or the faulty action.

The official documentation notes that deletion is backend administration and may offer only broad choices such as clearing the cache or deleting old output; portable Bazel flags do not provide a universal selective quarantine API.1 Your backend's precise invalidation and audit abilities are therefore part of the security qualification. 6.2.4 Wrong Cache Result Diagnostics continues from the retained evidence to determine the blast radius and recover shared state.

Test the policy with negative cases, not just a successful trusted build. A developer credential should fail to write. An untrusted presubmit should fail to write into the postsubmit population. A valid credential paired with the wrong instance should be denied by authorization rather than merely routed elsewhere. After revocation, the old credential should fail while a separately trusted producer can republish a controlled result. Those observations prove the boundaries that configuration names alone cannot.

key takeaway

Secure cache reuse requires two independent decisions: whether the action identity represents equivalent work, and whether an authorized producer may bind that identity to a result. Authenticate every principal, enforce separate read, write, namespace, and recovery permissions at the service boundary, and never treat a REAPI instance name or a read-only client flag as authorization.

CI-only writing narrows risk but does not make unreviewed CI trusted. Keep presubmit and postsubmit authority separate, make every accepted write attributable, and pre-plan revocation, bounded quarantine, evidence-preserving rebuild, and selective recovery. Revoking a writer stops new poisoning; it does not validate or erase results the writer already published.

Check your understanding · 2 questions

1.What makes a REAPI instance name a real security boundary?

Select one answer

2.After a cache writer credential is compromised, which response steps preserve a bounded investigation?

Select all that apply

0 of 2 answered

Footnotes

  1. Remote Caching — cache contents, read-only client configuration, CI-only writer guidance, authentication prerequisites, and backend-dependent deletion 1 2 3 4

  2. Understanding Bazel remote caching — accidental and malicious poisoning, AC write authority, presubmit abuse, auditability, and CAS disclosure/availability risks 1 2 3 4 5

  3. How Bazel built its CI system on top of Buildkite — stateless workers and separation of trusted and untrusted CI infrastructure

  4. Fast Builds, Secure Builds. Choose Two. — trusted remote-execution publishers, action-cache poisoning, and the need to isolate untrusted actions from publishing authority