6.3.7 Remote Execution Storage

An action can finish successfully on a remote worker and still leave no usable remote result. Its output bytes may not have reached storage, a directory tree may be incomplete, or result metadata may point at data that cannot be read. Treat remote storage as a publication chain, not as a side effect of successful execution: an input closure must be available before work starts, and an output closure must be reachable before a result is announced as reusable.

6.3.4 Remote Action Tracing separated command completion from output upload and the terminal client observation. 6.3.5 Remote Execution Architecture placed the storage role on its own data path. Here the question is narrower: which evidence proves that the bytes and metadata together form a usable remote result?

A result is metadata plus a reachable closure

The Action Cache (AC) answers a lookup with result metadata. The content-addressable store (CAS) holds bytes addressed by digest. They are different services in the Remote Execution API, alongside the transfer operations used to move blobs.¹ 6.2.1 How Remote Cache Keys Work established how that AC/CAS identity connects an action to output metadata and bytes. An ActionResult normally describes output files, directories, and stdout/stderr through CAS digests and tree digests. A response may additionally include requested OutputFile.contents, stdout_raw, or stderr_raw bytes, but those are optional hints that may coexist with their digests or be omitted. A complete result therefore has every populated output-file, output-tree, stdout, and stderr digest reachable in CAS. Inline bytes only let the current recipient use that value without a separate read; they never replace digest validation.

This gives one action three distinct storage facts to test:

Fact	A useful witness	What it does not prove
Inputs are available	The input-root, command, tool, and file digests were accepted or found in the CAS for this request.	That a compatible worker was selected or that its environment can run the command.
Execution completed	The remote attempt has a terminal execution result.	That its captured outputs reached the CAS or that an AC entry exists.
A cacheable result was published	An AC lookup returns the expected ActionResult, and every populated output-file, tree, stdout, and stderr digest is readable from CAS. Any included inline bytes may serve this recipient directly.	That a client has downloaded or materialized every output locally.

Keep the action digest, result identity, and the digests for the input and output closures in the same trace. A bare “remote succeeded” event loses the join needed to distinguish an executor failure from an unavailable input or an incomplete publication.

The Action Cache is not a universal terminal witness. An action marked do_not_cache cannot have its result cached; a terminal execution response can still carry its ActionResult.¹ For that case, verify the terminal result and every populated digest in its intended-consumer CAS closure—not a later AC lookup. Any included inline bytes can save the current recipient a read, but cannot replace that check. For a cacheable action, also record whether the selected server policy is expected to publish a reusable AC entry before treating an absent entry as a publication failure.

Make the input closure available before execution

Remote execution begins with an immutable description of the command and its Merkle input tree. Before submitting an action, the client can determine which digests are absent and upload only those blobs; the REAPI CAS and ByteStream contracts are the portable boundary for those operations.²

An executor then needs the complete closure that the action references: not only a top-level input tree, but also its files, nested directories, command, and tools. If one required digest cannot be read, record it as a storage/input availability failure with the missing digest and service endpoint. Do not call it a matching failure merely because no worker process started, and do not call it an executor failure merely because a worker was assigned.

The publication chain deliberately does not choose how the worker presents those available inputs. Ordinary CAS materialization and lazy or remote filesystem presentation are the decision in 6.3.10 Remote Input Presentation. This article only requires a checkable answer to the earlier question: can this action's declared input closure be read by the execution path?

Publish outputs in dependency order

After the command finishes, output capture starts a second closure. A worker may have files in its execution root, but they are not yet remotely reusable. It must upload every captured output blob and the directory-tree objects needed to describe output directories, and make the CAS object for every populated output-file, tree, stdout, or stderr digest reachable. Requested OutputFile.contents, stdout_raw, and stderr_raw are optional response hints: the server may omit them, and when present they may coexist with the digest they save this recipient from reading. Only then can a cacheable ActionResult safely name the intended-consumer closure for an AC lookup.

The safest operational invariant is simple: do not make a cacheable result visible as a usable cache result until every CAS object its intended consumer needs is reachable. Test every populated digest in the ActionResult, even when the current response also contains inline bytes. Those optional values can avoid a read for this recipient, not remove the referenced object from the CAS closure. This is a consistency target, not a claim that every backend exposes one transaction. REAPI defines the client/server messages; it does not prescribe a backend's storage layout, upload ordering, retry policy, or recovery from a partially completed publish.¹ Pin those choices to the backend revision you operate.

A useful backend-specific trace records the steps separately:

1. The executor captured a terminal command result and enumerated outputs.
2. For every populated output-file, directory-tree, stdout, and stderr digest in the result, its CAS object was written or already present.
3. A read-after-write or equivalent storage witness established that every populated digest in the intended-consumer closure was reachable through the path consumers will use. Any inline value is optional response convenience, not a substitute for that witness.
4. For an action that the server may cache, the service published the ActionResult to the AC.
5. For that cacheable path, a fresh lookup returned the result and every populated digest was readable; otherwise the terminal response and the same closure check are the publication witness. Included inline values may spare the current recipient a read but do not change the pass condition.

Steps 2–5 are deliberately separate. A failure after capture but before CAS reachability is a failed publication, even if the process exit status is zero. A failure after blob upload but before the AC write can leave unreferenced CAS data; its lifecycle is a storage-policy decision, not a completed cache hit. An AC entry that appears before all of its referenced data is readable is a dangling result and must not be accepted as a successful publication.

Diagnose the first broken edge

Use the first witness that separates two explanations instead of treating the whole remote system as one failure domain.

Observation	First hypothesis to test	Do not infer
A required input digest is missing before a process starts	The input-upload or CAS-read path is incomplete or unavailable.	That scheduling selected the wrong worker.
A cacheable action expected to be published has no AC result	Output upload, reachability verification, or result publication failed.	That the command output was never produced.
An AC result exists but one required output, tree, stdout, or stderr digest cannot be read	A dangling result, retention/visibility failure, or inconsistent storage path exists.	That local output download is misconfigured.
A fresh lookup returns a result and every populated digest is readable from CAS	The result is remotely publishable for that consumer. Any included inline value can serve this response directly.	That every consumer has materialized the outputs it needs.

The last distinction matters. Accessing, downloading, and preserving a result for IDEs, tests, CI, and later actions begin only after publication, in 6.3.11 Remote Output Availability. Long-term retention, replication, recovery class, and loss testing for the stored data are addressed in 6.3.6 Remote Execution State and Recovery and 6.2.6 Remote Cache Lifecycle. Do not use an output-download symptom to skip the publication checks above.