6.3.6 Remote Execution State and Recovery

A scheduler restarts while one action is queued, another has a worker lease, and a third may have uploaded output. “Restore the cluster” is not yet a plan. The three actions depend on different state, and restoring an old queue record can be less safe than rebuilding it. Recovery starts by inventorying each state population, deciding what truth it represents, and classifying it from an explicit service promise and measured recovery cost.

6.3.5 Remote Execution Architecture located the storage, scheduler, executor-control, executor, coordination, and evidence roles. This article asks what survives when one of those roles or its store is lost. The answer is not encoded in the name “cache,” “database,” or “lease.” It comes from the pinned implementation contract and the recovery objective for this deployment.

State has a recovery contract, not a component name

Inventory a concrete population, classify it against a declared objective, then prove the class by losing it.

Examples of remote-execution state include CAS blobs and Action Cache relationships, Operation and queue or lease knowledge, coordination data, and executor-local caches. None has a universal recovery class. A pinned implementation contract and measured recovery cost determine whether a state population must be durable, can be reconstructible, or is disposable. A component-loss test starts with a baseline witness, injects the named loss, applies the declared action, and verifies recovery or accepted loss with a fresh client. Ambiguous in-flight work stays unknown and its evidence is quarantined.

1. Inventory concrete state and its relationships

Bytes and published resultsCAS blobs, AC entries, ActionResult closure

In-flight execution knowledgeOperation, queue, assignment, lease

Coordination and worker warmthBackplane, registration, executor-local caches

2. Use the pinned implementation contract and measured recovery cost

DURABLE

Required truth survives the named loss

Meet the stated correctness or availability objective.

Protect: replicate or perform a tested restore.

RECONSTRUCTIBLE

Derive safely from named authority

Fit reconciliation and cold demand inside the declared bound.

Rebuild: reconcile evidence and measure capacity.

DISPOSABLE

Remove only after an explicit safe condition

Loss preserves required truth, isolation, and service promises.

Discard or recreate: measure the cold-start cost.

3. Prove one named class at a time in an isolated loss test

BASELINE WITNESS

Preserve typed joins

Digests, Operation.name, queue or lease correlation, worker and backend revision.

INJECT LOSS

Remove one component or state class

Do not infer recovery from a process restart.

DECLARED ACTION

Protect, rebuild, or discard safely

Use the action chosen for this concrete recovery class.

MEASURED OUTCOME

Fresh client sees recovery or accepted loss

Required reads, joins, and cold capacity stay within the declared gate.

Ambiguous in-flight work: retain or quarantine the evidence as unknown. Never relabel it as a cache miss or let an old lease prove current ownership.

Acronyms do not classify state. The deployment contract, recovery objective, named authority, and measured loss test do.

State is not one thing

Start with six separate populations. Record relationships as well as objects: an Action Cache entry refers to an ActionResult, that result refers to CAS blobs and trees, an Operation refers to current or terminal execution state, and backend queue, assignment, lease, and worker records may describe the same work from different control-plane viewpoints.

REAPI defines only a limited portable retention contract. The Action Cache lifetime is implementation-specific. When GetActionResult returns a result, the server should ensure that its referenced CAS blobs are available then and for some period afterward. WaitExecution may return NOT_FOUND because an Operation no longer exists, and the client is told to call Execute again. The service also need not guarantee at-most-one execution; equivalent actions may run more than once, even in parallel.¹

Those statements do not define a scheduler database, queue persistence, lease fencing, worker registry, storage replication, backup, or restart algorithm. A lost Operation is a lost client-facing handle or observation. A lost CAS blob is missing content that may be needed to execute an action or materialize a result. Treating those losses as equivalent erases the very distinction recovery needs.

Inventory at least these state populations before selecting a recovery mode:

State population	Authority question	Relationship or cost that must stay visible
CAS blobs and storage metadata	Which bytes does the service currently promise to make reachable, and through which loss domains?	Input and output digests identify bytes; they do not preserve them. Record every blob and tree needed by a protected result.
AC entries and ActionResult metadata	Is this entry merely reusable metadata, or part of a promise that a published result remains obtainable?	An entry is useful only while its referenced CAS closure can be served. Loss may force recomputation; stale survival may create a dangling result.
Long-running Operation records	Does the service promise continuation, terminal-history lookup, or only best-effort current observation?	Loss can hide progress or a terminal outcome even when executor work or output bytes survive.
Queue, assignment, and lease state	Which record is authoritative for accepted, queued, assigned, or possibly running work?	Old records can describe work that is still running, already completed, or no longer owned. Time and fencing semantics matter.
Worker registration, heartbeat, and executor-local caches	Which data is live coordination, and which is only warm materialization?	Replacement may be correct yet overload CAS, image distribution, network, or preparation capacity.
Coordination database or backplane	Does it contain an irrecoverable authoritative mapping, a derivable index, or expiring observations?	The answer is schema-specific. The storage technology alone does not determine the recovery class.

Classify the instance, not the acronym

Use three recovery classes. They describe a contract, not an intrinsic property of a component.

• Durable state must survive the named failure domain to meet the declared correctness or availability objective. Specify its consistency boundary, recovery point, recovery time, and restore witness. “On persistent disk” is not enough.
• Reconstructible state can be derived safely from named authoritative evidence. Specify the derivation, how ambiguity is represented, and the measured execution, storage, network, and operator cost. If the rebuild load exceeds the protected capacity envelope, the state is not reconstructible for that objective.
• Disposable state may be discarded without losing required truth or violating a service promise. Its loss may still have a measured cold-start cost. Disposal is a deliberate lifecycle decision, not a synonym for “cache.”

The following decision table turns the inventory into a classification. A real deployment must choose one class for each concrete population and attach its backend version and evidence; the conditions below are not universal defaults.

State instance	Durable when…	Reconstructible when…	Disposable only when…
CAS blobs	the current availability promise requires those bytes to survive the tested loss	every byte has a named trusted source and re-upload or re-execution fits the measured recovery bound	no active input, published result, retained evidence, or required consumer depends on the bytes
AC result	consumers are promised that this published result remains obtainable through the loss	it can be invalidated or recreated without returning an incomplete hit, and recomputation cost is acceptable	losing the hit is visible and safe, all surviving references are checked, and no retention promise depends on it
Operation record	clients are promised continuation or retained terminal history	action, backend, worker, and result evidence can produce an explicitly known or unknown client-visible state	its observation and terminal outcome are outside the retained service contract
Queue or assignment record	accepted work must survive and the pinned backend can restore it with current ownership and fencing intact	retained admission plus live worker and completion evidence can safely reconcile it	loss becomes an explicit visible abandonment rather than silent disappearance
Current lease, registration, or heartbeat	never solely because an old record exists; current authority must be renewed under the backend contract	liveness and fencing create a new current observation	expiry removes its authority and retained audit evidence still covers ambiguous work
Executor-local cache	it unexpectedly contains the only copy of required state, which is a design defect to repair	a pinned implementation rebuilds it from CAS, image, and configuration within the measured cold bound	it is only warm state and cold replacement preserves correctness, isolation, and capacity
Coordination/backplane record	it is authoritative and cannot be derived within the objective	named retained sources and a tested reconciliation recreate it without resurrecting stale work	it is an expiring hint whose absence cannot hide accepted or completed work

Pinned implementations make the method concrete without making it portable. At revision fd06ca3, Buildfarm's Shard instance uses a Redis backplane for worker registration, AC data, Operation queues and monitors, and a CAS index. Its configuration gives AC, Operation, action-to-execution, and CAS-index records explicit TTLs, and its dispatched monitor requeues operations after worker leases expire.² That supports a Buildfarm Shard recovery contract. It does not make Redis, those TTLs, or that requeue rule part of REAPI, nor does it prove that replaying a Redis backup would recreate current lease truth.

At revision 91a3580, a Buildbarn deployment example gives CAS and AC separate block-device data and persistent-state paths. Its worker configuration instead bounds the build cache, file pool, and directory cache by size and count.³ This is evidence that the deployment distinguishes stored service data from worker warmth. It is not evidence of replication, an off-host backup, or a tested restore procedure.

At revision 3f1fb5d, NativeLink's worker directory cache has a temporary default root and bounded size and reconstructs input directories from CAS. A scheduler test separately demonstrates a backend policy in which the first worker disconnect requeues work and a later disconnect beyond the configured retry cap completes it with an error.⁴ The first fact can justify classifying that worker cache as disposable after measuring cold rebuild. The second is loss-test evidence for one NativeLink scheduler policy; it is not a portable rule for ambiguous attempts. The retry and accepted-result decision itself continues in 6.3.13 Remote Action Failure Handling.

Choose backup, replication, rebuild, or discard deliberately

Classification becomes operational only when every row has an owner and a recovery mechanism. Keep the mechanisms distinct:

• Replication maintains live copies for a stated failure domain. Record write acknowledgement, lag, failover, and read behavior. A second live copy is not automatically a recovery point for deletion or corruption.
• Backup is a separately retained, restorable recovery point. Record what is captured together, encryption and access, retention, restore tooling, and the restore test. A backup of queue rows without compatible lease, Operation, and worker evidence can restore stale claims rather than useful truth.
• Rebuild derives state from named authoritative inputs. Record the algorithm, version, invalidation rules, uncertainty state, and measured cold demand.
• Discard removes state whose authority has expired or whose loss is safe. Record the condition that makes deletion safe and the capacity consequence of returning cold.

A compact recovery worksheet should contain these columns:

Field	Question to answer
State population and owner	Which exact schema, namespace, backend revision, and team own it?
Authority and relationships	What truth does it represent, and which action, result, blob, Operation, assignment, lease, or worker identities join it?
Recovery class and objective	Durable, reconstructible, or disposable for which failure domain, recovery point, and recovery time?
Mechanism	Replicate, restore from backup, rebuild from named evidence, or discard after a stated condition?
Recovery cost and gate	What storage, network, execution, image, and operator demand was measured, and what gate stops an unsafe cold storm?
Witness	Which pre-loss and post-recovery reads, joins, statuses, and capacity observations prove the claim?

Preserve the witness before mutating state: action, result, and blob digests; Operation.name; backend queue, assignment, and lease correlations; worker and image identity; backend revision; and the source clock for each observation. If evidence cannot distinguish queued, still running, completed, and published work, record that work as unknown. Do not relabel it as a cache miss or use a restored lease as proof of current ownership.

Normal input availability, output upload, and safe result publication are developed in 6.3.7 Remote Execution Storage. This recovery plan uses their relationship as a pass gate, but does not invent a transaction or repair algorithm. AC/CAS retention and cold-cache lifecycle policy are developed in 6.2.6 Remote Cache Lifecycle.

Prove the plan by losing one component

Run loss tests in an isolated deployment or namespace, one state class at a time. Each case needs a baseline witness, an injected loss, a declared safe outcome, the recovery action, and a fresh-client observation. Do not accept “the process restarted” as a recovery result.

Loss test	Evidence to retain before loss	Pass criteria to declare before the test
CAS member, tier, or storage path unavailable	Protected input/output digest set, AC references, replica/tier identity, successful reads	The promised blobs remain readable through the stated path, or dependent results become explicit misses/unavailable results; no usable result silently points at missing required bytes.
AC population removed	Action/result/blob closure, hit observation, recomputation cohort and load baseline	A fresh client sees the documented miss/rebuild behavior, referenced bytes are handled coherently, and measured rebuild demand stays inside its gate.
Scheduler or coordinator restart	Accepted Operations, queue/assignment/lease joins, worker liveness and backend revision	Queued and in-flight work reconcile to known or explicit unknown states; stale leases are not treated as current execution; normal admission resumes only after the declared invariant holds.
Executor terminated and replaced	Worker/image identity, assigned-work correlation, local-cache state, CAS/image/network baseline	Worker-local state is recreated or safely discarded, isolation remains intact, and cold fetch/preparation load stays within the declared bound.
Coordination store isolated or restored	Schema/version, authoritative mappings, queue/lease/worker witnesses, backup or rebuild identity	Restoration or reconstruction does not resurrect stale ownership, hide accepted work, or erase the evidence needed to reconcile it.
Wider deletion or corruption recovery	Backup identity, capture boundary, retention and restore procedure	The restored point recreates every relationship in its stated scope and its age and omissions match the declared recovery objective.

These are component/state-class tests, not incident command or an upgrade sequence. Planned mixed-version draining and rollback use this inventory in 6.3.17 Remote Execution Upgrades. Coordinating a real cache, execution, BES, network, or region incident across services belongs to 6.6.6 Build-Service Incident Recovery.