6.3.16 Action Deduplication and Racing

extra

Two clients can miss the Action Cache for the same action at nearly the same time. Without another coordination layer, both can ask for execution and both requests may consume workers even though either result would satisfy the same declared action. Deduplication saves that repeated work; hedging deliberately adds some of it back when sharing one slow attempt would make too many callers wait.

These are scheduler policies, not properties of Bazel's target graph or a portable REAPI feature. Start after 6.3.8 Scheduling Actions on Remote Executors has made admission, compatible queues, dispatch, and capacity accounting observable. The decision here is when several requests may share work, when a second attempt is justified, and how the backend prevents two completions from becoming two conflicting accepted results.

Separate the four behaviors

The vocabulary varies across backends, so define behavior before adopting a product label.

Behavior	What the backend does	Main benefit	Main risk
Independent execution	Gives each request its own Operation and execution path.	Simple fallback with no shared in-flight dependency.	Repeats compute while no completed cache result exists.
Duplicate suppression	Declines to dispatch another attempt for an action already in flight.	Avoids immediate duplicate worker use.	A suppressed caller needs a visible, durable way to await or retry; silently dropping it is not reuse.
In-flight reuse, coalescing, or action merging	Attaches a later request to a primary execution and delivers its result to more than one waiter.	Saves work and lets a later caller inherit progress already made.	One stuck or flaky primary affects every attached caller.
Speculation or hedging	Starts an additional attempt after a trigger and races it against existing work.	Bounds exposure to a slow or stuck primary.	Consumes extra capacity and creates cancellation, accounting, and result-selection work.

Action merging is a common backend name for the third row, not a separate wire-level primitive. BuildBuddy's implementation, for example, maintains a mapping from an action identity to a primary execution while the work is pending; later requests wait for that execution instead of immediately dispatching another one.¹ A BazelCon case calls the same family of behavior deduplication, merging, or coalescing and illustrates why concurrent CI builds create the opportunity: the second request can miss the cache simply because the first execution has not published a result yet.²

REAPI defines actions, execution requests, Operations, and results, but it does not require a scheduler to merge equal requests or guarantee at-most-once execution. Equivalent actions may execute more than once, including in parallel.³ Independent execution is therefore the portable baseline. Every merge registry, attachment rule, hedge trigger, and accepted-result rule must be attributed to the pinned backend.

Equal actions create a coordination opportunity, not one shared identity

Keep each request and Operation visible while the backend coordinates primary, follower, and optional hedged attempts.

Portable REAPI identities remain separate

request A · Operation A

Action digest D · policy P · caller A

request B · Operation B

Action digest D · policy P · caller B

request C · Operation C

Action digest D · policy P · caller C

Candidate merge key: digest D, then verify instance, policy, authorization, platform, trust, and do_not_cache = false

Backend-specific coordination contract

PRIMARY ATTEMPT

One worker execution owns progress

Registry, lease, cancellation, and expiry remain observable backend state.

FOLLOWERS

Operations B and C stay attached

Each waiter retains identity, accounting, detach, and terminal delivery state.

OPTIONAL HEDGE

A measured trigger spends a bounded attempt

The pinned backend defines attachment, budget, racing, and cleanup.

ACCEPTED RESULT

Publish one reachable completion

Deliver the selected result to every still-attached Operation.

LOSING ATTEMPT

Cancel, observe, and account

A late or conflicting completion is evidence, never a replacement result.

Independent fallback: ineligible requests — including do_not_cache = true — keep their own execution path. Merging is optional; ordinary execution is the portable baseline.

Share backend work only after eligibility checks. Never collapse request, Operation, attempt, and result identities into the Action digest.

Use action identity, but do not overstate it

Deduplication needs an equivalence key. At the portable boundary, the action digest identifies the serialized Action, which refers to the command and input root. An invocation ID identifies a client journey; an Operation.name tracks a client-visible execution operation; a backend execution or lease ID tracks an implementation attempt. They are not interchangeable.

Identity	Suitable use here
Action digest	Candidate key for deciding that requests describe the same declared action.
Invocation or request identity	Count and detach callers without confusing them with the shared work.
`Operation.name`	Let one caller continue observing its client-visible operation.
Backend execution, attempt, or lease identity	Attribute actual dispatch, capacity, cancellation, and completion.
Result and output digests	Decide which completed result became accepted and whether late results agree.

Equal action digests are necessary for safe merging, but operationally they are not enough. The action must also be eligible under the same instance, authorization, cache policy, platform routing, trust boundary, and compatible request-level execution policy. ExecuteRequest.execution_policy is not part of the serialized Action; its priority is a server-interpreted scheduling hint. A pinned backend must therefore define whether a follower inherits the primary's scheduling state, changes its effective priority, remains separately accounted, or is ineligible to merge across policy classes. REAPI does not choose among those behaviors.³

A backend must not merge two requests merely because a convenient subset of their fields looks alike. Nor does digest equality repair a bad action contract: a hidden input, nondeterminism, or a flaky test can make one shared execution amplify an incorrect or unlucky outcome. BuildBuddy explicitly reports that a stuck primary can stall all merged callers and that a flaky primary can propagate the same failure to every waiter.¹

Before enabling merging for a class, preserve the readiness and compatible placement evidence from 6.3.1 Making Actions Work Remotely and 6.3.2 Remote Executor Matching. Exclude classes whose result is intentionally repeated, tenant-specific, or not trustworthy across the proposed sharing boundary unless the pinned backend documents and tests an appropriate narrower identity. Treat the REAPI Action.do_not_cache field as a stricter boundary: when it is true, the result cannot be cached and in-flight requests for that Action may not be merged. A backend-specific identity or policy cannot relax that protocol contract.³

Treat followers as owned state

A merged request has not disappeared. It has become a follower whose outcome depends on shared state. The scheduler needs a record joining:

the merge key and primary attempt;
every waiting request or invocation still interested in the result;
the registry's creation, refresh, expiry, and deletion rules;
the primary's queue, lease, worker, and terminal state; and
the result selected for each follower.

Stale registry state is dangerous because it can attach new work to an execution that no longer exists. BuildBuddy uses short-lived registry entries, extends their lifetime after a worker claims the primary, refreshes them while the executor remains healthy, and removes the state after the result reaches the cache.¹ That is one implementation contract, not a TTL recipe for other systems. Your backend needs its own evidence for process loss, scheduler restart, registry partition, lease expiry, and result-publication failure.

Cancellation requires follower-aware ownership. If the primary invocation is cancelled while another invocation still waits, cancelling the shared attempt would turn an optimization into a correctness or availability failure. BuildBuddy addresses that case with waiter reference counting and cancels only when no waiting action remains; its documented shortcut can also leave a primary running after all original invocations cancel.¹ This exposes the real trade-off: exact waiter lifecycle reduces wasted work but adds state and reconciliation complexity.

Do not duplicate the cross-layer retry policy here. Use 6.3.13 Remote Action Failure Handling to decide how client retries, backend attempts, cancellation, late completions, and local fallback interact. Deduplication adds one required fact to that ledger: which requests shared which primary, and when each attachment ceased to be authoritative.

Hedge only against a measured tail problem

Merging concentrates risk. If a primary deadlocks or encounters a very slow dependency, every follower inherits the same tail. Hedging starts another attempt after a stated trigger—for example, elapsed time or accumulated followers—so later work is not bound indefinitely to one execution. BuildBuddy documents a backend-specific design in which a hedged attempt races the primary toward result publication; it also documents an important limit: the hedge can unblock invocations arriving after its result is cached while callers already attached to the stuck primary may remain attached.¹

That limit shows why “start two and take the fastest” is not a sufficient contract. Specify all of these before enabling a hedge:

Trigger: which action class, elapsed-time distribution, follower count, worker signal, or health observation justifies another attempt?
Budget: how many extra attempts and how much compatible capacity may one action, invocation, tenant, and pool consume?
Attachment: do existing followers move to the winning attempt, or does the hedge help only future requests?
Acceptance: which terminal result becomes authoritative, and what happens when attempts disagree, fail differently, or cross trust boundaries?
Cancellation and cleanup: how are losing attempts requested to stop, how is actual termination observed, and how is orphan capacity accounted?
Publication: which result may be published, and how does the backend prevent a late completion from replacing or conflicting with it?

The result must still satisfy the reachability and publication checks in 6.3.7 Remote Execution Storage. A fast completion with missing output blobs is not the winner. A late completion with different output digests is evidence of nondeterminism or an identity/trust defect, not a harmless duplicate to discard without investigation.

think

Decide: Ten CI invocations are attached to one long-running test. Its runtime has passed the normal tail for this action class, but the pool is near its protected capacity limit. Should the scheduler hedge immediately?

Reveal

Not from elapsed time alone. Check the class-specific hedge trigger, current compatible-pool budget, primary worker and lease evidence, and the cost of letting all ten callers wait. Hedge only if the declared tail-latency benefit outweighs the protected capacity and cancellation cost. If no budget remains, keep the delay visible or apply the scheduler's bounded overload behavior; an unbudgeted hedge can amplify the incident it is meant to repair.

Qualify the policy with a version-pinned experiment

Run the experiment on representative action classes and a pinned backend release and configuration. Compare three paths: independent execution, merging without hedging, and merging with the proposed hedge policy. Hold the action digests, compatible pool, worker environment, cache state, input/output shape, and offered request pattern constant.

For each path, retain:

request count, distinct primary and hedged attempts, dispatches, and completed worker executions;
saved worker time and added speculative worker time by action class;
follower wait and end-to-end latency distributions, especially the tail;
compatible-pool occupancy, queue age, and admission or rejection caused by hedges;
cancellations requested, attempts confirmed stopped, and orphan work;
accepted, late, conflicting, incomplete, and unjoinable results; and
behavior after primary cancellation, worker loss, scheduler or registry restart, and registry expiry.

Correctness and complete evidence are gates, not performance counters. Abort the rollout if a follower loses a terminal result, a late attempt can overwrite the accepted result, registry loss silently drops work, or trust and tenant boundaries cannot be reconstructed. Then judge savings and tail improvement per action class. A global deduplication rate can hide that short actions pay coordination overhead while long, frequently overlapping actions provide most of the value.

Keep an explicit off switch. With merging and hedging disabled, each request must return to ordinary independently observable execution without relying on the registry. Include this fallback, its capacity demand, and its cache behavior in 6.3.18 Remote Execution Production Readiness rather than assuming the optimization is required for correctness.

key takeaway

Action deduplication is a backend scheduler optimization over identical in-flight work. Use the action digest as the candidate equivalence key, then preserve instance, policy, platform, trust, request, Operation, attempt, and result identities. In-flight reuse saves duplicate execution but couples every follower to one primary; hedging deliberately spends bounded extra capacity to reduce that shared tail.

REAPI permits redundant parallel execution and does not promise merging or hedging. Accept either optimization only after a pinned experiment proves follower lifecycle, cancellation, result publication and selection, failure recovery, capacity accounting, and the ordinary non-merged fallback. Saved work is valuable only when every caller receives one trustworthy, reachable result and late attempts cannot change that decision.

Check your understanding · 3 questions

1.Match each repeated-action policy to the behavior a caller observes:

Drag each answer onto the matching prompt, or click an answer and then click a prompt

Answers

Independent execution

Duplicate suppression

In-flight reuse or action merging

Speculation or hedging

2.Two Execute requests carry the same action digest. Which facts can still make merging them unsafe or backend-dependent?

Select all that apply

3.An Action has do_not_cache set to true. What may a backend do when an equivalent request arrives while it is executing?

Select one answer

0 of 3 answered

Action Merging — BuildBuddy-specific primary/follower registry, TTL refresh, cancellation reference counting, stuck-primary behavior, hedging, and documented limitations ↩¹ ↩² ↩³ ↩⁴ ↩⁵
Action Deduplication: Faster and Cheaper Remote Builds Without Lifting a Finger - Christian Scott — concurrent cache misses, action registry/coalescing model, CI overlap, and clustered flaky-test outcomes ↩
Remote APIs — protocol contracts for caching and remote execution — portable Action, Execute, Operation, and result boundary; scheduler merging and hedging remain implementation-specific and execution is not at-most-once ↩¹ ↩² ↩³

← PreviousMulti-Region Remote Execution Next →Remote Execution Upgrades