6.4.2 Dynamic Execution

Dynamic execution is a latency hedge: for an action eligible for both paths, Bazel starts remote and local branches, accepts the branch that finishes first, and cancels the other. The winning time can be attractive, but it is not free. Until cancellation takes effect, both branches may consume CPU, memory, executor slots, network, and tool capacity.¹

That makes dynamic execution a selective operating policy, not a faster replacement for every strategy. 6.4.1 Choosing Execution Strategies establishes which action classes are valid candidates for local and remote execution. Here the decision is narrower: for which of those classes does a race reduce user-visible tail latency enough to repay duplicate work and cancellation risk?

Treat the two branches as one action contract

The local and remote branches are two attempts to realize one declared action. They must therefore agree on the output contract. Strategy choice must not change the command's intended semantics or its output bytes; otherwise a race turns an existing correctness defect into a placement-dependent build.²

The scheduler cannot make unlike environments equivalent. Before enabling a class, prove that representative actions succeed locally and remotely under the same declared inputs, tools, environment, execution platform, and expected outputs. A dynamic success is not that proof: the consistently faster branch could hide failures or different results in the other branch.

Use three separate cohorts first:

1. Local-only establishes local correctness and its latency distribution.
2. Remote-only establishes remote correctness, queue and transfer cost, and the remote output under both cache-hit and cache-miss conditions.
3. Dynamic establishes whether racing those already-qualified paths improves the end-to-end distribution and whether losing work is contained.

Eligibility remains action-specific. If an action cannot use the local branch, Bazel executes it remotely without a race; if it cannot use the remote branch, the applicable local strategy runs normally.¹ Record these one-sided actions separately instead of counting them as dynamic winners with zero loser cost.

This separation matters because Bazel intentionally fails the action when the first branch to finish reports a failure. It does not wait for the other branch to rescue an inconsistent environment. That behavior makes local/remote drift visible rather than silently choosing whichever result happens to be successful.¹

Choose candidates from distributions, not anecdotes

Dynamic execution is most plausible when local and remote paths have complementary latency. A persistent local compiler may be excellent for a short incremental action while the remote fleet is better for a broad clean build. Similarly, a normally fast remote path may occasionally queue or suffer network delay, allowing a local hedge to cap the long tail. The original dynamic execution work was motivated by exactly this combination: remote capacity for large builds and fast local workers for edit-build cycles.³

For each mnemonic or finer action cohort, capture at least:

• local-only and remote-only latency distributions, including p50, p95, and p99;
• remote cache-hit, queue, input-transfer, execution, and output-transfer time;
• dynamic winner identity and the time at which each branch actually started;
• losing-branch runtime before cancellation, plus work that continued after the winner was known;
• local CPU and memory, remote executor time, network bytes, and worker-pool occupancy; and
• action failures, output mismatches, cancellation failures, and fallback.

The useful comparison is the complete build or developer journey, not the isolated action that won. Racing many long local actions can starve the few local workers that make incremental builds fast. The official guidance describes a profile in which non-worker dynamic actions consumed resources and delayed Javac workers; restricting the dynamic cohort improved the balance.¹ That is why “enable it for every mnemonic and inspect the average” is a weak experiment.

Build a decision table before the canary:

Action cohort	Expected hedge	Main loser cost	Initial decision
Short incremental compile with a warm worker	Avoid remote queue and round-trip tail	Remote attempt and transfer	Canary dynamically
Long CPU-bound compile that is reliably faster remotely	Rare local win	Sustained local CPU and memory	Keep remote-only
Tiny action dominated by remote overhead	Local completion before dispatch pays off	Small remote request overhead	Canary only if frequent enough to affect the critical path
Action with unproven local/remote equivalence	None that is safe to claim	Incorrect accepted output or branch failure	Do not race

The entries are hypotheses, not universal rules. Your measurements may reverse them because topology, cache quality, workstation size, worker support, and the remote queue all change the break-even point.

Delay is an admission policy for duplicate work

Starting both branches at the same instant maximizes the chance of taking the minimum latency, but also maximizes overlap. Bazel's --dynamic_local_execution_delay can delay the local branch, but it is not an unconditional timer for every race. In current Bazel, the configured delay is applied only after remote execution has already been faster during the current build—described by the dynamic-execution guide as observing a remote cache hit. Earlier actions in the invocation may therefore start locally without that delay.¹

Once active, delaying the local branch gives the remote path time to return likely cache hits before local resources are spent. Bazel also exposes per-mnemonic local and remote branch strategy controls. Its guidance recommends tuning the delay from observed cache-hit time rather than copying a global number.¹

Reason about the delay as a budget:

• A shorter delay increases local starts and may reduce remote-tail exposure, but consumes more workstation capacity.
• A longer delay reduces duplicate work and protects local workers, but leaves more of the remote tail visible to the user.
• A delay tuned only to cache hits may still be wrong for cache misses, queued actions, or users with different network round trips.

Tune one eligible cohort at a time, and record whether the conditional delay was active for each observed race. Hold the revision, targets, configuration, platform, cache condition, client class, and remote pool comparable. Change the delay, measure the whole journey, and keep a rollback configuration. If the dynamic cohort improves its own winner latency while build p95 gets worse or remote executor-hours rise beyond the budget, the hedge failed operationally.

Cancellation is part of correctness

The losing branch does not vanish when the winner is selected. It receives a cancellation request and may need time to stop a subprocess, finish a worker request, clean a sandbox, or abandon remote work. Some work may be impossible to reclaim after dispatch. Historical scheduler work also found that aggressive local process churn could expose bugs in surrounding filesystem infrastructure, illustrating that cancellation stress can escape the scheduler's obvious metrics.³

Qualify cancellation with a conflicting-completion test. Choose a deterministic action whose local and remote executions overlap long enough to force both completion orders. Repeat with local winning, remote winning, and cancellation arriving near completion. For every run, require:

• exactly one accepted result for the Bazel action;
• outputs matching the local-only and remote-only baselines;
• no partial loser output visible in the final output tree;
• termination or a bounded, explained cleanup period for the loser;
• correctly attributed logs and branch/winner evidence; and
• no contamination of a later action through a persistent process or reused filesystem state.

This is a deployment-specific evidence requirement, not a portable REAPI test recipe. Cancellation through the Operations contract is best effort, and the portable REAPI contract exposes no result-publication barrier.⁴ Your backend must therefore supply an attempt identity, cancellation acknowledgement followed by terminal or cleanup evidence, and a documented publication boundary. If it cannot expose those observations, record cancellation and publication as unknown and do not promote the cohort.

This test should also include a branch failure. Because the first completed failure fails the action, the canary must show that the failure is observable and diagnosable rather than hidden as “the other side won.”¹ General remote retry and attempt policy remains with 6.3.13 Remote Action Failure Handling; the dynamic-specific question is whether the paired local and remote attempts can race and be cancelled without corrupting the one accepted action outcome.

Operate it as a bounded canary

Start with one mnemonic and one representative user or CI cohort. Disable silent local fallback in the remote-only baseline so a green control cannot masquerade as remote evidence. In the dynamic cohort, retain execution logs, profiles, and backend observations that identify branch starts, winner, cancellation, and remote work. The JSON trace profile can expose local worker requests that continue finishing after they lose a race.¹

Define promotion and stop conditions before the run. A sound promotion gate requires equivalent accepted outputs, no branch-specific failure regression, a measurable p95 or p99 improvement in the chosen journey, and duplicate local and remote consumption within explicit budgets. Stop on output divergence, unbounded loser cleanup, unexplained branch failures, local worker starvation, remote queue damage, or a journey-level latency regression.

After promotion, watch winner share together with resource cost. A sudden run of local wins can mean remote queueing or network trouble; a sudden run of remote wins can mean local contention or worker churn. Winner share is a symptom, not a health objective. If one branch almost never wins but regularly consumes meaningful capacity, return that cohort to a single strategy. Coordinate the resulting local and remote resource limits with 6.4.6 Execution Concurrency rather than compensating for a poor cohort by raising every concurrency control.