6.4.1 Choosing Execution Strategies
An action is not “local” or “remote” merely because a flag names that mode. Bazel first selects an applicable execution strategy, and that strategy may still find a cache result, reuse a persistent process, race another branch, or fall back to a different executor. Operating a mixed fleet therefore requires two separate decisions: which strategy should be attempted for an action class, and what evidence proves what actually happened.
2.5.1 Strategies Abstraction introduced strategies as interchangeable ways to realize a spawn. At infrastructure scale, the useful unit is not one global default but a cohort of actions with the same mnemonic, tool behavior, platform, input and output shape, and service journey.
Compare complete execution paths
The main strategies expose different cost and isolation boundaries:
| Strategy | Where the command runs | What it can save | Cost or risk to measure |
|---|---|---|---|
local | A subprocess on the Bazel host | Remote queueing and transfer | Host contention and ambient-state exposure |
sandboxed | A locally isolated subprocess | Remote overhead while checking declared-input discipline | Sandbox setup, filesystem work, and platform-specific overhead |
worker | A reused local tool process | Process startup, JIT, parsing, and tool initialization | Retained memory, pool fragmentation, request-state leakage, and fallback |
remote | A compatible remote executor | Shared capacity and host independence | Cache lookup, input upload, queueing, execution, output upload, and materialization |
dynamic | Competing local and remote branches | Tail latency when either side occasionally wins | Duplicate resource use, cancellation, and output coordination |
Bazel's command documentation describes --spawn_strategy as the broad
selection and --strategy=<mnemonic>=<strategy> as a per-mnemonic override.
It also distinguishes local, sandboxed, worker, and remote; remote is
applicable only when a remote executor is configured.1 A comma-separated
strategy list is a priority order of applicable implementations, not a race.
Dynamic execution is the explicit racing mechanism and is treated separately in
6.4.2 Dynamic Execution.2
Do not compare only command duration. For the same action cohort, measure the complete path from readiness to usable output:
- local resource admission or remote queue time;
- process or worker startup, sandbox preparation, and input transfer;
- command execution;
- output upload, download, and materialization; and
- the action's contribution to invocation critical path, resource pressure, and service cost.
This prevents a common false conclusion: a remote command can execute faster yet finish later for its consumer because queueing and data movement dominate. Conversely, a locally fast action class can make the invocation slower when it occupies CPU and memory needed to drive remote work. Profiles should be compared between matched clean or incremental cohorts because cache state changes the amount and kind of work performed.3
Persistent workers deserve the same end-to-end treatment. They help when repeated actions have substantial startup cost or benefit from cross-action caching, but only rules and tools that implement the worker contract can use them.4 Pool sizing, recycling, and fragmentation continue in 6.4.3 Operating Persistent Worker Pools.
Build an action-class strategy matrix
Start with representative mnemonics and split them further when one mnemonic hides materially different platforms, tools, or artifact shapes. For each row, record:
| Field | Question it answers |
|---|---|
| Correctness gate | Do candidate strategies produce equivalent required outputs under isolated reruns? |
| Applicability | Does the action support sandboxing, workers, and the selected remote platform? |
| Latency distribution | Which path improves median and tail completion, including queue and transfer? |
| Resource demand | Which client CPU, memory, disk, network, and remote capacity does it consume? |
| Reuse | Does it obtain a cache result or reuse worker initialization, and how often? |
| Failure behavior | Which fallback is permitted, and can it hide an infrastructure or action defect? |
| Placement evidence | Which client and backend observations prove the selected path? |
Use a narrow override for the canary rather than changing every spawn. For example, a pinned configuration may test a candidate strategy for one mnemonic:
build:strategy-canary --strategy=Javac=remote
The exact mnemonic and applicable strategies come from the analyzed actions and the pinned Bazel and ruleset versions; do not copy a mnemonic from another repository and assume it represents the same workload. A broad default remains useful as a fallback policy, but the high-value exceptions should be justified by measured rows in the matrix.
The canary must preserve target scope, revision, configuration, execution platform and toolchain, client and service versions, cache temperature, and output-download policy. Change one strategy decision at a time. Require output correctness and complete evidence before accepting a latency or cost win, and define a rollback that restores the previous strategy map without clearing shared state.
A four-step flow separates configured strategy intent, Bazel strategy selection, actual execution or reuse, and joined evidence. A matrix shows the distinct roles and limits of execution logs, profiles, Build Event Protocol data, and backend telemetry for cache hits, local execution, persistent workers, remote execution, and fallback.
no joined executor attempt
Prove placement instead of inferring it
A successful build proves that Bazel obtained required outputs. It does not say whether an action executed locally, remotely, through a worker, or not at all. A remote cache hit is especially easy to mislabel: reusable remote bytes are not evidence of remote execution.
Join observations at three layers:
The runnable placement-evidence harness creates isolated cohorts for each outcome. Its verifier joins Bazel records to backend state and makes the negative assertions explicit, rather than treating a configured strategy as observed placement.
- Bazel-side action identity: mnemonic, target context, command and input identity, plus an execution log or other spawn-level record. Bazel documents tool-friendly execution-log outputs for executed spawns.1
- Invocation timing: a profile shows scheduling, queueing, worker, transfer, and critical-path effects. Treat it as timing evidence, not by itself as a backend executor identity.
- Invocation result context: BEP supplies structured command, target, result,
artifact, and completion evidence that lets fleet tooling retain the cohort's
outcome. It does not replace spawn-level placement evidence. In particular,
ActionExecutedevents cover failed actions by default; publishing them for every action requires explicit configuration. 5.5.1 Build Event Protocol (BEP) develops the event graph and completeness rules.5 - Service-side identity: cache lookup/result, remote operation, scheduler attempt, and executor or worker-pool observations joined to the same action.
The evidence expected for each case differs:
| Claimed outcome | Minimum distinguishing evidence |
|---|---|
| Remote cache hit | Reused result and referenced blobs, with no executor attempt for that action |
| Local subprocess or sandbox | Bazel spawn record plus local strategy/process evidence, and no joined remote operation |
| Persistent worker | Bazel worker request/process evidence for the action, not merely the mnemonic |
| Remote execution | A remote operation and executor attempt joined to the action, followed by a reachable result |
| Fallback | Both the failed or unavailable preferred path and the accepted alternate path, with the fallback reason |
| Dynamic winner | Both branch attempts, winner and accepted outputs, loser cancellation or completion, and consumed resources |
Absence of one backend record is not enough to prove local execution: telemetry may be incomplete or joined with the wrong identity. Conversely, a remote AC or CAS request only proves remote data access. 6.1.4 Action and Result Lifecycle provides the control, data, and result transitions needed to disambiguate these cases.
Diagnose: A canary configured with --strategy=CppCompile=remote is green,
and backend logs show many CAS reads but no joined executor attempts for the
sampled actions. Did the compiler run remotely?
Reveal
Not necessarily. The strongest current hypothesis is remote cache reuse: CAS traffic proves remote data access, not execution. Check the Bazel execution log for whether those spawns executed, join their action or result identities to AC and CAS observations, and run a controlled miss or cache-bypass cohort. If fallback is permitted, also retain the preferred-path failure and local attempt; the configured strategy alone cannot distinguish these paths.
Promote a strategy only with bounded fallback
Fallback trades availability for diagnostic clarity and sometimes for correctness. During qualification, disable or tightly bound it so a green result cannot hide remote incompatibility, worker rejection, or sandbox failure. In normal operation, document which action classes may fall back, the alternate strategy, the triggering failures, and the evidence retained from both paths.
Promote the canary only when the strategy matrix shows correct outputs, better end-to-end outcomes for its named journey, acceptable local and remote resource cost, and observable placement. Roll back on output divergence, missing placement evidence, unexpected fallback, or a protected latency, reliability, or capacity regression. Concurrency limits are a separate control developed in 6.4.6 Execution Concurrency; increasing them is not a repair for a poor strategy choice.
Choose execution strategy per measured action class, not as one fleet-wide preference. Compare local, sandboxed, worker, remote, and mixed paths from action readiness through usable output, including queueing, startup, transfer, materialization, resource pressure, correctness, and failure behavior.
Configuration expresses intent; it does not prove placement. Join Bazel's spawn and timing evidence with BEP invocation outcomes and cache, remote-operation, scheduler, worker, and executor observations. A cache hit is not remote execution, a green build is not placement proof, and fallback is safe only when its scope, trigger, alternate path, and evidence are explicit. Promote a bounded canary only after its action-class matrix passes correctness and end-to-end outcome gates with a reversible strategy map.
Check your understanding · 3 questions
1.A canary sets a per-mnemonic remote strategy and finishes successfully. What can the operator conclude immediately?
Select one answer
2.Match each observation to the strongest conclusion it supports:
Drag each answer onto the matching prompt, or click an answer and then click a prompt
3.Which evidence should a controlled fallback canary retain before the alternate path is accepted as understood?
Select all that apply
Footnotes
-
Commands and Options — broad and per-mnemonic strategy selection, local/sandboxed/worker/remote distinctions, and execution-log options ↩1 ↩2
-
Dynamic Execution — explicit local/remote racing, applicability constraints, resource trade-offs, cancellation, and profile-based diagnosis ↩
-
Breaking down build performance — separating clean and incremental cohorts and interpreting execution and resource metrics ↩
-
Creating Persistent Workers — worker applicability, avoided startup cost, cross-action caching, and the worker protocol boundary ↩
-
Build Event Protocol Glossary — invocation result events and the default publication boundary for
ActionExecuted↩