6.4.6 Execution Concurrency

Increasing --jobs can expose more parallel work, yet make the build slower. The client may admit more remote requests while local actions compete for memory, persistent-worker pools multiply, downloads contend for the same network and disk, and the critical path waits behind work that is not urgent. Concurrency is therefore one budget spanning several execution paths, not one number to maximize.

6.4.1 Choosing Execution Strategies established which action classes may run locally, remotely, through workers, or dynamically. This article turns that placement matrix into a measured concurrency policy. Capacity inside the remote service is treated separately in 6.3.12 Remote Execution Capacity; here the question is how much work one Bazel invocation should offer to all of those paths together.

Read each control at its real boundary

--jobs limits concurrent jobs admitted by Bazel. It is not a promise that all of them are consuming a CPU at the same instant. Local resource controls apply another gate to locally running actions and do not limit the Bazel server's own analysis and orchestration work.¹ A remote action does not consume a local execution CPU, but it still occupies client-side concurrency and may use local CPU, memory, network, and filesystem capacity while preparing inputs or receiving outputs.

Persistent workers introduce another multiplicative boundary. Pool instances are separated by worker keys, so the process count and memory footprint depend on the set of active keys as well as the limit for any one pool. Multiplexing may reduce process duplication, but only after the concurrency and isolation contract from 6.4.5 Multiplex Worker Qualification passes.

Model the effective limit for an action class as the tightest gate on its path:

Execution path	Admission and contention to observe
Local subprocess	Bazel jobs, declared local CPU/RAM, host CPU, memory, disk, and sandbox work
Persistent worker	Bazel jobs, pool availability, worker-key count, process memory, and worker queue time
Remote execution	Bazel jobs, client RPC/transfer work, remote queue, eligible workers, and output delivery
Dynamic execution	Both local and remote branches plus duplicate work, cancellation, and the shared output lock

This table explains why local and remote capacity are not independent. Raising client admission can shorten a remote queue only when the backend has unused eligible capacity; otherwise it can lengthen that queue and increase client overhead. Enlarging worker pools can remove worker-borrow waits while causing host swapping or OOMs. Backend-specific scheduler policies can matter: Buildbarn, for example, uses a size-class cache and estimates action duration to prioritize long-running work.² That is not a portable REAPI guarantee. Inspect the scheduling contract and telemetry of the backend you operate; when it exposes separate eligible pools or size classes, a fleet-wide utilization average can hide starvation in the pool that serves the critical path.

Locate the saturated gate before tuning

Begin with a comparable workload cohort from 6.1.2 Measuring Builds Fairly. Keep revision, target scope, configuration, platform, cache temperature, strategy policy, client and service versions, and output-download policy fixed. Record correctness and required-result completeness before considering speed.

Use the Bazel profile and execution evidence to separate at least these intervals:³

• actions ready but waiting for Bazel or local resources;
• persistent-worker borrow, setup, and execution time;
• remote setup, queue, execution, and download time;
• local CPU, memory pressure, OOMs, disk and network saturation;
• remote pool eligibility, queue depth, worker occupancy, and scale-out delay;
• the action classes and dependencies on the critical path.

The critical-path profile snippet is a small local-only exercise in separating structural concurrency from the critical path. Its verification script checks two independent action chains that overlap before joining at a final action, and its captured output records the profile and aquery correlation. It does not model worker borrows, remote queues, or fleet capacity; use it to practice reading the local action graph before joining that evidence to production telemetry.

Do not use one busy percentage as the diagnosis. Low average CPU can coexist with a memory-bound linker, a saturated worker pool, or a remote queue for one platform. High remote utilization can be healthy throughput or a queue that is already beyond its latency objective. Machine configuration and leaked resources can also degrade long-lived runners independently of Bazel's nominal job limit.⁴

A useful first statement is causal and falsifiable: “JVM compile actions wait for worker instances while host memory remains below its bound; adding one instance per observed worker key should reduce borrow time without increasing memory pressure.” “CPU is only 60%, so raise jobs” is not a hypothesis because it does not identify the waiting work or the gate that should move.

Change one gate and watch the whole path

Build a small experiment matrix around the suspected boundary. A concurrency experiment should name:

1. the journey and action class being changed;
2. one primary control, such as --jobs, the local resource budget, a worker pool limit, or remote capacity for one eligible pool;
3. matched baseline and candidate cohorts;
4. correctness, completion, and placement gates;
5. end-to-end latency or throughput plus the stage metric expected to move;
6. stop signals for OOM, queue growth, retries, fallback, tail regression, or displaced load; and
7. the exact configuration rollback.

Sweep a few bounded candidate values instead of searching for a universal ratio. Builds differ in graph width, action cost, worker keys, outputs, and environment, which is why concurrency settings that help one target set or machine may hurt another.⁵ Compare distributions by action class and the complete user journey; a micro-action benchmark cannot show whether downloads, analysis, a serial dependency, or pool scale-out dominates the invocation.

For example, suppose increasing --jobs reduces the time remote actions wait for client admission, but remote queue time rises by more, local peak memory crosses the safety margin, and end-to-end presubmit latency regresses. The experiment did not reveal “insufficient jobs.” It demonstrated that admission moved pressure to a remote or local gate. Restore the baseline, then test the next specific hypothesis: increase eligible remote capacity for that action class, reduce local competition, or leave concurrency unchanged and shorten the critical path.

Promote a policy, not a lucky number

Keep separate named configurations for materially different environments and journeys. A developer laptop, a memory-limited CI container, and an RBE-backed presubmit do not share one useful job count. Store the selected jobs, local resource budgets, worker limits, strategy routing, and output policy together so a later change does not silently invalidate the experiment.

Promote gradually and retain the baseline cohort. Require the candidate to preserve output correctness, intended placement, and required result delivery; improve the declared latency, throughput, or cost outcome; and remain within local and remote queue, memory, retry, and fallback bounds. Re-run the sweep after changes to graph shape, toolchains, worker implementations, client versions, runner sizes, remote pools, or download policy.

Avoid depending on proposed runtime improvements as an operational guarantee. Work on virtual threads and dynamic concurrency has aimed to decouple remote action concurrency from platform-thread count and react to memory pressure, but the published BazelCon material describes goals and experiments rather than a portable promise that manual workload qualification is obsolete.⁵

When no candidate improves the complete journey, concurrency is not the lever. The remaining constraint may be a sequential critical path, an oversized action, poor cache reuse, excessive transfer, or insufficient eligible backend capacity. Preserve that negative result: it prevents the next operator from repeating the same sweep or buying capacity that cannot affect the bottleneck. When the question becomes baseline, burst, scale-lag, or failure headroom across cache, scheduler, executor, BES, and CI stages, continue with 6.6.7 Planning Capacity for Build Infrastructure rather than treating one invocation's concurrency policy as a fleet-capacity plan.