6.4.3 Operating Persistent Worker Pools

A persistent worker trades repeated process startup for a long-lived tool process. Bazel starts the process once, sends it a sequence of work requests, and lets the tool retain useful initialization such as a warm runtime, parsed libraries, or internal caches.¹ That can be a large win for a JVM compiler and no win at all for a cheap native tool. The operating decision is therefore not “workers are faster,” but whether reuse pays for this action class after memory, pool fragmentation, and lifecycle failures are counted.

6.4.1 Choosing Execution Strategies established how to compare complete execution paths and prove which one ran. This article starts after worker is a valid strategy for the action. It shows how to decide whether to adopt it and how to keep the resulting process pools stable. Implementing the worker protocol is a rule-author concern; request-state containment and concurrent multiplex workers continue in 6.4.4 Worker State Isolation and 6.4.5 Multiplex Worker Qualification.

Prove that reuse repays the pool

A worker helps only when it avoids meaningful repeated work. Common candidates pay for runtime startup, JIT warm-up, compiler initialization, parsing, or reconstructing an internal cache on every action. A worker-capable tool may still be a poor candidate when one-shot startup is small, action parallelism is more valuable than warmth, retained state grows rapidly, or the action cohort is too sparse to reuse a process. Some TypeScript rules, for example, moved away from workers when short-lived native SWC processes better matched their per-file parallel workload.²

Run a correctness-gated comparison before tuning a pool:

1. Select one stable action cohort, finer than a target set when the same mnemonic covers different tools or configurations.
2. Run matched worker and non-worker cohorts with the same revision, targets, configuration, platform, machine class, cache condition, and output policy.
3. Require equivalent outputs and complete actions before comparing time.
4. Measure end-to-end invocation latency and the action cohort's critical-path contribution, not only compiler service time.
5. Record process starts, warm requests, worker RSS, host peak memory, strategy selection, and failures over clean and representative incremental builds.

The existing placement-evidence project makes the first proof concrete. Its worker configuration pins one EvidenceWorker instance and enables sandboxing and lifecycle output. The verifier then requires two actions to report the worker runner, one worker-creation identity, the same process ID, and the expected output bytes. That proves selection and reuse for the fixture. It does not provide a worker-versus- one-shot performance comparison, representative memory pressure, key fragmentation, recycle behavior, or worker-failure recovery; those remain measurements for your action cohort.

The warm cohort needs enough requests to expose the benefit and enough repeated builds to expose retained cost. The official worker guidance demonstrates why this distinction matters: its measured optimum differed between clean and incremental Java builds, and it explicitly warns that the best worker count depends on the workload and memory budget.¹ Those measurements are examples, not fleet defaults.

Reject worker adoption if correctness differs, the complete journey does not improve, or saved startup merely becomes host memory pressure and queueing. Keep a separate one-shot control in the canary so you can compare outputs and rerun the cohort without worker mode when you need to contain a worker defect. A strategy list such as --strategy=Javac=worker,local provides an applicability fallback: Bazel selects the first listed strategy that can execute the action. It does not generally retry with local after a selected worker crashes or returns a failed response. The actual mnemonic and one-shot strategy must come from the pinned ruleset and your isolation requirements.³

A mnemonic is not a pool identity

The most important capacity model is the WorkerKey. Bazel does not maintain exactly one pool per mnemonic. In the pinned Bazel 9.1 implementation, local key identity includes the startup command, environment, execution root, mnemonic, protocol, and worker modes such as sandboxing, multiplexing, and cancellation. Action configuration can therefore split one mnemonic into many keys, each of which can admit its own worker instances.^3,1,4

Tool-file freshness is related but distinct. The digest of the worker's tool files is deliberately not part of local WorkerKey equality. When Bazel borrows a pooled worker, it checks those files and rejects the existing process if their digest changed. A content-only tool update can therefore cause replacement and cold-start churn without creating a second local key; a changed executable path, startup argument, or environment can create a different key.⁴

Suppose a fleet permits four instances and the Javac cohort produces six distinct keys because heap flags, environments, protocols, or sandbox modes differ. The relevant upper-bound question is no longer “four Javac workers?” but “how many of those key-specific pools become live, and how many processes does each need?” Do not treat the arithmetic as a promise that every slot is eagerly created; treat it as a capacity envelope to verify under representative load.

Build a pool inventory that joins:

Observation	What it can reveal
Mnemonic and WorkerKey dimensions	Whether apparently identical actions actually share a compatible process
Active and idle instances per key	Useful concurrency versus stranded warm processes
Starts, shutdowns, and restarts	Cold-start churn or an unstable worker lifecycle
Requests per process and warm-request latency	Whether reuse is occurring and remains valuable
RSS and host peak memory over time	Stable warmth, an oversized pool, or retained growth
Worker log and exit reason	Tool crash, malformed protocol response, deliberate recycle, or host pressure
Selected strategy and action outcome	Applicability fallback versus a worker execution failure

Bazel supplies pieces of this evidence, not one stable report containing the whole table. --worker_verbose and stderr logs expose lifecycle details; an invocation profile and host process metrics cover other parts. Per-key active/idle counts, request histories, warm latency, exit attribution, and cohort fallback rates may require worker, host, or fleet instrumentation. Name the source of every field instead of presenting a joined inventory as a Bazel command output.

--worker_max_instances limits instances by worker key, not by mnemonic alone. The current command reference accepts a default or mnemonic-specific value and also supports machine-resource expressions, but none of those forms chooses the right value for your workload.³ Start with the smallest pool that keeps the measured cohort supplied, then increase one mnemonic at a time. Promote an increase only if reduced queue or critical-path time is worth the added cold starts and steady-state memory.

Diagnose fragmentation before raising limits

Low utilization beside many worker processes usually points to fragmentation, not insufficient capacity. Group the active WorkerKeys by the dimensions that split them. Look for configuration drift, varying startup or heap flags, protocol or sandbox-mode differences, or environment differences that do not represent an intentional compatibility boundary. --worker_extra_flag itself creates separate startup variants, so a diagnostic flag left in one configuration can split the pool it was meant to observe.¹

Do not collapse keys merely to save memory. A key boundary may protect a real tool or startup-configuration incompatibility. First decide whether the variation is required. Remove accidental variation at its source; retain intentional boundaries and budget each resulting pool.

Remote persistent-worker deployments make the same lesson more visible. A production JVM case study used dedicated Java and Kotlin compilation pools and reported worker thrashing when a tool change produced a new backend worker key. Its operators prepared upgraded pools and shifted traffic rather than expecting old warm processes to serve the new identity.⁵ This is backend-specific infrastructure: its tool-hash execution property is not Bazel's local WorkerKey object. The transferable rule is that any identity transition that creates a new remote pool needs planned coexistence, warmup, traffic movement, and retirement instead of being rolled out as if only bytes had changed.

Separate a leak from an oversized pool

High memory has two different shapes:

• Pool multiplicity: each process reaches a stable size, but too many keys or instances coexist. Reduce accidental keys or the per-key instance limit.
• Per-process growth: one process's RSS continues rising with requests or workload diversity. Preserve its logs, request history, heap or native-memory profile, and tool version before recycling it.

Repeated restarts can hide both causes. --worker_quit_after_build deliberately ends workers after an invocation and is documented mainly for debugging and profiling; using it continuously also removes the reuse the strategy was chosen to provide.^1,3 Use a bounded comparison with and without that control to distinguish retained cross-build state from same-build growth, then repair the responsible key, pool size, or worker implementation.

For diagnosis, --worker_verbose reports worker lifecycle activity, while worker stderr logs live beneath the output base's bazel-workers area. There may be more log files for a mnemonic than the configured instance limit because multiple WorkerKeys exist.¹ Retain those files together with the invocation profile and configuration identity; a process count without keys and request history cannot distinguish useful warmth from churn.

Define recycling as an observable policy rather than an emergency loop. Name the trigger—such as a proven version transition, a bounded RSS condition, or a specific worker failure—the maximum disruption, the rollback path, and the evidence retained before shutdown. After recycle, verify correct outputs, pool repopulation, memory stabilization, and restoration of the expected worker hit path. If growth returns under the same request sequence, route the defect to the worker or ruleset owner instead of shortening the restart interval.

Qualify the pool under representative load

A promotion load test should combine the adoption and lifecycle questions. Use the expected action mix and configuration diversity, not a single hot compiler loop. Demonstrate all of the following:

• the worker cohort remains output-equivalent to its non-worker control;
• the intended WorkerKey cardinality is explainable from required differences;
• active and idle instance counts settle within the host memory budget;
• warm requests improve the protected build journey without starving other local work;
• memory reaches a stable envelope or a diagnosed recycle policy contains it;
• starts, exits, logs, and strategy selection are attributable to a key and invocation; and
• a separately selected one-shot control still produces equivalent outputs.

Stop the canary on output divergence, unexplained new keys, repeated cold-start churn, monotonic unexplained memory growth, missing lifecycle evidence, or an alternate-strategy rate that makes the measured worker benefit ambiguous. Coordinate the accepted pool with host and remote concurrency in 6.4.6 Execution Concurrency; raising global jobs is not a repair for a fragmented worker population.