6.4.5 Multiplex Worker Qualification
extraA multiplex worker replaces several isolated worker processes with concurrent requests inside one long-lived process. That can reduce process count, repeated startup, JIT warm-up, and duplicated in-process caches. It also removes a useful accidental safety boundary: requests that previously ran in different processes now share threads, memory, a working directory, output streams, and shutdown behavior.1
Treat multiplexing as a new concurrency mode for an already-qualified worker, not as a harmless pool-size optimization. 6.4.4 Worker State Isolation establishes whether sequential requests preserve the action contract. The multiplex qualification adds four harder questions: can requests overlap, can responses and diagnostics be attributed, can one request be cancelled without damaging another, and can the chosen sandbox mode still isolate filesystem access?
Three Bazel proxies send concurrent requests into one multiplex worker process. Request 12 completes before request 11 while each response keeps the correct ID. Cancelling request 31 does not stop its peer. Each request uses a separate sandbox root, while stderr remains a shared process log. A failed qualification crosses a fallback boundary to singleplex or standalone execution.
sandbox_dir/11reads and writes for 11
sandbox_dir/12same basename, separate data
sandbox_dir/31no access after response
--noworker_multiplex or a tested standalone strategyPin the contract before testing it
Qualification applies to a specific Bazel version, ruleset version, worker
binary, protocol encoding, and strategy configuration. Record those together
with the action mnemonic and execution requirements. Multiplex support is
declared by the ruleset with supports-multiplex-workers; Bazel can be forced
back to non-multiplex operation with --noworker_multiplex. The feature remains
documented as experimental, so a result from another Bazel or ruleset release is
evidence to repeat the test, not proof that your combination is safe.1
The operational distinction is that overlapping requests have identities and may finish out of order. Your evidence must therefore join every response, exit code, and action-facing diagnostic to the initiating request instead of using completion order. The worker and rule obligations that make this possible—including protocol framing, response IDs, cancellation responses, and output routing—are developed in 4.4.4 Persistent Workers for Rule Authors.2
Build a qualification record with at least these cells:
| Cell | What to prove | Failure signal |
|---|---|---|
| Concurrent success | Overlapping requests produce their isolated baseline outputs | Output depends on overlap or completion order |
| Response identity | Each exit code and response body returns to its own request | Swapped, duplicated, missing, or malformed response |
| Diagnostic identity | User-facing output can be assigned to the responsible action | Shared log lines are treated as action-specific evidence |
| Cancellation | Cancelling one request does not stop, corrupt, or relabel another | Cross-request termination, late writes, or lost response |
| Filesystem isolation | Every read and write stays below that request's sandbox root | One request observes or overwrites another request's files |
| Fallback | Disabling multiplexing yields a working ordinary worker or non-worker path | Green multiplex mode is the only recoverable configuration |
Do not infer these properties from a compiler being described as thread-safe. Thread safety may protect compiler internals while a wrapper still reuses an argument parser, temporary filename, current directory, output buffer, or cancel token across requests.
Force overlap and make mistakes observable
A useful test needs concurrent requests that would expose incorrect sharing. Choose two or more representative actions and arrange for their worker requests to overlap. Give each request distinguishable inputs, expected output content, temporary names, and diagnostic markers. Include deliberately conflicting basenames so a worker that accidentally resolves paths against the process working directory cannot pass merely because every test uses unique names.
Run each action alone first to establish its output and diagnostics. Then repeat the concurrent run with varied start order and varied completion order. The accepted outputs must match the isolated baselines, and each response's request identifier, exit code, and response text must remain attached to the initiating action. Repeat under representative load: a two-request success does not expose queueing, shared-cache contention, or a rare response-framing race.
Worker stderr deserves special treatment. Bazel's multiplex documentation
warns that all proxies for the same process share a log file, so concurrent
messages may interleave. Require action-facing output to remain attached to its
response, and treat the shared worker log as process evidence only when request
identity makes correlation reliable.1,2
The runnable qualification project
turns those checks into two separate evidence paths. Its
named configurations
select multiplex, multiplex-sandbox, singleplex, and standalone cohorts, while
the worker's barrier and fault hooks
make incorrect request sharing observable. In one direct run, requests 11 and
12 stop at separate barriers; releasing both lets the shorter request 12 return
first. The verifier still joins BETA to 12 and ALPHA to 11, and deliberately
swapped IDs and shared temporary state must make the sensitivity checks fail.
Diagnose: Two concurrent compile requests produce correct files, but their warnings appear interleaved in one worker log without request identifiers. Has the worker passed multiplex qualification?
Reveal
No. Output correctness passed one cell, but diagnostic attribution did not.
The shared log cannot prove which action emitted a warning. Capture
action-specific tool output into the matching WorkResponse, or add reliable
request identity to process-level diagnostics, then repeat with reversed
completion order.
Cancel a request at the dangerous boundaries
Cancellation is not permission to abandon bookkeeping. The observable contract is that the cancelled request closes once, produces no late file activity, and does not corrupt or relabel another response. Bazel may clean request files as soon as a response is sent, so a late write is a qualification failure. The exact cancel-request and response shapes belong to the linked rule-author contract.2
Exercise at least three timings: cancellation while work is active, cancellation near normal completion, and cancellation after the worker has already responded. The last case must be ignored. Alongside the cancelled request, keep another request running and require it to finish with its baseline output and correctly matched response. Then submit a later request to detect a cancel token, interrupted thread state, partial cache entry, or temporary file that leaked forward.
The project's cancellation matrix drives all three boundaries without depending on a lucky wall-clock race. In the active case, request 31 is held at a barrier and cancelled while peer 32 continues; request 33 then probes for leaked state. The near-completion case releases request 41 before cancelling it, and the post-response case cancels 51 only after retaining its response. The evidence assertions require one terminal response per ordinary request, unchanged peer and probe outputs, and no second response or changed file after the late cancel.
Pin cancellation separately from multiplexing. Current worker documentation
requires the action to declare supports-worker-cancellation and Bazel to have
worker cancellation enabled. If that combination is not supported by the
pinned ruleset and Bazel release, record cancellation as an explicit limit and
do not use a strategy that depends on safely interrupting the worker.2
Qualify the sandbox path, not just the flag
Singleplex sandboxing can isolate one worker process. A multiplex process serves
several requests at once, so process isolation cannot give each request a
different filesystem view. A multiplex-sandbox-aware worker instead receives a
per-request sandbox_dir; operational qualification must prove that every read
and write stays within the request's assigned root. Path translation and the
rule-side declaration stay with that implementation contract.1
Configuration alone cannot retrofit multiplex sandboxing into the worker. Test the advertised mode with the same basename and different contents in two request sandboxes, plus an attempted read outside the assigned directory. Verify both output separation and the absence of cross-request reads in the worker's evidence. If dynamic execution is in scope, test that exact combination: Bazel documents that a non-sandboxed multiplex worker under dynamic execution falls back to sandboxed singleplex workers, so a green build may prove fallback rather than multiplex placement.1
That distinction is explicit in the project's
top-level qualification command:
the direct protocol sandbox test uses identical same/result.txt basenames in
different roots, but a separate Bazel cohort retains --worker_verbose and
execution-log evidence and refuses to label the cell multiplex merely because
the output bytes are correct.
Promote with an explicit escape path
Compare a matched singleplex and multiplex cohort after correctness gates pass. Measure end-to-end action and invocation latency, worker process count, peak and steady-state memory, throughput, response and cancellation failures, restarts, and fallback frequency. Fewer processes is not automatically a win if lock contention, garbage collection, shared-cache contention, or attribution gaps worsen the complete journey.
Start with one action class and a bounded client or CI cohort. Retain the pinned configuration and the evidence for overlap, response matching, diagnostics, cancellation, sandboxing, and fallback. Stop on any output divergence, unmatched response, unexplained worker restart, late write after response, cross-request state, or missing request identity. The rollback should disable multiplexing for the cohort without clearing worker or build state before the failure is captured.
Ordinary persistent workers remain the default escape path when multiplex qualification fails; a standalone or sandboxed non-worker strategy may be the safer fallback when the worker implementation itself is suspect. Pool sizing and memory lifecycle remain with 6.4.3 Operating Persistent Worker Pools, while 6.4.6 Execution Concurrency handles the wider local and remote resource budget after this action class is known to be safe.
Enable multiplex workers only for a pinned Bazel, ruleset, worker, protocol, and sandbox combination that passes adversarial concurrency tests. Force requests to overlap and finish out of order; require isolated baseline outputs, atomic and correctly identified responses, attributable diagnostics, and no shared process or filesystem state that changes another request.
Cancellation must preserve request identity, produce exactly one response, stop
file access after response, and leave concurrent and subsequent requests
unchanged. Multiplex sandboxing is an implementation contract around each
request's sandbox_dir, not a property supplied by a flag. Promote only a
bounded cohort whose correctness, stability, and complete resource outcome beat
singleplex operation, with --noworker_multiplex or a non-worker strategy kept
as a tested fallback.
Check your understanding · 4 questions
1.Requests 11 and 12 overlap, and request 12 finishes first. What evidence proves that the worker preserved the protocol boundary?
Select one answer
2.Classify the required outcomes when request 31 is cancelled while peer 32 is still running.
Choose True or False for each sentence
3.Which observations are needed to qualify multiplex sandboxing rather than merely observe a successful shared process?
Select all that apply
4.A dynamic-execution build produces correct outputs after multiplex support is enabled. What is the strongest next step before calling the multiplex qualification successful?
Select one answer
Footnotes
-
Multiplex Workers — concurrent request routing, response identifiers, shared output, enablement, dynamic-execution fallback, and per-request sandbox directories ↩1 ↩2 ↩3 ↩4 ↩5
-
Creating Persistent Workers — worker request and response contracts, cancellation exchange, exactly-once response requirement, and post-response filesystem rule ↩1 ↩2 ↩3 ↩4