6.4.5 Multiplex Worker Qualification

extra

A multiplex worker replaces several isolated worker processes with concurrent requests inside one long-lived process. That can reduce process count, repeated startup, JIT warm-up, and duplicated in-process caches. It also removes a useful accidental safety boundary: requests that previously ran in different processes now share threads, memory, a working directory, output streams, and shutdown behavior.1

Treat multiplexing as a new concurrency mode for an already-qualified worker, not as a harmless pool-size optimization. 6.4.4 Worker State Isolation establishes whether sequential requests preserve the action contract. The multiplex qualification adds four harder questions: can requests overlap, can responses and diagnostics be attributed, can one request be cancelled without damaging another, and can the chosen sandbox mode still isolate filesystem access?

One process serves many requests, but each request keeps its own boundary.
Route by request ID, not completion order. Cancellation closes only its matching request.

Three Bazel proxies send concurrent requests into one multiplex worker process. Request 12 completes before request 11 while each response keeps the correct ID. Cancelling request 31 does not stop its peer. Each request uses a separate sandbox root, while stderr remains a shared process log. A failed qualification crosses a fallback boundary to singleplex or standalone execution.

Qualification question: Can concurrent work finish, cancel, and access files independently even though it shares one worker process?
ONE MULTIPLEX WORKER PROCESS
Bazel proxy
Concurrent worker lane
Matched response
Request 11marker ALPHA
11 remains activeLonger work may finish later
Response 11 · secondALPHA stays with 11
Request 12marker BETA
12 completes earlyCompletion order may change
Response 12 · firstBETA stays with 12
Cancel 31peer 32 keeps running
31 closes onceNo late writes or shared cancel state
32 still succeedsLater requests stay clean
sandbox_dir/11
reads and writes for 11
sandbox_dir/12
same basename, separate data
sandbox_dir/31
no access after response
Shared process boundary: worker stderr can interleave. Treat it as process evidence unless every line carries reliable request identity.
Any boundary failsSwapped ID, leaked cancel state, outside-root access, or unproven placement
Cross the fallback boundaryUse --noworker_multiplex or a tested standalone strategy

Pin the contract before testing it

Qualification applies to a specific Bazel version, ruleset version, worker binary, protocol encoding, and strategy configuration. Record those together with the action mnemonic and execution requirements. Multiplex support is declared by the ruleset with supports-multiplex-workers; Bazel can be forced back to non-multiplex operation with --noworker_multiplex. The feature remains documented as experimental, so a result from another Bazel or ruleset release is evidence to repeat the test, not proof that your combination is safe.1

The operational distinction is that overlapping requests have identities and may finish out of order. Your evidence must therefore join every response, exit code, and action-facing diagnostic to the initiating request instead of using completion order. The worker and rule obligations that make this possible—including protocol framing, response IDs, cancellation responses, and output routing—are developed in 4.4.4 Persistent Workers for Rule Authors.2

Build a qualification record with at least these cells:

CellWhat to proveFailure signal
Concurrent successOverlapping requests produce their isolated baseline outputsOutput depends on overlap or completion order
Response identityEach exit code and response body returns to its own requestSwapped, duplicated, missing, or malformed response
Diagnostic identityUser-facing output can be assigned to the responsible actionShared log lines are treated as action-specific evidence
CancellationCancelling one request does not stop, corrupt, or relabel anotherCross-request termination, late writes, or lost response
Filesystem isolationEvery read and write stays below that request's sandbox rootOne request observes or overwrites another request's files
FallbackDisabling multiplexing yields a working ordinary worker or non-worker pathGreen multiplex mode is the only recoverable configuration

Do not infer these properties from a compiler being described as thread-safe. Thread safety may protect compiler internals while a wrapper still reuses an argument parser, temporary filename, current directory, output buffer, or cancel token across requests.

Force overlap and make mistakes observable

A useful test needs concurrent requests that would expose incorrect sharing. Choose two or more representative actions and arrange for their worker requests to overlap. Give each request distinguishable inputs, expected output content, temporary names, and diagnostic markers. Include deliberately conflicting basenames so a worker that accidentally resolves paths against the process working directory cannot pass merely because every test uses unique names.

Run each action alone first to establish its output and diagnostics. Then repeat the concurrent run with varied start order and varied completion order. The accepted outputs must match the isolated baselines, and each response's request identifier, exit code, and response text must remain attached to the initiating action. Repeat under representative load: a two-request success does not expose queueing, shared-cache contention, or a rare response-framing race.

Worker stderr deserves special treatment. Bazel's multiplex documentation warns that all proxies for the same process share a log file, so concurrent messages may interleave. Require action-facing output to remain attached to its response, and treat the shared worker log as process evidence only when request identity makes correlation reliable.1,2

The runnable qualification project turns those checks into two separate evidence paths. Its named configurations select multiplex, multiplex-sandbox, singleplex, and standalone cohorts, while the worker's barrier and fault hooks make incorrect request sharing observable. In one direct run, requests 11 and 12 stop at separate barriers; releasing both lets the shorter request 12 return first. The verifier still joins BETA to 12 and ALPHA to 11, and deliberately swapped IDs and shared temporary state must make the sensitivity checks fail.

think

Diagnose: Two concurrent compile requests produce correct files, but their warnings appear interleaved in one worker log without request identifiers. Has the worker passed multiplex qualification?

Reveal

No. Output correctness passed one cell, but diagnostic attribution did not. The shared log cannot prove which action emitted a warning. Capture action-specific tool output into the matching WorkResponse, or add reliable request identity to process-level diagnostics, then repeat with reversed completion order.

Cancel a request at the dangerous boundaries

Cancellation is not permission to abandon bookkeeping. The observable contract is that the cancelled request closes once, produces no late file activity, and does not corrupt or relabel another response. Bazel may clean request files as soon as a response is sent, so a late write is a qualification failure. The exact cancel-request and response shapes belong to the linked rule-author contract.2

Exercise at least three timings: cancellation while work is active, cancellation near normal completion, and cancellation after the worker has already responded. The last case must be ignored. Alongside the cancelled request, keep another request running and require it to finish with its baseline output and correctly matched response. Then submit a later request to detect a cancel token, interrupted thread state, partial cache entry, or temporary file that leaked forward.

The project's cancellation matrix drives all three boundaries without depending on a lucky wall-clock race. In the active case, request 31 is held at a barrier and cancelled while peer 32 continues; request 33 then probes for leaked state. The near-completion case releases request 41 before cancelling it, and the post-response case cancels 51 only after retaining its response. The evidence assertions require one terminal response per ordinary request, unchanged peer and probe outputs, and no second response or changed file after the late cancel.

Pin cancellation separately from multiplexing. Current worker documentation requires the action to declare supports-worker-cancellation and Bazel to have worker cancellation enabled. If that combination is not supported by the pinned ruleset and Bazel release, record cancellation as an explicit limit and do not use a strategy that depends on safely interrupting the worker.2

Qualify the sandbox path, not just the flag

Singleplex sandboxing can isolate one worker process. A multiplex process serves several requests at once, so process isolation cannot give each request a different filesystem view. A multiplex-sandbox-aware worker instead receives a per-request sandbox_dir; operational qualification must prove that every read and write stays within the request's assigned root. Path translation and the rule-side declaration stay with that implementation contract.1

Configuration alone cannot retrofit multiplex sandboxing into the worker. Test the advertised mode with the same basename and different contents in two request sandboxes, plus an attempted read outside the assigned directory. Verify both output separation and the absence of cross-request reads in the worker's evidence. If dynamic execution is in scope, test that exact combination: Bazel documents that a non-sandboxed multiplex worker under dynamic execution falls back to sandboxed singleplex workers, so a green build may prove fallback rather than multiplex placement.1

That distinction is explicit in the project's top-level qualification command: the direct protocol sandbox test uses identical same/result.txt basenames in different roots, but a separate Bazel cohort retains --worker_verbose and execution-log evidence and refuses to label the cell multiplex merely because the output bytes are correct.

Promote with an explicit escape path

Compare a matched singleplex and multiplex cohort after correctness gates pass. Measure end-to-end action and invocation latency, worker process count, peak and steady-state memory, throughput, response and cancellation failures, restarts, and fallback frequency. Fewer processes is not automatically a win if lock contention, garbage collection, shared-cache contention, or attribution gaps worsen the complete journey.

Start with one action class and a bounded client or CI cohort. Retain the pinned configuration and the evidence for overlap, response matching, diagnostics, cancellation, sandboxing, and fallback. Stop on any output divergence, unmatched response, unexplained worker restart, late write after response, cross-request state, or missing request identity. The rollback should disable multiplexing for the cohort without clearing worker or build state before the failure is captured.

Ordinary persistent workers remain the default escape path when multiplex qualification fails; a standalone or sandboxed non-worker strategy may be the safer fallback when the worker implementation itself is suspect. Pool sizing and memory lifecycle remain with 6.4.3 Operating Persistent Worker Pools, while 6.4.6 Execution Concurrency handles the wider local and remote resource budget after this action class is known to be safe.

key takeaway

Enable multiplex workers only for a pinned Bazel, ruleset, worker, protocol, and sandbox combination that passes adversarial concurrency tests. Force requests to overlap and finish out of order; require isolated baseline outputs, atomic and correctly identified responses, attributable diagnostics, and no shared process or filesystem state that changes another request.

Cancellation must preserve request identity, produce exactly one response, stop file access after response, and leave concurrent and subsequent requests unchanged. Multiplex sandboxing is an implementation contract around each request's sandbox_dir, not a property supplied by a flag. Promote only a bounded cohort whose correctness, stability, and complete resource outcome beat singleplex operation, with --noworker_multiplex or a non-worker strategy kept as a tested fallback.

Check your understanding · 4 questions

1.Requests 11 and 12 overlap, and request 12 finishes first. What evidence proves that the worker preserved the protocol boundary?

Select one answer

2.Classify the required outcomes when request 31 is cancelled while peer 32 is still running.

Choose True or False for each sentence

Request 31 receives exactly one terminal response and performs no file access after that response.
Request 32 preserves its isolated baseline output and correctly matched response.
A later request should run to reveal cancellation state or temporary data that leaked forward.
A cancel request arriving after an ordinary response should produce a second cancellation response.

3.Which observations are needed to qualify multiplex sandboxing rather than merely observe a successful shared process?

Select all that apply

4.A dynamic-execution build produces correct outputs after multiplex support is enabled. What is the strongest next step before calling the multiplex qualification successful?

Select one answer

0 of 4 answered

Footnotes

  1. Multiplex Workers — concurrent request routing, response identifiers, shared output, enablement, dynamic-execution fallback, and per-request sandbox directories 1 2 3 4 5

  2. Creating Persistent Workers — worker request and response contracts, cancellation exchange, exactly-once response requirement, and post-response filesystem rule 1 2 3 4