6.3.8 Scheduling Actions on Remote Executors

An action can be ready for remote execution and still wait forever. Its inputs, tools, and execution properties may describe a compatible worker correctly, yet the compatible pool can be busy, a queue can favor other work indefinitely, or the service can keep accepting work long after it can give a useful response. That is a scheduling problem, not proof that the action was mismatched.

6.3.2 Remote Executor Matching established which destinations can fulfill an action's request. Start this investigation with that already-valid match set. The scheduler's job is to decide whether to admit the action, where to hold it while it waits, and when to give one compatible executor permission to begin.

The portable state is not the scheduler policy

REAPI gives the client an execution request and a long-running Operation to observe. It deliberately does not prescribe the server's queues, admission rules, priority-policy mapping, leases, worker protocol, or fairness rule.¹ QUEUED, EXECUTING, and COMPLETED can therefore describe useful client observations without answering the operational questions below. In particular, EXECUTING is portable evidence that the remote execution system currently reports worker execution. Backend evidence is still needed to identify and reconcile the specific worker, current dispatch, assignment or lease, queue, and policy path.

Question	Evidence that can answer it	Do not infer it from
Was this work admitted?	A backend admission decision, reason, and cohort or request correlation	A non-terminal Operation alone
Where can it wait?	The implementation's compatible queue or pool identity	The action's platform properties alone
Which worker or claim is current?	A backend dispatch, assignment, or lease joined to an executor	EXECUTING alone
Is it receiving fair service?	Per-cohort wait and dispatch evidence over time	A fleet-wide average or queue depth
Is the service overloaded safely?	A stated defer, reject, or backpressure outcome and its reason	The fact that requests still receive Operations

That separation prevents a common bad diagnosis: treating a stale assignment as the current worker or claim for an action that reports EXECUTING, or calling a large pool "available" when none of its workers can run the queued class. Keep the portable Operation name and stage and the backend queue, dispatch, lease, worker, and policy-revision identifiers as different fields in the same trace. The full portable-to-backend trace is developed in 6.3.4 Remote Action Tracing.

Schedule the compatible set, not the whole fleet

For every accepted action, make the control path observable:

1. Admission decides whether this request may consume a bounded service population now. Record the cohort, limits consulted, and an explicit accepted, deferred, or rejected result.
2. Queue selection places accepted work only where its already-established compatible executors can obtain it. A global queue can still work, but the scheduler must expose how it preserves compatibility.
3. Dispatch or lease gives one executor the current right to work. That right needs an implementation-specific expiry, loss, and reconciliation contract; it is not a REAPI guarantee.
4. Completion accounting releases the relevant limit only when the backend's chosen witness says the attempt is no longer consuming it. Do not silently count an expired lease as healthy running work.

This is why a generic request load balancer is often a poor substitute for a scheduler. Build actions have highly uneven duration and resource demands, so queue length is not a reliable prediction of wait time. One implementation uses late binding: it offers work to several executor-local queues and lets the first eligible executor obtain an exclusive claim.² That is useful implementation evidence, not a portable algorithm to copy.

Priority needs a limit and fairness needs a witness

REAPI can carry an optional priority hint, but the server decides the supported range and how that hint maps to scheduling policy. Priority in that policy says which eligible work should go first during contention. It does not decide how much work a cohort may inject, how long lower-priority work may wait, or when the service must stop accepting work. Without those other controls, a continuous stream of urgent requests can turn a lower-priority queue into invisible starvation.

Choose backend policy per named workload cohort—for example, interactive, presubmit, release, or background work—and make three promises testable:

• Quota or concurrency limit: the largest named share a cohort may occupy. It protects other work from one runaway build without claiming that the queue itself supplies tenant isolation.
• Fairness rule: the service it promises among eligible cohorts, such as a bounded wait target or a documented weighted/round-robin rule. Measure the result per compatible pool and cohort, not only across all workers.
• Backpressure rule: what happens before the service becomes an unbounded waiting room. It may defer, reject, or ask an upstream submitter to slow down; record the decision and reason so callers can distinguish overload from a failed action.

These mechanisms are backend choices. For example, one BuildBuddy deployment uses tenant queues with round-robin service and priority queues, while its executors obtain an exclusive lease before running a dequeued action.³ That demonstrates a possible combination of fairness, priority, and leases; it does not make its queue structure or flags part of Bazel or REAPI.

Do not turn this into an organization-wide ranking debate. This article needs the scheduler contract that enforces a chosen policy. Deciding who may declare a cohort urgent and governing that authority require an organization's policy process. Likewise, one queue or quota is not an end-to-end security boundary; 6.3.14 Multi-Tenant Remote Execution connects identity, storage, executor, and result controls into that larger guarantee.

Diagnose an overload with a joined trace

Suppose interactive actions in a Linux pool are slow while the dashboard shows idle macOS workers and a modest global queue. Neither observation disproves overload. The macOS workers are not matchable capacity for these actions, and the global depth hides which Linux cohort is waiting.

Start with a small cohort and time window. For representative actions, join the action and Operation identifiers to the admission decision, compatible pool, queue-entered time, every dispatch or lease transition, executor identity, and terminal attempt outcome. Then compare the following facts in one clock domain:

Signal	It helps distinguish	It cannot establish alone
Queue age by compatible pool and cohort	A protected cohort waiting behind usable demand	Command runtime or total fleet health
Admission deferrals or rejections by reason	Intentional bounded overload from an unexplained disappearance	Whether a rejected action was ever executed
Dispatch/lease age and expiry or loss count	Stalled or stale ownership from fresh execution	That the command completed or outputs published
Matchable idle workers and resource state	A scheduling/policy issue from merely nominal capacity	Whether the pool is large enough for the workload
Per-cohort wait percentiles and starvation count	Whether the stated fairness promise is being met	Whether all tenants are isolated end to end

Queue age is especially useful when paired with the service's limit. An incident with a long horizontal queue-age line turned out to be an internal one-hour reject limit after a single build submitted a vast number of actions; the useful follow-up was a shorter timeout, build cancellation, and per-build guardrails, not a claim that the scheduler's queue was broken.⁴ The same source notes that careless proxy backpressure can keep a finished worker from appearing available, so "busy" must be attributed to the actual backend state rather than guessed from a client stream.⁴

If the trace shows enough eligible work and a sustained lack of matchable capacity, take the measured arrival, duration, resource, and scale-lag evidence to 6.3.12 Remote Execution Capacity. Do not enlarge a pool merely because a global queue was nonempty. If it shows a bad mapping or no compatible executor, return to 6.3.2 Remote Executor Matching instead.