6.3.12 Remote Execution Capacity

Adding executors because a dashboard shows a queue is a tempting response—and often an expensive one. An action can wait because no compatible worker has the right memory, device, image, or isolation mode; because the worker is fetching inputs; or because a shared CAS, network path, or scheduler is saturated. Conversely, a pool can show spare average CPU while its memory-heavy or device-bound actions starve. Capacity is therefore a prediction about a named action class reaching a compatible, usable executor, not a fleet-wide executor count.

6.3.8 Scheduling Actions on Remote Executors established the queue and matching evidence needed before this point. Start with actions that are already admitted and matchable. If a request has no compatible destination, fix the property-to-pool contract in 6.3.2 Remote Executor Matching; buying another incompatible worker cannot help. Heterogeneous fleets make this distinction concrete: an embedded test pool may be the only place a hardware test can run, while ordinary compilation continues elsewhere.¹

Model an occupied executor, not just a command

For each action class and compatible pool, make a small capacity record. An action class is a workload with similar execution requirements and behavior—not merely a mnemonic. A Java compile and a simulator UI test can have quite different memory, device, input, and lifetime demands even when both happen to use one CPU at a particular instant. Simulator tests are an especially clear case: a UI test may need an exclusive simulator lease while logic tests can share one.²

Record	What the model needs	Why an average hides it
Arrival and burst shape	Arrivals by cohort and action class, including concurrent build starts and retry/fallback traffic	A quiet hourly average erases a short presubmit surge that fills the only compatible pool.
Occupancy distribution	Assignment-to-release stages: input/CAS fetch, execution-root and isolation preparation, command time, output upload, cleanup, and any deliberate warm-state work	A short command can hold a worker for a long fetch or upload; a long tail can consume the last suitable slot.
Resource distribution	CPU, memory, local disk and I/O, network, device or license leases, and image/runtime constraints at the concurrency actually observed	One worker is not one interchangeable slot, and average CPU does not reveal memory or I/O pressure.
Usable supply	Executors that are healthy, matchable, correctly provisioned, and able to accept this class, plus their effective concurrency	A nominally idle macOS, ARM, or small-memory worker is not spare capacity for an incompatible request.
Change and failure time	Measured request-to-usable scale lag, cold-image/input effects, a burst horizon, and the capacity lost during one declared failure scenario	A pod reported as created is not yet an executor that can successfully claim the next action.

Keep the records joined to the action, attempt, pool, and worker identities from 6.3.4 Remote Action Tracing. The worker stages matter here: input presentation, isolation startup, command execution, output upload, and cleanup are distinct occupancy causes, not all “execution.” 6.3.9 Remote Executor Environments and Isolation explains how to attribute those stages. A Snowflake deployment found low IOPS in network-attached worker and cache storage, along with an undersized scheduler, and used action traces to distinguish execution, queuing, upload, and download while right-sizing memory-different pools.³

A useful model has no universal executor-count formula. For each pool, project the arriving work during the scale-lag and burst interval against the distribution of time it will occupy each required resource. Then compare that demand with usable compatible supply after reserving the failure headroom the service promises. State the prediction in observable terms: which class will queue, which resource or stage limits admission or release, how much burst can be absorbed before scale-out is useful, and what changes after a worker or availability domain is unavailable. The model must retain uncertainty rather than convert a thin sample into a precise-looking threshold.

Treat autoscaling as a delayed capacity change

Autoscaling can move capacity; it does not remove the need for a model. Measure the whole lag from the backend's scaling decision through instance allocation, image or runtime readiness, registration, eligibility, and the first successful claim by the intended action class. Include the extra fetch and network demand created by cold workers. A fixed pool that cannot meet bursts and an autoscaler that arrives after the burst are equivalent for that protected journey.

Choose scale signals from the prediction, not from a generic CPU target. For example, sustained queue age and queued resource demand within a compatible pool can be evidence to add that pool; a rising CAS-fetch or network stage may instead show that scaling workers would worsen the bottleneck. BuildBarn's automatic resource management learns that an action needs a larger worker after an observed small-worker failure, and routes later executions accordingly. That is useful evidence for an implementation-specific class-sizing mechanism, not a portable rule for every REAPI backend.⁴ BuildBarn's related size-class case also shows why broad “large” annotations can spend most compute time on unnecessarily large workers.⁵

Scale-down needs the same care. Do not remove an executor simply because its instantaneous CPU is low if it is the only warm compatible capacity, holds a scarce device, or would leave no failure headroom. One production incident combined overly aggressive scale-down with slow scale-up, producing recurring queue spikes even though the fleet sometimes looked healthy.⁶ Record the chosen drain behavior, the capacity it temporarily removes, and the fallback load it can create. Cancellation, retries, and local fallback have their own correctness and amplification rules in 6.3.13 Remote Action Failure Handling; do not quietly count their extra work as free headroom.

Make the model predict a load test

The test is the decision point, not a decorative benchmark. Select a representative workload cohort with the relevant action-class mix, platforms, input/output shapes, cache temperature, and burst pattern. Hold the configured pool vocabulary and autoscaling policy known. Then write down the model's prediction before the run: expected queue-age shape, which compatible pool should fill first, which stage should saturate, how long added workers will take to become useful, and the point at which the stated failure headroom is spent.

During the test, join client, scheduler, worker, CAS, and network evidence in one clock domain. Check correctness and complete output publication first; a fast failed attempt is not capacity. 6.3.7 Remote Execution Storage defines what proves that remote result is actually reachable. Then compare prediction and observation:

Observation	Capacity conclusion it supports	Next move if it disagrees with the model
Compatible-pool queue age rises while suitable workers are fully occupied	The pool's usable executor/resource supply is short for this workload	Revisit the class resource and occupancy distributions, then evaluate a pool change or scale policy.
Workers begin commands slowly despite idle executor slots	A fetch, image, CAS, network, scheduler, or preparation stage may be the limiter	Trace the stage; do not call the number of executors the saturated resource.
Added workers register after the demand has drained	Scale lag, cold start, or signal timing is ineffective for this burst	Change the policy or protect warm/failure headroom, then rerun the same cohort.
Some queues grow while global capacity remains idle	Capacity is unmatchable or the class/pool routing is wrong	Return to matching evidence rather than increasing fleet size.
The predicted pool saturates but another stage fails first	The model omitted a dependency or mismeasured its distribution	Revise the model from the residual; keep the failed prediction as evidence.

This test should include a bounded failure case—such as losing the declared worker slice or preventing a scale-out from becoming usable—so the recorded headroom is demonstrated rather than assumed. It should also distinguish a warm run from cold input and output paths. Large, repeatedly transferred artifacts can saturate network delivery even when commands themselves have available CPU; Snowflake encountered this when large statically linked test binaries had to reach many workers.³

Do not turn an autoscaling demonstration into a generic cluster tutorial. The result needed here is narrower: a revised, action-class and compatible-pool model that correctly predicts a representative load test and identifies the stage that actually saturates. 6.3.10 Remote Input Presentation and 6.3.11 Remote Output Availability deepen the input and output choices that change worker occupancy. 6.6.7 Planning Capacity for Build Infrastructure extends this same evidence discipline to cache, scheduler, BES, and CI services; it does not replace the executor-pool model.