6.6.7 Planning Capacity for Build Infrastructure

A fleet can have spare CPU and still make every presubmit wait. The constrained resource may be cache metadata throughput, CAS bandwidth, scheduler matching, one memory-heavy executor pool, BES ingestion, or the finite number of CI runners that can start builds. Capacity planning for Bazel infrastructure means predicting which stage will saturate for a named workload, burst, and failure case—not choosing a machine count from a utilization average.

6.6.1 Tracing a Build Across Services provides the identities that connect an invocation to its actions, transfers, operations, events, and CI result. Capacity planning aggregates those correlated observations into arrival, service-time, resource, and queue distributions. 6.3.12 Remote Execution Capacity develops this model for compatible remote-execution pools; here we apply the same discipline to the whole build-service path.

Model stages, not one system-wide queue

Start with a journey and a cohort. An interactive build, a presubmit burst, and a postsubmit batch do not have interchangeable latency or completion needs. Within a journey, preserve dimensions that change the route or resource demand: action class, execution platform, configuration, cache state, artifact size, client and service versions, and the pool or region used.

Then describe each stage with the quantities that govern it:

Stage	Demand to retain	Evidence of saturation
Action Cache lookup	eligible lookup arrival rate, hit/miss mix, metadata request latency	lookup queue or latency rises while downstream work has not yet increased
CAS transfer	missing bytes and blobs, concurrency, object-size distribution, read/write direction	transfer queue, bandwidth, I/O, or connection limits rise
Scheduler	admitted actions by compatibility class and requested resources	compatible-pool queue age rises or work remains unmatched
Executor	full occupancy from input fetch through setup, command, upload, and cleanup	usable compatible slots or a required resource are exhausted
Output materialization	required output bytes, fan-out, client locality, fetch policy	builds finish remotely but clients still wait for outputs
BES and result consumption	events and referenced artifacts per invocation, ingestion and consumer throughput	required results arrive late or incompletely after build work finishes
CI runner	build arrivals, runner setup time, local Bazel state, checkout and orchestration occupancy	runnable jobs wait even though downstream Bazel services have headroom

This table is a deployment observation contract, not a list of metrics that Bazel or the Remote Execution API guarantees. REAPI supplies portable protocol surfaces such as Action Cache and CAS requests, Execute operations, standard operation stages, and execution metadata. Compatible-pool queues, admission and matching decisions, worker-slot accounting, setup and cleanup, BES-consumer backlog, and CI-runner occupancy come from the named backend or orchestrator. Record the producer and version for those signals. If the deployment cannot join an action to a queue, pool, worker, or downstream result, that stage's capacity conclusion is unknown; do not replace the missing join with a label, mnemonic, or nearby timestamp.¹

The stages are coupled. Adding cold executors can increase CAS reads and image pulls. A cache outage can turn reuse into synchronized execution demand. More CI runners can multiply the arrival rate seen by the scheduler and cache. A BES backlog can leave CI occupied after actions are complete. Treating each service as an independent autoscaling target misses these load transfers.

Remote execution illustrates why compatibility matters. A global executor total does not reveal whether a queued action can use those executors. Multi-region RBE also changes where capacity is usable: a self-contained regional pool can execute locally, while a cache-only edge may still depend on central storage and workers.² Capacity must therefore be recorded per action class, pool, region, and limiting resource before it is aggregated.

Describe workload as distributions

Averages erase the events infrastructure is built to survive. For every protected cohort, retain at least:

• the arrival distribution, including correlated starts when many developers or CI jobs react to the same revision;
• the burst duration and recurrence, not only an hourly request rate;
• the service and occupancy distributions at every stage;
• resource demand distributions for CPU, memory, disk I/O, network, scarce devices, and connection or stream limits;
• the measured delay from a scale decision to capacity that successfully serves the intended class;
• retry, fallback, cache-miss, and recovery traffic as real demand;
• one explicit component, pool, zone, or region loss that the service promises to tolerate.

The distinction between arrivals and duration is essential. A high action duration percentile does not state how many such actions arrive together. An arrival spike does not state how long each action occupies memory, a worker, or a transfer channel. You need both to predict queue formation.

Fixed fleets make this visible. Lyft described a roughly hundred-machine macOS CI fleet shared by ten applications; broad builds from only a small number of pull requests could consume the machines before test and lint jobs were added. The team reduced demand through affected-target selection because that scarce fleet could not simply expand on every burst.³ This is a capacity action: reduce admitted work while preserving required results. It is not evidence that target selection is always safer than adding supply; the correct decision depends on the completeness contract described in 6.5.6 Fail-Closed Target Selection.

Make scale lag part of supply

Nominal capacity is not usable capacity. For a runner or executor, measure from the decision to add it through allocation, image readiness, registration, checkout or input warming, compatibility admission, and the first successful unit of representative work. CI capacity guidance likewise distinguishes queue wait, runner readiness, warm state, finite-resource health, and the delay until Bazel begins useful work.⁴ If that interval is longer than the burst, the new instance does not protect that burst.

Signals should identify the constrained stage. CPU can remain low while work is waiting for a compatible large-memory runner, storage I/O, or network transfer. One production CI account instead scaled from recent queue length because its traffic followed synchronized developer submissions rather than CPU; its exact four-minute window was a locally tuned heuristic, not a portable Bazel default.⁵ The durable lesson is to measure the relationship among arrival intent, queue age, scale lag, and the first useful completion, then tune the policy against that evidence.

Scale-down also changes supply before demand disappears. Draining a runner or executor may strand work, discard valuable warm state, remove the only compatible slot, or reduce failure headroom. Some Mac hosts have a lengthy scrubbing interval after termination, making daytime downscale unable to return capacity promptly.⁶ Record drain and replacement time in the model rather than treating a stopped instance as instantly reversible.

Reserve headroom for a declared failure

“Thirty percent spare” is not a failure model. Name what is lost and which remaining route must absorb the work. Examples include one cache shard, one executor pool, one availability zone, one BES consumer partition, or the preferred CI runner class. Then project the resulting demand transfer:

1. How much nominal and compatible supply disappears?
2. Which requests retry, fall back, or recompute?
3. Which storage, transfer, and scheduler stages receive that extra work?
4. How long does replacement capacity take to become useful?
5. What correctness or completeness gate must hold while service is degraded?

This is why a cache flush or broad invalidation is a capacity event as well as a correctness and recovery event. It can synchronize recomputation, CAS traffic, and output publication across many invocations. Recovery planning in 6.6.6 Build-Service Incident Recovery decides how to contain and restore the service; the capacity model predicts whether the chosen degraded route can carry the load without creating a second incident.

The existing build-service recovery rehearsal makes that narrow transferred-load gate executable. Its replacement-pool negative control rejects a plan whose projected CAS demand exceeds its declared budget. This is a deterministic contract verifier, not an RBE backend, queueing simulator, or substitute for a production-shaped load test; use it to test the decision record before validating the real service under representative load.

Predict a representative load test

A capacity model becomes useful when it risks being wrong. Choose a representative cohort and state the prediction before the test:

• which stage and resource will saturate first;
• the expected queue-age and throughput shape;
• which action or build classes will wait;
• when scaled capacity will become usable;
• what happens when the declared component fails;
• which end-to-end latency and result-completeness signals must remain valid.

Run warm and cold variants when cache and local state materially change demand. Inject the bounded failure rather than merely subtracting capacity in a spreadsheet. Correlate the client, cache, scheduler, executor, BES, consumer, and CI observations so a queue at one stage is not confused with its upstream cause. Preserve correctness and complete results as gates: higher throughput of failed or unreported builds is not capacity.

Compare observation with prediction in a residual record:

Residual	What to revise
Queue forms earlier than predicted	arrival burst, occupancy tail, or usable supply was understated
Executors remain idle while actions wait	compatibility, admission, or pool topology was modeled incorrectly
More workers increase latency	CAS, image, network, scheduler, or materialization demand was omitted
CI runners stay occupied after execution	result delivery, artifact availability, or orchestration occupancy was omitted
Failure case collapses the service	fallback amplification, replacement lag, or reserved headroom was understated

Do not tune the model until it matches one convenient run. Repeat the same cohort, retain uncertainty ranges, and explain persistent residuals. The output is a decision record: add or reshape supply, reduce safely admitted demand, change routing, protect warm capacity, or improve the bottlenecked dependency. The next test should be able to disprove that decision.