6.6.8 Build Infrastructure Cost and Impact

A remote cache can save millions of action executions and still be a poor investment for one workload. Remote execution can reduce a wide build's critical path and still increase cost for small incremental builds. The useful question is therefore not “what is our hit rate?” or “how many CPU hours did we save?” It is: for a defined workload, which design produces the required correctness and reliability at the best combination of user wait, throughput, and total cost?

6.6.7 Planning Capacity for Build Infrastructure supplies the baseline, burst, scale-lag, and failure headroom that must be paid for. 6.6.3 Measuring Build-Service Latency and Reliability supplies comparable user journeys and service-stage measurements. This article turns those two views into an economic decision without pretending that every saved machine-minute becomes a saved developer-minute.

Count the whole service, not one invoice line

A build-service cost model needs the same boundaries as its lifecycle trace. For each invocation and action cohort, attribute the resources consumed by:

• Action Cache metadata operations and storage;
• CAS blob storage, replication, reads, writes, and retention;
• network transfer, including cross-zone or cross-region traffic and output materialization;
• scheduler, executor, and CI-runner compute;
• repeated work caused by retries, failed attempts, and unusable cached results;
• warm capacity, idle capacity, and failure headroom;
• the control and evidence planes, including BES ingestion and telemetry.

This list prevents two common accounting errors. First, “cache cost” is not just stored bytes. Publishing and retaining a result can incur CAS writes, replication, storage, and Action Cache updates. Consuming a hit instead incurs an Action Cache lookup plus whatever CAS reads, downloads, materialization, and egress that consumer needs. Replication during a read belongs in the reuse cost only when the measured backend actually performs read repair or read-triggered replication; REAPI does not require it. Second, spare capacity is not free merely because no action ran on it. Warm runners and reserved executor headroom buy lower startup latency and resilience to bursts or failures; their idle time is part of the design's cost.

The model should preserve a consistent normalization contract. Useful units include cost per completed invocation, per required presubmit result, per eligible action, or per user journey. The unit, cohort, eligibility definition, and observation window must match. Designs may complete different counts within that shared contract: divide each design's attributed cost by its own completed count, and retain the count or completion rate as a separate throughput outcome. A common population and observation window are necessary, but they do not make unlike units interchangeable. Keep a resource vector, or convert every quantity through an explicit versioned rate: vCPU-s × $/vCPU-s, GiB-month × $/GiB-month, transferred GiB × $/GiB, requests times their request rate, and operator hours times their labor rate. Add fixed, idle, and failure-headroom cost through a declared allocation policy, such as measured usage share or reserved capacity share, then divide the monetary total by the chosen completed-result denominator. Keep latency, throughput, reliability, and correctness outside that sum.

Allocation is not realization. Avoided execution or freed capacity becomes a cash reduction only when marginal usage, purchased capacity, or a commitment can actually be reduced. Otherwise record it as available headroom or displaced work, not money saved.

An evaluation methodology for remote caching and execution illustrates the importance of these boundaries: it uses a production-representative workload, holds the hosting platform constant, runs configurations continuously, and isolates their cloud accounting rather than extrapolating from an unrelated sample repository.¹ A shadow workload can be a useful comparison, but its omissions must remain visible. If it builds but does not test, for example, it does not measure test execution, result delivery, or their costs.

Separate useful reuse from headline hit rate

A hit-rate denominator answers a protocol question, not an investment question. Suppose two action cohorts each report 80% cache hits. In one, hits avoid ten-minute link actions and return small outputs. In the other, hits avoid short actions but download large artifacts over an expensive network path. The ratios match; the compute avoided, bytes transferred, user wait, and cost do not.

Measure reuse per cohort with at least these quantities:

Quantity	What it reveals
Eligible lookups	The population that could have reused a result
Usable hits	Results whose referenced content was available and accepted
Avoided execution	Work not recomputed because of those usable hits
Reuse transfer	Metadata and bytes read to consume the result
Publication cost	Execution, upload, and storage needed to make the result reusable
Repeated work	Retries, failed uploads, misses, and recomputation
Critical-path effect	Change in the journey's actual wait, not the sum of all action durations

Remote caching is most valuable when one execution can be reused across many developers or CI jobs over the result's useful lifetime.² Yet the value of that reuse depends on action cost, stability of the action key, artifact size, network path, and whether the action lies on the critical path. Do not add all avoided action CPU durations and label the total “developer time saved.” Parallel actions overlap, background CI consumes throughput without a developer waiting synchronously, and faster execution outside the critical path may not change the journey at all.

A practical cohort record can therefore look like this:

Cohort	Useful reuse	Recompute avoided	Transfer and service cost	User outcome
Interactive compile, small outputs	usable hits per eligible lookup	executor time and queue demand avoided	AC/CAS reads plus output bytes	median and tail edit-build wait
CI integration test, large logs	reusable test results per eligible test	runner or executor work avoided	result, log, and artifact transfer	presubmit throughput and time to result
Release link, low reuse	successful reuse within its valid cohort	expensive link attempts avoided	publication/retention separately from reads and download	release completion and reliability

The entries are a schema, not invented benchmark data. Populate them from the same correlated action, result, blob, transfer, executor, and journey evidence used by operations. A cache miss is not automatically waste: a changed input may require recomputation. A hit is not automatically value: fetching and materializing it may cost more or take longer than executing locally, a tradeoff that 6.4.1 Choosing Execution Strategies must account for.³

Compare designs as complete systems

Do not evaluate “remote cache,” “warm CI,” “remote execution,” and “faster local machines” as isolated features. Each is a complete path with a different fixed cost, marginal cost, failure mode, and latency shape.

• Cache-only retains local or CI execution while sharing reusable results. It pays for cache service and transfer, but can avoid scheduler and remote executor complexity.
• Warm CI pays for persistent state and idle runner capacity to preserve Bazel's analysis and filesystem state. Its economics depend on assignment, health, revision locality, and demand variation.
• Remote execution adds scheduling, CAS traffic, executor pools, and operational complexity. It is strongest when expensive, wide work can be matched to remote capacity; using it merely to cover poor determinism can increase both work and cost.⁴
• Local investment buys developer-machine capacity or reduces the work Bazel requests through graph, rule, toolchain, and action improvements. It can outperform a remote path for small incremental journeys while doing little for wide CI workloads.

For each candidate, build a decision table with the same cohorts and observation window:

Dimension	Required comparison
Wait	End-to-end journey distribution and critical-path stages
Throughput	Completed eligible journeys or required results per interval
Reliability	Failure, incomplete-result, retry, and degraded-mode behavior
Correctness	Hermeticity and result-integrity constraints that may not be traded away
Cost	Fixed, variable, transfer, storage, retry, idle, headroom, and operating cost

Correctness is a gate, not a discount. A cheaper system that serves results from incompatible action keys or depends on undeclared machine state is not an economic optimization. Hermeticity makes reuse defensible; it does not by itself prove that reuse changes developer wait or pays for the service.

Test uncertainty instead of hiding it

Every decision model contains assumptions: arrival rate, reuse lifetime, artifact size, network price, action duration, burst shape, failure rate, scale-up lag, and the fraction of measured wait that an intervention can change. Record each assumption as a range, not a convenient point estimate.

Then vary one or two uncertain inputs across plausible bounds. If remote execution wins only when every cache miss is expensive, egress is negligible, and the fleet is continuously utilized, the recommendation is fragile. If cache-only remains preferable across low and high demand, different retention windows, and a measured failure interval, it is a stronger decision. The same analysis exposes break-even observations worth measuring next: output bytes by action class, reuse count before eviction, unmatched queue time, or idle runner hours.

Use a controlled, production-shaped experiment rather than comparing a warm incumbent with a cold candidate. Hold revision, target scope, configuration, platform, and workload schedule comparable; include warm-up explicitly; and separate interactive, presubmit, postsubmit, and release cohorts. Continuous shadow pipelines can reveal normal workload variation, while isolated cloud accounts or equivalent cost allocation make configuration costs distinguishable.¹

Treat reliability events as part of the experiment. A design that is cheapest only when every component is healthy omits the cost of retries, cold recovery, fallback load, and reserved failure headroom. Conversely, capacity held for a tested degraded mode has a purpose; label it as resilience cost rather than calling it waste.

The economics decision contract makes this normalization inspectable with deterministic teaching data. Its fixture separates publication and replication from per-hit reads, converts each transfer path with a dated rate-set identity, varies demand, reuse, transfer, failure, and rates across explicit low/base/high scenarios, allocates fixed and headroom cost, and excludes a cheaper but incorrect design. The contract tests also reject a mismatched normalization unit, a missing sensitivity dimension or transfer-path rate, an absent allocation policy, and claimed cash savings without reduced capacity or commitment. Replace the teaching values with your measured quantities and rates; the fixture is a decision schema, not a benchmark or universal price model.

Keep salary arithmetic in its proper place

Multiplying measured wait by an average loaded salary can illustrate why a problem deserves investigation. It is not evidence that every minute becomes productive engineering time, that the organization can realize the amount as cash, or that users would pay that amount for the intervention.

If you include such an illustration, label every assumption: how much wait is synchronous, how many people are affected, whether waits overlap with other work, which percentile and cohort the estimate represents, and what fraction the candidate can actually remove. Present a range and keep it separate from observed infrastructure spend, journey latency, throughput, reliability, and rework. Use this evidence as an input to the decision rights and budget process in H.8.1 Decision Rights. The build-service model makes that decision auditable; it does not approve the investment by itself.