6.6.9 Bottleneck-Driven Optimization

A measured bottleneck deserves a matching intervention, not the most impressive optimization name. The useful question is: which stage consumes the protected journey's time, memory, or input-processing work, and which mechanism can change that stage without silently changing correctness, coverage, or reuse? This turns an advanced optimization from a fleet-wide bet into a bounded, reversible experiment.

6.6.1 Tracing a Build Across Services supplies the stage and identity evidence. If that evidence cannot distinguish loading, analysis, execution, transfer, or output materialization, the optimization decision is not ready.

Route the symptom before choosing the mechanism

Start from a trace or profile, not a feature list. Record the affected journey, revision, targets, configuration, platform, Bazel and ruleset versions, cache temperature, and client or worker cohort. Then locate the first stage where the candidate and control diverge.

Measured symptom	Where to investigate next	Plausible intervention class	What would reject it
Retained Bazel heap grows across incremental builds	5.6.5 Memory Diagnostics	retain less evaluator state, or bound the active working set	the pressure is action, worker, or process RSS rather than Bazel's retained graph
Loading or analysis allocates repeatedly in user Starlark	5.6.6 Starlark Memory Profiling	repair copied transitive data, eager expansion, or another attributed rule hotspot	the profile does not attribute material cost to the proposed code path
Many configured targets appear for work that should be equivalent	4.7.4 Transition Boundaries & Gotchas	reduce configuration cardinality or unnecessary transitions	the configurations are required and the bottleneck is elsewhere
Cold analysis dominates after server or configuration changes	5.9.1 Skyframe Data Model	preserve compatible state, or evaluate a versioned analysis-state cache	state lookup, validation, or transfer costs at least as much as recomputation
Client hashing or input scanning dominates before lookup or upload	6.2.5 When a Cache Hit Is Slower	reduce the declared input set, change its granularity, or qualify a digest-preserving input presentation	the delay is transfer or execution, or the alternative cannot prove that each digest identifies the intended bytes
Remote workers wait on input bytes or metadata	6.3.10 Remote Input Presentation	change eager versus lazy input presentation, locality, or materialization policy	scheduler or execution time dominates, or the alternative cannot preserve the same declared input closure
Execution dominates after input setup	6.4.1 Choosing Execution Strategies	change eligible placement, concurrency, worker operation, or remote capacity	the apparent execution interval is actually queueing, input fetch, upload, or client materialization

The row is a routing rule, not a prescription. For example, a slow remote build does not justify a remote filesystem if the trace attributes the delay to scheduler matching. A high Bazel heap does not justify dropping incremental state if the protected journey depends on fast repeated builds. Configuration trimming means removing unused configuration fragments; it can reduce distinct configurations and improve cache reuse, but only configuration evidence can show whether that is the limiting dimension.¹

Rule-level structural defects also belong before fleet machinery. Repeatedly copying a transitive list or flattening a depset at every rule can create quadratic time or memory as a graph grows; constructing transitive depset nodes preserves sharing until a final consumer needs a flat sequence.² 4.1.5 depset vs list shows the implementation pattern. Infrastructure should not compensate for a profiled rule defect by buying more memory indefinitely.

Treat named optimizations as versioned cases

An optimization name is incomplete without its maintained user-facing control, applicability boundary, and fallback. The following cases illustrate how to qualify a mechanism; they are not promises that every current Bazel release supports the same surface.

Skyfocus: retained state for a declared working set

The mapped memory documentation describes experimental Skyfocus as retaining state needed for incremental changes within a declared working set and reclaiming state outside it. Its documented trade-off is equally important: changes outside that set are disallowed until the Bazel server is restarted or cleaned.³ This makes it a candidate for measured retained Skyframe heap during a stable, narrow development journey, not for peak memory in a clean CI build, action memory, or arbitrary repository-wide edits.

Canary it on developers or jobs whose working sets can be named. Compare post-GC Bazel heap, repeated-build latency, restart frequency, rejected out-of-set changes, and correctness against a matched control. Fallback is a server restart without Skyfocus; removal is triggered when working-set violations, forced restarts, or latency outweigh the retained-memory benefit. See 5.6.5 Memory Diagnostics for the diagnostic and command details.

Configuration trimming: remove irrelevant configuration, not evidence

Configuration trimming retains only configuration fragments a target needs. Treat it as a rule, transition, and configuration-design intervention—not a fleet switch that makes every analysis smaller. It is a candidate only when configuration evidence shows that irrelevant fragments or unnecessary transitions multiply configured targets or invalidate work.¹

Canary one rule family or transitioned subtree. Compare configured-target cardinality, analysis time and memory, action identity, outputs, and tests with the existing configuration path. Restore the previous fragment and transition declarations as the fallback. Abort if a required setting disappears, a configured dependency changes meaning, outputs diverge, or the measured bottleneck remains elsewhere. 4.7.4 Transition Boundaries & Gotchas provides the diagnostics for configuration spread and duplicate configurations.

Dormant dependencies: avoid eager analysis of optional edges

The Bazel 9 State of the Union presented dormant dependencies for very large builds whose expensive dependencies are determined on demand.⁴ That is a rule-author mechanism with a narrow semantic contract, not a switch that makes general analysis lazy. See 4.12.4 Dormant Dependency Materialization for its API and provider constraints.

Consider it only when analysis evidence identifies eagerly resolved optional dependency edges and the rules can express the selection correctly. Canary the specific rule family and graph shape. Verify providers, configured targets, actions, outputs, and tests against the eager implementation. The eager rule path is the fallback; any missing edge, changed output, or unsupported Bazel version aborts the rollout.

Remote analysis state: compare lookup with recomputation

A 2024 BazelCon design described serializing configured targets, aspects, and related analysis state into a key-value store so compatible Bazel instances could avoid cold analysis. The same source described invalidation by Bazel, BUILD and transitive Starlark content, and top-level configuration, and called the implementation in progress rather than a general supported service.⁵ Treat those results as design evidence, not a current compatibility guarantee.

This case is plausible only after cold analysis—not action execution or remote input transfer—is measured as the bottleneck. A canary must measure lookup, validation, bytes transferred, analysis wall time, heap, miss and rejection reasons, and output equivalence for explicit producer and consumer versions. The fallback is local analysis from source. Remove or disable the experiment if validation gaps, version skew, unavailable state, or storage latency erase the benefit. Do not call it a remote action cache: it stores a different kind of state and has different invalidators.

Lazy or snapshot input views: change presentation, not identity

For remote execution, an implementation-specific lazy, CAS-backed, FUSE-like, or snapshot view can defer some byte and metadata work until access. It does not change the declared Merkle/CAS input closure, and the Remote Execution API does not guarantee a particular filesystem presentation. The portable fallback is materializing that same closure before execution, as explained in 6.3.10 Remote Input Presentation.

Use this class only when traces attribute material delay to input presentation and access patterns show that eager materialization fetches substantial unused data. Canary by action class and worker image. Observe mount or view creation, first access, bytes and metadata operations, command start, cleanup, reuse, view or CAS loss, and output identity. A fallback that adds undeclared host files is not a fallback; it changes the action's correctness boundary.

External repository digests: identity proof, not a shortcut

“External repository digest” is not one portable Bazel optimization. It may refer to a repository-contents cache, or to implementation-specific filesystem metadata that lets a client obtain a content digest without rereading a large file. AssetFuse, for example, described a FUSE manifest and extended attributes that supplied digests while content remained remote.⁶

Qualify such a proposal only after naming its maintained implementation and client version. The contract must say who establishes the digest, which algorithm and size it covers, how the corresponding bytes become available, what invalidates the metadata, and how the ordinary materialized path resumes after metadata or remote content is unavailable. A timestamp, pathname, or untrusted manifest cannot replace content identity merely because hashing is expensive. No mapped source establishes “external digests” as a general current Bazel control, so this article does not recommend one by that name.

The same rule applies to labels such as “snapshots” and “remote filesystem.” Require the proposal to identify whether it means evaluator state, a revision graph snapshot, precomputed content metadata, or an executor input view. If no maintained surface and independent operational contract can be established, remove it from the decision record.

Write a falsifiable intervention record

Every candidate should fit on one reviewable record:

Protected journey and cohort:
Measured bottleneck and evidence identity:
Mechanism documentation and maintained implementation/version:
Mechanism expected to change that stage:
Correctness and completeness invariants:
Canary boundary and matched control:
Success metric and minimum useful effect:
Abort signal:
Fallback path:
Rollback steps and state compatibility:
Removal trigger and review date:
Rejected alternatives and evidence:

For example, a Skyfocus candidate could produce this inspectable record without inventing a performance result:

Protected journey and cohort:
  Repeated edit-builds whose intended changes stay in a declared working set.
Measured bottleneck and evidence identity:
  Matched profiles show growing post-GC Bazel heap retained by Skyframe;
  action, worker, and process RSS are not the attributed pressure.
Mechanism documentation and maintained implementation/version:
  Skyfocus controls verified for the pinned Bazel release; see 5.4.6.
Mechanism expected to change that stage:
  Retain evaluator state for the declared working set and reclaim state outside it.
Correctness and completeness invariants:
  The control and canary build the same targets and tests with identical required outputs;
  every intended edit remains inside the declared working set.
Canary boundary and matched control:
  One developer or CI cohort, matched by revision, targets, configuration, and workload;
  the control keeps ordinary retained state.
Success metric and minimum useful effect:
  A predeclared reduction in post-GC Bazel heap without a material repeated-build
  latency regression or additional forced restarts.
Abort signal:
  An intended edit is rejected as out of set, outputs differ, restarts increase beyond
  the declared limit, or the heap improvement misses its predeclared threshold.
Fallback path:
  Restart the Bazel server and run the same workload without Skyfocus.
Rollback steps and state compatibility:
  Stop admitting the canary, restart affected servers, and rebuild from source-managed
  inputs; do not treat retained evaluator state as portable across the rollback.
Removal trigger and review date:
  Remove the experiment after repeated working-set violations, missed heap benefit,
  unsupported Bazel upgrade, or the recorded review date.
Rejected alternatives and evidence:
  More remote executors were rejected because action placement cannot reduce the
  client Bazel server's attributed retained Skyframe heap.

The success metric must measure the symptom the mechanism claims to change. Retained heap is not peak heap. Analysis time is not end-to-end wait. Fewer downloaded bytes are not faster completion. A feature can improve its local metric while moving cost to storage, validation, restarts, or operators.

Run the canary through 6.1.5 Safe Infrastructure Rollouts. Keep the baseline available, change one intervention at a time, preserve version and cohort identity, and test the fallback before promotion. Correct outputs and complete required results are gates, not metrics that may regress in exchange for speed. If the mechanism is experimental, its removal condition matters as much as its enablement condition.

Finally, do not infer a threshold from repository size alone. The rough scale indicators in H.10.3 Service Capacity are navigation hints; language mix, graph shape, configurations, workload journeys, and service topology determine the actual bottleneck. Nor should a system “autonomously delegate builds” be taught as a Bazel optimization unless a maintained implementation defines what is delegated, how identity and results remain correct, and how the operator can bound and reverse it.