5.6.6 Starlark Memory Profiling

A build can run out of Java heap during loading or analysis even when every action is small. In that case, an action timeline is the wrong starting point: the question is which Starlark call paths retained allocations while Bazel was constructing the configured graph. Bazel's Starlark memory tracker records that attribution and emits a pprof profile. Its CPU profiler answers the related but different question of where Starlark threads spent processor time.¹

Use 5.6.7 bazel dump — Internal State Inspection for the complementary snapshot of state retained by the current Bazel server.

Use this workflow on a representative invocation, not on an artificially tiny target. Inefficient transitive aggregation often becomes visible only when a rule is used across a wide, overlapping dependency graph.¹

Prepare a dedicated profiling server

Memory tracking in Bazel 9.0.0 requires the Java Allocation Instrumenter agent and a system property. Download the java-allocation-instrumenter 3.3.4 JAR from Maven Central, record its checksum according to your normal tool-download policy, and pass both settings as startup options—before build, info, or dump:¹

bazel \
  --host_jvm_args=-javaagent:/absolute/path/java-allocation-instrumenter-3.3.4.jar \
  --host_jvm_args=-DRULE_MEMORY_TRACKER=1 \
  build --nobuild //app:release

The options configure the long-lived Bazel server. Repeat them on every command in the session. Omitting them changes the startup options and restarts the server, losing the state you intended to inspect.¹ Use a separate output base if normal development must continue concurrently, and remove any profiling settings from routine .bazelrc configurations after the investigation.

--nobuild is deliberate when the symptom is loading or analysis memory: it analyzes the requested target without executing its actions. Preserve the real top-level targets, configuration flags, platform, and repository state. Also record whether the baseline was a fresh server or an incremental invocation. those are different experiments.

Allocation instrumentation adds work and memory of its own. Treat the profiled run as attribution evidence, not as an unbiased latency benchmark. Measure the fix later with the agent removed.

Capture heap attribution from the same server

First obtain a coarse total after garbage collection and, when useful, a rule-class breakdown:

bazel <startup-options> info used-heap-size-after-gc
bazel <startup-options> dump --rules

Then dump the Starlark allocation profile:

bazel <startup-options> dump --skylark_memory=/tmp/starlark-heap.pb.gz

The spelling deserves attention. Bazel calls the language Starlark, but the Bazel 9.0.0 dump option still uses its historical name, --skylark_memory. A local Bazel 9.0.0 check accepts that spelling and rejects --starlark_memory. Check the producing binary when using another release. bazel dump is explicitly a developer debugging interface, so pin the Bazel version beside the capture rather than building durable automation around it.

The dump is not a conventional JVM heap dump. It is a pprof-compatible profile whose samples attribute tracked Starlark allocations to call stacks and source locations.¹ It therefore complements, rather than replaces, a total heap measurement: not all Bazel heap belongs to Starlark objects, and a large total alone does not identify a rule implementation.

Read pprof from broad stacks to exact lines

Current pprof provides a local browser UI:

pprof -http=:0 /tmp/starlark-heap.pb.gz

Open the Flame Graph and Top views. Wide frames identify call paths associated with many sampled bytes. The table separates flat allocation at a location from cumulative allocation in that location and its callees. A large cumulative value with a small flat value points down-stack. A large flat value focuses attention on that frame itself.²

For a reviewable text artifact, use:

pprof -text -lines /tmp/starlark-heap.pb.gz

Start with the largest relevant user-defined .bzl locations, then follow their call stacks. Native frames such as a built-in rule or glob can account for large totals without naming the user code that caused them. Cumulative stacks and nearby user frames provide the path back to code you can change.¹ Do not interpret the ranking as proof that the top line is faulty. It reports where tracked allocation was attributed, not what the rule was supposed to compute.

Older Bazel documentation suggests pprof -flame. Current google/pprof documentation does not list -flame among its report formats. It documents the Flame graph as a view in the -http web interface. The separately documented -text and -lines options can be combined for a stable line-oriented report.^1,2

Turn a hot line into a code hypothesis

Inspect the data shape crossing the hot call path before editing it. Three patterns are especially worth testing:

• Copied transitive lists. A provider that publishes local + child_all at every node duplicates information across the dependency graph. Publish rule-local values in a list if convenient, but aggregate transitive values as one depset with direct and transitive members. Whether the resulting cost is quadratic depends on the graph and requested target set. The profile is the evidence for this invocation, not a universal complexity label.¹
• Repeated flattening or deep depset chains. Calls to to_list() at many nodes or across overlapping top-level targets discard structural sharing. Likewise, rebuilding a depset around the previous depset in a loop creates deep nesting. Collect child depsets and construct one depset instead. The depset semantics and ordering constraints remain the design boundary described in 4.1.5 depset vs list.¹
• Eager command-line strings. Joining a transitive file set or concatenating a fresh "--flag=" + file.path string for every configured target retains expanded strings during analysis. Pass files and depsets to ctx.actions.args() and use add_all, format, or format_each so expansion is deferred. Keep the action's real inputs declared as depsets rather than flattening them for ctx.actions.run. 4.2.2 Actions supplies the action correctness model.¹

These are diagnostic categories, not mechanical rewrites. A plain list of values local to one rule can be entirely appropriate. A depset is for shared transitive aggregation, not a blanket replacement for every collection.¹

Use the CPU profile for CPU questions

When the symptom is slow loading or analysis rather than retained allocation, capture Starlark CPU samples on the representative build:

bazel build --nobuild \
  --starlark_cpu_profile=/tmp/starlark-cpu.pb.gz \
  //app:release
pprof -http=:0 /tmp/starlark-cpu.pb.gz

In Bazel 9.0.0, --starlark_cpu_profile writes a pprof profile of CPU usage by all Starlark threads.³ A CPU hotspot may overlap a heap hotspot—for example, repeated flattening can consume both—but the sample units are different. Never describe allocated bytes as CPU time or infer retained memory from CPU samples.

Verify the repair outside the profiler

Close the loop with two comparisons:

1. Repeat the instrumented heap capture with the same Bazel version, targets, flags, repository state, server state, and output-base policy. Confirm that the suspect line or stack loses bytes without merely moving the allocation to another equally expensive path.
2. Restart without the allocation agent and repeat the actual user workflow. Compare end-to-end time, peak or post-GC heap as appropriate, and correctness tests across several comparable runs. Keep the change only if it improves the real symptom and preserves providers, action inputs, command lines, and outputs.

If total JVM heap remains high while tracked Starlark stacks do not explain it, stop forcing a rule-level conclusion. The remaining memory may live in other Bazel state, and a general JVM or Skyframe investigation is a different workflow. Likewise, return to 5.4.2 Critical Path when the expensive work is execution rather than loading or analysis.