5.6.5 Memory Diagnostics

recommended

When a large build runs out of memory, first identify which memory is exhausted. The long-lived Bazel server is one JVM, while compilers, linkers, tests, and persistent workers are separate processes. Increasing Bazel's Java heap can help the first case and make the second worse by leaving less RAM for actions. Diagnose the process before changing a limit.

When retained Skyframe state is the verified pressure source, 6.6.9 Bottleneck-Driven Optimization covers the experimental working-set mechanism and its operational constraints.

Establish whether the Bazel JVM is under pressure

Start with Bazel's own measurements:

bazel info server_pid
bazel info used-heap-size-after-gc
bazel info committed-heap-size
bazel info max-heap-size
bazel info gc-count
bazel info gc-time

used-heap-size-after-gc is the most useful single heap reading because it asks the JVM to collect first. gc-count and gc-time are cumulative for the lifetime of the server, so record them before and after the same representative invocation rather than interpreting one absolute value. Process RSS from ps, a container monitor, or the operating system is a different measurement: it includes committed heap pages and non-heap memory, and should not be expected to equal the live Java heap.¹

For a live JVM view, get the PID from Bazel rather than guessing from a process listing, then use jcmd from a JDK compatible with the JVM running Bazel:

pid="$(bazel info server_pid)"
jcmd "$pid" GC.heap_info
jcmd "$pid" GC.class_histogram

Run the attachment as the same operating-system user as the Bazel server. GC.heap_info is a snapshot of heap layout and use. A class histogram is more intrusive and is useful when repeated snapshots show that retained heap keeps growing. If jcmd is unavailable, install an appropriate JDK toolset or use bazel info first. Do not confuse this with --host_jvmopt, which configures Java binaries used as build tools rather than the Bazel server itself.

For a reproducible build-level record, add --memory_profile=memory.json to the build command. Bazel writes measurements at phase boundaries and computes a stable-heap value at the end. It can also publish peak post-GC heap through the Build Event Protocol. That is better evidence for comparing two revisions or target sets than a single observation from a live server.¹

Read GC evidence, not empty space

Capture a JSON trace with --profile=profile.json.gz and inspect its Garbage Collector lane. Minor and major GC pauses are explicit events. They should not be inferred merely from unexplained gaps elsewhere in the timeline. A few short collections are normal. Long or repeated pauses that consume a meaningful share of the invocation, together with high post-GC heap and increasing gc-time, support a heap-pressure diagnosis.²

Even then, a larger heap is only one hypothesis. Compare the same target, flags, server state, and machine after changing one factor. A growth in packages loaded, configured targets, or actions created can explain both analysis time and memory growth. A rule implementation that retains duplicated values can do the same. 5.4 Performance develops the broader measurement workflow.

Change the server heap deliberately

--host_jvm_args is a startup option, so it appears before the command:

bazel --host_jvm_args=-Xmx6g build //app:all

Multiple occurrences accumulate JVM arguments. Changing startup options causes Bazel to start a server with the new startup configuration, so compare measurements only after noting that the first invocation may have cold in-memory state. Choose -Xmx below the machine or container limit with room for Bazel's non-heap memory and for local actions. 6g is an example, not a generally correct default.³

If Bazel throws an OOM, --heap_dump_on_oom is a build option and writes a per-invocation HPROF file under the output base. It covers Bazel's manual OOM detection as well as JVM-thrown OOMs, unlike relying only on -XX:+HeapDumpOnOutOfMemoryError.³

bazel --host_jvm_args=-Xmx6g build \
  --heap_dump_on_oom \
  //app:all

A JVM heap dump can contain strings and objects derived from BUILD files, command lines, paths, and rule data. Handle it as potentially sensitive and capture it only when someone will inspect it.

Do not misdiagnose the analysis cache

The analysis cache is in the Bazel server's memory, but ordinary heap pressure does not imply that Bazel silently evicted the whole cache. Bazel commonly discards analysis state because relevant build options changed, because the user requested it, or because the server stopped. A warning such as “build option ... has changed, discarding analysis cache” points to a configuration transition, not proof of insufficient RAM.⁴

Also separate re-analysis from re-execution. Losing analysis state makes Bazel load or analyze work again. The on-disk action cache can still prevent actions from executing when their inputs and keys remain valid. Profiles, configured-target counts, and the discard warning diagnose analysis. Execution logs and cache evidence diagnose action reuse.

Trade incrementality for a one-shot memory bound

For an ephemeral worker or a genuinely one-off invocation, Bazel documents this combination:

bazel test \
  --discard_analysis_cache \
  --notrack_incremental_state \
  --nokeep_state_after_build \
  //...

The flags solve different parts of the problem:⁵

--discard_analysis_cache releases analysis data after analysis, reducing memory during execution. It does not lower analysis-phase peak memory.
--notrack_incremental_state avoids retaining graph edges needed for later invalidation and reevaluation.
--nokeep_state_after_build drops retained in-memory build state when the command ends. By itself, it does not reduce the current build's high-water mark.

The next invocation must reconstruct in-memory state, although the disk action cache can still avoid much execution. This is therefore a poor default for an interactive developer server or a sequence such as build-then-test. It is a targeted exchange: lower retained memory for deliberately cold later analysis.⁵

Keep a focused incremental graph with Skyfocus

Skyfocus offers a different trade-off: retain incremental state for declared active directories while reclaiming Skyframe state outside that working set. In Bazel 9.0 it remains experimental, and the current command-line syntax is --experimental_active_directories with workspace-root-relative, comma-separated paths:³

bazel test //app/... \
  --experimental_enable_skyfocus \
  --experimental_active_directories=app,shared/protos \
  --experimental_skyfocus_dump_post_gc_stats

The active-directories option is stateful: once set, it persists for subsequent invocations until redefined. Treat that as part of the server's diagnostic state, record it with the reproduction, and explicitly replace the set when the developer changes focus. The post-GC statistics flag forces collections around focusing and reports the measured heap reduction, but adds focusing latency. Use it to evaluate the technique, not automatically on every build.³

Skyfocus is not a transparent global memory cap. Its value depends on how much graph state lies outside the active working set, and changes outside that set are outside the incremental contract. Pin and verify the exact flags against the Bazel version in use: older documentation uses --experimental_working_set, while Bazel 9's command reference names --experimental_active_directories. 6.6.9 Bottleneck-Driven Optimization covers broader ways to reduce the graph and analysis workload rather than pruning retained state after the fact.

key takeaway

Separate Bazel JVM heap from whole-process and action memory, then measure the server with bazel info, GC deltas, a trace's Garbage Collector lane, and—when needed—jcmd or --memory_profile. Put --host_jvm_args=-Xmx... before the command and leave headroom outside the Java heap.

Use the three one-shot flags only when later incrementality is expendable. Use experimental Skyfocus when development is confined to declared active directories and fast incremental rebuilds there matter. Verify its syntax against the pinned Bazel version.

Check your understanding · 4 questions

1.A local build is killed near the machine's RAM limit while several linker actions run. Bazel's post-GC heap remains moderate. What is the best next step?

Select one answer

2.Which observations together provide direct evidence that the Bazel JVM is under heap pressure during a representative invocation?

Select all that apply

3.Match each one-shot memory flag to its distinct effect:

Drag each answer onto the matching prompt, or click an answer and then click a prompt

Answers

--discard_analysis_cache

--notrack_incremental_state

--nokeep_state_after_build

4.A developer repeatedly edits app/ in a large monorepo and wants lower retained memory without making every rebuild cold. When is Skyfocus an appropriate experiment?