5.6 Debugging

bazel test --config=ci //payments:contract_test fails on an upgrade, but the first facts you receive may point in different directions: exit code 3, a long profile lane, an out-of-memory message, or “no matching toolchains found.” Each fact is useful. None is yet the cause. The friction in difficult Bazel debugging is deciding which observation deserves the next experiment without destroying the state that made the problem reproducible.

The connecting model for this section is an evidence ladder. Classify the outcome, observe the failing or expensive layer, isolate the variable that changes the result, and inspect internal state only when the ordinary evidence leaves a specific question unanswered. Each rung should make the next question smaller. A more detailed artifact is not automatically stronger evidence if it describes the wrong command, configuration, server, or process.

Route The Symptom Before Reaching For Depth

Begin with the outcome and the invocation that produced it. 5.6.1 Exit Code Taxonomy explains how to read an exit code with its command rather than treating the number as a root cause. This is the shortest entry for CI triage: preserve the raw status and diagnostics, distinguish a build failure from a test result or launched program, and nominate retries only under an explicit policy. It tells you where to route the investigation, not what repair to make.

When the symptom is elapsed time rather than a failure, 5.4.1 Timing Profile Analysis moves from the invocation into a trace. Its role is to separate observed loading, analysis, execution, remote-transfer, and garbage-collection work without turning lane names into causal claims. The important comparison is between matched workloads—cold, warm no-op, or warm incremental—not between two conveniently available profiles that exercise different states.

One failure class has an unusually precise local probe. 5.6.4 Toolchain Resolution Debugging is the jump path for a missing or surprising toolchain. The resolution trace lets you test registration, target-platform, execution-platform, constraint, and ordering hypotheses for one configured consumer. Read it after the platform model in 4.6.3 Toolchain Resolution. use it instead of widening a general trace or guessing at constraint changes.

These articles form the first diagnostic cluster:

outcome and command -> observed phase or failure class -> narrow next probe

The discipline is classification without overclaiming. An exit code can route CI handling. A profile can locate where time went. A toolchain trace can explain candidate selection. None of them should be stretched into a claim about a different layer.

Turn “It Changed” Into A Boundary

Some investigations begin with stronger evidence: the same focused command has one result on one side of an upgrade and another result on the other side. At that point, the best next move is often not a larger log but a smaller change boundary.

Use 5.6.2 Bisecting Bazel Version Issues when the variable is the Bazel binary. It develops the endpoint checks, Bazelisk's automatic range search, reverse bisection for a fix, and the fallback to git bisect when candidates cannot be tested. The result is the first tested Bazel change where the predicate flips—not yet proof of why that change causes the symptom.

Use 5.6.3 Bisecting Dependency Issues when the Bazel version stays fixed and a ruleset or external dependency changes. A local override keeps the consumer pointed at one clone while git bisect moves that clone through its history. This sibling path also makes an important operational distinction: preserving a warm server can reuse unaffected work, but changing dependency definitions may still trigger broad reevaluation.

Both bisection workflows depend on the same contract: one small command must reliably distinguish known-good from known-bad endpoints under controlled conditions. Infrastructure failures and unavailable candidates need an “untestable” path rather than being silently classified as the regression. Do not add cleanup merely to make every trial look fresh. Reset state only when the proven predicate requires that state.

Inspect Retained State Only For A Named Question

Memory and graph-state problems invite the deepest tools in this section, but the useful order still begins with observable evidence.

5.6.5 Memory Diagnostics first separates Bazel's JVM heap from process RSS and from the memory used by actions and workers. Read this branch when the symptom is an OOM, sustained garbage-collection cost, or excessive retained memory. It establishes the measurements needed before changing a heap limit or trading away incrementality.

When those measurements isolate allocations in user Starlark code, the extra 5.6.6 Starlark Memory Profiling branch captures allocation stacks for pprof attribution and verifies a proposed repair with a matched capture. It is not the starting point for every Bazel memory problem: action, worker, native, and JVM heap symptoms must first be separated by the general diagnostic workflow.

Once the question is specifically about state retained by the Bazel server, 5.6.7 bazel dump — Internal State Inspection provides focused snapshots of caches, packages, rules, Skyframe nodes, and tracked Starlark allocations. Start with summaries and counts. A dump is internal, version-sensitive evidence from one populated server. It is not a stable query interface and cannot reconstruct tracking that was never enabled.

5.6.8 Skyscope — Skyframe Visualizer is the visual escalation when raw Skyframe topology is too large or opaque to explore as text. It helps reveal internal neighborhoods and dependency paths progressively. Its edges describe retained evaluation dependencies, not execution order or invalidation history, so a useful visual clue should lead back to a focused supported probe such as cquery, aquery, a profile, or BEP evidence.

This cluster moves from measurement to internals:

process and workload measurements -> bounded server snapshot -> visual topology

Skipping directly to the last rung usually produces an impressive picture with weak provenance. Keep the Bazel version, output base, startup options, workload, configuration, and cold-or-warm server state attached to every internal capture.

Choose The Shortest Useful Reading Path

For general incident triage, start with Exit Code Taxonomy and Timing Profile Analysis. They establish how to preserve command context and how to turn one invocation into a narrower question. Then jump by symptom:

• For “no matching toolchains found” or a surprising selected toolchain, go directly to Toolchain Resolution Debugging.
• For a regression bounded by two Bazel versions, read Bisecting Bazel. For a ruleset or dependency boundary, read Bisecting Dependencies instead.
• For an OOM or unexplained process growth, begin with Memory Diagnostics. Add the extra Starlark profiler only when user-code allocation is the bounded domain, Bazel Dump after you can name a retained-state question, and the extra Skyscope view only when internal graph topology will answer it.

Read all eight in order only if you are building a broad debugging practice. During a live investigation, depth should be earned: collect the least invasive evidence that can distinguish the current hypotheses, preserve the reproducing state, and escalate only when you can say what the next artifact must resolve.