5.1.1 Evidence-Led Investigation Contract

When a Bazel build surprises you, the first explanation that comes to mind is usually plausible—and that is exactly why it is dangerous. An evidence-led investigation does not begin by proving that explanation. It defines what was observed, keeps several explanations alive, and chooses the smallest comparison that can make at least one of them less plausible.

This method works before you know which Bazel subsystem is responsible. It is the common shape behind investigations of correctness, performance, caching, configuration, and execution.

Start by drawing an observation boundary

An incident is not “Bazel is slow” or “the cache is broken.” Those are interpretations. Begin with an observation that another person could recognize:

For revision R, one invocation of target T finished in the expected time on environment A, while a later invocation reported by environment B did not.

That sentence deliberately leaves blanks to fill. Record, as far as the available evidence permits:

• the exact command and target pattern;
• the repository revision and whether the working tree had local changes;
• the Bazel version;
• the host and target platform facts relevant to the build;
• the rc files, named configs, and explicit flags that shaped the invocation;
• the output base, server process identity, and whether its in-memory analysis state had prior relevant work;
• the on-disk output-base state, including its local action cache and outputs;
• any separate disk or remote cache endpoint, read/write policy, and relevant history; and
• the observed result, including when and where it was observed.

The list is not an assertion that every field caused the symptom. It is the provenance envelope: the boundary of what you can legitimately compare. Bazel may load options from several rc layers, with later and more specific settings affecting the final invocation, so two command strings alone do not prove equivalent configuration.¹ A long-lived Bazel server preserves in-memory package and dependency-graph metadata between invocations, and its identity follows the output base.² That process state is not the same thing as the action cache and outputs stored under the output base, nor as an explicitly configured disk or remote cache.³

Keep three categories separate in your notes:

Category	Example	What it licenses you to say
Observation	“Invocation B executed work that A reused.”	The two observed outcomes differed.
Provenance fact	“The recorded Bazel versions differ.”	Version is a candidate variable.
Interpretation	“A version change caused the extra work.”	Only a hypothesis until a comparison tests it.

This separation prevents a familiar debugging error: quietly turning a guess into a fact while retelling the incident.

Make hypotheses compete

Write at least two explanations that predict different evidence. For example:

• H1 — different effective configuration: an rc or flag difference changed what Bazel planned;
• H2 — different in-memory evaluation state: the invocations reached different server processes or one server retained relevant analysis state;
• H3 — different output-base state: local action-cache entries or outputs existed for one invocation but not the other;
• H4 — different configured cache: disk/remote endpoint, read/write policy, credentials, or cache history differed;
• H5 — different real inputs: revision, generated files, tools, environment, or platform facts differed even though the target label did not.

These are useful hypotheses because they can lose. “Bazel behaved strangely” cannot lose, so no observation can refine it. Build investigations become tractable when a broad complaint is replaced by predictions: if H1 is true, the effective option record should differ; if H2 is true, the comparison must isolate the server process and its in-memory graph; if H3 is true, the output-base-local records should differ; if H4 is true, configured cache provenance or a cache-policy-controlled rerun should differ; if H5 is true, the input or environment record should expose a difference.

Bazel's build-data sources answer different questions, so selecting evidence by the question is more reliable than collecting every available artifact.⁴ Later Level 5 articles introduce specialized probes. The method here comes first: name the suspected boundary, then choose evidence that crosses it.

Choose a discriminating next probe

A good next probe separates hypotheses. It does not merely produce more data. Ask:

1. Which two explanations are still viable?
2. What result would be expected under one but surprising under the other?
3. What is the smallest safe observation or controlled rerun that can reveal that result?
4. Which variables must remain fixed so the comparison still means what you think it means?

Suppose H1 and H2 are both viable. Comparing active rc contributions while keeping revision, target, platform, output base, server process, and cache policy fixed tests configuration without also changing retained state. To test H2, keep the output base and configured caches fixed and compare only a named server-process boundary whose prior in-memory analysis history is known. That does not test H3: restarting a server leaves on-disk output-base state in place. To test H3, preserve the command and external cache policy while comparing explicitly identified output-base-local state. To test H4, preserve the server and output base while changing only one disk/remote cache policy or endpoint. A generic “warm versus fresh” rerun mixes these surfaces and cannot tell you which one mattered.

Prefer a read-only probe when it can discriminate. Destructive “fixes” such as clearing caches or stopping a server erase the very state you may need to explain. Preserve the original evidence first; then make any state-changing experiment explicit in the record.

Falsification is progress

An investigation is not a sequence of attempts until the symptom disappears. After each probe, update the hypothesis table:

Hypothesis	Prediction	Evidence	Status
Different effective configuration	The invocation records differ in an output-affecting option.	Record the comparison or its absence.	supported / weakened / open
Different in-memory evaluation state	A server-process-controlled comparison changes the symptom.	Record the server identity and analysis-state boundary.	supported / weakened / open
Different output-base state	An output-base-local comparison changes the symptom.	Name the output base and on-disk surface.	supported / weakened / open
Different configured cache	A disk/remote-cache-controlled comparison changes the symptom.	Record endpoint, policy, and relevant history.	supported / weakened / open
Different real inputs	Source, tool, environment, or platform identity differs.	Name the observed difference.	supported / weakened / open

“We did not find a difference” is not automatically proof of equality. State which surface was compared and what it could not see. Bazel can only reason about inputs it records; host tools or environment facts that escape the model can make apparently identical builds different.⁵

A hypothesis that survives one probe is not yet a diagnosis. It has merely survived that probe. Confidence grows when a prediction succeeds and plausible alternatives fail under controlled comparisons.

End with a bounded diagnosis memo

The useful output is not a triumphant root-cause sentence. It is a small memo that another person can inspect and continue:

Observed:
  What differed, for which invocations, and under what boundary?

Provenance:
  Command/targets, revision, Bazel version, rc/config, platform,
  server process/in-memory history, output-base-local state,
  disk/remote cache policy and history, time and environment.

Competing hypotheses:
  H1 ... predicts ...
  H2 ... predicts ...

Probe and controlled variables:
  What was compared? What was intentionally held constant?

Evidence:
  Which observation supports or weakens each hypothesis?

Bounded diagnosis:
  What can the evidence justify? What remains unknown?

Next action:
  Repair, verification, or the next discriminating probe.

The word bounded matters. “The compared rc records differ in option X, and a matched rerun reproduces the outcome when only X changes” is stronger than “configuration caused it,” because it names both the evidence and its limits. If the evidence does not separate the remaining hypotheses, the honest diagnosis is “unknown,” followed by the next probe.