5.1.2 First Investigation: Two Builds That Look Identical

Two developers report the same command, bazel build //app:bundle, but only one invocation reuses the expected work. We will investigate this as a staged case. Each stage reveals a small piece of evidence; before continuing, we will decide which hypotheses it weakens and which probe is justified next.

The cards below are a teaching case, not captured terminal output. Values such as R17, OB-A, and RC-A are case identifiers, not strings Bazel prints.

Stage 0: the report

Invocation A

reported command: bazel build //app:bundle
reported result: expected work was reused

Invocation B

reported command: bazel build //app:bundle
reported result: app actions executed again

Apply 5.1.1 Evidence-Led Investigation Contract before naming a cause:

For //app:bundle, two reported invocations had different reuse outcomes. Their equivalence has not yet been established.

The visible command and outcome are observations. “Remote cache failure,” “Bazel regression,” and “non-hermetic rule” are interpretations. We keep four competing hypotheses:

• H-source: the revision or dirty working tree differs;
• H-config: rc files or explicit options produce different effective configurations;
• H-local-state: the server process or the on-disk output-base state differs;
• H-shared-cache: disk/remote cache endpoint, policy, or history differs.

Stage 1: establish source and tool identity

The developers attach records made with these ordinary commands:

git rev-parse HEAD
git status --short
bazel --version

We store the returned values in the incident system rather than inventing terminal output here:

Field	Invocation A	Invocation B
revision	R17	R17
git status --short	empty	empty
Bazel version	V8	V8
host / target platform record	P-linux / P-linux	P-linux / P-linux

This evidence weakens H-source for the surfaces inspected. It does not prove that every real input is equal: an undeclared host tool or environment value could still differ.¹ But changing the revision is no longer the most discriminating next probe.

The best next comparison is effective configuration. It is read-only with respect to the retained-state hypotheses and tests a common reason that matching command tails are misleading.

Stage 2: reveal the invocation Bazel received

Each developer repeats the original request with rc announcements enabled:

bazel build --announce_rc //app:bundle

We preserve the announcements and reduce them to this case record:

Field	Invocation A	Invocation B
workspace rc	RC-workspace	RC-workspace
additional rc	none	RC-user
named config	none	none
compilation mode contributed by rc	fastbuild	opt
other recorded build options	matched	matched

Bazel reads defaults from layered rc files, and later contributions can change the effective invocation even when the visible command matches.² We have now found a concrete prediction for H-config: if compilation mode is the relevant difference, the reuse outcome should follow that option under an otherwise matched invocation.

We have not proved H-config yet. The original invocations may also have different server, output-base, or shared-cache histories. Changing the option on those two original machines would leave those alternatives entangled.

Stage 3: separate the three state surfaces

Before the controlled rerun, the developers record local and configured cache provenance:

bazel info output_base
bazel info server_pid

The incident cards use aliases for the actual returned paths and process IDs:

Surface	Invocation A	Invocation B	Meaning
server process / in-memory analysis state	SERVER-A, prior relevant invocation	SERVER-B, no recorded prior invocation	Different process histories
output base / on-disk action cache and outputs	OB-A, populated	OB-B, sparse	Different workspace-local histories
configured disk cache	disabled	disabled	Matched policy
configured remote cache	CACHE-C, read-only	CACHE-C, read-only	Matched endpoint and policy

These are three different surfaces. The server holds in-memory package, dependency-graph, and analysis state across commands; its identity is selected through the output base.³ The output base also contains on-disk local state, including the action cache and outputs, which does not disappear merely because the server process stops. An explicitly configured disk or remote cache is a third store with its own endpoint and read/write policy.⁴

Therefore the original A/B pair cannot test H-config cleanly. It differs in the server process and output-base-local history, even though its remote-cache policy matches. A generic “make B fresh” experiment would be worse: replacing the server and output base together would change two evidence surfaces.

Stage 4: choose one controlled probe

We use one controlled environment and keep these fields fixed:

revision=R17; dirty=false; Bazel=V8; platform=P-linux
server=SERVER-C; output_base=OB-C
disk_cache=disabled; remote_cache=CACHE-C/read-only
target=//app:bundle

First we establish the fastbuild baseline in that environment. Then, without changing the server process, output base, cache endpoint, cache policy, source, platform, or target, we vary only the compilation mode contributed by the rc layer. The completed probe card is:

Controlled run	Only varied field	Observed reuse outcome
C1	compilation_mode=fastbuild	expected reuse
C2	compilation_mode=opt	affected app actions execute
C3	compilation_mode=fastbuild	expected reuse returns

This is an A/B/A probe. C3 matters: it checks that the outcome follows the selected mode rather than a one-way change in server, output-base, or external cache history. The contents of OB-C naturally evolve as each run completes; the reversal does not pretend to freeze them. It tests whether the baseline result remains recoverable after the opt run instead of confusing monotonic cache population with the option change. Compilation-affecting options are part of the action identity; changing such an option can correctly require different work rather than indicating a broken cache.⁵

H-config is supported for this case. The probe weakens a server-replacement or cache-policy change as the explanation of the controlled difference because those named surfaces remained fixed across C1–C3. It does not prove that local cache contents had no interaction with the sequence, nor that H-local-state was impossible in the original A/B pair; its state histories were not matched.

Stage 5: write the bounded conclusion

Observed:
  The original A and B invocations reported different reuse for //app:bundle.

Provenance limit:
  Their source, Bazel version, and platform matched, but their server processes
  and output-base-local histories did not.

Discriminating evidence:
  Rc announcements exposed fastbuild versus opt. In a controlled A/B/A probe,
  only compilation_mode changed; server process, output base, disk/remote cache
  policy, source, platform, and target remained fixed. The reuse outcome followed
  the mode and returned with the baseline.

Bounded diagnosis:
  For R17, V8, P-linux, //app:bundle, and the C1–C3 environment, the unexpected
  comparison was explained by different effective compilation modes, not by a
  failure of the fixed remote-cache policy. The original pair alone cannot
  establish whether its differing local state also affected either result.

Next action:
  Decide whether RC-user should override the workspace mode. After changing the
  policy, repeat the matched comparison rather than clearing retained state.

We reached a useful answer without query, profiles, or execution logs. Those tools become appropriate when the provenance matches and the remaining question is about graph shape, time distribution, or individual action reuse. The first investigation skill is deciding whether two invocations deserve that deeper comparison.