5.10.1 Unexpected Rebuild Walkthrough

An unexpected rebuild is not yet a diagnosis. In this worked case, two matched invocations rebuild a large part of the same target after a new schema file that should be private to one consumer is added. We will use the smallest evidence that separates the live explanations, stop when the evidence supports one bounded claim, and record what remains unknown.

The dossier below is synthetic. Its labels, counts, and observations form a teaching case assembled from documented query and execution-log behavior; they are not copied from a real invocation and should not be read as benchmark data.

Establish the comparison before explaining it

The team reports that adding pricing/schema/private.tbl for one local consumer causes “most of checkout” to rebuild in CI. The affected workload is bazel build //checkout:release.

The first pair of CI invocations differs in several ways:

Field	Earlier invocation	Later invocation
revision	before the file addition	after the file addition
Bazel version	the same pinned version	the same pinned version
target and flags	recorded	recorded, apparently equal
worker image	image A	image B
remote cache	warm shared cache	warm shared cache

That pair cannot isolate the file addition. Image drift is a competing explanation, and “apparently equal” flags do not establish equal effective configuration. Apply the provenance checklist from 5.1.2 First Investigation: Two Builds That Look Identical before choosing a specialized tool: record command, rc sources, revision, Bazel version, platform, server state, and cache context.

The team therefore creates a controlled pair on one CI worker image. Both runs use a newly started Bazel server, the same pinned Bazel version, target, rc files, platform, and cache policy. The only deliberate difference is the source revision. This does not make every hidden influence disappear. It makes the file addition the useful variable for the next comparison.

Keep more than one explanation alive

Before collecting another artifact, write hypotheses that predict different observations:

1. Broad declared dependency reach. The schema directory legitimately feeds a large part of //checkout:release.
2. Configuration drift. The two builds analyze different configured graphs despite the recorded command looking equal.
3. Changed action input or command. One early action changes identity and invalidates downstream work.
4. Unrecorded influence. The same recorded action state produces different bytes.

A timing profile would show where time was spent, but all four hypotheses can produce a long execution phase. BEP would provide structured invocation outcomes, but it would not by itself identify the first changed spawn. Sandbox tracing would be valuable if undeclared runtime access became the leading branch. None is the first discriminating probe here.

Bound the possible blast radius

Start with the declared graph because it is cheap and answers whether broad reach is even plausible:

bazel query \
  'allpaths(//checkout:release, //pricing/schema:public_headers)' \
  --output=label

5.2.1 bazel query — Static Graph Analysis explains why this is unconfigured target-graph evidence: it can establish declared paths, but it cannot show the active select() branch or the actions that actually ran. In this dossier, the result contains two paths. One goes through //checkout:pricing_adapter; the other through a packaging target that gathers release inputs.

This observation rejects “the header generator is structurally unrelated to the release target.” It does not prove that the breadth of execution was necessary, nor does it choose among the other hypotheses. A cquery branch would become useful if configuration remained in dispute. Here the matched provenance records and the next artifact give a more direct discriminator, so we do not collect it merely because it is available.

Compare execution evidence at the first divergence

The controlled runs record compact execution logs. See 5.7.2 Execution Log Analysis for how to collect and compare them; here we use the result as one step in the case. A structural comparison reports this synthetic sequence:

Order	Spawn observation	Interpretation
first divergence	header-generation spawn gains pricing/schema/private.tbl as an input	recorded action state changed
next	generated public schema artifact changes	downstream consumers now see different bytes
cascade	compile and packaging spawns rerun	effects, not independent causes

The first row falsifies the strongest version of the unrecorded-influence hypothesis: the logs already contain a changed declared input before any changed output appears. It also moves the investigation away from generic cache failure. The relevant producer had different recorded state, so a cache miss is expected for that action identity.¹

Now inspect the rule and macro that construct the header-generation target. The new file was intended for one local consumer, but a broad file collection added the entire pricing/schema directory to the header generator's declared inputs. The generator therefore gained a real input and produced a changed public schema artifact. That is a target-boundary problem: the declared collection does not express the file's intended consumer boundary.

Notice what the evidence does not justify. It does not prove that every rerun in the invocation descends from this producer. It does not measure the best possible build time. It does not establish a fleet-wide cache problem. The case supports a narrower claim about this target, these revisions, and this controlled pair.

Write a diagnosis that fits the evidence

A useful consultant memo separates observation, inference, limit, and next test:

Workload: //checkout:release, two controlled CI invocations across the private-schema-file addition.
Observation: the declared graph contains paths from the release target to //pricing/schema:public_headers. The first execution-log divergence is a header-generation spawn whose declared inputs gain pricing/schema/private.tbl; changed outputs and downstream reruns follow.
Bounded diagnosis: for this pair, the unexpected rebuild begins at an over-broad declared input boundary, not at an unexplained output difference.

Evidence limit: the comparison does not quantify all invocation cost or prove behavior on other worker images and configurations.
Next action: split or narrow the generator input set, repeat the matched pair, and expect the first divergence and its dependent cascade to disappear.

The repair is not accepted because the next build is green. It is accepted if the repeated experiment changes the predicted evidence. If the generator input no longer changes but the cascade remains, reopen the hypotheses at the new first divergence.