5.5 Observability

bazel test //... --build_event_json_file=/tmp/bep.json can turn one build into structured data. A dashboard can turn thousands of such builds into a trend. Neither transformation automatically explains why a build slowed down, lost cache reuse, or produced the wrong result.

The useful mental model is controlled zoom. Begin with a trustworthy record of one invocation, group only comparable invocations, then zoom back into the smallest evidence that can distinguish the remaining explanations. Each step preserves context for the next. Without that chain, a precise event becomes an ambiguous metric, and an alarming metric becomes a guess about cause.

From Record, To Signal, To Evidence

5.5.1 Build Event Protocol (BEP) establishes the observation boundary. The Build Event Protocol is the machine-readable account of an invocation: its command, configurations, results, artifacts, metrics, and completion state. Read this first because the rest of the section depends on knowing what the source stream does—and does not—claim. It also explains why a BEP consumer must follow event identities and handle incomplete streams instead of treating the data as a flat, guaranteed log.

5.5.2 Metrics & Monitoring moves from individual records to shared history. Its role is not merely to list dashboard counters. It supplies the measurement contract that makes comparisons meaningful: classify builds by workload, platform, configuration, origin, and warm or cold state. Retain detailed invocation evidence, and export bounded aggregates for trends and alerts. This is where isolated observations become operational signals without erasing the dimensions needed to interpret them.

5.5.3 Remote Cache Diagnostics completes the zoom. Cache behavior provides a concrete case where an aggregate can reveal a problem but cannot name its cause. The article moves from a cohort-level change to paired execution logs and, for suspected wrong reuse, to matched independent builds. Read it as the section's worked diagnostic method: preserve the observation, narrow the hypotheses, and demand stronger evidence as the claim becomes stronger.

A Concrete Product View

BuildBuddy illustrates controlled zoom without changing the evidence contract: a BES-fed invocation page preserves one build, comparable invocations expose a duration or cache trend, and the trend links back to a representative timing profile or action detail.¹ Other BEP consumers can preserve the same chain with different interfaces and storage choices; the product is an example, not a Bazel requirement.

At this level, use that view to investigate one bounded cohort and retain the route back to invocation evidence. Designing authentication, retention, storage, shared telemetry pipelines, and cross-service correlation is a Level 6 responsibility; 6.6.1 Tracing a Build Across Services develops the identity and governance model for that shared system.

Observability Is A Chain Of Claims

The surprising difficulty is not collecting more build data. It is keeping the claim at each layer no stronger than the evidence.

A BEP record describes what one invocation reported. A monitoring series says how a defined population changed. A profile or execution-log comparison can help explain a representative invocation. These artifacts complement one another because they answer different questions. None is a universal build record. The operator's job is to maintain the links between them: an alert names its cohort, the cohort leads to representative invocation IDs, and those invocations lead to the detailed artifact that can test a causal hypothesis.

This also explains two recurring mistakes. Mixing unlike builds can manufacture or hide a regression, while jumping directly from a changed metric to a root cause confuses detection with diagnosis. A cache-hit-rate drop, for example, can justify investigation. It cannot by itself distinguish changed action identity, disabled reuse, or an incorrect cached result.

Take The Shortest Useful Path

For a bounded observability investigation, read all three articles in order. Start by writing BEP to a file, define the cohort and drill-down identity needed for the question, then use the cache case to test whether the evidence chain can distinguish the live explanations.

If an existing dashboard is producing noisy alerts, begin with Metrics & Monitoring and return to BEP when you need to verify the provenance, completeness, or semantics of a field. If the immediate symptom is unexpected rebuilding or a suspected bad cache result, jump to Remote Cache Diagnostics, but use its comparison workflow rather than treating the current dashboard percentage as proof.

The temptation to avoid is instrumenting everything before deciding what question the data must answer. Start with an operational decision—detect a latency regression, compare graph work, investigate lost reuse—and preserve a route from the aggregate back to the invocation-level evidence.

Footnotes

1. Finding and Fixing Strange Builds with BuildBuddy — BES-fed trends, dimensional filtering, invocation drill-down, and timing-profile diagnosis ↩