5.5.2 Metrics & Monitoring

A dashboard is useful only when every point can be interpreted as a comparable build. A five-minute CI build, a thirty-second local incremental test, and a cold release build are three different workloads. Averaging them into one line can hide a regression rather than reveal it. Build monitoring therefore starts with a measurement contract, not with Grafana.

For each invocation, retain enough dimensions to identify its workload: command, requested target pattern or workload class, repository revision, CI versus local origin, platform, Bazel version, relevant configuration, success status, and a clean/cold versus incremental/warm classification. Bazel's performance guidance explicitly recommends separating clean and incremental builds because they represent different user experiences and cache states. It also warns that even CumulativeMetrics.num_analyses is only a broad classifier: changed flags or targets can make an apparently incremental server behave like a clean build.¹

Build the Pipeline Around Invocation Records

5.5.1 Build Event Protocol (BEP) supplies the structured invocation stream. A practical pipeline has two paths:

1. Bazel publishes BEP events to a BES backend for near-real-time processing, or writes a binary/JSON BEP file for batch ingestion.
2. An ingest service validates and enriches the invocation, then stores both the raw record and a bounded set of derived metrics.
3. A time-series system such as Prometheus holds low-cardinality aggregates for dashboards and alerts. A warehouse retains invocation-, target-, and test-level history for cohort comparisons and drill-down.
4. A dashboard links an aggregate anomaly back to representative invocation IDs and the richer evidence needed to diagnose it.

BEP is designed for programmatic consumption, and a BES transports its events as opaque bytes. Bazel can also emit length-delimited binary messages or JSON/text files. The event graph is not guaranteed to be complete after a Bazel crash or a failed transport, and summary events may follow BuildFinished. Ingestion must therefore record completeness instead of treating every partial stream as a successful sample.²

Prometheus integration is a transformation step, not a property of BEP itself: the exporter parses events, selects fields, attaches controlled labels, and exposes time-series observations.³ Do not put target labels, commit SHAs, invocation IDs, user names, or arbitrary flags into Prometheus labels. Their unbounded cardinality belongs in the warehouse or raw invocation store. Keep the invocation ID as a drill-down key outside the metric label set.

Choose Metrics That Separate Work From Delay

Start with a small scorecard whose metrics answer different questions:

The BEP build metrics expose PackageMetrics.packages_loaded, TargetMetrics.targets_configured, ActionSummary.actions_created, ActionSummary.actions_executed, and BuildGraphSummary.outputArtifactCount. These counts are useful proxies for work: they can explain a latency movement even when wall time is noisy. Actions created can include actions that were never executed, so it must not be presented as execution volume.¹

Cache data needs the same precision. Distinguish action-cache lookup hits from CAS blob transfers and distinguish local from remote reuse when the source data permits it. A single blended "cache hit rate" loses the boundary needed for diagnosis. Invocation tools consequently present Action Cache and CAS activity separately and drill from misses into individual actions.⁴ This article uses that distinction to make monitoring honest. The miss investigation workflow is covered in 5.5.3 Remote Cache Diagnostics.

Compare Cohorts, Not Isolated Numbers

Use distributions rather than one average. Track a median for the normal case and an upper percentile for the slow tail, together with sample count and failure rate. Compare like with like: the same representative workload, cache state, platform, configuration, and origin. When a repository grows, graph-work counts help distinguish expected added work from infrastructure slowdown.

Shared history also changes prioritization. Count affected users, hosts, projects, and invocations rather than ranking failures only by occurrence. Ten thousand failures generated by one loop and two thousand failures spread across many developers demand different responses. The same history makes flaky tests observable as repeated pass/fail disagreement for a comparable test cohort.⁵

Alert on a Decision

An alert should name the cohort, the baseline, the required persistence, and the owner who can act. Prefer conditions such as "the warm CI p95 for the representative test workload exceeded its trailing baseline for three windows" over a universal duration threshold. Require a minimum sample count, suppress known maintenance windows, and include links to example invocations.

Cache-rate drops, duration regressions, action-count growth, failure clusters, and flaky-test spikes are useful alert candidates only after their denominators and cohorts are stable. Start an alert in dashboard-only mode, inspect its false positives, then page or gate CI only when the signal has a reliable response. The monitoring loop is complete when an operator can move from the trend to a representative profile, action graph, execution log, or test history and verify the proposed repair under matched conditions.