6.6.3 Measuring Build-Service Latency and Reliability

A dashboard can report that the remote executor is busy while developers still wait on cache lookup, queueing, output download, or an incomplete result stream. A Bazel-aware service-level indicator (SLI) turns one of those observations into a precisely defined measurement: it names the journey, lifecycle interval, population, denominator, missing-data rule, and service layer that can act on it.

6.6.1 Tracing a Build Across Services supplies the identities and joins needed to follow one invocation across services. An SLI aggregates those correlated observations without erasing where they came from. This is different from the controlled before-and-after experiments in 6.1.2 Measuring Builds Fairly: an experiment tests a change, while an operational SLI continuously describes a bounded user outcome or service transition.

Start With the Outcome, Not the Available Counter

The easiest metrics to collect are often component metrics: executor CPU, scheduler memory, cache storage occupancy, RPC request rate. They are useful for operating those components, but none directly states how long a developer waited or whether CI produced a complete result.

Start with a question that has one observable answer:

• How long did an interactive invocation take from request acceptance to usable top-level outputs?
• How long did an eligible remote action wait between admission and assignment to a compatible worker?
• What fraction of eligible cache lookups returned a usable result whose referenced content was available?
• What fraction of required presubmit results became available to the result consumer before the workflow deadline?

Bazel provides several evidence surfaces rather than one universal metrics stream. BEP, trace profiles, execution logs with spawn metrics, and execution graph logs answer different questions about an invocation and its actions.¹ The JSON trace profile, for example, exposes phase-level context and the action critical path, while execution evidence can separate remote setup or fetching from process time.² A service SLI may join these with scheduler, cache, executor, CI, and result-consumer telemetry, but it must preserve the meaning of each boundary.

Write the Indicator as a Contract

An SLI specification should be reviewable without opening its dashboard. Use a compact contract with these labelled fields:

• Outcome: The user-visible result or service transition being measured.
• Population: Journey and cohort dimensions: revision class, target scope, configuration, platform, cache state, action class, and relevant versions.
• Start and end: Named correlated events, including which clock or duration source defines the interval.
• Numerator: Observations satisfying the stated condition.
• Denominator: All observations eligible for that condition, including the fate of partial observations.
• Exclusions: Explicitly justified cases, recorded separately rather than silently dropped.
• Missing data: How absent starts, ends, joins, or required results change the indicator and its validity.
• Owner: The layer able to investigate or change the measured transition.

Here is a complete queue-latency contract for one deliberately narrow cohort. It is an inspectable design fixture, not captured production telemetry: the full contract and its three evidence cases make the boundary executable.

The filled contract is one vertical record:

• Outcome and population: A CppCompile action in a presubmit reaches the linux-compile pool; the cohort pins Linux x86_64, client 9.1.0, and executor image compile-v4.
• Start and end: scheduler.admission.accepted through scheduler.assignment.recorded, joined by (invocation_id, action_digest).
• Numerator and denominator: Count eligible joined assignments at most 60 seconds in the numerator; retain all eligible admissions in that cohort in the denominator.
• Missing-data rule: A cancelled action is reported separately. An action still unmatched when the window closes is incomplete; a missing trustworthy join is invalid. Neither becomes a short queue sample.
• Validity and owner: Publish incomplete and invalid counts beside the percentile. The scheduler and linux-compile pool own the transition.

Consider a remote-action queue-latency indicator. Its population is not “all remote actions.” It could be compile actions admitted to one compatible Linux pool for interactive invocations under a specified client and executor-image cohort. Its interval begins at accepted admission and ends at worker assignment. Actions rejected before admission, cancelled by the client, unmatched because no compatible pool exists, or missing one boundary need explicit dispositions. If they simply disappear from the dataset, the remaining percentile becomes faster precisely when the system loses difficult work.

The same discipline applies to ratios. A cache reuse indicator needs a denominator of eligible lookups, not total actions. Its numerator must mean a usable result, not merely an Action Cache metadata response when referenced content cannot be materialized. A result-completeness indicator needs the set of required results as its denominator; counting only events that arrived makes delivery loss invisible.

Separate a Journey From Its Stages

Keep at least one end-to-end indicator for each protected journey. An interactive edit-build loop, presubmit, postsubmit, and release build have different start events, required outputs, and tolerable failure modes. A single “build duration” distribution mixes those contracts and can move merely because the workload mixture changed.

Then define stage indicators that can explain the journey:

Invocation tooling demonstrates why this decomposition matters: a successful build can break a long action into input fetch, queue wait, process execution, and output upload rather than presenting one anonymous remote duration.³ Historical build analysis also becomes more useful when operators can select a slow cohort and compare its dimensions with the baseline instead of treating a high percentile as an explanation.⁴

Stage indicators are diagnostic evidence, not competing definitions of the user outcome. Lower execution latency does not establish a faster journey if queueing or materialization grew. Higher executor utilization is neither a latency improvement nor a reliability result. Use utilization to form a capacity hypothesis, then test it against the relevant queue, throughput, and end-to-end indicators.

Keep Cohorts Comparable Without Exploding Cardinality

Percentiles are meaningful only for a named population. Partition by a dimension when it changes the service path or leads to a different operational decision: journey, target or action class, target and execution platform, compatible pool, cache outcome, client/backend version, or result requirement. Long-term build analysis can expose regressions by selecting slow invocations and finding dimensions disproportionately represented in them.⁴

Do not promote every label, target, digest, user, or invocation ID to a metric dimension. Those identities are valuable for drilling from an aggregate into a trace, but unbounded dimensions make metric storage costly and dashboards hard to interpret. Keep bounded cohort dimensions in the metrics system; retain high-cardinality identities in trace or event storage with suitable access and retention controls.

When a cohort changes, report that change rather than presenting the new distribution as continuous with the old one. A shift from warm developer servers to fresh CI clients, a new executor image, or a different target mix can move the indicator without any service regression. 6.1.2 Measuring Builds Fairly shows how to construct matched cohorts when you need to attribute that movement to an intervention.

Make Missing Evidence a First-Class Signal

Telemetry completeness is part of the measurement. For every indicator, count observations with a start but no end, an end but no trustworthy join, required results that never reached the consumer, and records rejected because their cohort identity is unknown. Publish those counts or ratios next to latency and success distributions.

Choose the missing-data treatment before looking at the result:

• Count as an unsuccessful outcome when absence directly violates the contract, such as a required result not reaching its consumer.
• Report as incomplete and invalidate the indicator when the missing boundary prevents a defensible classification.
• Exclude only by a declared rule when the observation was never eligible, while preserving the excluded count and reason.

This prevents a telemetry outage from manufacturing a reliability improvement. It also keeps the next diagnostic honest: 6.6.4 Locating Build-Service Failures can only assign a failing layer as far as the correlated evidence permits.

Use a Small, Connected Indicator Set

A practical specification needs few indicators, each tied to a decision. For a presubmit journey, that might mean end-to-end usable-result latency and required- result completeness, supported by cache-lookup usability, queue latency, execution latency, transfer latency, and telemetry completeness for bounded action cohorts. Keep correctness as a gate rather than celebrating a fast wrong result.

Level 6 defines what these measurements mean and whether their evidence is fit for diagnosis. Organization-wide targets, error budgets, paging thresholds, and escalation rules are governance decisions. The fast feedback assumed by H.7.1 Integration Models depends on trustworthy build-service indicators, but this article does not choose that organization's acceptable latency or reliability target.