6.1.2 Measuring Builds Fairly

A build that finishes faster after an infrastructure change has not yet proved that the change helped. The new run may have used a warmer cache, a different revision, an easier action mix, or another platform. A fair Bazel comparison holds those workload dimensions steady, changes one intended condition, and keeps enough identity to reproduce both cohorts.

6.1.1 Bazel as a Shared Service introduced journeys and action classes because a shared service serves several kinds of work at once. Measurement turns those names into comparison boundaries. Instead of asking whether “builds got faster,” ask whether one named journey, under one reproducible workload definition, improved without crossing its correctness or completeness gates.

Define the Observation Before Reading the Number

Start with the outcome you intend to compare. Interactive edit-build latency, presubmit completion time, action execution time, cache lookup time, and queue time are different observations. A component can improve while the user-facing journey does not.

Bazel exposes different evidence for these observations: BEP, trace profiles, execution logs with local and remote spawn metrics, execution graph logs, and controlled bazel-bench comparisons.¹ Do not collapse those signals into one explanation merely because they all appear in a slow invocation.

Write the observation as an interval or ratio with an explicit denominator:

• End-to-end latency: from the named journey's start event to its complete result, for invocations in the stated cohort.
• Action latency: from the same action lifecycle boundary to the same later boundary, for one action class.
• Reuse: eligible lookups that returned a usable result, not all actions or all invocations unless that is genuinely the intended population.
• Throughput: completed, trustworthy units of the named workload per stated interval, not requests merely accepted by a service.

This is not a lesson in general statistics. The important Bazel-specific move is to keep unlike journeys, action classes, and lifecycle intervals out of the same denominator. 6.1.4 Action and Result Lifecycle provides the shallow map of cache, execution, materialization, and reporting transitions used by those intervals.

Build Cohorts That Differ Only Where You Intend

Treat a cohort as a reproducible workload specification, not as a dashboard filter chosen after the result looks surprising. A production-representative remote-services study should choose its workload, platform, and accounting boundaries before comparing configurations.² Record at least these dimensions:

Dimension	What must be named	Why it can change the result
Journey	interactive, presubmit, postsubmit, release, or maintenance	Each starts and finishes at different boundaries and values different results.
Revision and requested scope	source revision plus target or test universe	A different graph slice can change both the amount and shape of work.
Configuration	Bazel client version, effective configuration, relevant flags, and rules/config revision	These can change configured targets, actions, and execution behavior.
Platform	target and execution platform, toolchain, and relevant executor image or pool identity	The same label can produce materially different action populations.
Cache temperature	which local and shared states are absent, retained, seeded, or reused	A first producer, a clean consumer, and a repeatedly warm client answer different questions.
Action mix	counts or proportions by meaningful class, with critical work identifiable	A fast class replacing a slow class can move an aggregate without improving either class.

REAPI supplies portable action-digest, CAS, result, capability, and platform- requirement records, but it does not prescribe one backend topology or a universal operational identity scheme.³ Retain the action digest and relevant protocol records, then add the implementation identities needed to explain the cohort: invocation and producer, client and backend versions, worker pool or image, cache instance, and CI context.

Cache temperature deserves an operational definition. “Warm” is not a feeling based on the second run being quicker. State which Bazel server or output-base state, local caches, and shared cache entries may survive, and which are fresh. Cache reuse and changed execution placement remain two separate interventions; use the same separation in an experiment.

If a dimension is intentionally changed, put it in the hypothesis. If it changes accidentally, the cohorts are no longer comparable. Do not repair that problem by averaging the incompatible runs together.

Preserve the Action Mix

An invocation average can hide a changed workload. Suppose one period contains mostly cacheable compile actions and another contains more links, tests, or large-output actions. Their aggregate latency can move even if no action class became faster.

Partition only as finely as the decision requires. Useful boundaries are the ones that can explain a different service path or cost: action mnemonic or class, platform or compatible worker pool, cache outcome, output shape, and whether the action is on the journey's critical path. NativeLink's repository keeps CAS, scheduler, executor, and metrics surfaces separate, with independent cache and remote-execution integration tests.⁴ That separation is a useful measurement discipline: a cache lookup, scheduler wait, worker execution, and output transfer should not become one anonymous “remote” duration.

Do not invent a universal utilization threshold. Healthy average CPU does not show that the action class on the critical path had a compatible executor, nor does high utilization prove useful work completed. Use the cohort's queue, placement, action, and completion evidence to decide whether capacity is even the constrained stage.

Write a Reproducible Experiment Contract

A useful before/after report lets another operator reconstruct why each invocation belongs in either cohort. Continuous shadow pipelines and isolated configuration accounting make those cohort choices auditable over time.² Keep this compact contract with the results:

1. Question and hypothesis: name the journey, action class or invocation population, intended change, and the lifecycle stage expected to move.
2. Cohort identity: record revision, target scope, configuration, platform, cache state, action mix, client/backend versions, and time window or run set.
3. Intervention: state the one intended difference. List any concurrent change that could not be controlled.
4. Observation: define start, end, numerator, denominator, exclusions, and how missing evidence is represented.
5. Gates: require trustworthy outputs and complete required results before interpreting speed, throughput, reuse, or cost.
6. Abort criteria: stop when correctness or completeness fails, telemetry cannot place observations into cohorts, workload identity drifts, or the protected journey exceeds its stated bound.
7. Evidence: retain invocation IDs and the action/result/version identities needed to drill from the aggregate into representative and anomalous runs.

A fair experiment should attribute a change to an explicit service boundary instead of stopping at “remote was slower.”

Compare in Layers, Decide Once

Read the result in this order:

1. Confirm that both cohorts satisfy their correctness and completeness gates.
2. Confirm that every observation belongs to the declared cohort and uses the declared denominator.
3. Compare the end-to-end journey outcome.
4. Use action-class and service-stage evidence to explain the movement or lack of movement.
5. Record where the evidence cannot distinguish two explanations.

This order prevents a component win from becoming an unsupported service claim. A higher hit rate with unchanged presubmit latency may still be valid reuse evidence, but it is not evidence that presubmits became faster. Lower executor time with higher queue or materialization time may identify a displaced cost, not a net improvement.

The output of this article is a trustworthy comparison, not an infrastructure choice. Take matched workload evidence to 6.1.3 Choosing Remote Cache or Execution when deciding whether remote caching, remote execution, both, or neither addresses the measured bottleneck.