6.6.2 Action-Internal Telemetry

A remote action can dominate a build profile while remaining opaque where the most useful diagnosis begins: inside the compiler, test runner, or other tool. Action-internal telemetry fills that gap by collecting events and system metrics from within the remote sandbox. It is an execution-service integration, not a standard Bazel tracing mode or a standard Remote Execution API facility.

Escalate only when the action is the unknown

Start with the portable lifecycle and least invasive evidence that can answer the question. 6.3.4 Remote Action Tracing follows the standard Execute and Operation stages; 6.6.1 Tracing a Build Across Services establishes the explicit join keys and governance needed when evidence crosses services. A Bazel JSON trace shows when an action ran and how its span relates to the invocation. 5.2.3 bazel aquery — Action Graph shows the registered command, inputs, and outputs, while 5.7.2 Execution Log Analysis helps exhaust cache and reproducibility questions. Instrument inside an action only when those views isolate an expensive action but cannot distinguish causes such as:

• time in the tool's own phases.
• CPU stalls or hardware-counter signals.
• process-level CPU, memory, or I/O pressure.
• kernel scheduling, filesystem, or syscall behavior.

This is especially useful for a long-running action whose duration varies on otherwise comparable remote workers. It is usually disproportionate for short actions: collector startup, trace traffic, storage, and analysis can cost more than the span being investigated.

The boundary also avoids confusing two different traces. Bazel's --profile records the Bazel invocation. An application trace produced inside a remote action describes the tool's work. A system trace describes the remote worker and sandbox during that action. Perfetto can visualize several trace formats, but the viewer does not make them one automatically correlated data source.¹

A concrete private-executor design

A 2023 BazelCon case study described Stripe's private remote execution service. In that environment, a custom stripe/tracing execution property could request HTTP or gRPC addresses. The executor interpreted the property, created servers at those addresses, accepted Chrome Trace Event JSON emitted by an instrumented action, and exported the trace separately from the action result.²

That description is deliberately environment-specific:

Bazel does not interpret a platform's exec_properties. It forwards the string map through the remote-execution platform field.³ Therefore a key such as stripe/tracing, its address syntax, and the server behavior have meaning only when the selected executor implements that contract. Copying such a property to another service does not enable tracing.

The same case study described executor-started Perfetto collection for hardware performance counters, per-process CPU/memory/I/O statistics, kernel ftrace events, and system statistics. Unlike application-level Chrome events, this system collection did not require the action tool to emit trace events. It did require the worker environment and executor to support and authorize the collectors.²

The talk's Q&A suggested that tracing in remote actions might someday be a candidate for a future Remote Execution v3. That was a proposal-shaped idea, not a specification or announced Bazel capability. Design against the protocol and executor versions you actually deploy. 6.3.4 Remote Action Tracing develops the standard action/Operation evidence chain. Treat any tracing extension beyond it as a private, versioned service contract.

Keep observations outside the cached result

Trace events commonly contain timestamps and may vary with scheduling, worker load, sampling, and collector behavior. If a trace file is declared as an action output, those observational differences become part of the output set. Repeated executions of the same action key can then upload different trace blobs to the CAS and produce nondeterministic ActionResult values for that key. Changed trace artifacts can also propagate to downstream consumers. The Stripe design instead exported traces through a side channel so tracing did not alter the declared action outputs.²

Out-of-band collection must preserve more than the output list. A safe design keeps the action's executable, arguments, environment, declared inputs, and declared outputs unchanged, enables collection in the executor, and associates the resulting telemetry with execution metadata outside the cached action result. This preserves cache identity only when telemetry is selected through a genuinely separate private executor-side control outside the hashed Action and Command. Standard exec_properties are encoded as Command platform properties: changing one changes the Command digest referenced by the Action, and therefore changes the Action digest and cache key. A service cannot preserve standard REAPI identity merely by "normalizing" that property. An identity-preserving diagnostic control must travel through a separate executor request or operations channel that is not part of either proto.

A cache hit can reuse the artifact, but it cannot produce a new trace of an execution that did not occur. To obtain a fresh trace, use a diagnostic request that bypasses cache lookup and define whether its result may populate the cache. Those are execution-service semantics, not a consequence of exporting the trace out of band.

This separation also clarifies the relationship to 5.5.1 Build Event Protocol (BEP). BEP can carry invocation and action-completion context for observability, but it does not by itself reveal the tool's internal phases. A private observability pipeline could use identifiers carried by its own systems to join BEP records, Bazel profiles, executor metadata, and internal traces. This is a design inference, not a correlation guarantee from the cited Aspect source. Define the join explicitly and report ambiguity. Timestamps alone are a fragile identity scheme.

Bound cost, exposure, and interpretation

Action-internal traces can expose target and output names, command structure, process trees, paths, environment-derived values, timing, resource use, and possibly application-specific event arguments. System tracing can reveal activity from the worker or sandbox beyond what an individual build author expects. Before collection, define who may request it, which fields are redacted, where traces leave the worker, who can read them, and how quickly they expire.

Use an explicit operating policy:

1. Ad hoc diagnosis first. Enable tracing for a narrow target or action class, a short time window, and a controlled worker pool.
2. Measure observer cost. Compare traced and untraced executions for wall time, CPU, memory, I/O, network volume, and trace size. A trace that changes the suspected bottleneck is weak evidence about normal execution.
3. Cap collection. Set duration and byte limits, reject unsupported event volume, and ensure collector failure cannot fail or hang the action unless that is an intentional policy.
4. Sample deliberately. For regression monitoring, select a stable sampling unit—such as a small fraction of eligible executions—and record the sampling decision. Cache hits provide no new action-internal trace, so sampled data is inherently biased toward executions that actually run.
5. Compare like with like. Separate action versions, worker types, execution properties, cache state, and tracing configurations before interpreting a regression.

The 2023 case study reported ad hoc debugging at the time and described always-on sampling only as a future plan.² Sampling is therefore an observability-system design problem, not a behavior to infer from the presented executor implementation.