6.6.1 Tracing a Build Across Services

A slow or failed build can leave evidence in six places: the CI system, Bazel, the remote cache, the execution service, BES, and a result consumer. No one identifier is guaranteed to appear in all six. To trace the build, you preserve a chain of smaller joins and say explicitly where that chain breaks.

This extends the control, data, and result paths from 6.1.4 Action and Result Lifecycle. The goal is not to put every event into one giant log. It is to answer a narrower operational question: which observation in the next service belongs to the same build, action, result, or blob?

Start with typed identities

“Build ID” and “action ID” are dangerously vague names. A useful trace keeps each identity's scope and producer visible:

• CI run, job, and attempt
  • Follows: source revision, trigger, runner, retries, and pipeline stages.
  • Boundary: Bazel and remote services do not have to understand the CI model.
• Bazel invocation ID
  • Follows: one Bazel command across stderr, BEP, and remote requests.
  • Boundary: it does not identify a configured target, action, result, or CI retry.
• BEP event ID
  • Follows: one event in an invocation's ID-linked event graph.
  • Boundary: it is unique only within that BEP stream, not a remote-operation handle.
• Action digest
  • Follows: declared action content used for cache lookup and remote execution.
  • Boundary: several attempts or invocations can refer to the same digest.
• REAPI request action_id
  • Follows: related cache, CAS, and execution RPCs when clients and services preserve it.
  • Boundary: it is optional metadata, not the digest or a universal backend trace ID.
• Remote Operation.name
  • Follows: one execution operation and subsequent waits or polls.
  • Boundary: it is not a portable scheduler assignment, lease, or worker-attempt ID.
• Result and blob digests
  • Follows: exact ActionResult-referenced content and CAS objects.
  • Boundary: equal bytes do not identify producer, consumer, reason, or time of use.
• Backend attempt, assignment, or worker ID
  • Follows: a scheduler- or executor-specific attempt.
  • Boundary: it has meaning only in that pinned backend and deployment.
• JSON trace profile artifact
  • Follows: invocation-local timing evidence archived with its producer version.
  • Boundary: the mutable latest-profile alias is not a stable invocation identity.
• Optional execution log artifact
  • Follows: captured spawn or action observations stored with their invocation.
  • Boundary: entries do not automatically identify a remote operation or backend attempt.

Bazel's --invocation_id accepts a UUID supplied by the caller and publishes it to stderr, BEP, and the remote execution protocol. If CI sets it, CI must ensure uniqueness.¹ This makes it a strong bridge from a CI attempt into Bazel and REAPI request metadata, but it still does not replace the more specific identities below it.

BEP adds another graph, not a flat sequence. Each event has an identifier, announces child identifiers, and carries a payload; its event identifiers are unique within the invocation.² 5.5.1 Build Event Protocol (BEP) develops that graph and its completeness rules. For cross-service tracing, retain the invocation ID beside every ingested event and store the event ID as a scoped pair such as (invocation_id, bep_event_id). Never join two streams on a bare event ID.

At the remote-action boundary, keep the action digest, request metadata, and Operation.name in separate columns. 6.3.4 Remote Action Tracing explains why they identify different things. A digest can join a cache lookup to an execution request; an operation name follows one service operation; optional backend identities may then join that operation to queue, assignment, lease, and worker evidence. Converting all of them into one field called action_id throws away exactly the distinctions an incident needs.

Bazel names the default JSON trace profile command-$INVOCATION_ID.profile.gz in the output base, while command.profile.gz is a symlink to the latest command's profile. Archive the invocation-named file together with the invocation ID and Bazel version; do not resolve the mutable alias later and assume it still belongs to the incident. Bazel retains only a bounded number of default profiles, so the trace pipeline must ingest the artifact before that retention window expires.³

Treat an execution log as another optional, separately captured artifact. Store its producer version and invocation beside it. If the available format or deployment cannot prove which log entry maps to an action digest, request correlation field, or remote operation, record that join as missing. A mnemonic, target label, or nearby timestamp can narrow candidates, but cannot manufacture the identity link. Bazel's evidence formats index overlapping action evidence differently, so this gap must remain visible.⁴

Make every boundary an explicit join

Treat the trace as a sequence of handoffs. For every handoff, specify:

1. the identity written by the producer;
2. the identity read or generated by the consumer;
3. the system that records the mapping;
4. the retention and access period for that mapping;
5. what conclusion is allowed when either side is missing.

For example, a CI job can generate one invocation UUID, pass it to Bazel, and record (ci_run, job, attempt) -> invocation_id. Bazel can attach the invocation and action correlation metadata to remote RPCs. The execution frontend can record (invocation_id, action_digest, request_action_id) -> operation_name, while its scheduler records operation_name -> backend_attempt -> worker. BES ingestion records the invocation ID alongside the received BEP stream, and a result consumer records which invocation and scoped BEP result caused its update.

That is an architectural pattern, not a promise that every Bazel client, backend, or CI product exposes those exact mappings. Request correlation fields can be absent, and different Bazel logs index the same action differently; a source focused on Bazel tracing calls this cross-log correlation out as a real tooling gap.⁴ Inspect the versions you actually operate and record an unsupported join as a gap rather than reconstructing it from names or timing.

Timestamps are supporting evidence, not identity. Two events occurring at 10:14:03 can belong to different concurrent builds. Clock skew, batching, upload retries, and asynchronous consumers can also reorder observations. Use time only after an identity join, or to produce a bounded candidate set that another field must confirm.

Follow one seeded delay

Suppose a presubmit is slow only when a particular remote worker image is in the fleet. Seed a controlled delay in one representative compile action on a canary worker pool. The following is a normalized investigation record, not captured Bazel output:

1. CI → Bazel
   • Join: CI attempt mapping to invocation UUID.
   • Observation: attempt 2 started invocation I-42 for the expected revision and targets.
   • Confidence: direct mapping; attempt 1 remains a different trace.
2. Bazel → remote request
   • Join: invocation ID plus action digest.
   • Observation: invocation I-42 submitted action digest D-9.
   • Confidence: direct client record; a target label alone would be ambiguous.
3. Execution frontend
   • Join: request metadata plus digest to operation.
   • Observation: D-9 became operation O-17.
   • Confidence: direct frontend mapping.
4. Scheduler → executor
   • Join: operation to backend attempt and worker.
   • Observation: O-17 attempt A-3 ran on canary worker W-8, image img-v2.
   • Confidence: backend-specific mapping; image identity comes from deployment telemetry, not REAPI.
5. Executor → terminal result
   • Join: attempt and operation to result.
   • Observation: the command interval contains the seeded 400 ms delay; the operation completes successfully with result digest R-5.
   • Confidence: direct worker and terminal-operation evidence.
6. Bazel → BES
   • Join: invocation ID and scoped BEP IDs.
   • Observation: BES receives completion for I-42.
   • Confidence: arrival proves receipt of that event, not completeness of every announced BEP child.
7. BES → result consumer
   • Join: ingestion record to consumer update.
   • Observation: the consumer publishes a result for I-42 80 ms later.
   • Confidence: direct mapping; artifact-byte availability must be checked separately.

The trace supports a narrow conclusion: the seeded delay occurred during the executor attempt on the canary image and propagated into the observed invocation latency. It does not prove that all slow builds use that image, that the scheduler caused the delay, or that BES delivery is complete. Repeating the experiment with a matched control pool turns this trace into comparative evidence; 6.6.3 Measuring Build-Service Latency and Reliability defines how to aggregate such observations without hiding missing data.

If the scheduler records only O-17 -> W-8 but not the backend attempt, report “operation assigned to worker; attempt-level join unavailable.” If BES contains the invocation but an expected result event was announced and never received, report an incomplete result-plane trace. Missing evidence lowers attribution confidence; it is not evidence that the missing stage did no work.

Design telemetry that remains operable

A trace schema should preserve the smallest identities needed for later joins, plus their type, producer version, and observation status. Record at least the client and service versions that control serialization or correlation behavior. Without producer versions, a rollout can silently change the meaning or presence of a field and make old and new traces appear comparable.

Do not promote every label, digest, path, argument, or operation name into an unbounded metrics dimension. Keep high-cardinality identities in trace or log records that support indexed lookup; aggregate metrics should use bounded dimensions such as journey, action class, platform cohort, service stage, and result class. The detailed record explains an incident. The bounded aggregate reveals which incident deserves investigation.

Retention must cover the diagnostic window across all joined systems. Keeping CI metadata for 30 days does not help if the operation-to-worker mapping expires after ten minutes. Document each join's shorter retention, sampling, and loss behavior, and test that a known trace remains reconstructable after the normal delay before investigation begins.

Access deserves the same explicit design. Command arguments, environment data, labels, repository paths, source revisions, user identities, and artifact references can reveal secrets or sensitive project structure. Prefer opaque correlation IDs, allowlist collected fields, redact at the producer where possible, and separate broad aggregate access from restricted trace-detail access. Never copy credentials into telemetry merely to explain which request used them; record a safe credential or principal identity instead.

Test the joins, not just the dashboards

A dashboard can render plausible spans while its joins are wrong. Qualify the trace pipeline with seeded events whose identities and expected boundaries are known:

• run two concurrent CI attempts for the same revision and action digest, then prove they remain separate by invocation and attempt;
• route one action to a canary worker with a seeded delay or controlled failure and follow it through the terminal result;
• interrupt BES delivery after an event announces children and verify the consumer marks the stream incomplete instead of silently excluding it;
• remove one correlation mapping and verify the trace displays an explicit gap rather than inventing a timestamp join;
• roll one producer version and prove the schema and version distinction survive mixed old/new traffic;
• exercise access controls with sensitive-looking arguments and paths and prove they are redacted or restricted.

The result is not “one trace ID everywhere.” It is a reproducible evidence chain with typed identities, known join owners, explicit loss semantics, and a confidence statement at each boundary. That chain supplies the evidence needed to locate a failing service layer in 6.6.4 Locating Build-Service Failures without mistaking the place where a symptom appeared for its cause.