6.5.8 Complete CI Results with BEP/BES

A green CI tile is trustworthy only when it accounts for all intended work and all evidence needed to interpret that work. Bazel's Build Event Protocol (BEP) supplies structured invocation and test events, while the Build Event Service (BES) transports those events to a backend. Neither one knows whether your distributed CI plan was complete. Your result collector must join the event evidence to the partition manifest and refuse to turn a gap into a pass.

Start from an Intended-Work Ledger

6.5.7 CI Work Distribution ends with a partition manifest: the jobs, labels, configurations, test shards, runs, and retry policy that CI intended to execute. Preserve that manifest under an immutable revision and configuration identity. It is the left side of the completeness comparison; events received from Bazel are the right side.

For each unit of intended work, record an identity precise enough to distinguish:

• the source revision and CI lane;
• the partition and Bazel invocation;
• the target and configuration;
• for tests, the shard, run, and attempt;
• the terminal result and the artifacts required by your consumer.

The dimensions matter because BEP emits a TestResult for every test attempt, shard, and run. Its TestSummary aggregates a test's results and includes attempt, shard, and run counts so a consumer can distinguish flaky from failed outcomes.¹ A collector keyed only by label can overwrite an earlier failure with the latest retry or collapse several shards into one apparent success.

A compact reconciliation record might look like this:

revision: <immutable-revision>
partition: linux-tests-03
invocation: <invocation-id>
unit: //search:index_test [configuration=<id>, shard=2, run=1]
attempts:
  - attempt=1, status=FAILED, artifacts=available
  - attempt=2, status=PASSED, artifacts=available
verdict: flaky

This is a design sketch, not a Bazel schema. Its important property is append-only attempt evidence. Policy may present the final verdict as flaky, failed, or passed, but it must not erase the history that produced that verdict.

Treat BEP as a Graph, Not a Log Tail

5.5.1 Build Event Protocol (BEP) introduces the event types for one invocation. At CI scale, completeness depends on their graph semantics. Every BEP event except the first is announced by an earlier event. When an invocation completes normally, every announced event is eventually posted; after a Bazel crash or failed transport, announced events can remain missing.² A collector can therefore detect a protocol hole by retaining announced event identifiers and checking that each was fulfilled.

Do not equate seeing BuildFinished with consuming the last useful event. BuildFinished carries the command's authoritative exit code, but summary events such as metrics or profiling data may follow it.³ Nor does a successful BuildFinished prove that every CI partition was launched. It establishes the outcome of one Bazel command, not the coverage of the distributed plan.

Classify each invocation independently before rolling it into a lane verdict:

An Aborted payload replaces the normal payload under an announced event identifier and explains why that event was not produced normally.⁴ Preserve it as terminal negative evidence for that event. Absence is different: it means the collector cannot establish what happened.

Delivery order must not become hidden correctness logic. BEP's parent-child relationship does not require every parent to be posted before its child, while referenced NamedSetOfFiles events have their own ordering rule: they appear before a TargetComplete or another named set that references them.⁵ Index events by identity, tolerate valid ordering and duplicate delivery, then evaluate the graph when sufficient evidence is available. “The most recent message wins” is not a reconciliation algorithm.

Separate Event Generation from Event Transport

Bazel generates BEP events. It can serialize them to a local binary, text, or JSON file, or publish them through its BES client. BES is a generic gRPC publishing protocol, independent of BEP, and transports BEP events as opaque bytes.⁶ A backend may add indexing, dashboards, retention, or consumer-specific completion state, but those are backend contracts rather than portable BEP semantics.

This gives an investigation three distinct checkpoints:

1. Generation: Did the Bazel invocation produce the expected event graph?
2. Transport: Did the configured file or BES path deliver that graph to the collector?
3. Consumption: Did the backend reconcile it against the correct immutable work ledger?

6.1.4 Action and Result Lifecycle makes the same separation for remote action execution and result publication. A successful test action can still be followed by a failed event upload, and a complete event upload can still be attached to the wrong revision or partition manifest. Keep those failures attributable instead of reporting all of them as “test missing.”

Test the transport boundary deliberately. Cover a normal success, a build failure, client interruption, upload failure, reordered and duplicate delivery, and a conflicting duplicate. The expected outcome is not that every case produces a complete stream. It is that only the normal complete case can satisfy the success predicate, while failures remain failed or unknown rather than becoming green.

The runnable BEP complete-result fixture captures real local BEP JSON streams for both a passing and an intentionally failing Bazel test, then checks each against an immutable partition, shard, run, and attempt manifest plus the actual test log. It uses the failing stream as a control: complete evidence of a failed test must be failed, not unknown-delivery. Deterministic collector-input mutations then remove terminal, result, summary, manifest-obligation, partition, shard, retry, or artifact evidence; replace an announced nonterminal event's normal payload with Aborted; and add a conflicting duplicate. Those mutations exercise the collector contract; they are not claimed to be a live BES deployment.

Events Can Be Complete While Artifacts Are Missing

BEP events commonly refer to files instead of embedding their bytes. TargetComplete points to one or more NamedSetOfFiles structures, which may recursively refer to other named sets. This sharing avoids repeatedly listing the same files, but a consumer must traverse the structure with a visited set and retain named sets until the stream is exhausted.⁵

Resolving the graph yields file metadata or a URI; it does not by itself prove that the bytes are available to the consumer. The official BES guidance calls out the boundary directly: test logs and XML files may exist only on the Bazel machine, which a remote BES server cannot access. Remote caching is one way to upload referenced outputs so that the BES server can fetch them.⁷

For every artifact required by your CI contract, track four separate facts:

• its reference was present in the event graph;
• the full named-set path was resolvable;
• the authorized consumer could retrieve the bytes;
• retention covers the period in which downstream consumers need them.

An authorization error is not proof that a blob is absent, and a resolvable URI is not proof that it exists. Report those states separately. Profiles, test logs, XML reports, coverage files, and build outputs may also have different retention and access rules; do not infer a shared lifecycle from their appearance in one event stream.

Define One Complete-Result Predicate

Make the gating condition explicit and machine-testable. For a revision and lane, a result is complete only when:

1. the intended-work ledger has an immutable identity;
2. every intended partition and invocation is accounted for;
3. every intended target, configuration, shard, and run has the required attempt evidence;
4. every invocation has a terminal success, failure, cancellation, or explicit incomplete state;
5. no required announced BEP event is unresolved and no conflicting event remains;
6. every required referenced artifact is resolvable, authorized, available, and retained under the lane's contract;
7. all evidence names the same revision, universe, and policy versions.

The predicate should return at least complete, failed, and unknown-delivery, not a single Boolean. A test failure with complete evidence is different from missing evidence. Both block a green result, but only the former proves that the tested code failed.

Exercise the predicate with adversarial fixtures. Remove one partition record, one shard result, one retry attempt, one announced event, and one referenced artifact in turn. Add a cancellation and two conflicting events with the same identity. Each mutation must prevent a false green while preserving the evidence that explains why. Include a fully delivered failing invocation as a control: the system should report a complete failure, not confuse any non-green result with incomplete delivery.

The output is an immutable technical handoff: revision identity, intended-work ledger, invocation results, event-graph verdicts, and artifact-availability verdicts. H.7.5 Merge Queue decides when that handoff permits queue advancement, and H.7.4 Change Admission decides which lanes and scopes are required. The collector supplies trustworthy evidence; it does not advance workflow state itself.