6.5.7 CI Work Distribution

Once target selection has produced a complete work set, CI still has to finish that work quickly enough to be useful. The tempting move is to call every kind of parallelism “sharding” and increase a single number. That hides three different schedulers with different units of work, identities, and completion conditions.

The safe design starts with an immutable partition manifest. It says which selected labels and configurations each CI job must invoke. Bazel may then split one test target into test-runner shards, repeat it for additional evidence, and send the resulting actions to remote executors. Those later layers can accelerate a partition, but they cannot repair a label that the CI partitioner omitted.

Keep the Three Distribution Layers Separate

Use the word partition for CI-level division and reserve test shard for Bazel's test protocol:

Layer	Unit assigned	Scheduler	Identity to retain
CI partitioning	selected top-level obligations under a configuration	CI orchestrator	request, partition, label, configuration
Bazel test sharding and repetition	one target's shard and run/attempt	Bazel plus the test runner	target, shard index/count, run or attempt
remote execution	actions created by analysis and test execution	Bazel and the remote backend	invocation and action identity

CI partitioning turns one established target set into several Bazel invocations. A practical implementation can enumerate test labels and use the CI system's parallel-job index to assign them to buckets; Bazel CI has used this shape to replace one large wildcard invocation with multiple jobs.¹

Bazel test sharding is narrower. When test sharding is enabled, the default explicit strategy honors a target's shard_count: Bazel launches the runner once per shard and supplies a zero-based shard index and total shard count. disabled turns sharding off, while forced=k overrides the count for a run.² A runner that does not update TEST_SHARD_STATUS_FILE makes Bazel fail the sharded test. A runner that signals support but ignores TEST_SHARD_INDEX and TEST_TOTAL_SHARDS can still repeat cases instead of dividing them. The runnable test-sharding snippet demonstrates both the handshake and index-based allocation in its runner. 1.2.5 Test Sharding teaches its operation in detail.

Repetition is different again. --runs_per_test requests multiple executions for evidence such as flake investigation; it does not divide the cases of one run.³ The execution and attempt ledger must therefore distinguish (target, shard, run, attempt) rather than collapse all executions under one label. The immutable manifest still records the selected (label, configuration) obligation. An infrastructure retry of a cancelled job is not a new logical test run or a new manifest member: it is another attempt to produce the required result.

Remote execution works below these identities. When its strategy is configured and selected, it schedules eligible action executions on backend workers; remote-cache hits execute nowhere, and locally selected actions remain local.³ It can make a large partition faster and let independent partitions share cached action outputs, but it does not know that five CI jobs collectively constitute the intended target set. Databricks' earlier horizontal CI split ran bazel test on many machines, yet suffered repeated builds, checkout overhead, and noisy-neighbor effects; adopting remote execution addressed action-level distribution rather than eliminating the need for CI orchestration.⁴

Make the Manifest the Reconciliation Boundary

The partitioner should consume the exact selector result and emit a deterministic manifest before launching jobs. Give the manifest an immutable identity and bind it to the same revision pair, target universe, configurations, and selector outcome established in 6.5.6 Fail-Closed Target Selection. Its unit of assignment is a configuration-qualified obligation: at minimum (label, configuration), plus any declared selector identity dimension that distinguishes an invocation.

An implementation-specific record might look like this:

{
  "manifest": "<immutable-id>",
  "revision": "<head-revision>",
  "configuration": "linux-release",
  "partitions": {
    "0": ["//app:unit_tests", "//lib:parser_test"],
    "1": ["//service:integration_test"]
  }
}

This is an interface sketch for one configuration, not a prescribed schema. Its invariant is exact set reconciliation over configuration-qualified obligations:

union(partition obligations) = selected obligations
intersection(partition i, partition j) = empty, for i != j

The first equality prevents omissions. The second prevents duplicate assignment accounting. It does not promise that Bazel will execute every transitive action only once. Separate Bazel invocations can share dependencies, and a cache hit or remote execution may avoid repeated computation without changing the fact that both partitions requested the dependency.

Uber observed this distinction when equal-count CI shards repeated shared dependency work. Assigning graph root targets reduced overlap because building a root already builds its dependencies, while a shared remote cache reduced remaining duplicate execution across shards.⁵ The transferable rule is not “always use roots.” It is to remove redundant top-level assignments only when the requested roots still cover the selected work, then measure duplicated action work separately from duplicate manifest membership.

Validate the manifest before uploading dynamic jobs. Reject unknown obligations, duplicate memberships, missing configuration identities, and any selected obligation absent from all partitions. An empty partition is harmless if represented deliberately; an unrepresented selected obligation is a correctness failure. Preserve the validated manifest for the result collector in 6.5.8 Complete CI Results with BEP/BES.

Balance from Evidence, Not Label Counts

Equal numbers of labels rarely produce equal jobs. One integration test can dominate dozens of unit tests, while two labels may share most of their action graph. Start with measurements from comparable revisions and configurations:

• elapsed time and resource demand per top-level label or stable group;
• fixed job cost such as checkout, Bazel startup, graph loading, and artifact upload;
• historical variance and timeout frequency, not only the mean;
• overlap in expensive actions or inputs across candidate partitions;
• constraints such as platform, credentials, exclusive resources, and runner class.

Use those estimates to place work into bounded bins, then keep an explicit bucket for labels with no trustworthy history. Newly selected work must never disappear because the timing database has no row. A conservative first weight, a dedicated discovery partition, or a broader default class are all valid if they remain in the manifest.

More partitions stop helping when fixed setup and duplicated graph work outweigh added concurrency. Uber's dynamic CI varied the number of shards with the selected target count, but separate containers also introduced repeated checkout, Bazel startup, and cold-state costs.⁶ That is why the tuning signal is end-to-end critical-path time and total resource cost for a cohort, not the theoretical 1/N reduction from creating N jobs.

Keep platform-incompatible work separate before balancing. Do not let a timing optimizer move a macOS-only test to a generic Linux queue or combine mutually exclusive credentials merely because their durations fit. Configuration and runner eligibility are hard constraints; timing is a soft optimization inside each eligible pool.

Account for Attempts Without Rewriting History

A partition manifest describes intended work. The attempt ledger describes what happened while trying to execute it. Keep them separate so retries and cancellations cannot mutate the original obligation.

For every partition, record:

1. the manifest identity and partition index;
2. each CI job attempt and its terminal state;
3. the Bazel invocation identity produced by that attempt;
4. expected target, test-shard, and run identities;
5. whether the attempt's evidence was accepted, superseded, or incomplete.

If a runner disappears after producing two of four test shards, retrying the CI job does not make the partial attempt vanish. The collector may accept a complete later attempt according to its policy, but it must retain the earlier cancellation and avoid counting the two old shards as extra successful coverage. Likewise, retrying one failing test is not equivalent to completing a missing partition: it changes attempt evidence for that test, not manifest membership.

Cancellation needs an explicit reason and scope. A fail-fast policy may cancel sibling partitions after one failure to save capacity, but the resulting build is failed with intentionally incomplete work, not evidence that all selected work ran. A superseding revision can cancel an older manifest, but results from the two revisions must not be combined.

Test this boundary with a small synthetic manifest:

1. generate partitions and assert exact union and disjoint configuration-qualified membership;
2. add a label with no timing history and prove it is still assigned;
3. cancel one partition after partial shard output and retry it;
4. retry a failing shard while another shard remains missing;
5. inject a duplicate job delivery and a stale attempt from another manifest.

The distribution layer passes when every selected obligation has one manifest home and every execution is attributable to the correct partition, revision, configuration, shard, run, and attempt. Whether those attributable results are complete enough for a green decision is the next article's job.