6.5.9 Reusing CI Runners Safely

Reusing a CI runner is not one optimization. It is a decision to retain several different kinds of state—source checkout, Bazel server and output base, explicit local caches, and credentials—each with its own compatibility and isolation rules. A runner is safe to reuse only when the next job can prove which state it inherited, why that state is compatible, and how contamination would be detected and recovered.

Inventory State Before Choosing a Lifecycle

A remote action cache does not make the CI runner stateless. Loading, repository evaluation, and analysis still happen in the Bazel server that coordinates the build. The server retains the in-memory graph, while the output base contains outputs and Bazel's internal working state. The repository cache and disk cache are separate stores, and the checkout has state that Bazel does not manage at all.¹

Start with an explicit retained-state inventory:

State	Why retain it	Isolation question
checkout and workspace files	avoid a full clone and reuse filesystem metadata	can one job observe files, generated data, or untracked changes from another?
Bazel server and output base	preserve the in-memory graph and incremental output state	can incompatible or concurrent invocations reach the same server and mutable tree?
repository and repository-contents caches	avoid downloading or reconstructing external repositories	which dependency, repository-rule, and producer inputs make an entry reusable?
local disk cache	reuse action results without a remote round trip	are writers trusted, entries namespaced, and corruption recoverable?
credentials and runner services	authenticate source, cache, BES, registry, and release access	can a later or less-privileged job read or use an earlier job's identity?

These rows must not collapse into one warm=true flag. For example, an ephemeral VM can attach a persistent cache volume, so its process memory is new while its cache is old. A persistent host can start every job with a fresh output base. A snapshot can clone memory and disk together. “Stateful” and “ephemeral” describe lifecycle choices, not reliable security or freshness properties.

The CI cache baseline demonstrates only one deliberately narrow layer of this inventory: fresh, separate output bases can share a remote cache, and a job shuts Bazel down before saving explicit caches. It does not qualify reuse of a complete runner allocation, tenant separation, or checkout cleanup; those missing boundaries are the point of the rest of this article.

3.6.2 Bazel Server Lifecycle in CI introduces why a cold server repeats loading and analysis. Here the operational question is whether the saved time is worth the larger compatibility and recovery contract. A case study from Uber illustrates the distinction: its jobs matched reusable containers using dimensions such as job name, Bazel version, image hash, agent identity, configuration files, and environment, while using read-only dependency caches and a remote action cache as separate layers.²

Give Every Concurrent Job an Exclusive Mutable Domain

Bazel serializes invocations that use the same output base. The output base also contains the lock that prevents multiple Bazel processes from mutating it concurrently. If two CI jobs need to run at the same time, assign each an exclusive output base—or deliberately queue them behind the same server. A different checkout path alone is not proof of isolation, because startup options can direct both jobs to the same output base.³

One safe assignment model is:

runner allocation
  checkout:    /work/jobs/<allocation-id>/repo
  output base: /work/jobs/<allocation-id>/output-base
  credentials: short-lived identity for <allocation-id>
  shared read-only state: versioned dependency seed
  shared writable state: only stores whose concurrency and trust contract allows it

The runner-reuse qualification fixture makes the allocation boundary concrete. Its qualification script builds two clean allocations with distinct output bases, admits a matching reuse, then quarantines an identity mismatch and an untracked checkout file before proving a clean recovery build. It models runner-owned lifecycle decisions around a real Bazel action; credential scoping and cross-tenant authorization remain deployment tests for the CI provider and secret system.

The allocation identity should be unguessable to other tenants and should own all mutable directories and local processes for the job. Do not reuse a Bazel server simultaneously between users merely because Bazel has a lock: waiting for a lock prevents concurrent output-base mutation, but it does not establish tenant authorization, credential separation, or workspace cleanup.

After the job, reconcile results before promoting any warm state. A successful process alone is not enough if the intended CI work or its artifacts are incomplete; use the complete-result boundary from 6.5.8 Complete CI Results with BEP/BES. Promotion should also reject a runner with leaked processes, mounted filesystems, unexpected writable files, exhausted disk or file descriptors, or a failed cleanup check. Persistent pools otherwise invite resource leaks that accumulate across jobs.⁴

Credentials deserve the strictest rule: inject the minimum identity for the current job, keep it outside reusable workspace and cache content, and revoke or remove it before returning the runner to the pool. A snapshot that contains a credential is a credential copy, even if the snapshot was created only to save an analysis cache. Release credentials should not enter an ordinary test-runner cohort at all.⁵

Reuse by Compatibility, Not by Git SHA Alone

An immutable source revision is necessary evidence, but it does not identify a complete Bazel invocation. Reuse can also depend on the Bazel binary and startup options, checked-in and injected rc files, rules and module state, repository inputs, supported platform and toolchain, runner image, and job identity. Define a compatibility record for each retained layer rather than one global cache key:

{
  "checkout": {"repository": "<repo-id>", "revision": "<commit>"},
  "bazel_state": {
    "bazel_version": "<version>",
    "startup_profile": "<startup-options-id>",
    "build_configuration": "<configuration-id>",
    "runner_image": "<image-digest>"
  },
  "dependency_seed": {"schema": 3, "producer": "<image-build-id>"},
  "security_domain": "presubmit-untrusted"
}

This is a design sketch, not a Bazel file format. Its purpose is to make the runner's claims reviewable. When one dimension cannot be proven compatible, discard or bypass only the affected layer if isolation permits it. A changed checkout may still use a trustworthy repository cache; a changed Bazel startup profile may require another output base; a changed security domain requires a new allocation regardless of cache warmth.

Bazel does invalidate retained analysis state for changes it knows affect the configuration. That protects Bazel's internal incremental model, but it cannot validate every surrounding CI promise. It cannot prove that a checkout cleanup removed an untracked secret, that an image was patched under the same tag, that a credential is still scoped correctly, or that an external cache volume came from the expected producer. The runner owns those lifecycle facts. The deeper Skyframe mechanics are covered by 5.9.4 Incrementality Mechanism.

Avoid routine bazel clean as a substitute for this model. Cleaning discards useful evidence and Bazel-managed incremental state while leaving unrelated runner state—credentials, untracked workspace files, leaked processes, mounted volumes, and external caches—untouched. Quarantine the allocation, capture its identity and diagnostics, then rebuild or discard the specific unsafe layers.

Choose a Lifecycle from Measured Trade-offs

There are four common lifecycle shapes:

1. Fresh runner, restored explicit caches. Startup and analysis are cold, but the mutable job boundary is simple. Cache archives or volumes still need provenance, namespaces, concurrency semantics, and credential-free contents.
2. Persistent runner or hot pool. Checkout, server, and disks can stay warm. The service must schedule compatible work, isolate concurrent allocations, health-check hosts, and recycle them before accumulated state becomes unsafe.
3. Prebuilt disk image or seed. Jobs start fresh from a versioned read-only dependency and tool seed. This reduces setup without treating the result of an arbitrary prior job as trusted; rebuilding and rotating the seed is part of the design.⁶
4. Cloned VM or remotely hosted Bazel server. A snapshot can preserve both memory and disk, including a warm server. This is powerful but makes snapshot keying, copy-on-write isolation, promotion, credential scrubbing, and corrupt snapshot recovery part of the service contract.⁷

These are variants of the same state problem, not unrelated tricks. Compare them on reproducible cohorts from 6.1.2 Measuring Builds Fairly. Measure cold startup, steady-state latency, tail latency after incompatible work, queueing, storage and transfer cost, and recovery time. Gate every result on correctness and isolation. “Persistent is faster” is not a conclusion if warm jobs queue for scarce compatible runners; “ephemeral is clean” is not a conclusion if every job restores the same contaminated archive.

Do not generalize vendor performance figures or “drop-in” claims into Bazel semantics. Firecracker snapshot cloning and hosted remote Bazel are named implementations with reported case-study results, not portable guarantees.⁸ Likewise, preemptible instances and disk cloning can lower a particular fleet's startup cost, but they do not remove the need to qualify the image, disk, and identity used by each job.⁹

Prove Reuse with Mutation and Recovery Tests

Qualification needs negative-path tests, not only a warm-build benchmark. Create a matrix that mutates one compatibility dimension at a time:

• change the Git revision and leave an untracked file in the old checkout;
• change Bazel version, startup options, build configuration, platform, toolchain, rules, or runner image;
• run two allocations concurrently with distinct users and credentials;
• corrupt or replace one explicit cache layer;
• kill the Bazel server or host during a job;
• exhaust disk space or leak a process, port, mount, or file descriptor;
• expire credentials while retained state remains warm.

For every mutation, assert which layers are reused, which are invalidated, and which cause the allocation to be quarantined. Then assert recovery: the next job must obtain a clean compatible allocation, complete its intended work, and publish evidence tied to its own revision and identity. Also retain a cold control cohort; without it, a pool that silently stopped reusing state could pass every isolation test while delivering none of the intended benefit.

Record the inventory and compatibility decision with each invocation. When a failure appears only on reused runners, operators can then compare checkout, output-base, server, cache, image, and credential provenance rather than wiping everything and hoping the symptom disappears. Repository checkout acceleration at larger scale continues in H.3.1 Git Workspace; the safety boundary here is that a faster snapshot or mirror must still resolve to the intended immutable source state.