Bazel Knowledge Hub

3.6.2 Bazel Server Lifecycle in CI

A developer who times a local incremental build at one second and then sees the same target take minutes in CI has not hit a CI misconfiguration — they have hit the gap between a warm Bazel server and a fresh one. Bazel is designed as a long-running server process that keeps BUILD files, dependency graphs, and analysis results in memory across invocations.1 A typical CI job is the opposite: a fresh runner, a fresh Bazel binary, an empty server. Once you see the 3.6.1 Basic CI Recipe through this lens, a lot of "why is CI slower than my laptop?" stops being a mystery and starts being a design decision to plan around.

Warm server vs cold runner — what Bazel keeps between builds
In-memory state lives with the server. Disk caches can persist beyond one server and across machines.
LOCAL (WARM SERVER)
Daemon reuses prior work
Same process, same output_base
In-memory Skyframe graph
Packages, configured targets, action graph
KEPT
JVM process with JIT-warmed code
Hot code paths from prior invocations
KEPT
Action-result cache
output_base/action_cache/ index intact
KEPT
Repository cache on disk
Downloaded archives reused as-is
KEPT
Next command reuses all of this
CI RUNNER (COLD START)
Fresh server starts empty
New process, new output_base
In-memory Skyframe graph
No graph to transplant across machines
✕ ABSENT
JVM process with JIT-warmed code
Brand-new JVM, no JIT warmup
✕ ABSENT
Action-result cache
Replayed via --disk_cache (separate CAS)
RECOVERABLE
Repository cache on disk
Restored via --repository_cache
RECOVERABLE
Analysis phase runs cold every time
same repo, different machine
In-memory state is lost on every fresh runner — persistent caches only help the execution side.

A fresh runner means a cold server

On a developer machine, the bazel you type is a client that talks to a server already running in the background, keyed by the workspace's output_base (1.1.3 Server/Client Architecture).1 Across consecutive commands, that server reuses the work it has already done: loaded packages, the Skyframe incremental graph from 2.4.1 Skyframe & Incrementality, and the in-memory action graph from 2.2.1 Loading, Analysis & Execution. That is why bazel query after bazel build is fast, and why an incremental rebuild is mostly a question of which Skyframe nodes became dirty.

On a CI runner, none of that is true on the first command. The runner starts empty, Bazelisk downloads the right Bazel binary, the client does not find a matching server for this output_base, and it spins up a new one.1 Inside that server, the JVM starts with no JIT warmup, the in-memory Skyframe graph is empty, and the action cache index under $(bazel info output_base)/action_cache/ from 2.4.2 Where Bazel Caches Things does not exist yet.2 Every later-phase optimization Bazel usually relies on — change pruning, analysis reuse, output-tree validation — has nothing to compare against. The cost of that emptiness is a full cold pass through loading, analysis, and execution.

Cold analysis is usually the loudest part of the log

The piece most maintainers notice first is the analysis phase, and the measurements are stark. For a six-minute CI build that already used remote cache and remote execution on GitHub runners, the analysis phase alone took about 1.5 minutes — roughly 25% of the entire wall-clock time — and dropped to under a second on a runner that reused a warm Bazel server.3 The same culprit shows up at the per-command level: the first Bazel build of a session starts a local server that initializes, loads dependencies, and builds the project's dependency graph, which for larger projects can take on the order of minutes.4

Analysis is also hard to accelerate with the easy cache layers. Remote cache and RBE help on the execution side, but the analysis phase still runs on the machine that kicked off the build. If your analysis cache has been lost, Bazel is doing that work from scratch regardless of where actions eventually run.4,5 The practical rule follows: for correctness-focused build systems like Bazel, persistent runners with a remote cache are the first CI optimization, and trying to compensate with RBE alone tends to increase costs without recovering the lost analysis time.5,6

When a CI run feels slow, the first diagnostic question is therefore not "is the cache broken?" but "is this runner cold?" The JSON trace profile separates analysis-phase events (the skyframe evaluator sections) from execution, which makes cold analysis visible instead of implied.4

What a maintainer can do today

The deep infrastructure patterns — warm runner pools, disk cloning, Skyframe-preserving microVM snapshots — belong to 6.5.9 Reusing CI Runners Safely. The maintainer-level actions are smaller but still meaningful.

Split "cold" from "colder" with persistent caches. A warm-server strategy is an infrastructure-level investment, but a persistent disk cache across CI runs recovers most of the execution-side cost of a cold server. That is what 3.6.4 Disk Cache in CI and, when disk alone isn't enough, 3.6.5 Simple Remote Cache Setup are for. They will not shrink the analysis phase, but they keep every other phase from starting empty.

Shut the server down at the end of each CI step. On ephemeral runners, running bazel shutdown at the end of a step releases the server's memory and file handles, stops the server from interfering with cache upload or log capture, and guarantees the log ends at a predictable point.7,8 The ci-cache-baseline workflow closes its CI job with exactly that step, gated on if: always() so the shutdown still runs on test failure. It also avoids accumulating idle servers when CI scripts drive multiple builds in different directories — a scenario the scripting docs call out specifically.8 For one-off scripts that should not keep a server around, --max_idle_secs=5 on the startup line is a more passive alternative: the server still exits on its own five seconds after the command finishes, so idle servers do not accumulate.8

think

Trace: A CI job runs bazel build //..., then bazel test //... on the same runner. The second log says WARNING: Build option ... has changed, discarding analysis cache. The Bazel server never exited. Which part is still warm, and which part did you just lose?

Reveal

The JVM process is warm, but the configured analysis state you wanted to reuse is gone. Disk-backed things such as downloaded repositories, outputs, and action-cache metadata may still help, but the Skyframe work from the first command has been invalidated by the flag drift.

That warning is therefore not harmless log noise. It is a performance regression with a cause: two adjacent commands asked Bazel for different configured worlds. Put the shared CI flags behind one named config and reuse it for every command in the job.

Keep flags stable across CI commands in one job. The analysis cache is discarded when Bazel sees a different flag set between invocations — the classic WARNING: Build option ... has changed, discarding analysis cache line.4,9 Even on a warm runner, slipping a bazel query with a different flag set between two bazel build invocations throws away the analysis work done by the first. Encoding the CI flag set once, for example behind a named config from 3.2.1 .bazelrc Hierarchy, and reusing it in every command makes this invisible regression much harder to introduce.

key takeaway

CI is slow on the first build because a fresh runner is a cold Bazel server: no loaded packages, no in-memory Skyframe graph, no output-tree action-cache index. That is an artifact of the daemon design, not a CI bug. The practical maintainer response is to plan for it — persistent 3.6.4 Disk Cache in CI or 3.6.5 Simple Remote Cache Setup to recover execution cost, stable flags and named configs so warm runners stay warm, and an explicit bazel shutdown at the end of each CI step. The architecture that actually eliminates cold analysis — warm runner pools, disk cloning, Skyframe-preserving microVM snapshots — belongs to 6.5.9 Reusing CI Runners Safely.

extra

Why a second bazel command is sometimes still cold

Even when a CI script runs two Bazel commands back to back, the second one can restart the server (1.1.3 Server/Client Architecture). Bazel restarts the server automatically if the client and server binary versions differ (for example, Bazelisk just fetched a new one via 0.1.1 Bazelisk & .bazelversion) or if any startup options changed — startup options are fixed when the Bazel server's JVM starts, so Bazel cannot change them on a running server.1,7 --announce_rc is useful here: it shows exactly which options .bazelrc handed to the startup line and the command line, which is usually where the surprise lives.

Check your understanding · 3 questions

1.A CI run shows high remote cache hit rates but still takes much longer than expected. What is the most likely explanation?

Select one answer

2.True or false about Bazel server lifecycle in CI:

Choose True or False for each sentence

A Bazel server restart is triggered if any startup options change between two consecutive bazel commands in the same CI job.
The repository cache and output-tree action cache survive a Bazel server restart.
Using --max_idle_secs=5 means the server will exit after 5 seconds of no build activity.

3.Why does keeping flags stable across all Bazel commands in a CI job matter for performance?

Select one answer

0 of 3 answered

Footnotes

  1. Client/server implementation — server as a long-lived process, output-base-keyed identity, version-mismatch restarts, and --max_idle_secs idle shutdown ↩1 ↩2 ↩3 ↩4

  2. The Many Caches of Bazel — in-memory Skyframe cache (glob results, action graph, actions themselves) is lost when the server dies. Output-tree action-cache index lives under output_base/action_cache/ ↩

  3. Reusing Bazel's analysis cache by cloning micro-VMs — analysis phase measured at 1.5 min out of a 6 min CI build on GitHub runners with remote cache + RBE, dropping to under 1 second on a warm cloned runner ↩

  4. Why is my Bazel build so slow? — first Bazel command starts a local server that loads deps and builds the graph. JSON profile skyframe evaluator sections reveal cold analysis. Flag changes discard the analysis cache ↩1 ↩2 ↩3 ↩4

  5. Estimating the effort to build a Bazel CI/CD — ephemeral CI runners are the opposite of what Bazel wants, repository rules re-execute, there is no way to export and restore the in-memory server state ↩1 ↩2

  6. Achieving the Promised 3x-10x Bazel Speedup — persistent runners with a remote cache as the first CI optimization, analysis results stored in JVM memory and easily discarded ↩

  7. Commands and Options — shutdown command, --max_idle_secs default (10800s / 3 hours), and --host_jvm_args applying only to server startup ↩1 ↩2

  8. Calling Bazel from scripts — scripts should call shutdown or set --max_idle_secs=5 so idle servers do not accumulate across automated builds ↩1 ↩2 ↩3

  9. Meet BuildBuddy Workflows — once RBE and remote cache remove network bottlenecks, the analysis phase typically becomes the next CI bottleneck. Warm hosted Bazel instances reuse the analysis cache across runs ↩

← PreviousBasic CI RecipeNext →CI-Specific Flags & Exit Codes