6.3.15 Multi-Region Remote Execution

Putting executors in several regions does not automatically make remote builds regional. The scheduler may still send an action to distant workers, its inputs may live in another CAS, and the consumer may have to fetch the outputs back across the same link. A multi-region design must place work and data together for the journeys it protects, then state what happens when a link or an entire region disappears.

This is a Bazel-specific placement problem built on ordinary distributed-systems design. 6.3.7 Remote Execution Storage supplies the publication invariant: the input closure must be available before execution, and a published result must refer to reachable output data. Here we decide in which region those conditions are satisfied and how they remain true during a partition or failover.

Start with the portable boundary

Bazel and the Remote Execution API give a deployment useful identities and contracts: an action and its Merkle input root, content digests, an instance name, remote service endpoints, and opaque platform properties. REAPI does not define geographic regions, choose the nearest one, replicate AC or CAS data, guarantee residency, or prescribe regional failover.¹

Keep the portable surface separate from the regional implementation:

A property such as region=europe-west is therefore not portable regional routing. It becomes meaningful only when a versioned backend configuration maps that value to a bounded compatible destination and visibly rejects or handles unsupported values. The translation and placement proof follow 6.3.2 Remote Executor Matching; adding a region does not relax OS, architecture, toolchain, image, or isolation compatibility.

An instance name is not a residency control either. It can select a logical namespace, but authorization, physical storage placement, replication, and deletion are deployment promises. Prove them through backend configuration, access tests, and storage observations rather than inferring them from the string sent by Bazel.

Choose a topology from the complete action journey

Draw one representative action from its client to its final consumer. Mark the client region, the selected frontend and scheduler, every CAS and AC read or write, the executor region, and the place where outputs are consumed. Then add the inter-region links explicitly. “Nearest executor” is useful only when the required inputs, compatible capacity, result publication path, and output consumer are also sufficiently near.

Four topology shapes expose different trade-offs:

The last two shapes are easy to confuse. A read-through edge can improve cache latency while still sending execution inputs to a central worker fleet. A self-contained regional island can execute against its own source of truth, but cannot silently borrow remote workers during a partition. Dropbox's published case calls the latter a “super-edge” and contrasts it with an edge cache whose central execution path needs additional completeness checks.²

That case also shows a paired-metro pattern: local reads, synchronous mirrored writes, remote lookup on a local miss, and repair back into the local region. Treat those choices as one measured implementation, not as REAPI defaults. In particular, synchronous mirroring moves distance into the write path; asynchronous replication moves it into freshness and failover uncertainty. There is no universally correct choice without the workload's latency, availability, and data-loss requirements.

Buildbarn's Kubernetes deployment provides inspectable frontend, storage, scheduler, runner, and worker roles that can be assigned to failure domains, but it does not itself define a multi-region placement or replication policy.³ This is the right way to use an implementation topology: map concrete components and owners, then add and test the regional contract instead of reading geography into a Kubernetes manifest.

Make locality a measured placement decision

For each important action class, score a candidate region using the whole path:

• where the client, source upload, and current input closure are located;
• whether compatible and matchable executor capacity is available, including protected headroom after a region loss;
• expected AC lookup, CAS transfer, queue, execution, output upload, and local or downstream materialization time;
• which consumers need the result and whether they can read it under their own identities;
• residency and trust constraints on inputs, outputs, logs, and execution; and
• inter-region transfer, replicated storage, idle headroom, and operational cost.

6.3.12 Remote Execution Capacity explains why nominal executors or average CPU are not usable supply. In a regional design, calculate supply again after removing one region and its links. A failover destination that lacks the correct worker class, image, device, memory, or CAS access is not capacity for the affected actions.

Compare matched cohorts rather than two conveniently available builds. Hold the revision, target scope, configuration, platform and toolchain, action mix, cache temperature, client behavior, and output policy constant. For both the local and cross-region paths, retain:

1. AC lookup latency and whether the result was local, replicated, or fetched remotely;
2. input bytes and metadata served locally versus across the link;
3. queue and compatible-placement time;
4. command, output-upload, and result-publication time;
5. output transfer or materialization to the real consumer; and
6. cross-region bytes, tail latency, errors, retries, and total service cost.

A shorter executor distance can lose if inputs and outputs cross the region boundary. A slower remote miss may be acceptable when it leads to necessary execution, while a small delay repeated across a large cached action graph can dominate an otherwise warm build. The Dropbox case observed precisely this asymmetry between its latency-sensitive cached path and its more tolerant executed path.²

Specify partition and region-loss semantics before failover

Replication is a mechanism, not a failure contract. For every regional state population, state which copy is authoritative, when a write is acknowledged, what lag is allowed, what remains readable during a partition, and how divergent or incomplete state is reconciled. Apply the rule separately to CAS blobs, AC results, Operations, queues or leases, and audit evidence; they do not have the same durability or recovery semantics.

In particular, an AC entry must not become usable in a region before every CAS object needed by that result is reachable through the promised path. A replicated digest proves the identity of bytes, not that every region currently stores or may disclose them. Preserve the action, result, and blob digests when a failover exposes a miss or incomplete result.

Choose a degraded action policy explicitly:

• stay regional and queue or reject until local service returns;
• fail closed when residency, trust, or release policy forbids movement;
• execute in another region only after proving compatible capacity, input availability, authorization, and allowed data movement; or
• fall back locally only for action classes whose correctness and trust contract permits it.

Failover may create a second attempt while the first is still running or merely unobservable. Preserve Operation and backend-attempt identities, bound retry amplification, and define which terminal result may be accepted. Those attempt and cancellation rules belong to 6.3.13 Remote Action Failure Handling; geography does not make duplicate work safe.

Qualify the design with loss tests

Run the normal local and remote paths first, then inject one boundary loss at a time. At minimum test an inter-region partition, a lagging or missing CAS object, loss of the regional scheduler, and complete loss of one execution region. For each test, declare beforehand:

• which clients and action classes remain admitted;
• which data may cross the boundary and which must remain resident;
• which capacity and latency objective is protected at reduced scale;
• how queued and running attempts become known, failed, or explicitly unknown;
• whether AC/CAS reads and result publication remain complete; and
• how service returns to normal without accepting stale ownership or creating a cross-region retry storm.

Prove recovery with fresh clients in each affected region and with the real output consumers, not only service health checks. Planned topology changes use the drain, canary, and rollback discipline in 6.3.17 Remote Execution Upgrades. A coupled production incident across cache, execution, network, and regions continues in 6.6.6 Build-Service Incident Recovery; the whole deployment's qualification record is assembled in 6.3.18 Remote Execution Production Readiness.