6.3.14 Multi-Tenant Remote Execution

Shared remote execution becomes multi-tenant when one team's build can no longer read another team's data, publish a result another team trusts, reach its credentials, consume its protected capacity, or inspect its evidence. That is an end-to-end containment promise, not a name on a request. An instance name, queue, container, and login can each help, but any one of them leaves other paths open.

6.2.3 Securing a Shared Remote Cache established the cache question: who may bind an action identity to a reusable result? 6.3.9 Remote Executor Environments and Isolation established that a worker boundary must contain files, processes, authority, and residue. Here, connect those decisions from the client identity to the result a consumer can see.

Define the tenant promise along the whole action path

Start by naming a tenant in terms the service can check: an authenticated principal, its workload or CI lane, and the data and capacity population it is allowed to use. Then make every transition answer the same question: which tenant is acting, and what is it allowed to read, write, consume, or observe?

Path	Tenant control to specify	Evidence that proves the control is active
Client admission	Authenticate the calling principal and authorize the requested execution, cache, and evidence operations.	Accepted and denied request records include principal, operation, policy revision, and tenant correlation.
Namespace and cache	Map authorized requests to an instance or namespace; separately enforce read visibility and trusted writer authority for AC/CAS data.	A cross-tenant read and write attempt is denied, while an allowed request reaches only its intended population.
Scheduling	Bind the admitted tenant to compatible pools, an admission limit or quota, a fairness rule, and explicit defer/reject behavior.	Per-tenant, per-compatible-pool queue age, dispatches, rejections, and resource use show the stated rule under load.
Executor	Select an allowed pool and constrain filesystem, process, network, credential, and cleanup boundaries.	A hostile action cannot observe a co-tenant marker, reach a control credential, or leave state visible to the next tenant.
Result and evidence	Authorize who may fetch outputs, logs, and audit joins; retain the producer, worker, and result identities needed to contain an incident.	The wrong tenant cannot retrieve the result or evidence, while an authorized investigation can join request, attempt, publication, and access events.

An instance name chooses an instance-scoped protocol namespace. It is not an ACL until the backend authenticates the caller and enforces the permitted operations for that instance. Likewise, a scheduler queue may keep work apart while the same credentials can still read a shared cache, or a strong microVM may contain a process while the action can still call a cache-writing endpoint. The Remote Execution API defines interoperable execution, storage, and optional priority fields, but leaves authentication, authorization, queue policy, tenant identity, quotas, and fairness to a backend and deployment.¹

Keep trust roles separate

The dangerous shortcut is giving the same authority to every participant that can submit a build. A developer or untrusted presubmit may be entitled to ask for work without being entitled to publish a reusable result. A worker may need service authority to fetch inputs and upload its captured output, while the arbitrary command inside that worker must not inherit that authority.

Stripe's remote-execution case makes both halves visible: only trusted sources may upload action results, and untrusted actions running there must be prevented from influencing co-tenants or the action-cache write path.² That separation turns remote execution into a controlled producer path; it does not make every action or every queue trusted. Keep cache reader, cache writer, execution requester, executor control plane, and incident responder as distinct roles, with credentials and audit records scoped to the smallest useful operation.

Result visibility needs the same treatment. Before a client downloads an ActionResult or its blobs, decide whether its identity may read that result, its stdout/stderr, and the associated execution evidence. A result that is correctly published but downloadable by the wrong tenant is still a disclosure. 6.3.7 Remote Execution Storage supplies the publication and reachability checks; 6.3.11 Remote Output Availability covers how an authorized consumer obtains the files. This article adds the authorization boundary around both paths.

Quotas protect capacity; fairness needs a witness

Separate queues can prevent one workload from sitting directly behind another, but they do not by themselves limit how much capacity either workload occupies. For each tenant and compatible pool, state four operational rules:

1. Admission: which requests are accepted, deferred, or rejected, and the reason recorded for each decision.
2. Quota: the bounded concurrent work, resource reservation, or other consumption limit the tenant may hold.
3. Fairness: the documented service rule among eligible tenants, such as weighted or round-robin dispatch, plus the wait or starvation signal that can falsify it.
4. Backpressure: what happens when the protected share is exhausted, rather than allowing an unbounded queue to hide overload.

These are scheduler contracts, not names in Bazel configuration. 6.3.8 Scheduling Actions on Remote Executors explains the compatible-pool trace that distinguishes a policy problem from absent capacity, and 6.3.12 Remote Execution Capacity sizes the supply that those rules protect. Do not use fleet-wide CPU or a global queue average as a fairness result: a tenant can starve in one compatible pool while another pool is idle.

The public BuildBuddy routing case is one useful, explicitly bounded example. For managed executors it describes separate tenant queues on each executor, round-robin service between them, priority queues, and action profiling for resource consumption.³ Those are documented fairness/routing and resource-accounting/resource-prediction capabilities for that implementation. They support an overload or abuse investigation only when the deployment also joins them to tenant correlation, explicit thresholds, and enforcement. They do not establish its cache ACLs, credential mediation, result-evidence visibility, or a complete tenant policy; qualify those separately before treating the deployment as isolated.

EngFlow's post-mortem supplies a different control set. A single build submitted 172,000 timing-out compile actions; its action items were to reduce timeouts, provide a mechanism to cancel a specific build, and consider per-build quotas.⁴ Queue age, admission/rejection observations, per-build resource use, cancellation status, and timeout cohorts can therefore reveal or bound overload. The case does not prove a particular authentication or cross-tenant storage design. It is evidence for guardrails, not a universal definition of fairness.

The two cases deliberately leave some boxes blank. A source that documents a tenant round-robin queue has not thereby documented result access control; a source that documents a runaway-build guardrail has not thereby documented tenant identities. This is not a claim that other maintained backends lack those controls. It means a deployment may claim only the controls and signals its pinned backend version, configuration, and tests actually establish.

Prove containment with two negative tests

A green build for each tenant proves only that both can use the service. Test the boundaries that a shared service must hold under failure and abuse.

First, run a cross-tenant access test. Give tenant A a deliberately distinctive output, log marker, and cacheable result; then use tenant B's credential to attempt the forbidden cache read, cache write, execution request against A's pool, result download, and evidence lookup. Retain the denied principal, operation, namespace, policy decision, and correlation IDs. Then repeat the authorized path so a failure is not mistaken for a broken service. If a test is intentionally allowed—for example, a shared read-only dependency cache—record that as an explicit exception and verify that it cannot become a cross-tenant writer path.

Second, run an overload containment test in one compatible pool. Tenant A submits enough representative work to hit its declared limit while tenant B continues a protected workload. The pass condition is not merely that A is slower. The service must show A's admission/defer/reject decision, held quota or resource use, queue and dispatch observations, and B's measured service under the stated fairness rule. Preserve cancellations and late attempts as well: an interrupted tenant-A build that keeps consuming capacity can still starve B. 6.3.13 Remote Action Failure Handling supplies the attempt and cancellation evidence needed to make that case unambiguous.

Both tests should produce a joined record rather than separate dashboards:

Join	Why retain it
Principal, tenant correlation, authorization decision, and instance	Distinguishes a denial from a routing error and shows which policy authorized the request.
Action digest, Operation.name, backend queue or lease, compatible pool, and executor	Shows whether a request crossed admission, scheduling, and execution boundaries without treating those identifiers as one universal ID.
Action-result and output digests, publication/access decision, and evidence location	Bounds a suspected disclosure or poisoned result to consumers and data.
Quota decision, queue age, dispatch count, rejection reason, and cancellation outcome	Separates intentional backpressure from hidden starvation or orphan work.

This evidence is technical containment, not a replacement for deciding which teams deserve which shares or who can approve exceptions. Those entitlements, approval paths, and lifecycle rules are organization policy; see H.9.3 Credentials for that governance layer. The technical service must still make the policy enforceable and observable.