6.3.4 Remote Action Tracing

A dashboard can label 600 milliseconds as “remote execution” while hiding the question you actually need to answer: was the action waiting for admission, queued for a compatible executor, preparing inputs, running its command, or publishing its result? Following one action means joining those transitions without pretending that every backend exposes the same stages.

Begin with the fresh remote branch from 6.1.4 Action and Result Lifecycle. The action class has already passed the readiness work in 6.3.1 Making Actions Work Remotely, its compatible executor contract is known from 6.3.2 Remote Executor Matching, and a pinned backend has passed 6.3.3 Choosing a Remote Execution Backend. We can therefore trace one successful attempt instead of reopening cache, local-execution, or product-selection decisions.

This trace stops at the remote action's service and executor boundaries. If the remaining question is inside the action's tool or process, continue with the private, versioned instrumentation boundary in 6.6.2 Action-Internal Telemetry rather than inventing finer REAPI stages.

One successful remote action, joined across four owners

Read left to right by responsibility and top to bottom by time. A colored observation says who can supply that evidence.

This swimlane timeline follows one successful remote execution. The client first confirms the action closure is ready, then sends Execute and receives an accepted non-terminal Operation. A backend-specific record admits the action, queues it, finds a compatible pool, and dispatches it with a lease to worker-23. The executor prepares inputs, runs the command, uploads outputs, and the client finally observes a terminal ExecuteResponse. Cyan observations are portable REAPI fields or messages. Gold observations are backend-specific scheduler and lease records. Blue observations name CAS storage activity. Green observations are executor work. Each timestamp belongs only to the owner that reported it.

portable REAPI observation backend-specific observation CAS storage activity executor work

This is one normalized successful attempt. Empty lanes mean this trace has no observation there, not that the subsystem did no work.

Time and transition

Client

Storage

Backend control

Executor

12:00:00.000ZReadiness

Action closure readyRecords the action digest before Execute

CAS availableAction, Command, and input root are present

00.010 to 00.015ZExecute and acceptance

Execute sentClient submits the action digest

Operation acceptedNon-terminal Operation has a name; queue state is not yet established

12:00:00.018ZAdmission and queue

Queue state observedQUEUED with queued_timestamp

Admitted to a queueBackend joins the operation to linux-x86_64-compile

12:00:00.032ZMatch

Compatible pool foundScheduler records a bounded matching result

00.041 to 00.042ZDispatch and lease

Assigned with a leaseBackend records exclusive lease acquisition

Worker receiptworker_start_timestamp records receipt time; backend assignment and ExecutedActionMetadata.worker name worker-23

00.044 to 00.084ZPrepare inputs

Inputs fetchedCAS supplies the declared closure

Preparation intervalinput_fetch_*_timestamp bounds input work

00.090 to 00.290ZRun command

Coarse progressEXECUTING may be reported while the worker runs

Command runtimeexecution_*_timestamp records 200 ms here

00.292 to 00.322ZUpload outputs

Outputs and logs uploadedCAS receives result-referenced digests

Upload intervaloutput_upload_*_timestamp bounds this work

00.325 to 00.330ZTerminal response

Terminal ExecuteResponsedone = true, OK, cached_result = false

Pipeline completedBackend may record executor completion at .325Z

Worker finishesworker_completed_timestamp ends executor work

CONTROL WAIT

Queue to worker receipt

Backend interval. The match and lease split is known only when this backend records it.

INPUT PREPARATION

Fetch start to fetch complete

Do not call this queue time or command runtime.

COMMAND RUNTIME

Execution start to execution complete

A timestamp-defined command interval, not the whole remote duration.

OUTPUT UPLOAD

Upload start to upload complete

Upload completion and terminal client observation are different boundaries.

Portable protocol surface: Remote Execution API Scheduler and lease rows: pinned backend evidence

A successful trace is not one “remote time.” Keep portable Operation evidence separate from backend scheduling evidence, and subtract timestamps only within a known clock domain.

Keep protocol state and backend work separate

REAPI gives the client a deliberately thin execution surface. Before calling Execute, the client places the Action, Command, and input closure in CAS and submits the action digest. Execute returns a stream of long-running Operation messages. An in-progress operation carries ExecuteOperationMetadata; its terminal response carries an ExecuteResponse with the ActionResult, execution status, and cached_result flag.¹

The portable stage vocabulary is only UNKNOWN, CACHE_CHECK, QUEUED, EXECUTING, and COMPLETED. These are coarse current states, not a complete event log. REAPI does not define ACCEPTED, ADMITTED, MATCHED, DISPATCHED, LEASED, PREPARING, UPLOADING, or PUBLISHED stages. It also does not require servers to report the standard stages in numerical order. COMPLETED is special: if a server reports it, the Operation must have done = true and the stream must end.¹

Admission, queue ownership, matching, dispatch, leases, runner preparation, and executor identity are backend observations. For example, the documented BuildBuddy implementation queues an action on selected executors; an executor that can take the work obtains an exclusive lease. The same implementation describes container-image preparation and input fetching as cold-start work that can dominate the command itself.² Those facts make a useful bounded example, but they are not a REAPI architecture. Other maintained systems expose different service and worker splits.^3,4

One successful joined trace

The trace below is a normalized teaching record, not copied command output. The clock values are illustrative so we can practice attribution; the message fields and portable states come from REAPI, while the admission, match, and lease rows are explicitly backend-owned observations. For the arithmetic below, assume a tracing system has normalized every displayed timestamp onto one established UTC clock domain. In a real investigation, retain the actual values, pin the client, backend, configuration, and instance, and label each clock source before joining or subtracting timestamps.

This trace follows one representative compile action. Its identity card is:

Identity	Role in this trace	What it is not
Action digest	Identifies the declared Action content referenced by ExecuteOperationMetadata	An execution-attempt ID; one action may be executed more than once
Operation.name	Follows the operation returned by Execute and is the handle for WaitExecution	A universal scheduler, lease, or worker ID
Request metadata action_id	When sent and recorded, joins this action's CAS, Action Cache, and Execution RPCs	Guaranteed to exist or to match a backend's internal trace ID
Backend assignment or lease correlation	Joins scheduler and executor-control evidence when this pinned implementation exposes it	A portable REAPI field or guaranteed identifier
ExecutedActionMetadata.worker	Names the worker that reports the completed execution, when populated	Proof of its image, isolation, or compatibility by itself

REAPI defines the optional request-correlation fields and the relationship between an operation name and action digest. It does not promise that every client sends them or every backend records them.¹ Keep the identities typed rather than collapsing all five into an ambiguous “action ID.”

Time	Transition and evidence	Owner	What this observation establishes
12:00:00.000Z	Action, Command, and input-root digests are present; client records CAS availability and its action digest.	Client + CAS	The declared closure is ready for Execute, not that any executor has accepted it.
12:00:00.010Z	Client sends Execute; at 12:00:00.015Z it receives a non-terminal Operation with a name and the same action digest.	Client + REAPI Execution service	The service created or accepted responsibility for an operation. This observation alone proves neither queue entry nor placement.
12:00:00.018Z	queued_timestamp is populated; the Operation reports stage QUEUED, and a backend admission record joins the operation/action to queue class linux-x86_64-compile.	REAPI metadata + backend scheduler	The action entered a wait for a machine and this backend's scheduling path. Queue name and admission checks are not REAPI stages.
12:00:00.032Z	Backend matching record identifies the bounded compatible pool selected from the request properties.	Backend scheduler	The backend found a compatible destination set. It has not yet assigned a concrete executor.
12:00:00.041Z	Backend assignment record names executor worker-23 and records exclusive lease acquisition; REAPI metadata reports worker_start_timestamp = 12:00:00.042Z.	Backend scheduler + executor control	A concrete worker received the action. The lease is implementation evidence, not a standard Operation state.
12:00:00.044Z–12:00:00.084Z	input_fetch_start_timestamp to input_fetch_completed_timestamp; the executor's pinned record covers its preparation work.	Executor + CAS	The worker prepared the action inputs. The interval is not queue wait or command runtime.
12:00:00.090Z–12:00:00.290Z	execution_start_timestamp to execution_completed_timestamp; Operation may report EXECUTING.	Executor + REAPI observation	The action command ran for 200 ms in this timestamp domain. EXECUTING alone is coarser and may also cover worker-side activity.
12:00:00.292Z–12:00:00.322Z	output_upload_start_timestamp to output_upload_completed_timestamp; result records output and log digests.	Executor + CAS	Output capture/upload completed for this trace. It does not by itself prove safe publication ordering or future retention.
12:00:00.325Z	worker_completed_timestamp; backend records completion of its executor pipeline.	Executor	The worker finished all reported stages, not that the client has observed the result.
12:00:00.330Z	Client observes the same Operation with done = true, an ExecuteResponse whose status is OK, cached_result = false, and its ActionResult; stream ends.	REAPI Execution service + client	The client observed a terminal result from execution rather than a cache-served result. It does not prove local output materialization.

The timestamp fields from queue entry through output upload are optional fields of ExecutedActionMetadata; absence means unknown, not zero duration.¹ Likewise, the scheduler timestamps exist only if the chosen backend records and exposes them. A maintained NativeLink integration test can prove that an uncached test used Bazel's remote strategy against one real implementation, but it does not expose this complete state-by-state record.⁵ Preserve that distinction instead of filling telemetry gaps with plausible events.

If the Execute stream remains connected, the server streams progress as if the client had called WaitExecution. WaitExecution takes the returned operation name, immediately reports current state, and keeps its stream open through completion; it may emit additional updates. The standard long-running Operations service also offers GetOperation for a point-in-time poll of that operation name.¹ Streaming or polling lets you continue following the same operation, but it does not manufacture scheduler or executor evidence that the backend never recorded.

Attribute intervals, not one “remote time”

Only subtract timestamps from the same clock domain or from a tracing system whose clock relationship is established. In the normalized record, the backend-generated times support these separate intervals:

• Queue to worker receipt: worker_start_timestamp - queued_timestamp. This includes backend waiting up to worker receipt; it is not process time.
• Input preparation: input_fetch_completed_timestamp - input_fetch_start_timestamp.
• Command runtime: execution_completed_timestamp - execution_start_timestamp.
• Output upload: output_upload_completed_timestamp - output_upload_start_timestamp.

virtual_execution_duration, when present, is another protocol-defined command duration. REAPI explicitly gives it no required relationship to the execution start/end timestamps, so report it by name rather than forcing it to reconcile with wall-clock subtraction.¹

The match and lease observations refine the queue-to-worker interval only because this example backend records them. If your evidence has queue entry and worker receipt but no match or lease timestamp, say “24 ms from queue entry to worker receipt; internal split unknown.” Do not allocate the gap by looking at an aggregate remote-duration graph.

Stop where the terminal evidence stops

This successful trace names output upload and terminal client observation only far enough to attribute time. Whether a backend separately published an ActionResult, whether every referenced blob was reachable before an ActionResult became usable, and what happens after a partial upload, belong to 6.3.7 Remote Execution Storage. Whether Bazel, an IDE, CI, or another consumer later downloads or materializes those outputs belongs to 6.3.11 Remote Output Availability. The process finishing, the worker completing, and the client observing a terminal operation are three different boundaries.

Queue admission and a lease are likewise observations here, not a scheduling policy. Fairness, priorities, quotas, backpressure, and overload are developed in 6.3.8 Scheduling Actions on Remote Executors. Executor images, isolation startup, and cleanup are developed in 6.3.9 Remote Executor Environments and Isolation. The next article, 6.3.5 Remote Execution Architecture, places the responsibilities in a deployable topology instead of treating this logical path as one.

Finally, one clean trace proves only that this pinned action reached these observed transitions. It does not establish cancellation, lease loss, retries, late completion, or fallback; those begin in 6.3.13 Remote Action Failure Handling. Nor does it qualify a fleet, which requires the broader gates in 6.3.18 Remote Execution Production Readiness.