5.8.2 Diagnosing Sandbox Issues

When a spawn fails only under sandboxing, the difference is useful evidence: the action may have relied on an ambient filesystem or process-environment property, or the sandbox backend may have hit a platform constraint. The right response is to preserve that environment, identify the exact failing spawn, and compare declarations with what the process tried to use. Turning sandboxing off can narrow the diagnosis, but it is not the repair.

Start with one failing spawn

Reduce the investigation to one target and keep the first execution error. Then rerun it with both diagnostic flags:

bazel build //config:bad_relative_path \
  --sandbox_debug \
  --verbose_failures

2.3.4 First-Response Debugging Flags introduces these as first-response flags. The consultant-level workflow here continues from their output into retained-tree inspection and action-contract reconciliation.

--verbose_failures prints the full command line for a failed spawn. --sandbox_debug does two things in Bazel 9: it prints additional sandbox diagnostics and leaves sandbox-root contents untouched after the invocation.¹ It is deliberately a short-lived investigation mode. Retained trees consume disk, and a later Bazel invocation may clean or reuse sandbox state, so copy any evidence you need before running another command.²

Do not begin by searching every temporary directory. Read the failing command that Bazel printed. It normally begins with a cd into the action's effective working directory, for example:

(cd /.../sandbox/processwrapper-sandbox/4/execroot/_main && \
  exec env - ... /bin/bash -c '...')

That path identifies the concrete backend, sandbox instance, and action execroot. The stable navigation anchors are:

bazel info output_base
bazel info execution_root

The shared execution root reported by bazel info execution_root is not the same directory as the per-action execroot in the retained sandbox. A sandboxed action runs in a directory that mimics the shared execroot so that declared inputs and outputs keep their exec paths.³ With the default sandbox base, retained local sandboxes are commonly below <output_base>/sandbox/. An explicit --sandbox_base=PATH moves them below that path.¹ Treat the path printed for the failed spawn as authoritative rather than depending on this implementation-shaped suffix.

Compare the attempted read with the declared action

Suppose the failure ends with:

cat: config/input.txt: No such file or directory

Inspect that exact path relative to the printed sandbox execroot. Also inspect its parents: a missing file, a missing generated directory, and a dangling symlink imply different repairs. In the retained tree for the genrule sandbox-path example, the genrule setup script was present under external/bazel_tools, while config/input.txt was absent. That is evidence about this spawn's prepared view, not yet proof of why the declaration is wrong.

Now inspect Bazel's action plan:

bazel aquery --output=text //config:bad_relative_path

For the example, the relevant fields are:

Mnemonic: Genrule
Inputs: [external/bazel_tools/tools/genrule/genrule-setup.sh]
Outputs: [bazel-out/k8-fastbuild/bin/config/bad_relative_path.txt]
Command Line: ... cat config/input.txt > .../bad_relative_path.txt

The command reads config/input.txt, but the action's input set does not contain it. That mismatch is the undeclared-input diagnosis. aquery shows the post-analysis action contract. It does not prove which files the process opened at runtime. The retained tree and error supply that runtime-side evidence, while 4.4.3 Action Execution Contract explains where rule authors must declare the missing file or tool.

Common mismatches include:

• a source or generated artifact read by the command but omitted from the action inputs.
• an executable, interpreter, shared library, or helper program used from an ambient PATH instead of declared as a tool.
• a guessed source-tree path for a generated artifact whose exec path is under bazel-out/....
• an absolute host path, home-directory file, or system header that happens to exist outside the sandbox, and
• a directory expected as an input even though only specific files were declared.

Sandboxing catches many of these by presenting only the staged execroot view, but it is not proof that every real input was declared. Depending on the backend and platform, an action may still read parts of the host filesystem.² 2.3.2 Sandboxing gives the enforcement boundary. This workflow uses a failure to locate a concrete contract violation.

Use a strategy comparison as a controlled test

After capturing the sandbox evidence, run the same focused target with the sandboxed strategy and with local:

bazel build //config:bad_relative_path \
  --spawn_strategy=sandboxed \
  --action_env=SANDBOX_PROBE=sandboxed

bazel build //config:bad_relative_path \
  --spawn_strategy=local \
  --action_env=SANDBOX_PROBE=local

For an action known not to inspect SANDBOX_PROBE, the probe values give the two runs distinct action keys so a successful result from one strategy cannot satisfy the other from the action cache. The variable is visible to the process and changes its action environment, so use a fresh name and do not call it behavior-neutral without checking the tool. Keep the target, revision, Bazel version, platform, toolchain, and all other configuration and meaningful environment matched. If the sandboxed run fails while the local run succeeds, the action depends on something exposed by the shared execroot or host but absent from the sandbox view. This narrows the hypothesis. It does not establish that every such difference is an undeclared file.

Classify the counterexamples before editing the rule:

• If both strategies fail identically, investigate the tool or command rather than blaming sandbox setup.
• If only one OS-specific backend fails, inspect platform policy, namespaces, mounts, or backend fallback as well as the input set.
• If a worker differs from a one-shot sandboxed spawn, inspect worker inputs and retained process state.
• If the failure is an attempted write, network access, permission denial, or process-isolation issue, the missing-input hypothesis may be wrong.

Fix the declaration or tool invocation, then restore the sandboxed strategy and rerun without the probe salt. Do not ship --spawn_strategy=local as the fix: it hides the evidence and leaves cache correctness dependent on ambient state.

Diagnose sandbox cost separately from correctness

A slow sandboxed action is a different incident from a failing action. Record a timing profile for the representative workload, identify whether staging, execution, output extraction, or cleanup is material, and compare matched repeated runs. Do not infer sandbox overhead from input count alone: filesystem latency, input-tree shape, tool-owned caches, workers, and actual file accesses can change the result.^2,4

In Bazel 9, --reuse_sandbox_directories already defaults to true for sandboxed non-worker execution.¹ To bound its benefit, compare the default against --noreuse_sandbox_directories under the same workload and cache state. Likewise, test --sandbox_base on a faster filesystem only after the profile implicates sandbox filesystem work. A tmpfs needs enough RAM and capacity for the action's intermediate files and outputs.¹ A single talk's benchmark or a single macOS workload is evidence for that workload, not a universal speedup: published results vary from near parity with an optimized strict sandbox to large penalties when the implementation or tool state differs.^4,5

Keep correctness and performance decisions separate. First make the action contract complete. Then retain sandboxing as the baseline and accept a narrower strategy exception only when repeated measurements and the environment's correctness requirements justify it.