3.1.10 Cloud Storage Dependencies

When dependency bytes already live in cloud object storage, you do not have to force them through generic HTTP rules or shell out from a genrule. A storage-specific ruleset can keep the fetch inside Bazel's repository machinery, so the downloader, repository cache, and credential-helper flow still apply.^1,2 A concrete provider-specific example is rules_gcs, so the examples below use GCS. The reusable maintainer pattern is "fetch through Bazel-aware repository logic, not ad hoc scripts."^1,2,3 That keeps this topic separate from 3.1.9 Offline / Air-Gapped Builds, which is about portability and no-network operation, and from 6.2 Shared Remote Cache, which is about sharing build outputs rather than dependency sources.

Use cloud storage rules when the bucket is the source of truth

rules_gcs gives GCS-native versions of the familiar HTTP repository rules: gcs_file for a single file and gcs_archive for an archive Bazel should download and extract.¹ They take gs://... URLs, translate them to HTTPS internally, and still go through Bazel's downloader instead of bypassing it.¹ That matters because Bazel's repository-rule API reuses the repository cache when sha256 or integrity is provided, then stores the downloaded file there after a successful fetch.⁴

gcs_archive = use_repo_rule("@rules_gcs//gcs:repo_rules.bzl", "gcs_archive")
gcs_archive(
    name = "magic",
    url = "gs://my_org_code/libmagic.tar.gz",
    sha256 = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
    build_file = "@//:magic.BUILD",
)

Use this pattern when the bucket is where the dependency is actually published: internal tool archives, test assets, or third-party blobs mirrored into your own storage.¹ If what you really need is better resilience for ordinary registry or HTTP downloads, the right tools are still mirrors, registry policy, and downloader config from 3.1.6 BCR Infrastructure rather than a new storage-specific ruleset.

One object is simple. Many objects are where the pattern pays off

The interesting part is not one archive. It is a bucket with many objects where most builds touch only a small subset. rules_gcs handles that with the gcs_bucket module extension: a hub repository reads a lockfile, then Bazel creates per-object repositories and fetches them only when a build actually needs them.¹ That directly contrasts with the "download everything up front" failure mode discussed in 3.1.5 Eager Fetch Anti-pattern.¹

gcs_bucket = use_extension("@rules_gcs//gcs:extensions.bzl", "gcs_bucket")
gcs_bucket.from_file(
    name = "trainingdata",
    bucket = "my_org_assets",
    lockfile = "@//:gcs_lock.json",
)

gcs_bucket can expose those objects through symlinks, aliases, copies, or an eager mode, but the maintainer decision is simpler: use the bucket extension when you need many independently addressed objects and want network traffic to stay proportional to what the build actually used.¹ If you want the mechanics underneath that design, 4.10 Authoring Module Extensions explains why extensions aggregate configuration and then call repository rules, while 4.9 Repository Rules covers the downloader-facing implementation model.

Private buckets need downloader-aware authentication

The fragile alternative is fetching bucket content outside Bazel, for example with wget inside a genrule. That approach falls apart for concrete reasons: Bazel cannot see the network edge for bazel fetch or bazel vendor, credentials end up in build logic, and you give up the downloader's normal caching behavior.² Cloud-storage rules work better because they keep the fetch inside repository rules or module extensions, where Bazel can still reason about dependency downloads.^1,2

For private GCS buckets, the concrete setup is a credential helper scoped to storage.googleapis.com in .bazelrc.¹ Bazel's helper configuration also supports exact host matches, wildcard host matches, and a default fallback helper, including S3-style host patterns such as bucket.s3.us-east-1.amazonaws.com or *.s3.us-east-1.amazonaws.com.³ Put differently: keep authentication in 3.2.1 .bazelrc Hierarchy, not in BUILD snippets or shell wrappers.

# GCS
common --credential_helper=storage.googleapis.com=%workspace%/tools/credential-helper

# S3-style hostname scoping
common --credential_helper=*.s3.us-east-1.amazonaws.com=/path/to/helper/for/aws-s3

The upstream adoption path keeps three separate boundaries visible: examples/tweag-credential-helper/ shows the helper selecting a local credential identity and returning request headers.^3,5 The helper does not grant object access: it supplies request credentials, and the storage service separately authorizes or rejects access. The examples/full/gcs_lock.json and its hash-check target independently verify that the downloaded bytes match the pinned content identity.⁵ Review and commit lockfile changes separately so that authorized access and approved, integrity-pinned bytes remain distinct.

The GCS example is the concrete one here, but the lesson generalizes cleanly: the storage backend may change, yet the Bazel-facing shape stays the same. Fetch through repository logic, scope credentials to the host that needs them, and keep hashes on the downloaded objects so the repository cache can do useful work.^1,3,4

The S3 counterpart shows where backend-specific diagnosis begins. rules_s3 documents S3-compatible endpoint selection alongside separate routes for credential-helper failures, HTTP 401/403 responses, checksum mismatches, and empty downloads.⁶ Start with its troubleshooting guide before inspecting private signing or download code: authentication, endpoint routing, and content integrity fail at different boundaries.