Cloud Destinations

Single-page tour of every destination Rivet ships with: what they have in common, where they differ, and what guarantees you can build downstream infrastructure on top of.

For the per-backend deep dive, read the dedicated pages:

For the credential matrix (env vars, profile files, identity providers), see docs/cloud-auth.md.

The supported destination set is exactly: AWS S3, Google Cloud Storage, Azure Blob Storage, the local filesystem, stdout — plus their loopback dev emulators (MinIO for S3, fake-gcs-server for GCS, Azurite for Azure), which CI exercises. Other S3-compatible services (Cloudflare R2, Wasabi, Backblaze B2), sovereign clouds, and custom-DNS endpoints are untested and not supported; the config-load endpoint guard rejects them by design.

Common output contract

Every non-streaming destination (Local, S3, GCS, Azure) produces the same three artefacts at the resolved prefix on a clean run:

The contract is atomic at write boundaries, not at the prefix: manifest.json is written before _SUCCESS, so an Airflow / CI sensor that polls for _SUCCESS never sees a half-built manifest. A HEAD _SUCCESS is cheap enough that downstream consumers should prefer it over GET manifest.json for the “data ready?” signal.

Schema and resume semantics:

• schema.json is reserved for a future release (per-run schema snapshot — see the roadmap). Today the schema fingerprint lives inside manifest.json under schema_fingerprint.
• _quarantine/<run_id>/ lands on resume when M9 finds an untracked or corrupt part. The original byte location is preserved under that subtree for forensics.

Authentication modes

Local

No credentials. Permissions come from the OS. Use path: (with optional {date}/{table}/{export}/{run_id} placeholders) to point at the output directory.

Amazon S3

region: is optional when the SDK can derive one from the profile or env vars; required otherwise. endpoint: overrides the resolved S3 endpoint. Only a loopback endpoint (MinIO) is a supported custom-endpoint path; any non-loopback endpoint (AWS GovCloud, Cloudflare R2, Wasabi, custom domains) is rejected at config load as an exfiltration guard. The one waiver is allow_anonymous: true — the anonymous-emulator escape, which sends no credentials at all; it is not an auth path for those services, which remain untested / not supported as Rivet destinations (see cloud-auth.md, “S3-compatible storage”).

Google Cloud Storage

bucket: is required. endpoint: overrides the default storage.googleapis.com.

Azure Blob Storage

account_key_env and sas_token_env are mutually exclusive — picking both is refused at config-load time with a message that names both fields. account_name is the prefix in <account>.blob.core.windows.net; an explicit endpoint: takes precedence over the derived URL, but only a loopback emulator endpoint (Azurite) is accepted alongside credentials — a non-loopback endpoint is rejected at config load unless allow_anonymous: true, which itself cannot be combined with credentials, so sovereign clouds are not reachable.

The Azure SAS-token body may be pasted with or without the leading ? — Rivet trims it transparently so sv=…&sig=… and ?sv=…&sig=… are both accepted.

Future Azure modes (Managed Identity, Service Principal, workload identity federation) are on the roadmap but not yet supported.

Manifest + _SUCCESS

The trust contract is the same on every cloud backend. See ADR-0012 for the formal invariants:

• M1: parts before manifest.
• M2: _SUCCESS carries the fingerprint of the exact manifest.json bytes — fingerprint drift means something else wrote that prefix.
• M5: with _SUCCESS present, every listed part exists at recorded size and content fingerprint.
• M6: legacy prefixes (no manifest.json) are surfaced as legacy_run: true; rivet validate returns success without certifying.
• M8: resume against a _SUCCESS-marked prefix is refused without --force; the verifier wants the operator to opt in to re-exporting over a completed dataset.
• M9: untracked or corrupt parts encountered on resume are moved under _quarantine/<run_id>/ rather than deleted.

Resume behavior

rivet run --resume walks the destination prefix, cross-checks against the state DB and the prior manifest, and decides per-chunk:

• skip — chunk already committed.
• rewrite — chunk was in-progress (or its part is missing); re-export the same key range.
• quarantine — untracked or corrupt object at the chunk’s part path; move to _quarantine/<run_id>/ and re-export.

Resume preserves both manifest.json and _SUCCESS only after the run finishes cleanly. An interrupted resume leaves the prior _SUCCESS in place so an external sensor polling _SUCCESS still sees the most recent verified dataset.

Validate behavior

rivet validate is the standalone counterpart to rivet run --validate (see docs/destinations/ for the per-backend nuance). It never queries the source: only HEAD / GET _SUCCESS / GET manifest.json against the resolved destination.

Validate flags:

• --date YYYY-MM-DD — resolve {date} against the given day instead of today. The flag a “did yesterday’s run land cleanly?” Airflow sensor needs.
• --run-id <RID> — substitute {run_id} in the destination template (composes with --date).
• --prefix <STRING> — bypass placeholder resolution and verify exactly that prefix. Refused with multiple exports — see the inline error.

The resolved physical prefix is surfaced in both --format pretty and --format json output (resolved_prefix) so it’s obvious which bytes were checked.

Reconcile and repair

Both honor the same placeholder resolver as run and validate.

Quarantine behavior

Path layout under the destination prefix:

<prefix>/
  part-000001.parquet
  part-000002.parquet
  manifest.json
  _SUCCESS
  _quarantine/
    <run_id>/
      part-XXXXXX.parquet   ← evicted object kept verbatim

Quarantine is best-effort: a successful copy followed by a failed delete leaves the object reachable at both paths. M9 re-trips on the next resume and the orphan eventually gets moved.

Support matrix

Known limitations

• Object lifecycle policies — Rivet does not configure retention, lifecycle transitions, encryption-at-rest, or replication rules on the destination. Manage those out-of-band (Terraform, console).
• Incomplete multipart uploads are not aborted on failure — when a streamed (large-part) upload fails mid-transfer, the multipart upload is left open: its already-uploaded parts are billed but invisible to listings, and Rivet currently has no abort call on that error path (a crash could never run one anyway). Configure an abort-incomplete-multipart-upload lifecycle rule on every destination bucket (S3: AbortIncompleteMultipartUpload, e.g. 7 days; GCS/Azure: the equivalent incomplete-upload cleanup) — this is load-bearing hygiene, not an optimization.
• Azure SAS expiry — SAS tokens are short-lived by design. Rivet reads the env var once at process start; a long-running export whose token expires mid-flight will fail at the next write. Pair short SAS validity with appropriately small exports, or use account-key auth.
• Eventual consistency on first list — S3 / GCS / Azure are read-after-write consistent for new objects but list operations can lag on some backends. This affects --resume only, which uses list_prefix; the validate path uses targeted HEAD requests and is consistent.
• Network egress costs — Rivet does not enforce a destination / source region affinity. Exporting a 100 GB table from us-east-1 Postgres to an eu-west-1 bucket goes the long way around at full egress price. Pin source and destination to the same region for any non-trivial workload.

Reporting trust-contract issues

A trust-contract violation is treated as a security-grade bug. See SECURITY.md for the disclosure channel. Examples:

• _SUCCESS present, but a part listed in manifest.json is missing.
• _SUCCESS fingerprint disagrees with the bytes of manifest.json.
• manifest.json references the wrong schema_fingerprint for the parts at the prefix.
• A credential ever appearing in any artefact (manifest.json, summary.json, journal events, log lines).