Cloud destination authentication

Rivet talks to S3 / GCS / Azure Blob Storage via opendal. Three supported AWS auth flows, three GCS flows, and three Azure flows are documented below, each with the exact rivet config + shell setup, plus a “what NOT to use” note for the common confused-by-AWS-CLI-v2 case.

If your auth path isn’t listed, the rivet error you’ll see most often is one of:

loading credential to sign http request, source: error sending request
for url (http://169.254.169.254/latest/api/token)

That’s the EC2 instance-metadata-service fallback — opendal didn’t find creds in the configured chain and is now trying IMDS, which is unreachable on a developer laptop or non-EC2 host. The fix is always “give opendal the right credentials before it falls through to IMDS”.

AWS S3

Path A — static IAM access key (long-lived)

The classical case: an IAM user has a long-lived (access_key_id, secret_access_key) pair (looks like AKIA...). No session token, no rotation worries. Best for CI, automation, dedicated rivet IAM users.

Shell:

export RIVET_AWS_ACCESS_KEY=AKIAxxxxxxxxxxxxxxxx
export RIVET_AWS_SECRET_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Rivet config:

destination:
  type: s3
  bucket: my-bucket
  region: eu-north-1
  access_key_env: RIVET_AWS_ACCESS_KEY
  secret_key_env: RIVET_AWS_SECRET_KEY

Path B — temporary credentials with session token (STS / SSO / IAM Identity Center / AssumeRole / MFA / IRSA)

If your access key starts with ASIA... rather than AKIA..., it’s a short-lived STS token and you MUST also pass the session token, otherwise S3 rejects every request.

This covers a lot of modern AWS setups:

• AWS IAM Identity Center / AWS Login (aws configure in AWS CLI v2 → “AWS Login”): credentials live in ~/.aws/login/cache/, not in ~/.aws/credentials.
• aws sts assume-role for cross-account access.
• MFA-protected sessions (aws sts get-session-token).
• EKS IRSA (IAM Roles for Service Accounts) / Pod identities.
• GitHub Actions OIDC / GitLab JWT-based AWS access.

Shell — bridge from any of the above to env vars rivet understands:

# AWS CLI v2 helper that prints export commands:
eval "$(aws configure export-credentials --profile default --format env)"

# Now in this shell session:
#   AWS_ACCESS_KEY_ID=ASIAxxxxxxxxxxxxxxxx
#   AWS_SECRET_ACCESS_KEY=...
#   AWS_SESSION_TOKEN=...
#   AWS_CREDENTIAL_EXPIRATION=2026-05-21T16:33:43+00:00

Rivet config — point all three env-name fields at the env vars the helper just exported:

destination:
  type: s3
  bucket: my-bucket
  region: eu-north-1
  access_key_env: AWS_ACCESS_KEY_ID
  secret_key_env: AWS_SECRET_ACCESS_KEY
  session_token_env: AWS_SESSION_TOKEN

Caveats:

• The token has a short lifetime (often 1 hour). When it expires re-run aws configure export-credentials … to refresh.
• For long-running pipelines that exceed the token lifetime, prefer Path A (static keys) or run a refresh loop in your scheduler.
• Rivet does NOT ship a daemon-mode that re-reads creds during a run — the token captured at startup is used throughout.

Path C — aws_profile (only for static-key profiles)

Rivet has a aws_profile: <name> config option that uses reqsign’s AwsDefaultLoader to read credentials from ~/.aws/config + ~/.aws/credentials.

This works only when the named profile carries plain static aws_access_key_id + aws_secret_access_key lines (the format AWS CLI v1 wrote, and AWS CLI v2’s “IAM user” mode still writes).

It does not work for AWS Login / SSO profiles that store short-lived sessions in ~/.aws/login/cache/ — reqsign 0.16’s loader doesn’t read that format and falls through to IMDS, hanging or failing with the error quoted above.

If you have an AWS Login profile, use Path B instead.

destination:
  type: s3
  bucket: my-bucket
  region: eu-north-1
  aws_profile: rivet-prod

What NOT to use

• Mixing aws_profile with access_key_env/session_token_env: the explicit env-var fields take precedence at the opendal level, but this leaves the reqsign default-chain still wired up and can trigger surprise IMDS lookups. Pick one path.
• AWS_PROFILE env var alone: rivet doesn’t read it. Either set aws_profile: in the config or use the env-var path.

Google Cloud Storage

Path A — Application Default Credentials (developer laptop)

If you ran gcloud auth application-default login, ADC writes a token to ~/.config/gcloud/application_default_credentials.json. Rivet auto-detects this and uses it transparently:

destination:
  type: gcs
  bucket: my-bucket
  prefix: exports/

No credentials_file: needed. See gcs_auth::try_authorized_user_loader in src/destination/gcs_auth.rs for the detection.

Path B — Service account JSON

For CI / production, point at a service-account key file:

destination:
  type: gcs
  bucket: my-bucket
  prefix: exports/
  credentials_file: /etc/rivet/sa.json

Or via env (GOOGLE_APPLICATION_CREDENTIALS):

export GOOGLE_APPLICATION_CREDENTIALS=/etc/rivet/sa.json

Rivet reads that key file itself and mints the access token in process — the RFC 7523 jwt-bearer grant (a claim set signed RS256 with the file’s own private_key, exchanged at https://oauth2.googleapis.com/token). The same credential is used for the GCS write and for a rivet load --target bigquery that follows, so both legs run as the SAME identity: the service account, not whatever human gcloud happens to be logged in as. Rivet logs which one it resolved at info level (GCS: using ADC service_account credentials as …@….iam.gserviceaccount.com), and BigQuery records it as user_email in INFORMATION_SCHEMA.JOBS_BY_PROJECT — check there, not in rivet’s output, if you need to prove it.

No Google Cloud SDK is needed on PATH for either leg. The one credential shape that still requires gcloud is external_account (workload identity), which needs an STS token exchange rivet does not implement.

destination:
  type: gcs
  bucket: my-bucket
  prefix: exports/

Path C — Anonymous / emulator

For fake-gcs-server / GCS emulator setups:

destination:
  type: gcs
  bucket: rivet-e2e
  endpoint: http://localhost:4443
  allow_anonymous: true

Rivet disables both VM metadata probing and the standard config-load chain when allow_anonymous: true so the emulator path works on a host that has unrelated GCS profiles configured.

Azure Blob Storage

type: azure uses Azure’s “container” terminology — the existing bucket: field carries the container name (rivet keeps a single field for the “top-level namespace inside the cloud account” across S3 / GCS / Azure).

Path A — Storage account name + account key

The primary, simplest auth flow. Account key is a long-lived secret string from the Azure portal (Storage account → Access keys → key1 or key2). Rotate it via the portal; rivet wipes the in-memory copy on drop via Zeroizing.

Shell:

export RIVET_AZURE_KEY="long-base64-key-from-portal=="

Rivet config:

destination:
  type: azure
  bucket: my-container            # Azure container name
  account_name: mystorageacct      # the `<acct>` in `<acct>.blob.core.windows.net`
  account_key_env: RIVET_AZURE_KEY

account_name is a plain string in YAML — it’s not a secret, it’s the public DNS-visible name of the storage account (same status as AWS region or GCS bucket name).

Rivet auto-derives the endpoint from account_name as https://<account_name>.blob.core.windows.net — operators only need to set endpoint: for a loopback emulator (Azurite). Both loopback shapes are accepted: with credentials (account_name: devstoreaccount1 + account_key_env holding the well-known dev key — the shape the live Azurite test in CI uses), or with allow_anonymous: true and no credentials at all (Path B below). Sovereign clouds (US-Gov, China-Mooncake) and custom DNS fronts are not currently reachable: a non-loopback Azure endpoint with credentials is rejected at config load, and allow_anonymous: true cannot be combined with credentials.

Path B — Azurite emulator / public-read containers

For local development against Azurite:

destination:
  type: azure
  bucket: rivet-e2e
  endpoint: http://127.0.0.1:10000/devstoreaccount1
  allow_anonymous: true

allow_anonymous: true skips both account_name and account_key_env. Use it only for emulators or genuinely public read-only containers; rivet will refuse to combine allow_anonymous: true with explicit credentials.

Path C — SAS token

A Shared Access Signature (SAS) token scopes access to a specific container and time window — useful when you can’t or shouldn’t hand out the full account key. account_key_env and sas_token_env are mutually exclusive; rivet refuses a config that sets both.

Shell:

export AZURE_STORAGE_SAS_TOKEN="sv=2021-08-06&ss=b&srt=o&sp=rwdlacupitfx&se=2026-06-01T00:00:00Z&spr=https&sig=..."

Rivet config:

destination:
  type: azure
  bucket: my-container
  account_name: mystorageacct
  sas_token_env: AZURE_STORAGE_SAS_TOKEN

The leading ? is stripped automatically if you paste the token straight from the Azure portal. Rivet parses the se= (signed-expiry) field at startup and fails fast on an already-expired token (and warns within 60 minutes of expiry) — see destinations/azure.md for the full preflight behaviour.

Not yet supported

These AAD-based flows are on the roadmap but not yet shipped; use Path A or Path C today:

• Service principal (tenant_id, client_id, client_secret_env) — for unattended automation.
• Managed identity — for rivet running inside Azure VMs / AKS / Functions.
• Connection string (connection_string_env) — the all-in-one DefaultEndpointsProtocol=https;AccountName=…;AccountKey=… blob.

To bridge a connection string today, extract the AccountKey value into an env var and use Path A.

S3-compatible storage (MinIO)

Same as AWS Path A above + an explicit endpoint: URL. Static keys only — STS / temporary credentials are an AWS-specific concept.

destination:
  type: s3
  bucket: rivet-test
  endpoint: http://localhost:9000
  region: us-east-1
  access_key_env: MINIO_ACCESS_KEY
  secret_key_env: MINIO_SECRET_KEY

Cloudflare R2, Wasabi, Backblaze B2 etc. are not supported: they require a non-loopback endpoint:, which Rivet rejects at config load (a committed custom endpoint redirects every upload — an exfiltration guard). Only the loopback MinIO shape above is a validated custom-endpoint path. Rivet has never been tested against R2 / Wasabi / B2; allow_anonymous: true technically waives the endpoint guard (static keys still sign), but the flag targets anonymous emulators and the combination is unvalidated — use it at your own risk, and verify the upload end-to-end if you do.

Troubleshooting

Recommended setups by use case

• Local dev → MinIO: Path A static keys, endpoint: http://localhost:9000.
• Local dev → real AWS S3: Path B (export creds via aws configure export-credentials …).
• CI / GitHub Actions → real AWS S3: Path B with OIDC-issued temporary creds (set the env vars from the GitHub aws-actions/configure-aws-credentials step output).
• Production / Airflow / Dagster → S3: Path A with a dedicated IAM user, key rotation handled by your secret store.
• Local dev → real GCS: Path A with gcloud auth application-default login.
• Production → GCS: Path B with a service account JSON.
• Local dev → Azurite: Azure Path B (allow_anonymous: true, endpoint: http://127.0.0.1:10000/devstoreaccount1).
• Production → Azure Blob Storage: Azure Path A with account_key_env sourced from your secret store (Key Vault, doppler, sops, etc.), or Path C with a scoped SAS token. Service Principal / Managed Identity are not yet supported.