S3 Destination (AWS S3 / MinIO)

Config block

destination:
  type: s3
  bucket: my-data-bucket            # S3 bucket name (must already exist)
  prefix: exports/daily/            # optional key prefix (folder-like path)
  region: us-east-1                 # AWS region (required for AWS S3)

Credentials

Rivet uses OpenDAL for S3 access. Credentials are resolved in this order:

Option 1: AWS default credential chain (recommended)

If you’re running on EC2, ECS, Lambda, or have ~/.aws/credentials configured, just set region:

destination:
  type: s3
  bucket: my-data-bucket
  region: us-east-1

Option 2: Environment variables

Set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY before running:

export AWS_ACCESS_KEY_ID=AKIA...
export AWS_SECRET_ACCESS_KEY=wJa...
rivet run --config export.yaml

Or reference them in the config:

destination:
  type: s3
  bucket: my-data-bucket
  region: us-east-1
  access_key_env: AWS_ACCESS_KEY_ID
  secret_key_env: AWS_SECRET_ACCESS_KEY

Option 3: AWS profile

destination:
  type: s3
  bucket: my-data-bucket
  region: us-east-1
  aws_profile: my-profile           # uses [my-profile] from ~/.aws/credentials

S3-compatible endpoints (MinIO / local emulators)

For a local S3-compatible emulator, add endpoint. A loopback endpoint (localhost / 127.x / ::1 — the MinIO case) is accepted as-is:

# MinIO (loopback — no extra flag needed)
destination:
  type: s3
  bucket: rivet-exports
  endpoint: "http://localhost:9000"
  region: us-east-1                 # required but can be any value
  access_key_env: MINIO_ACCESS_KEY
  secret_key_env: MINIO_SECRET_KEY

Non-loopback S3-compatible services (R2, Wasabi, B2) — not supported

Rivet rejects any non-loopback custom endpoint at config load, by design: a committed custom endpoint silently redirects every upload (a data-exfiltration / cleartext-credential risk), so only loopback emulators are accepted with credentials. Cloudflare R2, Wasabi, Backblaze B2 and similar services all require a non-loopback endpoint and are therefore not a validated Rivet destination — Rivet has never been tested against them. (allow_anonymous: true technically waives the endpoint guard and static keys are still used to sign, but that flag is meant for anonymous emulators; the combination is untested against real S3-compatible services and may break without notice. If you depend on it anyway, verify the full upload path — parts, manifest.json, _SUCCESS, and a re-read — yourself.)

Output keys

Files are uploaded as:

s3://{bucket}/{prefix}{export_name}_{YYYYMMDD}_{HHMMSS}_{mmm}.{format}

This is the single (non-chunked, non-keyset) runner’s naming: the timestamp includes a millisecond field, and its size-split parts append _part{N} before the extension. Chunked runs name parts {export}_{timestamp}_chunk{N}_{nonce}.{format} and keyset runs key part names off the run id — see the per-runner naming table in docs/cloud-destinations.md.

Example: s3://my-data-bucket/exports/daily/orders_20260406_120000_123.parquet

Streaming upload

Rivet streams data directly to S3 without buffering the entire file in memory. Peak RSS stays proportional to batch_size, not to the total export size.

Verify

rivet doctor --config export.yaml

Output:

[OK]  Destination S3(my-data-bucket)

Doctor labels the destination as S3(<bucket>); a passing check prints no detail suffix.

Troubleshooting

NoSuchBucket – The bucket must already exist. Create it first: aws s3 mb s3://my-data-bucket.

AccessDenied – Check IAM policy. Rivet needs s3:PutObject and s3:GetBucketLocation.

SignatureDoesNotMatch with MinIO – Ensure region is set (even for MinIO, e.g. us-east-1).

Keyboard shortcuts

Rivet