YAML Config Guide

📌 The exhaustive, always-current field reference is generated from the code: config-reference.md — rendered from rivet schema config (the schemars-derived JSON Schema), so it cannot drift and needs no manual verification. This page is the guide: the same options with defaults, rationale, and worked examples. For the guaranteed-current field/type/enum list, trust the generated reference.

The most-used options, grouped by section, with the why and examples.

Root

source

Connection approaches (mutually exclusive):

1. URL-based: provide exactly one of url, url_env, or url_file
2. Structured: provide host, user, database (+ optional port, password/password_env)

TLS

Example (production):

source:
  type: postgres
  url_env: DATABASE_URL
  tls:
    mode: verify-full
    ca_file: /etc/ssl/certs/rds-ca-2019-root.pem

Example (local dev only — no TLS):

source:
  type: mysql
  host: 127.0.0.1
  port: 3306
  user: dev
  password_env: DEV_PWD
  database: rivet
  tls: { mode: disable }       # explicit opt-out — silences the plaintext WARN

Example (SQL Server — sqlserver:// scheme, port 1433):

source:
  type: mssql
  url_env: MSSQL_URL           # sqlserver://user:pass@host:1433/database
  tls:
    ca_file: /etc/ssl/certs/your-sql-server-ca.pem   # private CA, or:
    # accept_invalid_certs: true                      # self-signed dev cert

SQL Server always encrypts the login handshake, so TLS is on regardless; the tls: block only controls how the server certificate is trusted. Supported export modes and types are listed in compatibility.md.

When tls: is omitted entirely, Rivet connects without TLS and emits a WARN so you notice. See reference/compatibility.md for which servers ship TLS-ready and Rivet’s dev-environment defaults.

Credentials & plan artifacts

A PlanArtifact (produced by rivet plan) is designed to be committed / reviewed; it must not carry plaintext credentials. Rivet enforces ADR-0005 PA9 (SourceConfig::redact_for_artifact):

• password: field → always stripped from the artifact (set to None).
• url: containing scheme://user:pass@… → userinfo rewritten to REDACTED.
• password_env / url_env / url_file → preserved as references so apply-time can re-resolve against the apply-environment.

When redaction runs, rivet plan logs:

WARN plan 'orders': plaintext credentials stripped from artifact —
     apply time must have equivalent env/file-based auth available

Recommendation: use password_env (or url_env) everywhere; only use plaintext password: for one-off local scripts. See ADR-0005 PA9.

exports[]

Each entry in the exports list defines one export job.

Compression profiles

compression_profile is the recommended way to pick a codec. It maps to a (codec, level) pair and takes precedence over any compression / compression_level fields.

exports:
  - name: events
    format: parquet
    compression_profile: balanced    # zstd level 3
    destination: { type: local, path: ./out }

If you need a specific codec that is not covered by the presets, use compression + compression_level directly and omit compression_profile.

Verification depth

verify controls how thoroughly --validate (and rivet validate) checks each part at the destination:

• size (default) — confirm each part exists at its recorded size_bytes, plus manifest self-consistency and _SUCCESS. Content is also MD5-checked for free whenever the store surfaces a checksum in its listing, but a part without one is accepted as size-only.
• content — require every part’s content MD5 to match the store’s listing checksum (no download). Any part that could only be size-verified fails validation with an actionable message.

How content verification works: Rivet computes each part’s MD5 before upload and records it in the manifest; GCS and Azure compute their own for a part uploaded as a single PUT and return it in object listings. --validate compares the two with no download. Parts large enough to stream as multipart / block-list get no checksum, and neither does S3 (its ETag is not an MD5 under SSE-KMS / SSE-C, so rivet does not trust it) or local FS. Under verify: content on GCS / Azure, set destination.oneshot_budget_mb (default 64 MB) comfortably above your part size: the budget is shared by concurrent uploads, so a part one-shots only if it fits what is free at that moment. S3 cannot meet verify: content.

The run report and rivet validate show coverage explicitly, e.g. 3 verified (2 md5, 1 size-only).

Parquet row group tuning

Parquet row groups affect memory usage during write, compression ratio, and downstream query performance (predicate pushdown, column skipping). When parquet: is omitted, Rivet uses the library default of 1,048,576 rows per group, which is optimal for narrow tables but can be large for wide tables.

exports:
  - name: events
    format: parquet
    parquet:
      row_group_strategy: auto          # auto | fixed_rows | fixed_memory
      target_row_group_mb: 128          # target Arrow buffer size per group (auto + fixed_memory)
      max_row_group_mb: 256             # optional upper bound (all strategies)

Examples:

# Auto-tune for a wide JSON table — groups sized to ~64 MB
parquet:
  row_group_strategy: auto
  target_row_group_mb: 64
  max_row_group_mb: 128

# Fixed row count — useful when downstream tooling requires exact group sizes
parquet:
  row_group_strategy: fixed_rows
  row_group_rows: 500000

Note: rivet plan shows the selected strategy and target in the Format section when parquet: is configured.

rivet init auto-generates this block for chunked exports and large full-mode tables, pre-selecting target_row_group_mb: 64 for wide schemas (≥ 5 text/JSON/bytea columns) and 128 for narrow ones.

exports[].on_schema_drift — schema drift policy

Controls what Rivet does when it detects a structural change in the output schema (column added, removed, or retyped) compared to the snapshot stored from the previous run.

fail is useful in CI pipelines where schema changes must be reviewed before the new shape is exported downstream.

exports:
  - name: orders
    on_schema_drift: fail

When fail triggers, behavior depends on the runner: in single, keyset, and parallel-Mongo modes the schema check runs post-extraction, so the output file has already been written to the destination (but no cursor advance or manifest commit occurs). In chunked mode the check runs pre-chunk from a scan-free type probe, so the run aborts before any chunk is written. Re-run after confirming the schema change is intentional, or switch to warn to accept it.

exports[].columns — per-column type overrides

Override the Arrow type Rivet infers for a specific column. Useful when:

• a NUMERIC / DECIMAL column has no explicit precision/scale in the source schema (beyond rivet init’s default decimal(38,18) placeholder), or
• you need a narrower precision for BigQuery NUMERIC compatibility.

columns:
  <column_name>: <type>          # applies to every captured table with the column
  "<table>.<column>": <type>     # applies to ONE table, wins over the bare key

Supported override types: decimal(p,s) / numeric(p,s) (both precision and scale required; precision > 38 produces Decimal256), integer widths (smallint/int/bigint/int16/int32/int64), floats (real/float4/double/float8), bool, text/string/varchar, uuid, json/jsonb, date, the timestamp* family (naive / tz / _ns variants), and binary (bytea/binary/varbinary/blob).

Overrides apply to batch and CDC identically (the same resolution surface). Key shapes: a bare column name applies to every captured table that has the column — on a multi-table CDC export that means ALL of them; a qualified "table.column" key targets one table and wins over the bare key there. A qualified key naming a table the export does not capture — or used on a query-shaped export — is a config error at load.

Example:

exports:
  - name: orders
    query: "SELECT id, amount, fee FROM orders"
    format: parquet
    destination:
      type: local
      path: ./out
    columns:
      amount: decimal(18,2)
      fee: decimal(18,6)

rivet init generates these automatically. When introspecting a table, rivet init reads numeric_precision and numeric_scale from information_schema.columns. If both are present, it emits a concrete override (decimal(p,s)). If the column is unbounded (NUMERIC without explicit precision), rivet init emits a working default decimal(38,18) plus a # REVIEW: YAML comment — the config header adds a # NOTE: line, and rivet init -o … prints a stderr reminder so you tighten precision when you know the real domain rules:

    columns:
      price: decimal(38,18)  # REVIEW: DDL has no numeric(p,s); edit to the real decimal(p,s) …

Type overrides are applied at export time and are reflected in rivet check --type-report output.

Some PostgreSQL types have no Arrow representation and cannot be exported directly. Rivet will report an error listing all unmappable columns before the run starts.

Use a SQL expression in your query field to work around any unsupported type:

exports:
  - name: locations
    query: >
      SELECT id, name, ST_AsText(geom) AS geom_wkt
      FROM locations
    format: parquet
    destination:
      type: local
      path: ./out

Rivet exports the WKT text as a Utf8 (string) column. Downstream tools (DuckDB, GeoPandas, QGIS) can reconstruct geometry from WKT.

Mode-specific fields

Incremental (mode: incremental):

Chunked (mode: chunked):

Time-window (mode: time_window):

exports[] — value-based partitioning

Splits a full, chunked or incremental export’s rows into Hive-style col=value/ destination sub-folders by a date column. See partitioning.md.

exports[].meta_columns

exports[].quality

Uniqueness tracking uses typed xxHash3-64 internally — numeric and binary columns are hashed directly from raw bytes without string formatting. unique_max_entries is the primary knob to control memory on very large tables.

Example:

quality:
  row_count_min: 100
  null_ratio_max:
    email: 0.05          # email must be <5% null
  unique_columns:
    - id
    - email
  unique_max_entries: 1000000   # stop after 1M unique values; warn if limit hit

Without unique_max_entries — tracking is unbounded. Safe for tables with hundreds of thousands of rows; may use significant RAM on tables with tens or hundreds of millions of distinct values.

With unique_max_entries — tracking stops at the limit and the run summary shows a warning. Duplicates found before the limit still fail the run; ones past it go unseen. Use when you want a best-effort uniqueness check without memory risk.

exports[].destination

The complete per-backend field list (local / s3 / gcs / azure / stdout) is in the generated config-reference.md (section exports[].destination). Per-backend setup, auth flows, and permissions: destinations/ — local · s3 · gcs · azure · stdout, plus the cloud auth matrix.

Path and prefix placeholders

The path (local) and prefix (S3 / GCS) fields support template placeholders, substituted at plan-build time:

destination:
  type: s3
  bucket: my-data
  prefix: exports/{date}/{export}/
  region: us-east-1

With an export named orders running on 2026-05-14, this resolves to exports/2026-05-14/orders/.

notifications

notifications.slack

Example:

notifications:
  slack:
    webhook_url_env: SLACK_WEBHOOK
    on: [failure, schema_change]

Environment variable interpolation

Any string value can reference environment variables:

source:
  url: "postgresql://${DB_USER}:${DB_PASS}@${DB_HOST}:5432/mydb"

Query parameters

Queries can use ${key} placeholders filled by --param key=value:

exports:
  - name: filtered
    query: "SELECT * FROM orders WHERE region = '${region}'"

rivet run --config export.yaml --param region=us-east