rivet init — config scaffolding

rivet init connects to PostgreSQL, MySQL, SQL Server, or MongoDB, introspects tables (collections on MongoDB), and prints a YAML scaffold you can save and edit before running rivet check / rivet run.

Generated configs use url_env: DATABASE_URL so secrets are not embedded in the file. Set DATABASE_URL (or switch to url: / structured credentials) before running exports.

Modes

Single table

Provide --table (optionally schema-qualified: public.orders on PostgreSQL, dbo.orders on SQL Server).

export DATABASE_URL='postgresql://user:pass@localhost:5432/mydb'
rivet init --source "$DATABASE_URL" --table orders -o rivet.yaml

# Qualified name (PostgreSQL)
rivet init --source "$DATABASE_URL" --table analytics.facts -o rivet.yaml

export DATABASE_URL='mysql://user:pass@localhost:3306/mydb'
rivet init --source "$DATABASE_URL" --table orders -o rivet.yaml

Rivet emits one export block: SELECT of all columns, a suggested mode (full, incremental, or chunked) from row estimates and column types, plus chunk_* or cursor_column when applicable. Every scaffold uses format: parquet and, by default, meta_columns with exported_at: true and row_hash: true (lineage and row fingerprinting in the output — see exports[].meta_columns in config). When the heuristic picks chunked, the scaffold also includes chunk_checkpoint: true (resumable runs, rivet run --resume, and reconcile/repair — see chunked mode).

Whole PostgreSQL schema

Omit --table. All base tables and views in the target schema are introspected; the file contains one export per object, sorted by name.

• --schema — PostgreSQL schema name (default: public).

export DATABASE_URL='postgresql://user:pass@localhost:5432/mydb'
rivet init --source "$DATABASE_URL" --schema public -o rivet_all_public.yaml

# Non-default schema
rivet init --source "$DATABASE_URL" --schema analytics -o rivet_analytics.yaml

The database itself comes from the connection URL path (/mydb).

Whole MySQL database

Omit --table. All base tables and views in the database are listed from information_schema.

• If the URL already includes the database (mysql://.../mydb), that database is used.
• If the URL has no database path, pass --schema <database> (same flag name as for Postgres; on MySQL it selects the database name for listing).

rivet init --source 'mysql://user:pass@localhost:3306/rivet' -o rivet_mysql.yaml

# URL without database — name it explicitly
rivet init --source 'mysql://user:pass@localhost:3306/' --schema rivet -o rivet_mysql.yaml

Heuristics (suggested mode)

The table above applies to the SQL engines (PostgreSQL / MySQL / SQL Server). MongoDB is schemaless — rivet init introspects no columns, primary keys, or cursor / chunk candidates — so every collection scaffolds mode: full (one export per collection), regardless of document count. MongoDB’s only batch mode is full; use --mode cdc for change capture.

Chunked exports and checkpointing

When the suggested mode is chunked, the scaffold always includes chunk_checkpoint: true. That enables resumable chunk runs after crashes or transient errors (rivet run --resume), chunk state in rivet state chunks, and reconcile/repair workflows. Set it to false only if you intentionally do not want checkpoint state on disk.

Meta columns (defaults)

The YAML scaffold enables exported_at and row_hash for every export. Set either to false, or remove the meta_columns block entirely, to turn them off.

Row estimates are cheap metadata (pg_class.reltuples on PostgreSQL, information_schema.TABLES.TABLE_ROWS on MySQL), not exact COUNT(*).

Always run rivet check --config <file> and adjust modes, destinations, and tuning before production runs.

DECIMAL / NUMERIC column overrides

Rivet reads numeric_precision and numeric_scale from information_schema.columns during introspection. When a NUMERIC or DECIMAL column has explicit precision and scale, the scaffold automatically emits a columns: block with the correct decimal(p,s) override — so exports don’t fail at runtime with an “unsupported type” error:

exports:
  - name: payments
    query: >
      SELECT id, amount, fee
      FROM payments
    mode: chunked
    chunk_column: id
    chunk_size: 100000
    chunk_checkpoint: true
    format: parquet
    columns:
      amount: decimal(18,2)
      fee: decimal(18,6)
    destination:
      type: local
      path: ./output

If the column is declared as plain NUMERIC (no precision / scale in the DDL), rivet init still emits columns: so exports run: it uses decimal(38,18) as a wide default (Decimal128 in Arrow), prefixes the YAML header with a # NOTE: pointing at these lines, and adds # REVIEW: inline on each such column — plus a rivet: note line on stderr when you write rivet init -o <file>. Replace the defaults with precision/scale from your domain (or constrain the DDL) before trusting the export:

    columns:
      price: decimal(38,18)  # REVIEW: DDL has no numeric(p,s); edit to the real decimal(p,s) or change the column type — values outside this bound may truncate or fail export.

Flags (summary)

Avoiding credentials on the command line

Shell history, process listings (ps, /proc/<pid>/cmdline), and container inspect logs all capture --source "postgresql://user:pass@host/db" verbatim. For anything beyond local dev, use --source-env or --source-file:

# Recommended — env var resolved inside the process only.
export DATABASE_URL='postgresql://user:pass@host:5432/db'
rivet init --source-env DATABASE_URL --schema public -o cfg.yaml

# File-based — useful when the URL is managed by your secrets mount.
rivet init --source-file /run/secrets/database_url --table orders -o cfg.yaml

Exactly one of --source, --source-env, --source-file must be provided (enforced by clap’s ArgGroup).

Discovery artifact (--discover)

rivet init --discover runs the same introspection but emits a machine-readable JSON document (schema described in src/init/artifact.rs). Intended consumers: external orchestration tools, code review, and automated config generators.

rivet init --source "$PG_URL" --schema public --discover -o discovery.json
rivet init --source "$MY_URL" --table orders   --discover    # pipes JSON to stdout

Per-table fields (tables[]):

The artifact is advisory — same policy as plan prioritization (ADR-0006): no runtime effect, no auto-application.

Docker Compose in this repository

The repo root docker-compose.yaml defines Postgres and MySQL (rivet / rivet users, database rivet) with the same schema as dev/postgres/init.sql and dev/mysql/init.sql.

docker compose up -d postgres mysql
export PG_URL='postgresql://rivet:rivet@localhost:5432/rivet?sslmode=disable'
export MY_URL='mysql://rivet:rivet@localhost:3306/rivet'

# One table
rivet init --source "$PG_URL" --table orders -o rivet_orders.yaml

# Whole PostgreSQL schema public
rivet init --source "$PG_URL" --schema public -o rivet_public.yaml

# Whole MySQL database from URL
rivet init --source "$MY_URL" -o rivet_mysql.yaml

To refresh many files at once (per-table YAMLs plus combined schema snapshots), run python3 -m dev.pytools.dev_scripts regen-docker-configs from the repo root after the DBs are up (and optionally seeded).

Warehouse scaffold: --bigquery-project / --bigquery-dataset / --gcs-bucket

With the three flags together, the generated config carries the warehouse half of the cycle — a top-level load: block (target: bigquery, pk: auto, cluster_by: auto, cleanup_source: true), a per-table partition: guess (the creation stamp — created_at / CreatedDate … — at granularity: day, never a mutation stamp, which would move a row between partitions on every update) and, for a mode that carries deltas (incremental, cdc), layout: base_buffer so rivet compact has a base to merge into. Every value is a guess from the catalog: review the block before the first load.

rivet init --source "$PG_URL" --table orders --mode incremental \
  --gcs-bucket my-bucket --bigquery-project my-proj --bigquery-dataset my_ds -o rivet.yaml
rivet run     -c rivet.yaml   # Parquet → gs://my-bucket/exports/orders/
rivet load    -c rivet.yaml   # → the base on the first pass, the buffer on later ones
rivet compact -c rivet.yaml   # MERGE the buffer into the base and drop it

--gcs-bucket is required with the BigQuery flags: rivet load reads GCS only, so a load: block over a local or S3 destination is a config its own next step refuses. For a whole-database CDC scaffold (one tables: stream with backfill: auto) the partition guesses are written on the stream’s load.tables.<table> blocks — the place the load reads them — not on the per-table recipes, which the load never reads.

Limitations

• Not a migration or DDL tool — only read-only introspection and YAML output.
• Views are included in schema-wide / database-wide runs; ensure each view is selectable for your user.
• Suggested modes are heuristics; large or sparse tables may need manual chunked / chunk_by_key / chunk_by_days tuning (see chunked mode).