Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Export modes — which one do I pick?

Every export declares a mode:. Start from the decision shortcut, then open the guide for the mode you land on.

Decision shortcut

  • Small-to-medium table, want a fresh snapshot every run → full
  • Append-only or has an updated_at, want only new/changed rows → incremental
  • Millions+ of rows, want speed and crash-resume → chunked
  • Only the last N days matter (event/log table) → time_window
  • Continuous low-latency replication from the WAL / binlog / oplog → cdc

When in doubt, rivet init inspects the table and picks a sensible default for you (small → full, large with an integer key → chunked).

At a glance

ModeUse whenKeeps state?Parallel?First run
fullcomplete snapshot each runno (stateless)noexports everything
incrementalonly rows past the saved cursoryes (cursor)noexports everything, then deltas
chunkedtables too large for one full scanyes (checkpoint, --resume)yesfull, split into ranges
time_windowrolling N-day windowno (recomputed each run)nothe window only
cdccontinuous log-based change captureyes (resume checkpoint)yes (multiplexed streams)optional initial snapshot, then stream

Notes worth knowing before you run:

  • incremental first run = full export. With no cursor yet, every matching row is exported, so the first run behaves like full — size batch_size accordingly (incremental.md).
  • chunked clean re-runs are NOT idempotent. A crash + --resume is at-least-once: a re-run chunk can be written twice (byte-identical), so de-duplicate downstream if you re-run (chunked.md).
  • Composite cursors are an incremental variant, not a separate mode — see incremental-coalesce.md when one timestamp column isn’t enough.
  • Source engine matters. The SQL sources (PostgreSQL, MySQL, SQL Server) support every mode. MongoDB, a document store, supports only full (with keyset / parallel / resume paging on _id) and cdc — see ../reference/mongodb.md.

Full configuration reference: ../reference/.