Low-Memory Runners

How to run Rivet reliably on hosts with 1–4 GB of RAM — containers, small VMs, and CI workers.

Numbers in this guide are measured, not estimated. The benchmark below was run against a content_items table: 200,000 rows, 12 columns (TEXT, JSONB, VARCHAR), average row size ~3 KB. All three scenarios exported the same 200,000 rows and produced identical row counts in the output.

The problem with defaults

Rivet’s balanced profile sizes each batch from a 32 MB memory target using the schema’s estimated row width (clamped 1,000–150,000 rows; the static 10,000 applies only when no schema is available). Narrow tables (IDs, timestamps, small text) fetch large batches, wide ones (TEXT/JSONB columns, average row ~3 KB) small — but a larger batch size can still push total RSS above 800 MB:

The safe baseline cuts RSS by 5.7× with no wall-time regression. The default profile does adapt to table shape, but its 32 MB target and 150k-row ceiling may still exceed a tight host budget — low-memory environments need explicit caps (max_batch_memory_mb, or a fixed batch_size).

The safe baseline

Start here for any host with less than 2 GB available:

source:
  type: postgres
  url_env: DATABASE_URL
  tuning:
    profile: safe
    max_batch_memory_mb: 64
    on_batch_memory_exceeded: auto_shrink
    memory_threshold_mb: 512

exports:
  - name: my_table
    query: "SELECT * FROM my_table"
    format: parquet
    destination: { type: local, path: ./out }
    parquet:                       # per-export setting — lives inside the export entry
      row_group_strategy: auto
      target_row_group_mb: 32
      max_row_group_mb: 64

What each setting does:

On a wide-text table (200K rows, avg 3 KB/row, 12 columns), this combination measured 154 MB peak RSS — compared to 878 MB without the cap. Actual RSS on your table will depend on row width and column count; use rivet metrics to validate after the first run.

512 MB host (container / CI)

source:
  tuning:
    profile: safe
    batch_size: 500
    max_batch_memory_mb: 32
    on_batch_memory_exceeded: auto_shrink
    memory_threshold_mb: 256
    throttle_ms: 1000

exports:
  - name: my_table
    query: "SELECT * FROM my_table"
    format: parquet
    destination: { type: local, path: ./out }
    compression_profile: fast    # per-export; snappy: lower CPU overhead than zstd
    parquet:                     # per-export setting — lives inside the export entry
      row_group_strategy: auto
      target_row_group_mb: 16
      max_row_group_mb: 32

On the wide-text benchmark table (200K rows, ~3 KB/row), this config measured 111 MB peak RSS — well within a 512 MB host. Wall time was identical to the uncapped run.

Trade-off: fast (snappy) compresses less aggressively than balanced (zstd-3). On the same table, snappy produced a 188 MB output file vs. 71 MB for zstd-3. If storage cost matters, use compression_profile: balanced even on a constrained host — the CPU cost difference is small (snappy is only ~5–10% faster on text-heavy data).

Wide-table export (TEXT / JSONB heavy)

Wide tables need a smaller batch size and tighter row group targets. Use batch_size_memory_mb to let Rivet calculate batch size from a memory budget rather than a row count:

exports:
  - name: events
    query: "SELECT id, payload, metadata FROM events"
    format: parquet
    destination: { type: local, path: ./out }
    tuning:
      batch_size_memory_mb: 32      # target ~32 MB per batch
      max_batch_memory_mb: 64       # hard cap; auto_shrink if exceeded
      on_batch_memory_exceeded: auto_shrink
    parquet:
      row_group_strategy: auto
      target_row_group_mb: 32

batch_size_memory_mb samples the first batch to estimate row width, then adjusts subsequent batches automatically. This is more reliable than guessing a row count for wide tables.

Parallel exports on memory-constrained hosts

When running multiple exports in parallel (--parallel-export-processes), each worker spawns its own OS process with an independent heap. Budget memory per-worker:

available_ram = total_ram × 0.7     # leave 30% for OS / filesystem cache
ram_per_worker = available_ram / workers
max_batch_memory_mb = ram_per_worker × 0.5   # batch is ~half of worker RSS

Example for 2 GB host, 2 workers:

available = 2048 × 0.7 = ~1400 MB
per worker = 1400 / 2 = ~700 MB
max_batch_memory_mb = 700 × 0.5 = ~350 MB

source:
  tuning:
    max_batch_memory_mb: 256    # conservative, with headroom
    on_batch_memory_exceeded: auto_shrink
    memory_threshold_mb: 512

auto_shrink guarantees and caveats

auto_shrink is the most reliable policy for low-memory environments because it adapts at runtime instead of failing.

Guarantees:

• Total row count is identical to a no-cap run — no rows are lost or duplicated.
• Cursor state and manifest are correct — each sub-batch writes atomically.
• Parquet schema is stable across sub-batches.

Caveats:

• Adds CPU overhead proportional to the split depth. A batch that splits 8 levels (256× fragmentation) adds measurable latency.
• If a single row is wider than max_batch_memory_mb, the split terminates at a 1-row batch and writes it as-is. This is correct but may produce many small files.
• For extremely wide rows (average row > max_batch_memory_mb), lower batch_size to 1–100 instead of relying on auto_shrink alone.

For tables where individual rows can be > 64 MB (BLOB-heavy schemas), set max_batch_memory_mb to match the expected maximum row size and accept single-row batches as the floor:

tuning:
  batch_size: 10
  max_batch_memory_mb: 256
  on_batch_memory_exceeded: auto_shrink

Monitoring RSS in production

Rivet always reports peak RSS in the run summary printed to the terminal:

✓ events  full  142,380 rows  1 files  71.2 MB  16.6s  RSS 154 MB

The RSS value is sampled by a background thread during the run, so it reflects the high-water mark rather than end-of-process RSS.

The same value is stored in the state DB and accessible via:

rivet metrics --config rivet.yaml --export events

Use these to validate that RSS stays within your host budget across different table sizes. The first run on a new table is the most reliable baseline — query planner cache, filesystem buffer, and jemalloc slab warmth all affect subsequent runs.

Quick-reference: settings by host RAM

Measured on a wide-text table (200K rows, ~3 KB avg row, 12 columns incl. TEXT and JSONB). Measured RSS is what you can expect on similarly shaped tables; narrow numeric tables will use less.

Rows marked ✓ are directly measured. Rows marked “est.” are extrapolated from the measured data points. Actual RSS will be higher on tables with larger average row width (e.g. JSONB blobs, long TEXT fields).

See also

• Resource-aware extraction — memory budgets, policies, RSS formula
• Parquet tuning — row group strategies and downstream implications
• Tuning reference — all tuning parameters and profiles