Compression Profiles

Rivet’s compression_profile is a high-level, intent-based way to choose a compression codec without knowing the specific codec name or level.

Profile-to-codec mapping

Choosing a profile

none — uncompressed

Use when:

• You are debugging output format or schema issues and want to open the file quickly.
• Downstream tooling re-compresses the file (e.g. S3 server-side compression).
• The file is temporary and will be deleted immediately after processing.

Avoid in production: uncompressed Parquet files are 3–10× larger than Zstd-3 on typical tabular data, increasing storage cost and upload time.

fast — Snappy

Use when:

• Throughput matters more than output size (large backfills, bulk loads).
• The extraction runs on a shared database or low-CPU runner where Zstd overhead is unwanted.
• Downstream query engines read the file frequently and benefit from fast decompression (Snappy is ~2–3× faster to decompress than Zstd).

Snappy produces files ~20–30% larger than Zstd-3 on typical tabular data.

balanced — Zstd level 3 (recommended default)

Use when:

• You want a sensible production default without thinking about the trade-off.
• Extraction runs on dedicated infrastructure (not shared OLTP database).
• Files are stored in S3/GCS and you want reasonable storage costs.

Zstd level 3 delivers ~60–70% compression ratio on typical tabular data with ~2–3× the CPU cost of Snappy. This is the right default for most pipelines.

compact — Zstd level 9

Use when:

• Storage cost or network transfer cost is a primary constraint.
• The pipeline runs infrequently (nightly, weekly) and has CPU to spare.
• Files are cold-stored and rarely read.

Zstd level 9 can deliver 5–15% better compression than level 3, at 3–5× the CPU cost. It is rarely worth using in real-time or latency-sensitive pipelines.

Precedence

compression_profile takes priority over the lower-level compression and compression_level fields. If you set compression_profile, any explicit compression or compression_level values on the same export are ignored.

# compression_profile wins — compression: snappy is ignored
compression_profile: compact
compression: snappy        # ignored

This ensures profiles are self-contained: once you pick a profile, you do not need to audit individual codec settings.

To use a codec not covered by the four profiles (e.g. Gzip, LZ4), omit compression_profile and set compression directly:

compression: gzip
compression_level: 6

CSV output

Compression profiles apply only to Parquet format. On a CSV export any compression_profile other than none is rejected at config-validation time (rivet check / doctor / run error out with “CSV output does not support compression_profile: …”). CSV files are always written uncompressed — omit the field (or set none) and compress after export with gzip, zstd, etc. if needed.

Configuration examples

Production default

format: parquet
compression_profile: balanced

Fast backfill from read replica

format: parquet
compression_profile: fast
tuning:
  profile: fast
  batch_size: 50000

Cold storage archive

format: parquet
compression_profile: compact
parquet:
  row_group_strategy: auto
  target_row_group_mb: 256

Benchmark expectations

Based on typical tabular data (mixed integer, text, timestamp columns):

Actual numbers depend heavily on data entropy. High-entropy data (UUIDs, hashes, random text) compresses poorly regardless of level. Low-entropy data (repeated values, sequential IDs, timestamps) compresses exceptionally well even at level 3.

Run the cross-tool harness (dev/bench/smoke.py, see docs/bench/README.md) against your own tables for concrete numbers.

See also

• Config reference — exports[].compression_profile
• Parquet tuning
• Resource-aware extraction