Testing matrix

Rivet’s test suite is organised into two tiers, selected by the standard #[ignore] convention. No test runner beyond cargo test is required.

Tiers

Both tiers run in CI (.github/workflows/ci.yml):

• Offline suite runs in the test / test-invariants / test-recovery / test-compatibility jobs on every push and PR.
• Live suite runs in two dedicated jobs:
  • test-type-golden — starts only Postgres + MySQL, runs --test live_type_golden -- --ignored. Named branch-protection gate for type-contract regressions.
  • e2e — full stack (Postgres, MySQL, SQL Server, MinIO, fake-gcs, Azurite, Toxiproxy, plus DuckDB/ClickHouse targets and the cdc / replica / pool compose profiles); seeds databases, builds a debug binary (deliberately — the e2e layer checks correctness, not throughput; the release profile is exercised by the separate release-build job), runs python3 -m dev.pytools.e2e, then runs all remaining --ignored tests except the MongoDB suites (live_mongo* / live_cdc_mongo), which run in the dedicated nightly mongo-versions matrix (4.4 → 8.0, nightly-live.yml).

Offline suite

Covers the full public API and every pure function in the crate. Runs in under two seconds on a developer laptop.

cargo test
# → example: cargo test: ~1360 passed, ~60 ignored (~32 suites, ~3s) — counts drift; check your local footer

Each integration file under tests/ maps to one domain:

Inline #[cfg(test)] mod tests blocks in src/ cover pure-function unit tests (config parsing, chunk math, cursor round-trip, format writers, destination capabilities, Slack payload formation) and benefit from pub(crate) access. cargo test --all-targets runs them automatically.

Live suite

Runs the full pipeline against the docker-compose stack. Every test carries #[ignore = "live: ..."] so the default offline run ignores them; invoking with --ignored activates them. If any service is unreachable, live tests fail with an actionable message naming the missing container and port (require_alive helper in tests/common/mod.rs).

docker compose up -d
cargo test -- --ignored
# → counts vary (~50+ ignored live tests). Check the cargo footer after `cargo test -- --ignored`.

Trust milestone: type golden round-trip

tests/live_type_golden.rs implements the roadmap contract database → Rivet (rivet run) → Parquet → Arrow read-back → exact assertions so type handling stays provable end-to-end, not only in unit tests (format_golden.rs covers writers in isolation).

Each test targets both engines where the contract applies:

CI runs these in the dedicated test-type-golden job (cargo test --test live_type_golden -- --ignored) as well as in the full e2e job. Local:

docker compose up -d
cargo test --test live_type_golden -- --ignored

Not yet in this matrix (future roadmap items): JSON logical metadata parity, unsupported-type strict failures, classified schema-drift variants as dedicated goldens (live_schema_drift.rs already covers drift telemetry for Postgres).

Test-only fault injection

A small env-var-driven hook in src/test_hook.rs lets the crash-matrix tests panic at precise pipeline boundaries without any cargo feature flag:

RIVET_TEST_PANIC_AT=after_file_write rivet run --config ... --export ...
# → process panics between dest.write() and record_file()

Valid point names are listed in src/pipeline/single.rs inline comments; see also dev/CRASH_MATRIX.md and ADR-0001. Cost when the env var is unset: one relaxed atomic load per call, roughly a nanosecond.

Live-test harness (tests/common/mod.rs)

Running from scratch

# Offline (default — used by PR-gate jobs):
cargo test

# Live (requires docker compose):
docker compose up -d
cargo test -- --ignored
docker compose down

Both command lines are what the corresponding CI jobs execute. If your cargo test diverges from the CI matrix, something is out of sync — check .github/workflows/ci.yml for the exact invocation.

Shell regression matrices

Binary-level regression guards under dev/matrices/ complement the Rust integration tests above. They drive the release rivet binary through fixture scenarios and diff stdout/stderr/exit codes, file layouts, EXPLAIN plans, and perf thresholds against committed baselines.

python3 -m dev.pytools.matrix_common setup-links   # one-time
python3 -m dev.pytools.matrices --tier=pr      # cli + cfg + path (PR CI)

See dev/matrices/README.md for the full taxonomy and tier map.

QA / roadmap alignment

Task IDs in tables above are historical QA labels. Trust & reproducibility (golden DB → Rivet → Parquet → Arrow read-back, Postgres and MySQL) lives in tests/live_type_golden.rs and is described above. Strategic tracking: rivet_roadmap.md §Phase 1 (Epic 14 / execution status).

CDC conformance gate

tests/cdc_conformance_gate.rs runs in plain cargo test and enforces two hard rules over the live CDC suite’s SOURCES:

1. Every engine × every conformance case (resume, idle-first-run, crash-before-ack, full type matrix, update/delete, initial snapshot, vanished anchor, mixed-transaction boundary, schema-qualified routing, non-UTC session, …) must have a live test — or an explicit NA("reason") in the matrix. A new engine cannot merge with a coverage hole; a new case must decide for all engines. Motivation: per-engine coverage drifts silently, and each engine’s missing case is exactly where a real bug lived (mixed-transaction existed only for MySQL, qualified-name only for PostgreSQL — ultrareview found the two matching bugs).
2. Every live CDC test that runs a capture must read back an outcome (manifest rows, a batch comparison, a destination listing, the state DB) — a bare exit-0 assertion would wave a 0-row silent success through, which is how three of the campaign’s worst bugs hid.

Mutation testing (nightly)

mutants-nightly in nightly-live.yml runs cargo-mutants on a rotating tier group per night (day-of-year modulo the group count: ledger/pipeline, CDC/value/integrity, planning/formats/gates, …) against the offline suite, and passes/fails on the diff against docs/mutants-baseline.txt — a NEW missed mutant fails the job; known misses live in the baseline and only ever shrink. A surviving mutant is a named test blind spot — the meta-gate that finds holes in the gates above.

Config-key composition gate (per PR)

config-key-composition in ci.yml diffs schemas/rivet.schema.json against the PR base: a NEW config key must be named by at least one test. The campaign’s worst bugs lived at the intersection of two individually-correct knobs (initial × vanished-slot, initial × skip_empty) — a new knob enters review with its interaction tests or not at all.

CDC gremlins (real faults)

tests/live/gremlin_cdc.rs (+ the capture-job stall in live_cdc_mssql.rs) injects REAL fault classes — SIGKILL, a TCP cut mid-binlog-stream, a hard destination outage, a failed checkpoint write, a stalled capture job — and asserts the at-least-once contract from the outside: the failure is loud, and after healing the union of all parts holds every source row (overlap fine, gap never). Panic-hook tests cannot cover these: a panic unwinds and runs Drop guards; none of the above do. Requires toxiproxy + fake-gcs from the compose stack; each case is a row in the conformance matrix.

Known-equivalent mutants (decimal canon)

The six surviving mutants in src/types/decimal.rs are all < → <= on branch guards where BOTH branches compute identical values at the boundary (e.g. scale < 0 vs scale <= 0: at scale 0 the negative-scale arm divides by 10⁰ = 1 — the same result as the plain path; frac.len() < scale vs <=: at equality the pad loop pads zero characters). They are mathematically unkillable; do not write pseudo-tests for them. Everything else in the file is caught (92) or timeouts-as-caught (4).