Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

ADR-0009: Reconcile and Targeted Repair Contracts

  • Status: Accepted
  • Date: 2026-04-18
  • Context: Epic F introduces partition-level reconciliation (rivet reconcile), Epic H introduces targeted repair (rivet repair). Both operate on completed chunked runs, produce structured reports, and interact with the progression table (ADR-0008) and the plan/apply channel (ADR-0005). The contracts below make the workflow auditable and composable with external tooling.

Scope

CommandInputOutputSide effects
rivet reconcile -c -eLatest chunk_run + committed files (manifest)ReconcileReport (pretty or JSON)May advance last_verified_* (ADR-0008 PG5)
rivet repair -c -e [--report …] [--execute]ReconcileReport (from --report <file>, or built fresh in-process against the latest chunk run when --report is omitted)RepairPlan (without --execute) or RepairReport (with --execute)With --execute: new output files; manifest entries (the repaired chunk’s originals marked superseded); no cursor/commit changes

Contract matrix

IDNameStatementEnforced by
RC1Reconcile requires a committed chunk runrivet reconcile bails when no chunk_run exists for the export. Operator must run the chunked export with chunk_checkpoint: true first.reconcile_chunked_inner — get_latest_chunk_run
RC2Partition SQL parity with extractionThe per-partition source COUNT(*) is built from the exact same build_chunk_query_sql shape used during extraction — same WHERE, same dense/range/by-days branch, same identifier quoting.reconcile_chunked_tasks
RC3Per-partition classificationEach chunk_task is classified as match (counts equal), mismatch (both counts known and differ), or unknown (either count missing). Unknown is always a repair candidate.PartitionResult::classify
RC4Reconcile scope v1Only chunked exports. time_window bails with a clear “use chunk_by_days” message; snapshot / incremental receive “use rivet run --reconcile”.reconcile_cmd::run_reconcile_command
RC5Report shape stabilityReconcileReport JSON fields (export_name, run_id, strategy, partitions[], summary) form a stable schema; new fields are additive only.plan::reconcile, serde defaults
RC6Verified advances only on full matchVerified boundary (ADR-0008 PG5) is written iff summary.mismatches == 0 && summary.unknown == 0.reconcile_cmd::reconcile_chunked
RR1Repair derives only from reconcileRepairPlan::from_reconcile is the single path that produces repair actions — no direct config paths, no operator-typed ranges.plan::repair::RepairPlan::from_reconcile
RR2Plan before executeWithout --execute, rivet repair prints the plan and exits; no destination files are written and nothing is re-exported. When --report is omitted it first builds a fresh reconcile in-process, issuing one read-only SELECT COUNT(*) per partition against the source (and, on a fully-clean result, advancing the verified boundary per RC6); only the --report <file> path is source-query-free.repair_cmd::run_repair_command
RR3Repair SQL parityRepair chunk queries use the same build_chunk_query_sql as extraction and reconcile — repair is apples-to-apples with the original run.run_chunked_sequential(ChunkSource::Precomputed)
RR4Committed boundary not moved by repairRepair re-exports chunks already covered by committed progression; last_committed_* is not re-stamped by repair. Operator advances verified by running rivet reconcile afterwards.repair_cmd::execute_repair (no record_committed_* call)
RR5Destination files are additiveRepair writes new files alongside originals using <export>_<ts>_chunk<idx>_<nonce>.<ext> naming; the 64-bit <nonce> makes the name collision-proof so a re-export of the same chunk never overwrites the original even when it lands in the same wall-clock second (the second-granularity <ts> alone would collide). Rivet does not delete or overwrite prior files. The manifest declares the replacement (amended 2026-09-26): the chunk’s previously committed part(s) are re-marked superseded, so row_count, part_count, column_checksums (the superseded parts’ contribution is re-read and subtracted), validate, and rivet load all see each row once. The superseded files stay on disk until opt-in load.gc_orphans collects them (never while a run is active on the prefix). When an original cannot be mapped to its chunk without guessing (a manifest from another run, a part name with no chunk index, a repair part the rename could not relabel), that chunk stays additive and repair warns. A warehouse that already loaded the old part without a primary key keeps those rows.chunked::chunk_part_filename, repair_cmd::superseded_parts
RR6Unparseable identifiers are skippedPartitions whose identifier does not match "chunk N [start..end]" with parseable i64 bounds are recorded in skipped[] — never silently dropped, never executed.RepairAction::from_identifier, execute_repair
RR7Strategy scope v1Repair requires mode: chunked. Other modes bail with a clear error (same policy as reconcile scope).repair_cmd::run_repair_command
RR8Report shape stabilityRepairPlan / RepairReport JSON is a stable additive schema (same policy as RC5).plan::repair, serde defaults

Interaction with other ADRs

ADRInteraction
ADR-0001 (state invariants)Reconcile does not write to state beyond progression (ADR-0008). Repair runs run_chunked_sequential which honors I1–I4 for the files it produces.
ADR-0005 (plan/apply)A reconcile or repair-report JSON is a peer of PlanArtifact: sealed, reviewable, auditable. No staleness check — reports are snapshots, not execution gates.
ADR-0006 (prioritization)Reconcile outcomes are not (yet) fed into prioritization; Epic I uses export_metrics, not reconcile reports.
ADR-0007 (cursor policy)Reconcile is chunked-only in v1 and does not touch incremental cursors. Coalesce mode is unaffected.
ADR-0008 (progression)RC6 is the sole writer of last_verified_*; RR4 documents that repair leaves last_committed_* untouched.

Failure map

ScenarioReconcile behaviorRepair behavior
No chunk_run for exportBails (RC1)Bails (RC1 via fresh reconcile) or error loading report
Non-chunked exportBails (RC4)Bails (RR7)
Source unreachableBails at first query_scalarBails at source::create_source before any chunk runs
Partition count mismatchPartition marked mismatch; verified not advanced (PG5)Repair action generated; user decides to --execute
Chunk task never completedPartition marked unknown; verified not advancedRepair action generated
Unparseable chunk keysPartition marked unknown (source count not attempted)Skipped with note (RR6)

Workflow example

# 1. Run chunked export with checkpoint
rivet run -c cfg.yaml

# 2. Reconcile — advances `last_verified_*` if all match
rivet reconcile -c cfg.yaml -e orders --format json -o reconcile.json

# 3. Repair plan (dry-run)
rivet repair -c cfg.yaml -e orders --report reconcile.json

# 4. Execute repair
rivet repair -c cfg.yaml -e orders --report reconcile.json --execute

# 5. Reconcile again to advance `last_verified_*`
rivet reconcile -c cfg.yaml -e orders

Out of scope (v1)

  • time_window and incremental per-partition reconcile.
  • Automatic repair execution (always opt-in via --execute).
  • Repair that rewrites or deletes prior destination files (superseded files are only collected by opt-in gc_orphans).
  • Hash-based partition verification (current v1 is COUNT(*) only).
  • Repair advancing last_committed_* (documented non-goal: commit = first successful extraction; repair is corrective, not commitment).

Test coverage

  • plan::reconcile::tests — classification, summary, round-trip JSON.
  • plan::repair::tests — identifier parsing, action derivation, summary counts.
  • pipeline::reconcile_cmd::tests — stubbed source closure exercises the full reconcile_chunked_tasks path without a DB.
  • pipeline::repair_cmd::tests — smoke test for the reconcile → plan derivation path.