Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

ADR-0018: Builder Facades for Runner-Level Invariant Ordering

Status: Accepted Date: 2026-05-30

Context

ADR-0001 defines eight state-update invariants (I1-I8) that govern the ordering of writes a runner makes around each part it commits. ADR-0008 PG2 defines a separate ordering invariant for the cursor + schema + progression tail. ADR-0012 M1 defines the manifest-parts contract on top.

Before the session that produced this ADR, those orderings were held by convention at every call site. Each of the (at that point) five runners — single, keyset, chunked/exec::run_chunked_sequential, chunked/exec::run_chunked_parallel, chunked/sequential_checkpoint, plus the now-added chunked/parallel_checkpoint — hand-wrote the per-part write block (I1 finalize → dest.write → I2/M1 manifest add → I7 file-log → counters → journal). They also hand-wrote the post-finalize cursor + progression block. Drift accumulated:

  • keyset never bumped files_committed and had no fault hooks.
  • parallel_checkpoint never populated summary.manifest_parts at all — the cloud manifest M1 contract was silently empty for every parallel>1 + chunk_checkpoint:true run. Documented in commit e9b0796.
  • parallel_checkpoint opened a fresh StateStore connection per chunk just to write file_log. Performance smell, also documented.
  • single’s incremental block (cursor + progression) and chunked record_chunked_commit disagreed on per-write failure semantics in their comments (and in one case, in their code).

The fix shipped over four commits (034fa64, 1db8eba, bb27336, e9b0796 for commit::record_part; 58c2c5d for RunStore). This ADR documents the architectural pattern those commits picked and why.

Decision

Two builder facades own the two ordered-write groups:

pipeline::commit::record_part — per-part commit ordering

Two-seam split keyed on the parallel-engine fork:

  • commit::write_part_file(dest, tmp_path, rows, file_name) -> Result<PartRecord> — ADR-0001 I1 (finalize) + dest.write + ADR-0012 M3 fingerprint. Worker-safe: takes no shared run state, can run off-thread.
  • commit::record_part(plan, summary, state, &PartRecord, kind) -> () — ADR-0001 I2 fault hook + counters (bytes_written, files_produced, files_committed) + ADR-0012 M1 manifest_parts.push + journal event (variant chosen by PartKind: RunEvent::FileWritten for File, RunEvent::ChunkCompleted for Chunk; Page reuses ChunkCompleted for journal-on-disk back-compat — there is no separate KeysetPageWritten variant) + ADR-0001 I7 state.record_file (warn-on-fail) + I3 fault hook. Parent-only: touches &mut summary and Option<&StateStore>.

Sequential runners call both inline per part; the parallel engine calls write_part_file in workers and pushes PartRecords through a shared Mutex<Vec<…>>, then the parent calls record_part on each during a post-scope drain (see ADR-0017 for why one variant of this splits further).

PartKind is a closed enum (File { part_index } for snapshot; Chunk { chunk_index } for chunked / checkpoint; Page { page_index } for keyset). The journal-event mapping is internal to record_part, keeping the per-call-site signature uniform.

pipeline::run_store::RunStore — post-finalize cursor + progression

Builder over the two ordered post-finalize writes:

#![allow(unused)]
fn main() {
RunStore::finalize(state, plan, summary)
    .with_cursor(last_val)                            // I3 — fatal on error
    .with_progression(Progression::Incremental {…})    // PG2 — warn-on-fail
    .commit()?;
}

commit() writes cursor first (fatal on error, returns directly without attempting the progression write — a half-finalized run would log a misleading progression boundary), then dispatches Progression::Incremental to state.record_committed_incremental or Progression::Chunked to chunked::record_chunked_commit (which walks chunk_task to pick the highest completed chunk_index — see ADR-0008 PG2). The after_cursor_commit test fault hook fires inside the facade so every runner inherits it.

Scope locked at cursor + progression. Schema (with drift policy) stays in single.rs::run_single_export because schema-drift detection

  • Continue/Warn/Fail policy is a runner-level state machine that does not generalize across modes. Metric writes (state.record_metric) stay in job.rs because they belong on the Coordinator layer (ADR-0003 L4), not on the runner-level Persistence layer (L3) the facade addresses.

Update (2026-06-18, ADR-0021): the schema-drift carve-out above was wrong. Adding on_schema_drift to the chunked modes (ADR-0021) showed the Continue/Warn/Fail policy does generalize — what differs between modes is only the column source: single mode resolves columns post-write from the sink’s data-derived Arrow schema; chunked resolves them pre-chunk from a scan-free type_mappings probe so fail aborts before any chunk writes. That difference is exactly one adapter each over a shared core, so schema-drift became the third runner-write facade, pipeline::schema_drift (check_from_sink_schema / check_from_type_mappings over a private check_and_persist), alongside commit::record_part and RunStore. The deletion test confirms its depth: inlining it back would re-duplicate the detect → policy → store state machine across single mode plus the four chunked Detect arms.

Update (2026-07, feat/parallel-keyset): a 7th runner landed — keyset_parallel (N row-percentile-range workers, per-range crash-recovery; see ADR-0017’s new row). It re-confirmed the standing gap: the facades make the per-part / cursor / drift logic live once, but calling them is still per-runner convention, and the completeness ledger (runner-coverage-matrix.yaml) models 4 runners for ~8 loops — so a runner that owns its loop can still forget a facade (iteration 1 of this branch shipped keyset_parallel without the drift gate; a human caught it, not a guard). The fix extends check_post_run_invariants (already the structural guard for the M1 manifest_parts gap) to the drift + Form-B facades: each leaves a telltale on RunSummary when it runs (schema_changed = Some(_); column_checksums populated or ..._incomplete set), and a state_backed success that committed parts with either telltale ABSENT panics in debug/test. This makes “you called the facade” machine-checked for the two write-groups the M1 assert didn’t cover — the runner-bypass class becomes RED-by-construction, not a matrix cell a reviewer maintains by hand. It does NOT reopen facade-vs-trait; the facades stay, only their invocation is now verified.

Why “builder” instead of one method with Option args

Three shapes were considered:

ShapeTradeoff
Builder (chosen)Callers chain only what they have. Chunked runners with no cursor skip with_cursor; snapshot runs skip both with_* and commit() is a no-op. Ordering enforced inside commit() regardless of chain order.
Single method + Option-structOne call-site, all writes visible. But chunked runners write Writes { cursor: None, progression: Some(…) } — noisy.
Type-stateCompiler-enforced ordering. Overkill for two optional writes; idiomatic Rust does not lean on this for this scale.
Multiple methods on a stateful handleOrdering becomes “the order you call methods in” — convention-at-call-site, just with a different surface. Defeats the point.

Builder is the balance: variable writes fit cleanly, ordering rule lives once in the impl, ceremony is bounded.

Why “facade” not “trait”

Neither commit::record_part nor RunStore is a trait. The runners share an implementation pattern (call the facade in the right place with the right args), not an interface contract. A trait would require a Run-level abstraction the runners can swap out at runtime — no such abstraction exists or is needed. The facade is a free function (for commit_part) or a builder struct (for RunStore); callers invoke it directly.

This matches ADR-0015’s “data-shape seam vs trait” reasoning at a different layer: the seam value comes from concentrating implementation logic, not from substitutability.

Trade-offs

Positive

  • Locality: ordering rules for I1→I3 and PG2 live in one implementation each. Per-write failure semantics (fatal vs warn-on-fail) is in the signature of the builder methods, not in comments at every call site.
  • Leverage: six runners (after the OPT-4 keyset and the parallel_checkpoint additions) share one body each. New runners inherit the contract by construction.
  • Drift prevention: the M1 gap that parallel_checkpoint had (silently empty manifest_parts) is structurally impossible under the facade — record_part always appends to manifest_parts when called. Documented as the retroactive guard provided by the cfg!(debug_assertions) coherence check in pipeline::finalize::finalize_manifest.
  • Fault-hook centralization: after_file_write, after_manifest_update, after_cursor_commit test fault points fire once per facade, not once per runner-specific re-implementation of the hooks.

Negative

  • The runner-write surface is now three facades, not one: commit::record_part (per-part), RunStore (post-finalize), and schema_drift (pre-chunk / post-write — added by ADR-0021). Metrics stay above on the Coordinator layer. A new contributor must learn where each ordered-write group lives instead of finding them all in one place.
  • Builder-with-fluent-chain may feel unidiomatic in Rust where most ordered writes are direct function calls. ADR-0017 explains the per-runner asymmetry that motivated the chain.
  • Test surface includes both seam-level unit tests (in commit.rs and run_store.rs) and runner-level live tests (the existing live_chunked_recovery, live_crash_recovery suites). New runners need both layers of coverage.

When this should be revisited

  • If a fourth ordered-write group emerges at the runner level (e.g., per-run lineage tracking, audit log of source queries) that does not fit cursor/progression or per-part: extend RunStore with a third with_* method rather than building a third facade. (Schema-drift, added later as its own facade per ADR-0021, is not a counter-example to this rule: it is a detect → policy → store decision keyed on column-source, run pre-chunk or post-write — not a post-finalize ordered write that fits RunStore’s cursor/progression shape. A true post-finalize fourth write should still extend RunStore.)
  • If a runner needs to dispatch between different per-part commit strategies at runtime (today every runner uses the same write_part_file → record_part): re-evaluate the facade-vs-trait choice. Until then, free functions are correct.

References

  • src/pipeline/commit.rs — write_part_file, record_part, PartKind definitions; module doc covers the in-step ordering rationale.
  • src/pipeline/run_store.rs — RunStore, Progression; module doc covers per-write failure model.
  • src/pipeline/summary.rs::RunSummary::check_post_run_invariants — runtime debug_assert that catches a runner bypassing the facade.
  • ADR-0001 — state invariants I1-I8.
  • ADR-0008 — export progression, PG2 ordering.
  • ADR-0012 — cloud manifest contract, M1 / M3.
  • ADR-0015 — data-shape seam vs trait (parallel reasoning at the source-introspection layer).
  • ADR-0017 — per-runner durability ordering map (when the facade is called sync per-part vs in a post-scope drain).
  • ADR-0019 — Governor extraction (similar deepening pattern at a different layer).
  • Session commits: 034fa64 (extract commit), 1db8eba (chunked migration), bb27336 (sequential_checkpoint migration), e9b0796 (parallel_checkpoint M1 gap fix), 58c2c5d (RunStore).

Amendment 2026-09-26: the invariant is RED in debug and at the release gate

The coherence check runs in every build. A debug or test build panics. The release binary logs run-integrity invariant violated at WARN and still exits 0, so a user’s run is never failed by it. The release oracle fails the gate on any occurrence of that line in any gated command’s output (dev/release_oracle/core.py, verify_no_invariant_violations). The drift telltale is not enforced on --resume runs.