ADR-0010: Two Parallel Execution Engines
Status: Accepted Date: 2026-05 Context: The architectural audit (2026-05) flagged that rivet runs two distinct parallel execution engines for what looks at first glance like the same job. This ADR documents why they coexist, what each is for, and the conditions under which we would unify them.
Decision
Keep two engines. Do not unify in v0.5.x.
| Engine | File | Use case | Parallel unit |
|---|---|---|---|
| In-process scoped threads | src/pipeline/chunked/exec.rs | Chunked export of a single table — split the row range into N chunks, run them concurrently against the same source DB. | std::thread::scope + per-thread Source connection |
| Subprocess fan-out | src/pipeline/parallel_children.rs | --parallel-export-processes — run many independent exports (different tables, different configs) concurrently as separate rivet child processes communicating via IPC. | std::process::Command + a JSON event stream |
Rationale
The two engines exist because they answer different questions:
-
In-process threads are the right tool when:
- Workers share one Arrow / parquet / OpenDAL runtime in process memory.
- Workers cooperate (semaphore, shared
Destination, shared progress bar). - Crash isolation is not a requirement — one panicking worker can take the whole process down because they share an export plan.
-
Subprocesses are the right tool when:
- Each export has its own config, plan, state, and OpenDAL runtime — sharing them would mean a much larger refactor of
Destination,StateStore, etc., for cross-export use. - Crash isolation matters: a failing export must not abort the others. A child process death is observable via exit code; a thread panic poisoning shared state is not.
- Memory pressure is per-process: a child that explodes its RSS hits the OOM killer alone.
- Each export has its own config, plan, state, and OpenDAL runtime — sharing them would mean a much larger refactor of
Unifying these into one engine would mean either:
- Pushing chunked exports into subprocesses → much higher overhead per chunk (cold connection, runtime spin-up, IPC for every progress event), losing the in-process semaphore and shared destination.
- Pushing multi-export concurrency into threads → losing per-export crash isolation and OpenDAL-runtime separation.
Neither trade is worth the refactor at v0.5.x scale. [Update: the sharing claim below held only for the in-process engine — it uses resource::Semaphore (kernel-parking, no busy-wait) and RetryClass (typed error classification); the subprocess fan-out engine uses neither, classifying child failures from exit codes over the IPC seam.]
What this ADR is not deciding
This ADR is not “we’ll never unify”. It is “we accept the duplication for now, and revisit when”:
- A third parallelism need arises (e.g. async source streaming) — duplication grows from N=2 to N=3 and the cost of keeping them in sync exceeds the cost of consolidation.
- The
Sourcetrait becomesSend + Sync(see ADR-0011). A shareableSourcewould unlock collapsing chunked workers to share one connection, which changes the in-process trade-offs. - A user-facing requirement forces a single execution model (e.g. cross-export coordination during chunked exports — currently impossible because subprocesses cannot observe each other’s chunk state).
Consequences
Cost
- Two retry loops, two progress UIs, two error-aggregation paths.
- New cross-cutting features (graceful shutdown, distributed tracing) must be implemented in both.
Benefit
- Each engine is small and focused; debugging chunked behaviour does not require understanding IPC, and vice versa.
- Subprocess engine inherits OS-level isolation for free.
- Different parallel semantics are not papered over with a
mode: enum.
Considered Alternatives
-
One engine via subprocesses only. Rejected: chunked exports of millions of rows on
--parallel 16would pay 16× cold connection latency at startup, and inter-chunk progress would be IPC traffic instead of a shared atomic. -
One engine via threads only. Rejected: a panic in one of N exports running multiple plans simultaneously would crash the whole
rivetprocess. The operator currently relies on the child-isolation property for long multi-export runs. -
Async/tokio for both. Rejected: rivet keeps its execution model sync. [Update:
tokiohas since become a direct dependency (rt-multi-thread/net/time) — the MongoDB and MSSQL drivers are async internally, bridged to the syncSourcetrait via a per-source runtime +block_on(ADR-0011); the pipeline itself remains sync.] Migrating Source to async I/O is a strictly larger refactor than this ADR is willing to scope. Decision deferred until the Source trait redesign in ADR-0011 lands.
When to revisit
Open a follow-up ADR if any of the following holds:
- The audit graph (
code-review-graph) shows new flows that cross the two engines — currentlypipeline-chunk(461 nodes) is one community, parallel-children lives insrc-batch. - A bug is reported that is impossible to express in one engine and trivial in the other — that asymmetry signals the abstraction is mis-cut.
- Multi-export with chunking (i.e. cross-product parallelism) becomes a real use case.