alexanderbianchi opened a new pull request, #26680: URL: https://github.com/apache/datafusion/pull/26680
## Which issue does this PR close? No linked issue. Stacked on #26679; review and merge that prerequisite first. This draft contains its commit because both PRs target Apache `main`. The parallel-only diff is https://github.com/alexanderbianchi/datafusion/compare/codex/substrait-input-hook...codex/substrait-parallel-consumer. Rebase this branch after the hook merges. ## Rationale for this change Independent Substrait JOIN/UNION inputs currently wait for catalog lookups one after another. A plan with many cold table reads accumulates those IO latencies even though the branches can be consumed independently. ## What changes are included in this PR? Override `consume_rel_inputs` in `DefaultSubstraitConsumer` to drive branch futures concurrently and restore input order before assembly. The trait's default remains sequential for custom consumers. Each branch copies enclosing outer-query and lambda scopes into independent stacks. Session state and extensions are borrowed; an explicitly shared `TableResolutionState` owns the per-table single-flight cells and semaphore. Branch construction lists every field to make the sharing decision explicit when state is added. Successful table resolutions, including missing tables, are cached within the consumer tree. Errors and cancelled initializations can be retried. Only a table-cell initializer acquires a permit; recursive traversal and duplicate waiters hold none. The limit is 32 active catalog loads per tree. An error queued behind this limit is observable only after admission; admitted failures cancel pending siblings. ## What is the testing strategy for this PR? Gated catalog tests cover overlapping JOIN reads, ordered wide UNION results, the shared limit across nested UNIONs, repeated-table single-flight, error cancellation, queued failures after the limit, and retry after cancellation. Scope tests verify that inherited outer-query/lambda stacks remain independent. The hook PR's trait-contract tests remain in the stack. `cargo fmt --all` and workspace `cargo clippy --profile ci --all-targets --all-features -- -D warnings` pass. The focused Substrait test suite is running. The extended workspace run on the hook prerequisite has passed core unit/integration tests and is still running fuzz tests. The standalone lint suite could not start because system Python lacks PyYAML. ## Are there any user-facing changes? The default consumer overlaps independent JOIN/UNION inputs and reuses providers within one consumer tree. Repeated reads therefore share the resolved provider rather than reloading it. No cross-consumer cache is introduced. Catalog lookups are bounded, and error timing may differ from sequential traversal. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
