alexanderbianchi opened a new pull request, #26603: URL: https://github.com/apache/datafusion/pull/26603
## Which issue does this PR close? - Closes #26602. ## Rationale for this change When a JOIN or UNION in a Substrait plan reads multiple tables with async schema providers, consuming their inputs sequentially makes planning wait for one table before starting the next. Independent metadata lookups should be able to overlap without mixing the consumer’s lambda or correlated-subquery state. ## What changes are included in this PR? - Add `SubstraitConsumer::consume_rel_inputs`, sequential by default so existing custom consumers keep their behavior. - Have JOIN and UNION assembly use that hook. The default consumer forks its traversal scopes for each independent input and restores the original input order after concurrent completion. - Reuse successful table resolutions (including missing tables) within one consumer, and bound simultaneous distinct-table loads across nested branches to 32. Errors and cancelled loads remain retryable. ## What is the testing strategy for this PR? - `cargo test -p datafusion-substrait --no-default-features`: 74 unit, 225 integration, and 3 doc tests pass (4 existing integration tests ignored). - Added deterministic gated-provider tests for JOIN overlap, ordered and bounded wide UNIONs, nested limits, single-flight reuse, error cancellation/retry, and independent traversal scopes. - `cargo fmt --all` and crate-scoped Clippy (`cargo clippy -p datafusion-substrait --all-targets --all-features -- -D warnings`) pass. - Full-workspace Clippy with `-D warnings` currently fails on `clippy::mutable_key_type` in unchanged `datafusion/core/src/dataframe/mod.rs`; it passes with only that lint suppressed at the command line (`-A clippy::mutable_key_type`). No source-level lint suppression was added. ## Are there any user-facing changes? Substrait JOIN/UNION planning can start independent table loads concurrently. `SubstraitConsumer` gains a defaulted trait method; existing implementations need no changes. A bounded table load can wait for a permit when other loads are in flight. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
