alexanderbianchi opened a new pull request, #26679: URL: https://github.com/apache/datafusion/pull/26679
## Which issue does this PR close? No linked issue. This extracts the extension point from a larger concurrent-consumer change; parallel default-consumer behavior will be proposed separately. ## Rationale for this change Custom Substrait consumers cannot overlap independent JOIN/UNION input consumption while reusing upstream operator assembly. Those helpers await each child directly. Downstream consumers currently need to override the operators and wrap already-consumed children just to control scheduling. ## What changes are included in this PR? Add `SubstraitConsumer::consume_rel_inputs(&[&Rel]) -> Result<Vec<LogicalPlan>>` and call it from JOIN and UNION assembly. The default consumes inputs sequentially and stops on the first consumption error. Overrides must return one plan per input in input order; assembly checks the count. This PR adds no branch cloning, concurrency, caching, semaphore, or dependency. JOIN/UNION assembly still handles predicates, qualification, and distinctness. Invalid JOINs with missing children return an error instead of panicking. ## What is the testing strategy for this PR? Tests exercise the public consumer trait: sequential default consumption with a gated first input, error short-circuiting, custom-hook dispatch for JOIN/UNION ALL/UNION DISTINCT, input order, and rejection of incorrect result counts. Existing Substrait integration tests cover operator semantics. `cargo fmt --all` and workspace `cargo clippy --profile ci --all-targets --all-features -- -D warnings` pass on base `6bb92bb3f`. The focused Substrait suite passed 70 unit tests, 228 integration tests (6 ignored), and 3 doctests before the base refresh; the hook code is unchanged. Extended workspace tests are running. The standalone lint suite could not start because system Python lacks PyYAML. ## Are there any user-facing changes? Adds a provided trait method, so existing consumer implementations remain source-compatible and keep sequential input consumption. Custom consumers can override scheduling without replacing operator assembly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
