alexanderbianchi opened a new issue, #26602: URL: https://github.com/apache/datafusion/issues/26602
### Is your feature request related to a problem or challenge? When a Substrait JOIN or UNION references several tables whose schema providers resolve metadata asynchronously, logical plan conversion waits for each input in sequence. Independent table lookups therefore add their latencies, even though neither input needs the other to start. ### Describe the solution you’d like Allow independent JOIN and UNION inputs to be consumed concurrently while returning their logical plans in input order. Isolate mutable lambda and correlated-subquery scopes per branch; limit simultaneous table loads across nested branches. Keep sequential input consumption as the default for custom Substrait consumers unless they opt in. ### Describe alternatives you’ve considered Concurrent consumption on a shared consumer could mix its traversal scopes. A concurrency limit per UNION node alone does not bound table loads across nested relations and can deadlock if permits are held while consuming child nodes. ### Additional context A deterministic test can block a schema provider and verify that both JOIN inputs start before either finishes; a wide UNION can verify the shared load limit and stable input order. Implementation draft: https://github.com/alexanderbianchi/datafusion/tree/codex/substrait-concurrent-consumer ### Willingness to contribute I plan to contribute to this feature independently. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
