andygrove commented on issue #23814: URL: https://github.com/apache/datafusion/issues/23814#issuecomment-5050726580
Thanks for creating this issue @jayshrivastava, and thanks for linking to https://github.com/apache/datafusion/pull/23282. My opinion (which I know may not be shared by others) is that it would be beneficial for "core" DataFusion to offer a choice of execution modes: 1. Eager execution of the whole query tree, streaming results immediately (current model) 2. A **distributed-style** execution, but without the distributed (networking, multi-process) part. Essentially this would model dfd/ballista style execution and allow for development of related capabilities in core. I would be fine with this being an optional crate. This was the idea I was trying to demonstrate in #23282 Expanding further on this, enabling a distributed-style execution path with optional persistence of data between stages could allow single process DF to scale more on certain workloads (even though performance may not be amazing). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
