thinkharderdev commented on issue #30: URL: https://github.com/apache/arrow-ballista/issues/30#issuecomment-1133741610
Great, thanks for the feedback everyone! I would summarize the key points as: 1. Based on this and previous discussions, the goal is not to create yet another general "distributed compute framework" and that Ballista should be something that works out of the box. 2. There is interest from the community for both batch processing (ala Spark) and a latency-sensitive distributed query engine with streaming execution. 3. Ballista shouldn't be inextricably tied to DataFusion as the underlying compute engine. This isn't the case right now really but Substrait can be an "engine agnostic" representation of a compute plan in the future and we shouldn't foreclose that future work with design decision in the more immediate term. Please chime in if anyone came to different conclusions :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
