yahoNanJing opened a new issue #1701: URL: https://github.com/apache/arrow-datafusion/issues/1701
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** <!-- A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] (This section helps Arrow developers understand the context and *why* for this feature, in addition to the *what*) --> Current Ballista implementation is more like a POC product for verification of whether it's able to run the Datafusion operators in a distributed way. It helps set up the whole framework and works well for just verification. However, it's a long way to introduce it to the production environment for real cases. This issue mainly raises several aspects we need to consider and to enhance for a more robust distributed execution framework. In big data era, there're many scenarios. Two common ones are query for interactive analysis and batch processing for ETL purpose. There's no silver bullet. Each scenario has its own characteristics and has its own needs. In the following, I'll describe some enhancement we can do for each scenario. For both interactive query and batch processing: - [Necessary] Able to access remote object store, like HDFS, S3, etc - [Necessary] Executor lost handling - [Necessary] Configuration management - [Nice to have] Schedule stages based on prioprity - [Nice to have] Cancel SQL/CancelJob - [Nice to have] Executor blacklist For interactive query: - [Necessary] Push-based task assignment - [Necessary] Better data exchange - [Necessary] Better result fetching For batch processing: - [Necessary] Task speculative scheduling - [Necessary] Shuffle fetch failure handling and retry - [Necessary] Stage reattempt -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
