mingmwang commented on pull request #1924: URL: https://github.com/apache/arrow-datafusion/pull/1924#issuecomment-1060683009
> I might be asking too much about this great initiative. Another plausible and low-hanging fruit in my mind is: multi-tenancy only inside the scheduler. By making executors one query entrance at a time, a "Mutex". We could have an easier route to achieve multi-tenancy. > > Another question, just out of curiosity, is the scheduler a single point of failure in the current design? If it is, have you considered a "multi" scheduler architecture? > > Thanks again for broadening Ballista's landscape! Actually the PR does not touch the existing scheduler part too much. The major purpose is to refactoring the configuration and make execution plan session/configuration aware. Of cause the Ballista scheduler server need to maintain a list of running sessions and their session state currently in a global RuntimeEnv. But since Ballista Client also maintains its own local session context and session state for optimization and planning, I think if we have multiple Ballista scheduler's in the system, the current design should not make the scheduler a single point of failure. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
