mingmwang commented on pull request #1924: URL: https://github.com/apache/arrow-datafusion/pull/1924#issuecomment-1060702963
> I wonder if we are missing a layer here. My intuition is that a global `RuntimeEnv` might be a problem and not really support real multi-tenancy but a session-level `RuntimeEnv` may be too fine-grained. We already have the concept of a namespace in the state configurations so does it make sense to namespace the `RuntimeEnv`. This could be away to support proper resource-level multi-tenancy. That is each namespace is a proper "slice" of the total cluster resources and there is some level of isolation so that one resource-hungry query by one client (in theory at least) can not use all cluster resources. It would probably be a little complicated to do dynamic registration of namespaces but should be possible in principle (and we probably don't have to tackle this right away). > > I have the same question as @yjshen with respect to mutli-scheduler environments. I think that will be a key consideration for Ballista in general. For cluster level resource isolations, I think it's the responsibility of the query scheduler and task scheduler. We should not put too much burden to the RunTimeEnv. Just make the RunTimeEnv as simple as possible. If future, we can add user level query profile or resource profile to the session configuration, if the plan is session configuration aware, the query scheduler and task scheduler can take the resource isolations and resource requirements into consideration, no need to leverage RuntimeEnv to make decision. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
